Big data analysis and early warning method and platform based on NLP algorithm
Through the big data analysis and early warning method based on NLP algorithm, the work order label data in urban management is refined and analyzed, and the risk scenario model is constructed, which solves the problem of insufficient timeliness of risk point discovery in traditional urban management systems, and achieves rapid and accurate Risk warning improves urban management efficiency.
Patent Information
- Application Number
- CN202410019816.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-05
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-01-05
AI Technical Summary
Traditional urban management systems rely on manual discovery and annotation methods, resulting in poor timeliness of risk point discovery, inability to quickly follow up on event development, lack of unified and effective quantitative means, resulting in passive risk management decisions.
Using the big data analysis and early warning method based on NLP algorithm, by obtaining work order data, refining geography, subject and topic tags, building multiple risk scenario analysis models, continuously analyzing work order tag data, calculating evaluation value, performing abnormal analysis and correlation Process, optimize risk identification and early warning.
It significantly improves the efficiency and accuracy of risk point discovery, promptly issue early warnings, and improves the efficiency and effectiveness of urban management.
Smart Images

Figure CN117744008B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a big data analysis and early warning method and platform based on NLP algorithm. Background Art
[0002] As cities expand in size, the complexity and difficulty of urban management are also increasing. Traditional urban management methods can no longer meet the needs of modern cities. Through smart city management research and analysis, we can better discover and solve problems in cities and improve the efficiency and effectiveness of urban management.
[0003] Currently, most urban management systems mainly use manual discovery and labeling. During system operation, when accepting, handling or returning visits, urban management system personnel find that a certain type of event occurs frequently in a certain area or time period, they manually label the case as a risky work order to achieve the effect of early warning.
[0004] The manual risk discovery and warning method has the following defects:
[0005] 1. Risk points cannot be discovered quickly after they occur, and the timeliness is not strong. Urban management system users will only discover the risk when processing work orders, and it is uncertain how long it will take for the risk point to be discovered, warned, and reported.
[0006] 2. Failure to quickly follow up the development and evolution of events. The development process of events is constantly changing, generating new information that affects the direction of events and brings more uncertainty. If risk decisions and plans cannot be adjusted in real time, risk management decisions will fall into a passive situation.
[0007] 3. In risk emergency management, it is more about decision-making management by decision makers, which has a certain degree of subjective consciousness and lacks unified and effective quantitative means.
[0008] Therefore, the technical problem that urgently needs to be solved is: how to provide a big data analysis and early warning method and platform based on NLP algorithm, quickly extract work order label data through NLP algorithm, and continuously import it into various risk scenario analysis models for continuous analysis, so as to timely and efficiently discover and warn of risk points, and improve the efficiency and effectiveness of urban management. Summary of the invention
[0009] The purpose of this application is to provide a big data analysis and early warning method and platform based on NLP algorithm, which can quickly extract work order label data through NLP algorithm, and continuously import it into various risk scenario analysis models for continuous analysis, so as to timely and efficiently discover and warn of risk points, and improve the efficiency and effectiveness of urban management.
[0010] To achieve the above objectives, the present application provides a big data analysis and early warning method based on NLP algorithm, which includes the following steps:
[0011] Get work order data;
[0012] Analyze work order data based on NLP algorithms and extract work order label data;
[0013] Store the work order tag data into the tag library;
[0014] Pre-build multiple risk scenario assessment models;
[0015] The work order label data is continuously imported into various risk scenario assessment models for continuous analysis to obtain abnormal analysis results.
[0016] The big data analysis and early warning method based on the NLP algorithm as described above, wherein the method further includes the following steps:
[0017] Obtain the scoring data of each risk scenario assessment model;
[0018] Process the scoring data of the risk scenario assessment model;
[0019] Based on the processed scoring data, calculate the evaluation value of each risk scenario analysis model.
[0020] The big data analysis and early warning method based on the NLP algorithm as described above, wherein the method further includes the following steps:
[0021] Perform correlation analysis on the abnormal analysis results, and merge the events corresponding to the abnormal analysis results.
[0022] As described above, the big data analysis and early warning method based on the NLP algorithm optimizes the risk scenario analysis model whose evaluation value is less than a preset evaluation threshold.
[0023] As described above, the big data analysis and early warning method based on the NLP algorithm, wherein the work order label data includes: geographic labels, subject labels and topic labels.
[0024] As described above, the big data analysis and early warning method based on the NLP algorithm, wherein the method of analyzing the work order data based on the NLP algorithm and extracting the work order label data includes:
[0025] Preprocess the work order data;
[0026] Decompose the pre-processed work order text information into words or phrases;
[0027] Through natural language processing technology, the geographic information, subject information and topic information in the work order text information are identified to obtain geographic tags, subject tags and topic tags.
[0028] In the above-mentioned big data analysis and early warning method based on NLP algorithm, the processing of the scoring data of the risk scenario analysis model includes:
[0029] Calculate the evaluation difference of all scoring data of each risk scenario assessment model respectively;
[0030] Based on discrete values, discrete data is eliminated from all scoring data of the risk scenario assessment model.
[0031] As a second aspect of the present application, the present application provides a big data analysis and early warning platform based on NLP algorithm, the platform comprising:
[0032] The first acquisition module is used to acquire work order data;
[0033] The extraction module is used to analyze the work order data based on the NLP algorithm and extract the work order label data;
[0034] The storage module is used to store the work order label data into the label library;
[0035] Construction module, used to pre-build multiple risk scenario assessment models;
[0036] The analysis module is used to continuously import work order label data into various risk scenario assessment models for continuous analysis to obtain abnormal analysis results.
[0037] The big data analysis and early warning platform based on the NLP algorithm as described above, wherein the platform also includes:
[0038] The second acquisition module is used to obtain the scoring data of each risk scenario analysis model;
[0039] Data processing module, used to process the scoring data of the risk scenario analysis model;
[0040] The data calculation module is used to calculate the evaluation value of each risk scenario analysis model based on the processed scoring data.
[0041] The big data analysis and early warning platform based on the NLP algorithm as described above, wherein the platform also includes:
[0042] The merging processing module is used to perform correlation analysis on the abnormal analysis results and merge the events corresponding to the abnormal analysis results.
[0043] The beneficial effects achieved by this application are as follows:
[0044] (1) Compared with the existing manual discovery, analysis and warning methods, in the operation process of the urban management system, the NLP algorithm of this application quickly extracts and analyzes and cooperates with various risk scenario analysis models to quickly analyze and discover risk points and issue warnings, which can greatly improve the efficiency and accuracy of risk point discovery and issue warnings in a timely manner, thereby improving the efficiency and effectiveness of urban management. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For those skilled in the art, other drawings can also be obtained based on these drawings.
[0046] Figure 1 This is a flowchart of a big data analysis and early warning method based on NLP algorithm in an embodiment of the present application.
[0047] Figure 2 This is a structural diagram of a big data analysis and early warning platform based on NLP algorithm in an embodiment of the present application. DETAILED DESCRIPTION
[0048] The following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.
[0049] Embodiment 1
[0050] like Figure 1 As shown, the present application provides a big data analysis and early warning method based on NLP algorithm, which includes the following steps:
[0051] Step S1, obtaining work order data.
[0052] Among them, the work order data records a certain type of event that occurs in a certain area during a certain period of time.
[0053] Step S2: Analyze the work order data based on the NLP algorithm to extract the work order label data.
[0054] Among them, the NLP algorithm identifies the geographic tags, subject tags and topic tags of the work order. The work order tag data includes: geographic tags, subject tags and topic tags.
[0055] Step S2 includes:
[0056] Step S210, pre-processing the work order data.
[0057] Specifically, preprocessing the work order data means preprocessing the work order text information.
[0058] Specifically, preprocessing includes removing stop words, punctuation marks, special characters, etc. This helps reduce noise.
[0059] Step S220, decomposing the preprocessed work order text information into words or phrases.
[0060] Step S230, using natural language processing technology, the geographic information, subject information and topic information in the work order text information are identified to obtain geographic tags, subject tags and topic tags.
[0061] Step S3, storing the work order label data into the label library.
[0062] It needs to be explained that a tag library is a tool for storing and managing tags. It can classify, organize and query tags to facilitate users to find and use them. In Java, the JSTL Standard Tag Library (JSTL) is a JSP tag collection that encapsulates the common core functions of JSP applications. JSTL supports common, structured tasks such as iteration, conditional judgment, XML document operations, internationalization tags, SQL tags, etc.
[0063] Step S4, pre-build multiple risk scenario assessment models.
[0064] Among them, the methods for constructing multiple risk scenario assessment models include:
[0065] Step S410, obtaining training data sets of multiple risk scenarios.
[0066] Specifically, the training data set includes geographic tags, subject tags, and topic tags corresponding to a known risk scenario.
[0067] Step S420, input the training data set of each risk scenario into the neural network basic model for training, and obtain the risk scenario analysis model corresponding to each risk scenario.
[0068] Among them, risk scenarios in urban management include: urban safety risks, urban planning risks, urban environmental risks, urban traffic risks, urban public service risks, etc. The risk scenario analysis model is used to analyze the work order tag data to determine whether there are risk points in the work order tag data. The risk scenario analysis model is used to identify, evaluate and control potential risks. If risks are found, timely alarms are issued to reduce the harm caused by risk events.
[0069] Step S5, continuously import the work order tag data into each risk scenario analysis model for continuous analysis to obtain abnormal analysis results.
[0070] This application constructs risk scenario analysis models and three types of labeling methods (geographic labels, subject labels, and topic labels) to intelligently identify risk points and issue warnings.
[0071] Step S6, obtaining the scoring data of each risk scenario analysis model.
[0072] Among them, the scoring data of each risk scenario assessment model includes the scores given by managers to the risk scenario assessment model or the scores given by citizens to the risk scenario assessment model.
[0073] Step S7, processing the scoring data of the risk scenario analysis model.
[0074] Step S7 includes:
[0075] Step S710, respectively calculate the evaluation difference of all scoring data of each risk scenario assessment model.
[0076] Among them, the discrete value calculation formula of the scoring data of the risk scenario analysis model is as follows:
[0077]
[0078] Where M represents the discrete value of the i-th scoring data of the risk scenario assessment model; H represents the control benchmark value of the scoring data; K represents the total number of scoring data of the current risk scenario assessment model; J i represents the value of the i-th scoring data; G j Represents the j-th scoring data of the current risk scenario assessment model.
[0079] Step S720: based on the discrete values, remove the discrete data from all the scoring data of the risk scenario analysis model.
[0080] Specifically, according to the calculated discrete values of the scoring data of the risk scenario analysis model, the scoring data with discrete values exceeding the preset threshold are selected for elimination, thereby improving the reliability of the scoring data of the risk scenario analysis model and avoiding malicious scoring or scoring errors.
[0081] It needs to be explained that by eliminating discrete data from all scoring data of the risk scenario assessment model, it is possible to eliminate unreasonable data and improve the calculation accuracy of the evaluation values of each risk scenario assessment model.
[0082] Step S8, calculate the evaluation value of each risk scenario analysis model based on the processed scoring data.
[0083] Among them, the calculation formula of the evaluation value of the risk scenario assessment model is as follows:
[0084]
[0085] Where W represents the evaluation value of the risk scenario assessment model; T represents the total number of scoring data sources; Y t represents the influence weight of the t-th scoring data source on the evaluation value; At represents the total number of scoring data contained in the t-th scoring data source; F h Represents the value of the hth scoring data.
[0086] Among them, the scoring data sources include from a certain platform, questionnaires, management platform, etc.
[0087] As a specific embodiment of the present invention, the risk scenario analysis model with an evaluation value less than a preset evaluation threshold is optimized to further improve the risk identification accuracy of the risk scenario analysis model.
[0088] Step S9, performing correlation analysis on the abnormal analysis results, and merging events corresponding to the abnormal analysis results.
[0089] Step S9 includes:
[0090] Step S910, obtaining data of abnormal analysis results.
[0091] Step S920: Calculate the correlation value of the events corresponding to the two abnormal analysis results according to the data of the abnormal analysis results.
[0092] The calculation formula for the correlation value of the events corresponding to the two abnormal analysis results is as follows:
[0093]
[0094] Among them, GL represents the correlation value of the events corresponding to the two abnormal analysis results; Indicates the correlation factor of the event types corresponding to the two abnormal analysis results; if the event types corresponding to the two abnormal analysis results are the same, then otherwise; Indicates whether the events corresponding to the two abnormal analysis results are within the preset distance range. If so, then otherwise δ1 represents the impact weight of the continuous abnormal value of the event; δ2 represents the impact weight of the event occurrence time; Kg1 represents the continuous abnormal value of the first type of event; kg2 represents the continuous abnormal value of the second type of event; Tmin represents the time difference between the occurrence of the first and second events; TY represents the maximum allowable limit of the time difference between two related events.
[0095] Among them, Kg1 and kg2 are calculated according to the following formula:
[0096]
[0097] Among them, ρ represents the impact weight of the duration of the first event; Tcx represents the duration of the first event; tf represents the time when the first event is discovered; Indicates the impact weight of the time it takes for the first event to be discovered.
[0098] The present application improves the calculation accuracy of the correlation value of events corresponding to two abnormal analysis results, thereby achieving merging and processing of events with correlation that meet the requirements, and improving the processing efficiency of events.
[0099] Step S930, comparing the correlation values of the events corresponding to the two abnormal analysis results with the preset correlation threshold. If the correlation values of the events corresponding to the two abnormal analysis results are greater than the preset correlation threshold, the events corresponding to the two abnormal analysis results are merged. Otherwise, there is no need to merge the events corresponding to the two abnormal analysis results.
[0100] Embodiment 2
[0101] like Figure 2 As shown, the present application provides a big data analysis and early warning platform 100 based on NLP algorithm, which includes:
[0102] A first acquisition module 10, used to acquire work order data;
[0103] The refining module 20 is used to analyze the work order data based on the NLP algorithm and refine the work order label data;
[0104] Storage module 30, used to store work order label data into label library;
[0105] A construction module 40 is used to pre-construct multiple risk scenario assessment models;
[0106] The analysis module 50 is used to continuously import the work order tag data into each risk scenario analysis model for continuous analysis to obtain abnormal analysis results.
[0107] The second acquisition module 60 is used to obtain the scoring data of each risk scenario analysis model;
[0108] The data processing module 70 is used to process the scoring data of the risk scenario analysis model;
[0109] The data calculation module 80 is used to calculate the evaluation value of each risk scenario analysis model based on the processed scoring data.
[0110] The merging processing module 90 is used to perform correlation analysis on the abnormal analysis results and merge the events corresponding to the abnormal analysis results.
[0111] The present application also provides a computer storage medium, which stores computer instructions, and when the computer instructions are called, they are used to execute the address mapping method of the large-capacity solid-state hard disk. The computer storage medium contains one or more program instructions, and the one or more program instructions are used by the processor to execute a big data judgment and early warning method based on the NLP algorithm.
[0112] The embodiments disclosed in the present invention provide a computer-readable storage medium, in which computer program instructions are stored. When the computer program instructions are executed on a computer, the computer executes the above-mentioned big data analysis and early warning method based on the NLP algorithm.
[0113] An embodiment of the present invention provides a processor for processing the above-mentioned big data analysis and early warning method based on NLP algorithm.
[0114] In the embodiment of the present invention, the processor may be an integrated circuit chip having the ability to process signals. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0115] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.
[0116] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.
[0117] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DRRAM).
[0118] The beneficial effects achieved by this application are as follows:
[0119] (1) Compared with the existing manual discovery, analysis and warning methods, in the operation process of the urban management system, the NLP algorithm of this application quickly extracts and analyzes and cooperates with various risk scenario analysis models to quickly analyze and discover risk points and issue warnings, which can greatly improve the efficiency and accuracy of risk point discovery and issue warnings in a timely manner, thereby improving the efficiency and effectiveness of urban management.
[0120] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.
[0121] In the description of the present application, the word "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.
[0122] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.
Claims
1. A big data analysis and early warning method based on NLP algorithm, characterized in that: The method comprises the following steps: Get work order data; Analyze work order data based on NLP algorithms and extract work order label data; Store the work order tag data into the tag library; Pre-build multiple risk scenario analysis models; Continuously import work order tag data into various risk scenario assessment models for continuous analysis to obtain abnormal analysis results; Conduct correlation analysis on the abnormal analysis results, and merge the events corresponding to the abnormal analysis results; Among them, performing correlation analysis on the abnormal analysis results and merging the events corresponding to the abnormal analysis results include: Get data on abnormal analysis results; According to the data of the abnormal analysis results, the correlation values of the events corresponding to the two abnormal analysis results are calculated; Compare the correlation values of the events corresponding to the two abnormal analysis results with the preset correlation threshold. If the correlation values of the events corresponding to the two abnormal analysis results are greater than the preset correlation threshold, the events corresponding to the two abnormal analysis results are merged for processing; otherwise, the events corresponding to the two abnormal analysis results do not need to be merged for processing; The calculation formula for the correlation value of the events corresponding to the two abnormal analysis results is as follows: Among them, GL represents the correlation value of the events corresponding to the two abnormal analysis results; Indicates the correlation factor of the event types corresponding to the two abnormal analysis results; if the event types corresponding to the two abnormal analysis results are the same, then otherwise; Indicates whether the events corresponding to the two abnormal analysis results are within the preset distance range. If so, then otherwise δ1 represents the impact weight of the continuous abnormal value of the event; δ2 represents the impact weight of the event occurrence time; Kg1 represents the continuous abnormal value of the first event; kg2 represents the continuous abnormal value of the second event; Tmin represents the time difference between the occurrence of the first event and the second event; TY represents the maximum allowable limit of the time difference between the occurrence of two related events; Among them, Kg1 and kg2 are calculated according to the following formula: Among them, ρ represents the impact weight of the duration of the first event; Tcx represents the duration of the first event; tf represents the time when the first event is discovered; Indicates the impact weight of the time it takes for the first event to be discovered.
2. The big data analysis and early warning method based on NLP algorithm according to claim 1 is characterized in that: The method further comprises the steps of: Obtain the scoring data of each risk scenario assessment model; Process the scoring data of the risk scenario assessment model; Based on the processed scoring data, calculate the evaluation value of each risk scenario analysis model.
3. The big data analysis and early warning method based on NLP algorithm according to claim 2 is characterized in that: Optimize the risk scenario analysis model whose evaluation value is less than the preset evaluation threshold.
4. The big data analysis and early warning method based on NLP algorithm according to claim 1 is characterized in that: The work order tag data includes: geographic tags, subject tags and topic tags.
5. The big data analysis and early warning method based on NLP algorithm according to claim 4 is characterized in that: Methods for analyzing work order data based on NLP algorithms and extracting work order label data include: Preprocess the work order data; Decompose the pre-processed work order text information into words or phrases; Through natural language processing technology, the geographic information, subject information and topic information in the work order text information are identified to obtain geographic tags, subject tags and topic tags.
6. The big data analysis and early warning method based on NLP algorithm according to claim 2 is characterized in that: The processing of the scoring data of the risk scenario analysis model includes: Calculate the evaluation difference of all scoring data of each risk scenario assessment model respectively; Based on discrete values, discrete data is eliminated from all scoring data of the risk scenario assessment model.
7. A big data analysis and early warning platform based on NLP algorithm, characterized by: The method according to any one of claims 1 to 6 is performed, wherein the platform comprises: The first acquisition module is used to acquire work order data; The extraction module is used to analyze the work order data based on the NLP algorithm and extract the work order label data; The storage module is used to store the work order label data into the label library; Construction module, used to pre-build multiple risk scenario assessment models; The analysis module is used to continuously import work order label data into various risk scenario assessment models for continuous analysis to obtain abnormal analysis results.
