Big data index management system and method

Abstract

The invention discloses a big data index management system and method, and the system comprises a big data index collection module which is used for collecting big data indexes; the format conversion module is used for converting the big data indexes collected by the big data index collection module into a unified format; the first storage module is used for storing the big data indexes converted by the format conversion module; the extraction module is used for performing feature extraction on the big data indexes to obtain index feature vectors; the index processing module determines whether the big data index is abnormal or not; and the management module is used for managing the big data indexes without abnormity. According to the method, the big data indexes are converted into a unified format, subsequent extraction, processing and management are facilitated, the big data indexes can be classified, stored and managed according to values, and valuable big data indexes are rapidly obtained.

Description

Technical Field

[0001] The present invention relates to a management system and method, and in particular to a management system and method for big data indicators. Background Art

[0002] With the rapid development of economy and information system, especially the rise of Internet, Internet of Things (IoT) and social media, more and more systems and software generate massive amounts of data. Big Data, or massive data, refers to data sets that cannot be effectively captured, managed and processed by traditional data processing application software. These data sets are usually massive, high-growth and diversified, and require new processing models to give them stronger decision-making power, insight discovery and process optimization capabilities. Big data indicators are a quantitative standard used to measure a certain phenomenon, feature or performance. They can reflect the quantity, proportion, trend, rate of change and other information of a certain phenomenon or feature, and are very important tools in management and decision-making.

[0003] The Chinese invention patent with the patent publication number "CN 112651594 A" discloses an indicator management system, method, corresponding equipment and storage medium. The management system adopts modules such as data source acquisition module, parsing module, statistical attribute determination module, saving module, selection module, verification module, indicator authority setting module and calculation module to quickly extract data from multiple data sources to establish an indicator system for indicator calculation and application.

[0004] The above patented technology has the following technical problems: First, there is no definition or unified format for the indicators, which makes it inconvenient for subsequent extraction, processing and management; second, there is no classification, storage and management of the indicators, the indicators are too scattered, and it is inconvenient to select valuable big data indicators. Summary of the invention

[0005] In view of the above situation, in order to solve the technical problems of the prior art, the present invention provides a management system and method for big data indicators.

[0006] The present invention solves the above technical problems through the following technical solutions: a management system for big data indicators, characterized in that it includes: Big data indicator collection module, used to collect big data indicators; The format conversion module converts the big data indicators collected by the big data indicator collection module into a unified format; The first storage module is used to store the big data indicators converted by the format conversion module; The extraction module is used to extract features from big data indicators and obtain indicator feature vectors; The indicator processing module is used to obtain the indicator feature vector. The indicator processing module is provided with an indicator processing model. The indicator feature vector is processed based on the indicator processing model to determine whether the big data indicator is abnormal. If the big data indicator is abnormal, the abnormality analysis is performed and the cause of the abnormality is determined. If the big data indicator is not abnormal, it is sent to the subsequent management module. The management module manages the big data indicators without abnormalities.

[0007] Preferably, the management module includes a value assessment module and a value comparison module. The value assessment module assesses the big data indicators without abnormalities and generates a value assessment degree. The value comparison module compares the value assessment degree with at least one preset value of the indicator and stores them by category.

[0008] Preferably, the management module includes a value priority management module and a value importance management module. The value priority management module sorts the big data indicators without abnormalities according to the priority requirements, and the value importance management module sorts the big data indicators without abnormalities according to the importance levels.

[0009] Preferably, the process of processing the indicator feature vector by the indicator processing model includes the following specific processes: presetting a threshold score, performing feature comparison on the indicator feature vector and the preset normal feature vector, and obtaining the abnormal score of the big data indicator; when the abnormal score is greater than or equal to the threshold score, it is determined that the big data indicator is abnormal; when the abnormal score is greater than or equal to the threshold score, it is determined that the big data indicator is not abnormal.

[0010] Preferably, the management module is connected to an operation module, and the operation module performs specific operations on the big data indicators without abnormalities.

[0011] Preferably, the big data index includes a data repeatability index and a data accessibility index. The data repeatability index is reflected by the data repetition rate, and the data repetition rate a is calculated using the following formula: a=(b / c)× 100% Among them, b represents the number of repeated values, and c represents the total amount of data; The data accessibility index is reflected by the data accessibility rate, and the data accessibility rate d is calculated using the following formula: d = (e / f) × 100% Where e represents the number of successfully accessed requests, and f represents the total number of requests.

[0012] The present invention also provides a method for managing big data indicators, which is characterized in that it includes the following steps: Step 1: Collect big data indicators; Step 2: Convert the collected big data indicators into a unified format; Step 3: Store the converted big data indicators; Step 4: extract features from the big data indicators in step 3 to obtain indicator feature vectors; Step 5: Obtain the indicator feature vector, process the indicator feature vector based on the indicator processing model, and determine whether the big data indicator has an abnormality; if the big data indicator has an abnormality, perform an abnormality analysis and determine the cause of the abnormality; if the big data indicator has no abnormality, send it to the subsequent management step; Step six: Manage the big data indicators without abnormalities.

[0013] Preferably, step six includes: evaluating the big data indicators without abnormalities through a value evaluation module and generating a value evaluation degree, and then comparing the value evaluation degree with at least one indicator preset value through a value size comparison module and classifying and storing them.

[0014] Preferably, the step six includes: sorting the big data indicators without abnormalities according to the priority requirements through a value priority management module, and sorting the big data indicators without abnormalities according to the importance level through a value importance management module.

[0015] The positive and progressive effects of the present invention are: First, the present invention converts the big data indicators collected by the big data indicator collection module into a unified format through a format conversion module, so as to facilitate subsequent extraction, processing and management; Second, the present invention uses a management module to classify, store and manage big data indicators according to their value, quickly obtain valuable big data indicators, and thus provide a basis for users, facilitate use and management, and facilitate the selection of valuable big data indicators. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a principle block diagram of the management system of big data indicators of the present invention.

[0017] Figure 2 The present invention is a flowchart of a method for managing big data indicators. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0019] Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0020] In the description of the present invention, it is necessary to understand that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0021] like Figure 1 As shown, the management system of big data indicators of the present invention includes: Big data indicator collection module, used to collect big data indicators; The format conversion module converts the big data indicators collected by the big data indicator collection module into a unified format; for example, the format conversion module can convert the big data indicators collected by the big data indicator collection module into json format data; The first storage module is used to store the big data indicators converted by the format conversion module; The extraction module is used to extract features from big data indicators and obtain indicator feature vectors; The indicator processing module is used to obtain the indicator feature vector. The indicator processing module is provided with an indicator processing model. The indicator feature vector is processed based on the indicator processing model to determine whether the big data indicator is abnormal. If the big data indicator is abnormal, the abnormality analysis is performed and the cause of the abnormality is determined. If the big data indicator is not abnormal, it is sent to the subsequent management module. The management module manages the big data indicators without abnormalities.

[0022] In this embodiment, the management module may include a value assessment module and a value size comparison module. The value assessment module assesses the big data indicators without abnormalities and generates a value assessment degree. The value size comparison module compares the value assessment degree with at least one indicator preset value and stores them by category, so as to facilitate the selection of valuable big data indicators and classify big data indicators of different values. Among them, if the value size comparison module adopts an indicator preset value, then if the value assessment degree is greater than or equal to the indicator preset value, it is stored in the second storage module, and if it is less than the indicator preset value, it is stored in the second storage module. By analogy, if the value size comparison module adopts two indicator preset values, namely the first indicator preset value and the second indicator preset value, and the first indicator preset value is greater than the second indicator preset value, then if the value assessment degree is greater than or equal to the first indicator preset value, it is stored in the second storage module, if it is less than the first indicator preset value and greater than or equal to the second indicator preset value, it is stored in the third storage module, and if it is less than the second indicator preset value, it is stored in the fourth storage module. If more than two indicator preset values ​​are adopted, by analogy, no further description will be given.

[0023] Among them, the first storage module, the second storage module, the third storage module, the fourth storage module and other storage modules can use a relational database or a NoSQL database or a distributed storage system according to the format converted by the format conversion module. For example, the format conversion module converts the big data indicators collected by the big data indicator collection module into json format data and stores it in a mysql database (mysql database is a relational database). Each storage module can be located in one server or multiple servers. Each storage module is used to store big data indicators for subsequent extraction and storage.

[0024] In this embodiment, the management module may include a value priority management module and a value importance management module. The value priority management module sorts the big data indicators without abnormalities according to the priority requirements, and the value importance management module sorts the big data indicators without abnormalities according to the importance, for example, by an Excel table. When in use, the value priority management module and the value importance management module can select a management module for sorting, or copy the data of the big data indicators into two copies, and sort them respectively through the value priority management module and the value importance management module, so as to facilitate management and screening.

[0025] In some embodiments, the management module is connected to an operation module, and the operation module performs specific operations on the big data indicators without abnormalities, including editing, copying, moving, collecting, displaying and other operations, which are convenient for use and operation.

[0026] As an example, the indicator processing model processes the indicator feature vector including the following specific process: presetting a threshold score, performing feature comparison between the indicator feature vector and the preset normal feature vector, and obtaining the abnormal score of the big data indicator. When the abnormal score is greater than or equal to the threshold score, it is determined that the big data indicator is abnormal; when the abnormal score is greater than or equal to the threshold score, it is determined that the big data indicator is not abnormal, which facilitates timely judgment.

[0027] In this embodiment, the big data index includes a data repeatability index and a data accessibility index. The data repeatability index is reflected by the data repetition rate. The data repetition rate a is calculated using the following formula: a=(b / c)× 100% Among them, b represents the number of repeated values, and c represents the total amount of data; For example, if b is 3 and c is 10, the data repetition rate a is 30%.

[0028] The data accessibility index is reflected by the data accessibility rate, and the data accessibility rate d is calculated using the following formula: d = (e / f) × 100% Where e represents the number of successfully accessed requests, and f represents the total number of requests.

[0029] For example, if e is 18 and f is 20, the data accessibility rate d is 90%.

[0030] like Figure 2 As shown, the present invention also provides a method for managing big data indicators, which includes the following steps: Step 1: Collect big data indicators; Step 2: Convert the collected big data indicators into a unified format; for example, the collected big data indicators can be converted into json format data; Step 3: Store the converted big data indicators; Step 4: extract features from the big data indicators in step 3 to obtain indicator feature vectors; Step 5: Obtain the indicator feature vector, process the indicator feature vector based on the indicator processing model, and determine whether the big data indicator has an abnormality; if the big data indicator has an abnormality, perform an abnormality analysis and determine the cause of the abnormality; if the big data indicator has no abnormality, send it to the subsequent management step; Step six: Manage the big data indicators without abnormalities.

[0031] Optionally, the management in step six may include the following steps: evaluating the big data indicators without abnormalities through a value assessment module and generating a value assessment degree, and then comparing the value assessment degree with at least one indicator preset value through a value size comparison module and classifying and storing them.

[0032] Optionally, the management in step six may include the following steps: sorting the big data indicators without abnormalities according to the priority requirements through a value priority management module, and sorting the big data indicators without abnormalities according to the importance level through a value importance management module.

[0033] In addition, an embodiment of the present invention provides an electronic device, which includes: a memory and a processor; The memory is used to store computer programs; The processor is used to implement the method in the above embodiment when executing the computer program.

[0034] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method in the above embodiment is implemented.

[0035] Based on the method in the above embodiment, an embodiment of the present invention provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0036] Based on the method in the above embodiment, the embodiment of the present invention further provides a chip, including one or more processors and an interface circuit. Optionally, the chip may also include a bus. Among them: The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or instructions in the form of software. The above processor can be a general-purpose processor, a digital communicator (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods and steps disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0037] The interface circuit can be used to send or receive data, instructions or information. The processor can use the data, instructions or other information received by the interface circuit to process it, and can send the processing completion information through the interface circuit.

[0038] Optionally, the chip further includes a memory, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM).

[0039] Optionally, the memory stores an executable software module or a data structure, and the processor can perform corresponding operations by calling an operation instruction stored in the memory (the operation instruction may be stored in an operating system).

[0040] Optionally, the interface circuit can be used to output the execution result of the processor.

[0041] It should be noted that the corresponding functions of the processor and the interface circuit can be implemented through hardware design, software design, or a combination of hardware and software, and there is no limitation here.

[0042] It should be understood that each step of the above method embodiment can be completed by a hardware-based logic circuit or a software-based instruction in a processor.

[0043] It is understandable that the size of the sequence number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. In addition, in some possible implementations, each step in the above embodiment can be selectively executed according to actual conditions, and can be partially executed or fully executed, which is not limited here.

[0044] It is understandable that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0045] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0046] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0047] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A management system for big data indicators, characterized in that: It includes: Big data indicator collection module, used to collect big data indicators; The format conversion module converts the big data indicators collected by the big data indicator collection module into a unified format; The first storage module is used to store the big data indicators converted by the format conversion module; The extraction module is used to extract features from big data indicators and obtain indicator feature vectors; The indicator processing module is used to obtain the indicator feature vector. The indicator processing module is provided with an indicator processing model. The indicator feature vector is processed based on the indicator processing model to determine whether the big data indicator is abnormal. If the big data indicator is abnormal, the abnormality analysis is performed and the cause of the abnormality is determined. If the big data indicator is not abnormal, it is sent to the subsequent management module. The management module manages the big data indicators without abnormalities.

2. The big data index management system according to claim 1, characterized in that: The management module includes a value assessment module and a value comparison module. The value assessment module assesses the big data indicators without abnormalities and generates a value assessment degree. The value comparison module compares the value assessment degree with at least one indicator preset value and stores them by category.

3. The big data index management system according to claim 1, characterized in that: The management module includes a value priority management module and a value importance management module. The value priority management module sorts the big data indicators without abnormalities according to the priority requirements, and the value importance management module sorts the big data indicators without abnormalities according to the importance levels.

4. The big data index management system according to claim 1, characterized in that: The process of processing the indicator feature vector by the indicator processing model includes the following specific processes: presetting a threshold score, performing feature comparison between the indicator feature vector and the preset normal feature vector, and obtaining the abnormal score of the big data indicator. When the abnormal score is greater than or equal to the threshold score, it is determined that the big data indicator is abnormal; when the abnormal score is greater than or equal to the threshold score, it is determined that the big data indicator is not abnormal.

5. The big data index management system according to claim 1, characterized in that: The management module is connected to an operation module, and the operation module performs specific operations on the big data indicators without abnormalities.

6. The big data index management system according to claim 1, characterized in that: The big data indicators include data repeatability indicators and data accessibility indicators. The data repeatability indicators are reflected by the data repetition rate. The data repetition rate a is calculated using the following formula: a=(b / c)× 100% Among them, b represents the number of repeated values, and c represents the total amount of data; The data accessibility index is reflected by the data accessibility rate, and the data accessibility rate d is calculated using the following formula: d = (e / f) × 100% Where e represents the number of successfully accessed requests, and f represents the total number of requests.

7. A method for managing big data indicators, characterized in that: It includes the following steps: Step 1: Collect big data indicators; Step 2: Convert the collected big data indicators into a unified format; Step 3: Store the converted big data indicators; Step 4: extract features from the big data indicators in step 3 to obtain indicator feature vectors; Step 5: Obtain the indicator feature vector, process the indicator feature vector based on the indicator processing model, and determine whether the big data indicator has an abnormality; if the big data indicator has an abnormality, perform an abnormality analysis and determine the cause of the abnormality; if the big data indicator has no abnormality, send it to the subsequent management step; Step six: Manage the big data indicators without abnormalities.

8. The method for managing big data indicators according to claim 7, characterized in that: The step six includes: evaluating the big data indicators without abnormalities through a value evaluation module and generating a value evaluation degree, and then comparing the value evaluation degree with at least one indicator preset value through a value size comparison module and classifying and storing them.

9. The method for managing big data indicators according to claim 7, characterized in that: The step six includes: sorting the big data indicators without abnormalities according to the priority requirements through a value priority management module, and sorting the big data indicators without abnormalities according to the importance level through a value importance management module.

Images