Enterprise industry and commerce judicial data analysis method and device, electronic equipment and storage medium
By collecting and pushing corporate industrial and commercial judicial data to a distributed message queue, using Spark streaming computing and engine for processing and storage, and analyzing it through data models, the problem that traditional databases are difficult to manage and utilize a large amount of corporate industrial and commercial judicial data is solved, and efficient management and utilization is achieved.
Patent Information
- Application Number
- CN202411767133.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-03-28
AI Technical Summary
Traditional relational databases are difficult to effectively manage and efficiently utilize the surge in corporate business judicial data.
By collecting corporate business judicial data and pushing it to a distributed message queue, the Spark streaming computing framework consumes data in real time, and the data is processed and stored in the big data warehouse through the Spark engine, and finally modeled analysis and processing through the data model.
It has achieved effective management and efficient utilization of corporate industrial and commercial judicial data, and improved the in-depth mining of data processing efficiency and data value.
Smart Images
Figure CN119719198A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of data analysis technology, and in particular, to a method, device, electronic device, and storage medium for analyzing corporate industrial and commercial judicial data. Background Art
[0002] Database technology is a core technology of information systems. It studies how to organize and store data, and how to efficiently obtain and process data. Database technology involves the unified organization and management of data, establishing corresponding databases and data warehouses according to the specified structure, and using database management systems and data mining systems to design application systems that can perform multiple functional processing on data in the database.
[0003] Databases can be divided into relational databases, non-relational databases and other types of databases according to different dimensions. Business and industrial judicial data of enterprises are generally stored in relational databases. With the rapid development of corporate business, the business and industrial judicial data of enterprises is also increasing rapidly. Traditional relational databases can no longer meet the needs of rapid processing of various types and huge amounts of business and industrial judicial data.
[0004] Therefore, how to analyze the ever-increasing corporate industrial and commercial judicial data has become a technical problem that needs to be solved urgently. Summary of the invention
[0005] In view of this, one of the technical problems solved by the embodiments of the present application is to provide a corporate industrial and commercial judicial data analysis method, device, electronic device, and storage medium, which can analyze corporate industrial and commercial judicial data, thereby achieving effective management and efficient use of corporate industrial and commercial judicial data.
[0006] In a first aspect, an embodiment of the present application provides a method for analyzing corporate industrial and commercial judicial data, the method comprising: collecting corporate industrial and commercial judicial data and pushing it to a distributed message queue; consuming the corporate industrial and commercial judicial data in the distributed message queue in real time through a Spark streaming computing framework, and parsing the corporate industrial and commercial judicial data into a fixed format and storing it in a big data warehouse; processing the corporate industrial and commercial judicial data in the big data warehouse through a Spark engine, and storing the processed corporate industrial and commercial judicial data in the big data warehouse; and performing modeling analysis and processing on the corporate industrial and commercial judicial data through a data model to form analysis result data and store the analysis result data in the big data warehouse.
[0007] In some optional embodiments of the present application, the method further includes: synchronizing the analysis result data to a relational database and a distributed search engine through a data synchronization tool.
[0008] In some optional embodiments of the present application, the method further includes:
[0009] The analysis result data is displayed by reading data from the relational database and the distributed search engine.
[0010] In some optional embodiments of the present application, the processing of the enterprise industrial and commercial judicial data in the big data warehouse by the Spark engine and storing the processed enterprise industrial and commercial judicial data in the big data warehouse includes: cleaning and filtering the enterprise industrial and commercial judicial data in the big data warehouse by the Spark engine, and storing the cleaned and filtered data in the big data warehouse; aggregating the cleaned and filtered data by the Spark engine, and storing the aggregated data in the big data warehouse.
[0011] In some optional embodiments of the present application, the big data warehouse includes: an operation data storage layer, a data warehouse detail layer, a data warehouse summary layer, and an application data storage layer; the Spark streaming computing framework consumes the enterprise industrial and commercial judicial data in the distributed message queue in real time, and parses the enterprise industrial and commercial judicial data into a fixed format and stores it in the operation data storage layer of the big data warehouse; the Spark engine cleans and filters the enterprise industrial and commercial judicial data in the big data warehouse, and stores the cleaned and filtered data in the data warehouse detail layer of the big data warehouse; the Spark engine aggregates the cleaned and filtered data, and stores the aggregated data in the data warehouse summary layer of the big data warehouse; using the trained data model, the enterprise industrial and commercial judicial data is modeled and analyzed to form analysis result data and store the analysis result data in the application data storage layer of the big data warehouse.
[0012] In some optional embodiments of the present application, the enterprise industrial and commercial judicial data is modeled and analyzed through a data model to form analysis result data and store the analysis result data in the big data warehouse, which also includes: converting the aggregated data of the data warehouse summary layer of the big data warehouse into features that can be used by a machine learning algorithm, and selecting a preset machine learning algorithm to use the features for training to obtain the data model.
[0013] In a second aspect, an embodiment of the present application provides an enterprise industrial and commercial judicial data analysis device, the device comprising: a collection module, used to collect enterprise industrial and commercial judicial data and push it to a distributed message queue; a consumption module, used to consume the enterprise industrial and commercial judicial data in the distributed message queue in real time through the Spark streaming computing framework, and parse the enterprise industrial and commercial judicial data into a fixed format and store it in a big data warehouse; a processing module, used to process the enterprise industrial and commercial judicial data in the big data warehouse through the Spark engine, and store the processed enterprise industrial and commercial judicial data in the big data warehouse; an analysis module, used to perform modeling analysis and processing on the enterprise industrial and commercial judicial data through a data model, form analysis result data and store the analysis result data in the big data warehouse.
[0014] In a third aspect, an embodiment of the present application provides an electronic device comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in the first aspect.
[0015] In a fourth aspect, an embodiment of the present application provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.
[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the method described in the first aspect.
[0017] The embodiment of the present application collects corporate industrial and commercial judicial data and pushes it to a distributed message queue, and then consumes the corporate industrial and commercial judicial data in the distributed message queue in real time through the Spark streaming computing framework, and parses the corporate industrial and commercial judicial data into a fixed format and stores it in a big data warehouse. The corporate industrial and commercial judicial data in the big data warehouse is processed by the Spark engine, and the processed corporate industrial and commercial judicial data is stored in the big data warehouse, and then the corporate industrial and commercial judicial data is modeled and analyzed through the data model to form analysis result data and store the analysis result data in the big data warehouse. The embodiment of the present application can analyze corporate industrial and commercial judicial data, thereby achieving effective management and efficient use of corporate industrial and commercial judicial data. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Hereinafter, some specific embodiments of the present application will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:
[0019] Figure 1 This is a flowchart of a method for analyzing corporate business judicial data according to an embodiment of the present application;
[0020] Figure 2 This is a flowchart of another method for analyzing corporate business judicial data according to an embodiment of the present application;
[0021] Figure 3 This is a flowchart of the steps of another method for analyzing corporate business judicial data in an embodiment of the present application;
[0022] Figure 4 This is a flowchart of step S3 of another method for analyzing corporate business judicial data in an embodiment of the present application;
[0023] Figure 5 This is a flowchart of the steps of another method for analyzing corporate business judicial data in an embodiment of the present application;
[0024] Figure 6 This is a flowchart of the steps of another method for analyzing corporate business judicial data in an embodiment of the present application;
[0025] Figure 7 This is a schematic diagram of the structure of a corporate industrial and commercial judicial data analysis device according to an embodiment of the present application;
[0026] Figure 8 A schematic diagram of the structure of an electronic device used in an embodiment of the present application. DETAILED DESCRIPTION
[0027] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be described clearly and in detail below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in the field based on the embodiments in the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.
[0028] Reference is made to the accompanying drawings in the following detailed description, which form a part of the detailed description and illustrate exemplary embodiments. In addition, it is to be understood that other embodiments may be utilized and structural and / or logical changes may be made without departing from the scope of the claimed subject matter. It should also be noted that directions and references (e.g., up, down, top, bottom, etc.) may be used merely to facilitate the description of features in the accompanying drawings. Therefore, the following detailed description is not to be understood in a limiting sense, and the scope of the claimed subject matter is limited only by the appended claims and their equivalents.
[0029] In the following description, numerous details are set forth. However, it will be apparent to those skilled in the art that the embodiments herein can be practiced without these specific details. In some cases, known methods and devices are shown in block diagram form, rather than in detail, to avoid blurring the embodiments herein. References to "embodiment" or "one embodiment" or "some embodiments" throughout this specification mean that the specific features, structures, functions or characteristics described in conjunction with the embodiment are included in at least one embodiment herein. Therefore, the phrases "in an embodiment" or "in one embodiment" or "some embodiments" appearing throughout this specification do not necessarily refer to the same embodiment. In addition, in one or more embodiments, specific features, structures, functions or characteristics may be combined in any suitable manner. For example, the first embodiment may be combined with the second embodiment in any case where the specific features, structures, functions or characteristics associated with the two embodiments are not mutually exclusive.
[0030] As used in the description and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
[0031] The terms "coupled" and "connected" together with their derivatives may be used in this article to describe the functional or structural relationship between components. It should be understood that these terms are not intended to be synonymous with each other. On the contrary, in a particular embodiment, "connected" can be used to indicate that two or more elements are in direct physical, optical or electrical contact with each other. "Coupled" can be used to indicate that two or more elements are in direct or indirect (with other intermediate elements between them) physical or electrical contact with each other, and / or that two or more elements cooperate or interact with each other (e.g., as in a cause-and-effect relationship).
[0032] As used herein, the terms "above," "below," "between," and "on" refer to the relative position of one component or material with respect to other components or materials where such physical relationship is noteworthy. For example, in the context of materials, a material or materials disposed above or below another material may be in direct contact, or may have one or more intervening materials. Also, a material disposed between two materials or materials may be in direct contact with both layers, or may have one or more intervening layers. In contrast, a first material or material "above" a second material or material is in direct contact with the second material / materials. Similar distinctions are made in the context of component assembly.
[0033] As used throughout the description herein and in the claims, a list of items connected by the term "at least one of" or "one or more of" may mean any combination of the listed items. For example, the phrase "at least one of A, B, or C" may mean A; B; C; A and B; A and C; B and C; or A, B, and C.
[0034] The term "circuit" or "module" may refer to one or more passive and / or active components that are arranged to cooperate with each other to provide a desired functionality. The term "signal" may refer to at least one current signal, voltage signal, or magnetic signal. The terms "substantially," "close," "approximately," "close to," and "approximately" generally refer to within + / - 10% of a target value.
[0035] An embodiment of the present application provides a radio frequency identification mobile collection vehicle.
[0036] The specific implementation of the embodiment of the present invention is further described below in conjunction with the accompanying drawings of the embodiment of the present invention.
[0037] See also Figure 1 , the present application embodiment provides a method for analyzing corporate business judicial data, the method comprising:
[0038] Step S1: Collect corporate industrial and commercial judicial data and push it to a distributed message queue.
[0039] Specifically, Kafka distributed message queue is used. Kafka is a distributed publish-subscribe message system. It was originally released by LinkedIn and written in Scala. Kafka is a high-throughput, persistent, distributed publish-subscribe message system. It is mainly used to process active data (data generated by user behaviors such as login, browsing, clicking, sharing, and liking).
[0040] The embodiments of the present application utilize the characteristics of Kafka such as high throughput, low latency, scalability, persistence, reliability, and fault tolerance to efficiently and reliably transmit massive amounts of corporate, industrial, and commercial judicial data.
[0041] Step S2: Use the Spark Streaming framework to consume the corporate business and judicial data in the distributed message queue in real time, and parse the corporate business and judicial data into a fixed format and store it in the big data warehouse.
[0042] The Spark streaming computing framework is a processing module based on streaming data in Spark. The Spark streaming computing framework splits the data stream into small batches by time slices. Each batch of data is processed into a resilient distributed data set, and then processed in batches through the Spark distributed computing framework. This processing method enables the Spark streaming computing framework to provide low latency while ensuring high throughput.
[0043] Step S3: Process the business, industrial and commercial judicial data in the big data warehouse through the Spark engine, and store the processed business, industrial and commercial judicial data in the big data warehouse.
[0044] Step S4: Through the data model, the enterprise's industrial and commercial judicial data is modeled and analyzed to form analysis result data and store the analysis result data in the big data warehouse.
[0045] The embodiments of the present application can analyze the business judicial data of enterprises, thereby realizing the effective management and efficient use of the business judicial data of enterprises.
[0046] In some specific implementations of the embodiments of this application, see Figure 2 , the method further comprising:
[0047] Step S5: Synchronize the analysis result data to the relational database and the distributed search engine through a data synchronization tool.
[0048] Specifically, data synchronization tools include: Sqoop and WaterDrop, which synchronize the analyzed and processed result data to the relational database Tidb and the distributed search engine ES respectively.
[0049] Sqoop is an open source tool that is mainly used to transfer data between distributed databases and traditional databases. It can import data from a relational database into a distributed database, and also import data from a distributed database into a relational database.
[0050] WaterDrop is a simple and easy-to-use data integration framework, which was officially renamed SeaTunnel on October 12, 2021. It aims to provide an ultra-high-performance distributed data integration platform that supports real-time synchronization of massive data.
[0051] The relational database Tidb is a fusion distributed database product that supports both online transaction processing and online analytical processing (HTAP).
[0052] The distributed search engine ES is a distributed search and analysis engine at the core of the Elastic Stack, which provides real-time search and analysis for all types of data.
[0053] The embodiment of the present application uses the synchronization service in the data analysis step S5 to synchronize the analysis result data to the relational database and the distributed search engine for user query, thereby improving the problem of low data query efficiency in the relational database.
[0054] In some specific implementations of the embodiments of this application, see Figure 3 , the method further comprising:
[0055] |Step S6: Display the analysis result data by reading data from the relational database and the distributed search engine.
[0056] Users can obtain the analysis results of corporate industrial and commercial judicial data by reading data from the relational database Tidb and the distributed search engine ES.
[0057] The embodiment of the present application obtains visualized analysis result data of corporate industrial and commercial judicial data, thereby further improving data query efficiency and realizing effective management and efficient use of corporate industrial and commercial judicial data.
[0058] In some specific implementations of the embodiments of this application, see Figure 4 , step S3, comprising:
[0059] Step S31: Clean and filter the business, industrial and commercial judicial data in the big data warehouse through the Spark engine, and store the cleaned and filtered data in the big data warehouse.
[0060] Step S32: Aggregate the cleaned and filtered data through the Spark engine, and store the aggregated data in the big data warehouse.
[0061] The embodiment of the present application utilizes the Spark engine to clean, filter and aggregate the corporate, industrial and commercial judicial data in the big data warehouse, thereby improving the data processing speed and achieving in-depth mining of the data value.
[0062] In some specific implementations of the embodiments of the present application, the big data warehouse includes: an operation data storage layer, a data warehouse detail layer, a data warehouse summary layer, and an application data storage layer.
[0063] The Spark streaming computing framework in step S2 parses the corporate industrial and commercial judicial data into a fixed format and stores it in the operational data storage layer (ODS) of the big data warehouse.
[0064] |The Spark engine in step S31 cleans and filters the corporate industrial and commercial judicial data in the big data warehouse, and stores the cleaned and filtered data in the data warehouse details (DWD) of the big data warehouse.
[0065] The Spark engine in step S32 aggregates the cleaned and filtered data, and stores the aggregated data in the data warehouse summary layer (DWS) of the big data warehouse.
[0066] In step S4, the trained data model is used to perform model analysis and processing on the enterprise's industrial and commercial judicial data to form analysis result data and store the analysis result data in the application data storage layer (ADS) of the big data warehouse.
[0067] Specifically, the big data warehouse in the embodiment of the present application is hive, which is used to extract, transform and load data. It is a tool that can store, query and analyze large-scale data.
[0068] The embodiment of the present application solves the problem of limited data storage resources in relational databases for massive amounts of corporate industrial and commercial judicial data through data hierarchical management, improves data processing efficiency, enhances data scalability, and makes data management and maintenance more convenient.
[0069] In some specific implementations of the embodiments of this application, see Figure 5 , step S4, before that also includes:
[0070] Step S7: Convert the aggregated data of the data warehouse summary layer of the big data warehouse into features that can be used by the machine learning algorithm, and select a preset machine learning algorithm to use the features for training to obtain a data model.
[0071] Specifically, the preset machine learning algorithms include: classification, regression, clustering, etc.
[0072] The embodiment of the present application establishes different data models through model training, such as a health index model, an operation index model, a five-dimensional scoring model, etc. to reflect the health status of enterprise development.
[0073] Those skilled in the art can establish data models as needed, thereby increasing the flexibility and scope of application of the embodiments of the present application. The embodiments of the present application can deeply mine the value of corporate business and judicial data through data modeling and data analysis and processing, providing an important basis for the healthy and sustainable development of enterprises.
[0074] To go through a specific application scenario, see Figure 6 , further illustrating the implementation of the embodiment of the present application.
[0075] Step 601: Collecting corporate business judicial data of multiple clients (client 1, client 2, ..., client n); wherein n is a positive integer greater than 1;
[0076] Step 602: Push the collected corporate industrial and commercial judicial data to the Kafka message queue.
[0077] Step 603: Consume the enterprise business and judicial data in the distributed message queue in real time through the Spark streaming computing framework, and parse the enterprise business and judicial data into a fixed format and store it in the operational data storage layer (ODS) of the big data warehouse.
[0078] Step 604: Clean and filter the business, industrial and judicial data in the big data warehouse through the Spark engine, and store the cleaned and filtered data in the data warehouse details (DWD) of the big data warehouse.
[0079] Step 605: Aggregate the cleaned and filtered data through the Spark engine, and store the aggregated data in the data warehouse summary layer (DWS) of the big data warehouse.
[0080] Step 606: Convert the aggregated data of the data warehouse summary layer of the big data warehouse into features that can be used by the machine learning algorithm, and select a preset machine learning algorithm to use the features for training to obtain a data model.
[0081] Step 607: Model analysis and processing are performed on the business and judicial data of the enterprise through the data model to generate analysis result data and store the analysis result data in the application data storage layer (ADS) of the big data warehouse.
[0082] Step 608: Use the data synchronization tools Sqoop and WaterDrop to synchronize the analysis result data to the relational database Tidb and the distributed search engine ES respectively.
[0083] Step S609: Display the analysis result data by reading data from the relational database and the distributed search engine.
[0084] For the above method, see Figure 7 The present application also provides a device for analyzing corporate business and judicial data, the device comprising:
[0085] The collection module 701 is used to collect corporate industrial and commercial judicial data and push it to a distributed message queue.
[0086] The consumption module 702 is used to consume the enterprise industrial and commercial judicial data in the distributed message queue in real time through the Spark streaming computing framework, and parse the enterprise industrial and commercial judicial data into a fixed format and store it in the big data warehouse.
[0087] The processing module 703 is used to process the enterprise industrial and commercial judicial data in the big data warehouse through the Spark engine, and store the processed enterprise industrial and commercial judicial data in the big data warehouse.
[0088] |Analysis module 704 is used to perform model analysis on the enterprise industrial and commercial judicial data through a data model, generate analysis result data and store the analysis result data in the big data warehouse.
[0089] |The embodiments of the present application can analyze the business judicial data of enterprises, thereby realizing the effective management and efficient use of the business judicial data of enterprises.
[0090] Electronic devices
[0091] In this embodiment, an electronic device 800 is provided, such as Figure 8 As shown, the electronic device 800 may include: a processor 801, a communication interface 802, a memory 803, and a communication bus 804. Among them:
[0092] The processor 801 , the communication interface 802 , and the memory 803 communicate with each other via the communication bus 804 .
[0093] The communication interface 802 is used to communicate with other electronic devices or servers.
[0094] The processor 801 is used to execute the program 805, and specifically can execute the relevant steps in the above-mentioned motion control method embodiment.
[0095] Specifically, the program 805 may include program codes, which include computer operation instructions.
[0096] The processor 301 may be a CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.
[0097] The memory 803 is used to store the program 805. The memory 803 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
[0098] The program 805 can be specifically used to enable the processor 801 to execute the above method in the above embodiment.
[0099] The specific implementation of each step in program 805 can refer to the corresponding description of the corresponding steps and units in the aforementioned data processing method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the aforementioned method embodiment, which will not be repeated here.
[0100] | Computer storage media
[0101] In this embodiment, a computer-readable storage medium is provided, which stores instructions for causing a machine to execute the data processing method as described herein. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code for implementing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device is enabled to read and execute the program code stored in the storage medium.
[0102] |In this case, the program code read from the storage medium itself can implement the functions of any one of the above embodiments, so the program code and the storage medium storing the program code constitute part of the present application.
[0103] |Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code may be downloaded from a server computer via a communication network.
[0104] Computer program product
[0105] In this embodiment, a computer program product is provided, including computer instructions, which instruct a computing device to perform any corresponding operation in the above-mentioned multiple method embodiments.
[0106] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.
[0107] |The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.
[0108] It should be noted that in the description of this application, the terms "first" and "second" are only used to facilitate the description of different components or names, and cannot be understood as indicating or implying a sequential relationship, relative importance, or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.
[0109] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.
[0110] It should be noted that although the specific embodiments of the present application are described in detail in conjunction with the accompanying drawings, this should not be construed as limiting the scope of protection of the present application. Within the scope described in the claims, various modifications and variations that can be made by those skilled in the art without creative work still fall within the scope of protection of the present application.
[0111] The examples of the embodiments of the present application are intended to concisely illustrate the technical features of the embodiments of the present application so that those skilled in the art can intuitively understand the technical features of the embodiments of the present application, and are not intended to be improper limitations of the embodiments of the present application.
[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for analyzing corporate industrial and commercial judicial data, characterized in that: The method comprises: Collect corporate industrial and commercial judicial data and push it to the distributed message queue; Consume the enterprise industrial and commercial judicial data in the distributed message queue in real time through the Spark streaming computing framework, and parse the enterprise industrial and commercial judicial data into a fixed format and store it in a big data warehouse; Processing the enterprise industrial and commercial judicial data in the big data warehouse through the Spark engine, and storing the processed enterprise industrial and commercial judicial data in the big data warehouse; The enterprise's industrial and commercial judicial data is modeled and analyzed through a data model to form analysis result data and store the analysis result data in the big data warehouse.
2. The method according to claim 1, characterized in that The method further comprises: The analysis result data is synchronized to the relational database and the distributed search engine through a data synchronization tool.
3. The method according to claim 2, characterized in that The method further comprises: The analysis result data is displayed by reading data from the relational database and the distributed search engine.
4. The method according to claim 1, characterized in that: The processing of the enterprise industrial and commercial judicial data in the big data warehouse by the Spark engine and storing the processed enterprise industrial and commercial judicial data in the big data warehouse includes: Cleaning and filtering the enterprise industrial and commercial judicial data in the big data warehouse through the Spark engine, and storing the cleaned and filtered data in the big data warehouse; The cleaned and filtered data is aggregated by the Spark engine, and the aggregated data is stored in the big data warehouse.
5. The method according to claim 4, characterized in that The big data warehouse includes: Operational data storage layer, data warehouse detail layer, data warehouse summary layer, application data storage layer; The Spark stream computing framework consumes the enterprise industrial and commercial judicial data in the distributed message queue in real time, and parses the enterprise industrial and commercial judicial data into a fixed format and stores it in the operational data storage layer of the big data warehouse; The Spark engine cleans and filters the enterprise industrial and commercial judicial data in the big data warehouse, and stores the cleaned and filtered data in the data warehouse detail layer of the big data warehouse; The Spark engine aggregates the cleaned and filtered data, and stores the aggregated data in the data warehouse summary layer of the big data warehouse; The trained data model is used to perform model analysis on the enterprise's industrial and commercial judicial data to generate analysis result data and store the analysis result data in the application data storage layer of the big data warehouse.
6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: performing model analysis on the enterprise's industrial and commercial judicial data through the data model to form analysis result data and storing the analysis result data in the big data warehouse, and the method further comprises: The aggregated data of the data warehouse summary layer of the big data warehouse is converted into features that can be used by the machine learning algorithm, and a preset machine learning algorithm is selected to use the features for training to obtain the data model.
7. An enterprise industrial and commercial judicial data analysis device, characterized in that: The device comprises: The collection module is used to collect corporate industrial and commercial judicial data and push it to the distributed message queue; A consumption module, used for consuming the enterprise industrial and commercial judicial data in the distributed message queue in real time through the Spark streaming computing framework, and parsing the enterprise industrial and commercial judicial data into a fixed format and storing it in a big data warehouse; A processing module, used for processing the enterprise industrial and commercial judicial data in the big data warehouse through the Spark engine, and storing the processed enterprise industrial and commercial judicial data in the big data warehouse; The analysis module is used to perform model analysis on the enterprise's industrial and commercial judicial data through a data model, generate analysis result data and store the analysis result data in the big data warehouse.
8. An electronic device, comprising: A processor, a communication interface, a memory and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute an operation corresponding to the method according to any one of claims 1 to 6.
9. A computer storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method according to any one of claims 1 to 6.
10. A computer program product, comprising computer instructions, wherein the computer instructions instruct a computing device to execute operations corresponding to the method according to any one of claims 1 to 6.