Big data enterprise environment public service system

Abstract

The embodiment of the invention relates to a big data enterprise environment public service system, and the system comprises a data collection module which is used for processing unstructured data through a webpage capturing algorithm and a multi-source AP I calling mechanism, and obtaining enterprise environment information; the data processing module is used for dynamically adjusting the task priority and the acquisition frequency of data acquisition by monitoring the change of a data source and a user query record in real time; the user interface module is used for receiving request information from a plurality of users and returning corresponding results according to the enterprise environment information; the request information comprises a Web browser, a mobile application, a desktop client and an AP I calling request; and the website function module is used for receiving the request information sent by the user interface module and carrying out certificate query, enterprise evaluation and point exchange according to the Web browser, the mobile application, the desktop client, the AP I calling request and the enterprise environment information.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a big data enterprise environment public service system. Background Art

[0002] In the existing technology, with the rapid development of industrialization, the demand for corporate environmental information from all walks of life is increasing day by day. The existing corporate environmental information service platform has many shortcomings, such as untimely information updates, single data source, lack of targeted services, etc. Summary of the invention

[0003] The purpose of the present invention is to provide a big data enterprise environment public service system to solve the problems existing in the prior art in view of the defects existing in the prior art.

[0004] To achieve the above objectives, the present invention provides a big data enterprise environment public service system, including:

[0005] The data collection module is used to process unstructured data and obtain enterprise environment information through web crawling algorithms and multi-source API call mechanisms;

[0006] The data processing module dynamically adjusts the task priority and frequency of data collection by real-time monitoring of changes in data sources and user query records;

[0007] A user interface module is used to receive request information from multiple users and return corresponding results according to the enterprise environment information; the request information includes a request through a Web browser, a mobile application, a desktop client, or an API call;

[0008] The website function module is used to receive the request information sent by the user interface module, and perform certificate query, enterprise evaluation and point redemption according to the web browser, mobile application, desktop client, API call request and the enterprise environment information.

[0009] In a possible implementation, the processing of unstructured data to obtain enterprise environment information specifically includes:

[0010] Obtain corporate information from corporate websites, public databases or social media using web crawler tools;

[0011] Obtain environmental monitoring data and corporate annual reports by calling the API interface provided by the company;

[0012] Get the file information uploaded by the user;

[0013] Preprocessing the acquired information; the preprocessing includes text cleaning, image processing, and audio-to-text processing;

[0014] Use NLP tools to identify key entities in the text and determine the positive or negative nature of the corporate environment information based on the sentiment of the text information; extract keywords and phrases from the text;

[0015] Use OCR technology to extract text information from images;

[0016] Using a deep learning model to identify specific objects in an image, wherein the specific objects include pollution sources and environmental protection facilities;

[0017] Extract table data from PDFs and images using table parsing tools;

[0018] Store the processed data in a relational database, create indexes for the data, and generate a big data index library.

[0019] In one possible implementation, the user interface module is also used to define different priority labels for the first type of data and the second type of data, and to sort the data before sending; the priority of the first type of data is higher than the priority of the second type of data; wherein the first type of data is highly real-time data; and the second type of data is data with low delay sensitivity.

[0020] In a possible implementation, the user interface module is further used to adjust the sending rate according to network feedback traffic and an adaptive traffic control algorithm; wherein the adaptive traffic control algorithm includes a slow start mechanism and a congestion avoidance mechanism of TCP.

[0021] In a possible implementation, the data processing module is also used to dynamically adjust the cache strategy according to the request frequency of user requests and the data update frequency; the cache strategy includes keeping high-frequency requested data in the cache and removing low-frequency requested data from the cache.

[0022] In a possible implementation, the data processing module is specifically used to:

[0023] Set up a message queue to monitor changes in the data source. When the data source is updated, receive the notification message sent by the message queue. Alternatively, configure Webhooks of the data source provider. When the data changes, receive the HTTP request notification sent by the data source provider. Alternatively, set up a scheduled task to regularly check the status of the data source and trigger the data collection task when there is a change.

[0024] Record each query request of the user, including the query time, query content, and query results; record the user's query history in the database, including query keywords and query frequency; push the user's query record to the message queue;

[0025] Dynamically adjust the collection frequency according to the frequency of change of the data source; data sources with high frequency of change have high priority and high collection frequency;

[0026] Dynamically adjust the priority of related data sources based on the keywords and frequency in user query records; data sources with high user query frequency have high priority and high collection frequency;

[0027] Train machine learning models to predict the changing trends of data sources and user query trends, and adjust task priorities and collection frequency based on the prediction results;

[0028] Preset a rule and dynamically adjust the task priority and collection frequency based on the rule.

[0029] In a possible implementation, the website function module includes manufacturer management and dealer management submodules for receiving manufacturer information and dealer information; and upon receiving a certificate query, generating a certificate confirmation message based on the manufacturer information and dealer information.

[0030] Points module, used to earn points through evaluation, sharing and other activities, and use the points to redeem goods or coupons;

[0031] Enterprise evaluation module: receiving evaluation information on the enterprise sent by the user terminal;

[0032] The social module receives green product information shared by user terminals.

[0033] By applying the big data enterprise environmental public service system provided by the embodiment of the present invention, a comprehensive, convenient and reliable enterprise environmental public service platform is provided. Through comprehensive and accurate enterprise environmental information management, the sustainable development of enterprises can be promoted, the ecological environment can be protected, and the overall environmental protection level of society can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of a big data enterprise environment public service system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments.

[0037] Figure 1 The following is a schematic diagram of the structure of a big data enterprise environment public service system provided by an embodiment of the present invention. Figure 1 , the technical solution of the present invention is described with specific embodiments. Figure 1 As shown, the big data enterprise environment public service system includes a data collection module 1, a data processing module 2, a user interface module 3 and a website function module 4.

[0038] The data collection module 1 is used to process unstructured data and obtain enterprise environment information through web crawling algorithms and multi-source API calling mechanisms.

[0039] Among them, enterprise environmental information refers to various information related to environmental protection generated by enterprises during the production and operation process. This information covers the impact of enterprises on the environment, environmental management measures, environmental performance and other aspects, and is an important basis for evaluating enterprises' environmental performance and sustainable development capabilities. Enterprise environmental information includes emission information, energy consumption information, resource utilization information, and environmental management information. Emission information includes air emissions, including waste gas emissions, types of pollutants (such as sulfur dioxide, nitrogen oxides, particulate matter, etc.) and their concentrations, water emissions, including wastewater emissions, types of pollutants (such as chemical oxygen demand, ammonia nitrogen, heavy metals, etc.) and their concentrations, solid waste emissions, including the amount of solid waste generated, types, and treatment methods (such as recycling, landfill, incineration, etc.). Energy consumption information includes electricity consumption, including enterprise electricity consumption and its sources (such as renewable energy, fossil fuels, etc.), fuel consumption, the amount of various fuels (such as coal, natural gas, diesel, etc.) used by enterprises and their uses, and heat energy consumption, including heat energy (such as steam, hot water, etc.) used by enterprises and their sources. Resource utilization information includes water resource utilization, including the amount of water used by the enterprise and its purpose (such as production water, domestic water, etc.), raw material utilization: the main types of raw materials used by the enterprise and their usage, and renewable resource utilization: the use of renewable resources (such as waste paper, waste plastics, etc.) by the enterprise. Environmental management information includes environmental management system: whether the enterprise has established an environmental management system (such as ISO 14001) and its operation. Environmental policy: including the environmental protection policy and goals of the enterprise. Environmental training: including the situation of employees receiving environmental training. Including emergency plans: the emergency plans and drills of the enterprise to deal with sudden environmental incidents. Environmental performance includes: Environmental performance indicators: environmental performance indicators set by the enterprise and their achievement, such as emission reduction targets, energy consumption targets, etc., Environmental audit: the results of environmental audits conducted by the enterprise and its rectification measures, Environmental report: the annual environmental report issued by the enterprise, including environmental performance, environmental management measures, etc.

[0040] Specifically, corporate information is obtained from corporate websites, public databases or social media through web crawler tools; for example, Scrapy, BeautifulSoup, etc. are used to capture relevant information from corporate websites, government public databases, social media and other sources.

[0041] Obtain environmental monitoring data and corporate annual reports by calling the API interface provided by the company;

[0042] Obtain the file information uploaded by the user; the file information includes but is not limited to PDF and Word;

[0043] Preprocess the acquired information; preprocessing includes text cleaning, image processing, and audio-to-text processing; data preprocessing includes text cleaning, removing HTML tags, special characters, stop words, etc., and retaining useful information. Image processing, using image processing libraries (such as OpenCV) to crop, scale, and denoise images. Audio-to-text: Use speech recognition technology, such as the Speech-to-Text API, to convert audio into text.

[0044] Use NLP tools to identify key entities in the text, and judge the positive or negative nature of the corporate environment information based on the sentiment of the text information; extract keywords and phrases in the text; among them, entity recognition is to use NLP tools (such as spaCy, Stanford NER) to identify key entities in the text, such as company names, dates, locations, etc. Sentiment analysis is to analyze the sentiment of the text and judge the positive or negative nature of the corporate environment information.

[0045] Use OCR technology to extract text information from images; for example, use Tesseract to extract text information from images;

[0046] Use deep learning models to identify specific objects in images, including pollution sources and environmental protection facilities;

[0047] Extract table data from PDFs and images using table parsing tools;

[0048] Convert the extracted data into a unified format, such as CSV, JSON, etc., unify the units, date formats, etc. in the data, ensure the consistency and comparability of the data, remove duplicate data records, and avoid redundancy.

[0049] Store the processed data in a relational database, create indexes for the data, and generate a big data index library.

[0050] Subsequently, you can calculate the statistics of various indicators, such as average, maximum, minimum, etc. Use time series analysis methods to analyze the changing trends of corporate environmental information. Use data visualization tools (such as Tableau, Power BI, Matplotlib) to display data in the form of charts to facilitate user understanding and analysis.

[0051] Data processing module 2 dynamically adjusts the task priority and frequency of data collection by real-time monitoring of changes in data sources and user query records.

[0052] Furthermore, the cache strategy is dynamically adjusted according to the request frequency of user requests and the update frequency of data; the cache strategy includes keeping the frequently requested data in the cache and removing the infrequently requested data from the cache.

[0053] The data processing module 2 is specifically used for:

[0054] Set up a message queue to monitor changes in the data source. When the data source is updated, receive the notification message sent by the message queue. Alternatively, configure Webhooks of the data source provider. When the data changes, receive the HTTP request notification sent by the data source provider. Alternatively, set up a scheduled task to regularly check the status of the data source and trigger the data collection task when there is a change.

[0055] Record each query request of the user, including query time, query content, and query results; record the user's query history in the database, including query keywords and query frequency; push the user's query record to the message queue;

[0056] Dynamically adjust the collection frequency according to the frequency of change of the data source; data sources with high frequency of change have high priority and high collection frequency;

[0057] Dynamically adjust the priority of related data sources based on the keywords and frequency in user query records; data sources with high user query frequency have high priority and high collection frequency;

[0058] Train machine learning models to predict the changing trends of data sources and user query trends, and adjust task priorities and collection frequency based on the prediction results;

[0059] Preset a rule and dynamically adjust the task priority and collection frequency according to the rule. For example, if a data source has changed many times in the past 24 hours and the user query frequency is high, increase the task priority and collection frequency of the data source.

[0060] User interface module 3, used to receive request information from multiple users and return corresponding results according to enterprise environment information; request information includes requests through Web browser, mobile application, desktop client, API call;

[0061] The request message may be a request message from a dealer terminal, an enterprise terminal or a public user terminal.

[0062] Specifically, the user interface module uses a binary format instead of a traditional text format (such as JSON). At the same time, a compression algorithm can be introduced to compress the message body to further reduce the amount of transmitted data, reduce the size of the data packet, and reduce the parsing overhead.

[0063] Furthermore, the user interface module is the basic architecture of a high-performance serialization / deserialization library, including core interfaces and data structures, which serializes any data object into a byte stream and then deserializes the byte stream into the original data object.

[0064] Defines commonly used serialization formats, such as binary format, JSON, Protocol Buffers, etc. Supports the extension of custom data types. Use SIMD instruction sets to optimize performance. Modern processors support SIMD (Single Instruction Multiple Data) instruction sets, which can process multiple data points in a single instruction, thereby greatly improving performance. Common SIMD instruction sets include Intel's SSE, AVX, and ARM's NEON.

[0065] Furthermore, by referring to the multiplexing mechanism of HTTP / 2, multiple data streams are parallelized on one TCP connection. This can effectively avoid the "head of line blocking" problem and improve the concurrent processing capability. Therefore, this application can transmit multiple requests or responses on one connection at the same time, reducing the time cost of establishing and disconnecting connections.

[0066] Specifically, each request or response is treated as an independent data stream. The data stream is divided into smaller units, called frames. Frame header: contains information such as frame type, length, and stream identifier. Frame data: actual data content. Reduce the amount of header information transmitted and improve transmission efficiency. The client and the server establish a TCP connection. Stream identifier: Each data stream has a unique identifier. Priority can be set for the data stream to ensure that critical data is transmitted first. The transmission rate of the data stream is controlled through the window update frame to avoid congestion.

[0067] In one example, the session is initialized first. The client sends an initial frame (such as setting the maximum frame size, the maximum number of concurrent streams, etc.), and the server replies with a confirmation frame. The client and server can send multiple data streams concurrently, and each data stream consists of multiple frames. Handling errors in a single data stream does not affect other data streams. Handling errors in the entire connection may require reestablishing the connection.

[0068] Furthermore, the user interface module 3 is also used to define different priority labels for the first type of data and the second type of data, and perform sorting before sending; the priority of the first type of data is higher than the priority of the second type of data; wherein the first type of data is real-time data; and the second type of data is delay-sensitive data.

[0069] Furthermore, the user interface module 3 is also used to adjust the sending rate according to the network feedback flow and the adaptive flow control algorithm; wherein the adaptive flow control algorithm includes the slow start mechanism and the congestion avoidance mechanism of TCP.

[0070] Among them, the slow start mechanism is: in the initial stage, the sending rate is increased rapidly until network congestion is detected. The congestion avoidance mechanism is: after detecting network congestion, the sending rate is gradually increased to avoid congestion again. The fast retransmission mechanism is: when three repeated ACKs are received, the lost data segment is immediately retransmitted. The fast recovery mechanism is: after fast retransmission, the sending rate is gradually increased to return to the level before congestion. In an example, the parameters are initialized first, the initial congestion window (Congestion Window, CWND): the initial value is set to 1 MSS (Maximum Segment Size). Initialize the slow start threshold (Slow Start Threshold, SSTHRESH): the initial value is set to a larger value, such as 64K. Initialize the receive window (Receiver Window, RWND): the maximum receive window of the receiver.

[0071] For the slow start mechanism, the congestion window is increased by 1 MSS for each ACK received. Check the congestion window: If the congestion window is greater than the slow start threshold, enter the congestion avoidance phase.

[0072] For the congestion avoidance mechanism, increase the congestion window: for each ACK received, increase the congestion window by 1 / CWND MSS. Check the congestion window: if the congestion window reaches the receiving window or the network congestion flag is triggered, reduce the congestion window.

[0073] For the fast retransmit mechanism, when three duplicate ACKs are received, the lost data segments are retransmitted immediately.

[0074] For the fast recovery mechanism, the slow start threshold is set to half of the congestion window, and then the congestion window is set to the slow start threshold and the congestion window continues to increase.

[0075] Then build a prediction model. Regularly collect network status information, such as RTT (Round-Trip Time), packet loss rate, bandwidth, etc. Use machine learning models (such as linear regression, decision tree, neural network, etc.) to predict future network status. According to the prediction results, adjust the sending rate in advance to avoid congestion.

[0076] The website function module 4 is used to receive the request information sent by the user interface module, and perform certificate query, enterprise evaluation and point redemption according to the Web browser, mobile application, desktop client, API call request and enterprise environment information.

[0077] The website function module 4 includes production enterprise management and dealer management submodules, which are used to receive the information of the production enterprise and the information of the dealer; and when receiving the certificate query, generate a certificate confirmation message according to the enterprise information and the dealer information.

[0078] Points module, used to earn points through evaluation, sharing and other activities, and use the points to redeem goods or coupons;

[0079] Enterprise evaluation module: receiving evaluation information on the enterprise sent by the user terminal;

[0080] The social module receives green product information shared by user terminals.

[0081] By applying the big data enterprise environmental public service system provided by the embodiment of the present invention, a comprehensive, convenient and reliable enterprise environmental public service platform is provided. Through comprehensive and accurate enterprise environmental information management, the sustainable development of enterprises can be promoted, the ecological environment can be protected, and the overall environmental protection level of society can be improved.

[0082] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0083] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0084] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A big data enterprise environment public service system, characterized in that: The system comprises: The data collection module is used to process unstructured data and obtain enterprise environment information through web crawling algorithms and multi-source API call mechanisms; The data processing module dynamically adjusts the task priority and frequency of data collection by real-time monitoring of changes in data sources and user query records; A user interface module is used to receive request information from multiple users and return corresponding results according to the enterprise environment information; the request information includes a request through a Web browser, a mobile application, a desktop client, or an API call; The website function module is used to receive the request information sent by the user interface module, and perform certificate query, enterprise evaluation and point redemption according to the web browser, mobile application, desktop client, API call request and the enterprise environment information.

2. The method according to claim 1, characterized in that: The processing of unstructured data to obtain enterprise environment information specifically includes: Obtain corporate information from corporate websites, public databases or social media using web crawler tools; Obtain environmental monitoring data and corporate annual reports by calling the API interface provided by the company; Get the file information uploaded by the user; Preprocessing the acquired information; the preprocessing includes text cleaning, image processing, and audio-to-text processing; Use NLP tools to identify key entities in the text and determine the positive or negative nature of the corporate environment information based on the sentiment of the text information; extract keywords and phrases from the text; Use OCR technology to extract text information from images; Using a deep learning model to identify specific objects in an image, wherein the specific objects include pollution sources and environmental protection facilities; Extract table data from PDFs and images using table parsing tools; Store the processed data in a relational database, create indexes for the data, and generate a big data index library.

3. The system according to claim 1, characterized in that The user interface module is also used to define different priority tags for the first type of data and the second type of data, and to sort the data before sending; the priority of the first type of data is higher than the priority of the second type of data; wherein the first type of data is real-time data; and the second type of data is delay-insensitive data.

4. The system according to claim 1, characterized in that The user interface module is also used to adjust the sending rate according to the network feedback flow and the adaptive flow control algorithm; wherein the adaptive flow control algorithm includes the slow start mechanism and the congestion avoidance mechanism of TCP.

5. The system according to claim 1, characterized in that The data processing module is also used to dynamically adjust the cache strategy according to the request frequency of user requests and the update frequency of data; the cache strategy includes keeping the data with high frequency requests in the cache and removing the data with low frequency requests from the cache.

6. The system according to claim 5, characterized in that The data processing module is specifically used for: Set up a message queue to monitor changes in the data source. When the data source is updated, receive the notification message sent by the message queue. Alternatively, configure Webhooks of the data source provider. When the data changes, receive the HTTP request notification sent by the data source provider. Alternatively, set up a scheduled task to regularly check the status of the data source and trigger the data collection task when there is a change. Record each query request of the user, including the query time, query content, and query results; Record the user's query history in the database, the query history including query keywords and query frequency; push the user's query record to the message queue; Dynamically adjust the collection frequency according to the frequency of changes in the data source; Data sources with high frequency of change have higher priority and higher collection frequency; Dynamically adjust the priority of relevant data sources based on keywords and frequency in user query records; Data sources with high user query frequency have higher priority and higher collection frequency; Train machine learning models to predict the changing trends of data sources and user query trends, and adjust task priorities and collection frequency based on the prediction results; Preset a rule and dynamically adjust the task priority and collection frequency based on the rule.

7. The method according to claim 1, characterized in that The website function module includes production enterprise management and dealer management submodules, which are used to receive information about the production enterprise and dealers; and when receiving a certificate query, generate a certificate confirmation message based on the enterprise information and dealer information. Points module, used to earn points through evaluation, sharing and other activities, and use the points to redeem goods or coupons; Enterprise evaluation module; Receiving evaluation information on the enterprise sent by the user terminal; The social module receives green product information shared by user terminals.

Images