Bridge data access and processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a bridge data access and processing method and device, electronic equipment and a storage medium, and belongs to the technical field of data processing.The bridge data access and processing method comprises the steps that target data corresponding to multiple bridges are obtained based on a data bus, the target data comprises temperature, humidity and pressure data collected by different data sensors on the single bridge; performing data stream conversion on the target data corresponding to the multiple bridges, and performing data cleaning and preprocessing on the target data corresponding to the multiple bridges after data stream conversion to obtain a target data set; the target data set is written into a monitoring database, monitoring results of the multiple bridges are determined based on the monitoring database, and the monitoring database determines the monitoring results of the multiple bridges based on edge calculation. According to the invention, while data access and processing of multiple bridges are realized, the complexity of data processing is reduced, the data processing efficiency is improved, and the real-time performance of data display is further improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for accessing and processing bridge data. Background Art

[0002] As the number of bridges connected to the platform gradually increases, the amount of access data and data processing requirements of the existing bridge group platform health monitoring platform are growing exponentially compared to a single bridge. The existing technology of obtaining data statistics from the data warehouse, then storing it locally and performing data batch processing can no longer meet the current bridge data processing needs.

[0003] At present, the single bridge realizes regular batch transmission of data by scheduling and coordinating various batch computing tools and sending them to the target regularly. This solution can only process data sets of limited size. The system data processing cycle is long, which increases the time loss and hardware storage in the data calculation process, resulting in problems such as feedback lag and high latency in the real-time display of data. Summary of the invention

[0004] In view of this, it is necessary to provide a bridge data access and processing method, device, electronic device and storage medium to solve the problem of long data processing cycle in the current bridge data access and processing solution.

[0005] In order to solve the above problems, the present invention provides a method for accessing and processing bridge data, comprising: Based on the data bus, target data corresponding to the multiple bridges are obtained, wherein the target data includes temperature, humidity and pressure data collected by different data sensors on a single bridge; Performing data stream conversion on the target data corresponding to the multiple bridges, and performing data cleaning and preprocessing on the target data corresponding to the multiple bridges after the data stream conversion to obtain a target data set; The target data set is written into a monitoring database, and monitoring results of multiple bridges are determined based on the monitoring database, wherein the monitoring database determines the monitoring results of multiple bridges based on edge computing.

[0006] In a possible implementation, performing data stream conversion on target data corresponding to the plurality of bridges respectively includes: The offline data in the target data corresponding to the multiple bridges are converted into a bounded data stream, and the real-time data in the target data corresponding to the multiple bridges are converted into an unbounded data stream.

[0007] In a possible implementation, the bounded data stream is bounded by the collection time of the offline data.

[0008] In a possible implementation, the data cleaning of the target data corresponding to the plurality of bridges after the data stream conversion includes: The missing values ​​and blank values ​​in the target data corresponding to the multiple bridges after the data stream conversion are eliminated, and data validity and data integrity tests are performed.

[0009] In a possible implementation, the preprocessing of the target data corresponding to the plurality of bridges after the data stream conversion includes: The target data corresponding to the multiple bridges after data stream conversion are normalized.

[0010] In a possible implementation manner, writing the target data set into a monitoring database includes: The target data set is written into the monitoring database based on the Sink operation in Apache Flink.

[0011] In a possible implementation, determining the monitoring results of the plurality of bridges based on the monitoring database includes: Data is classified on the monitoring database to obtain monitoring results of multiple bridges, and the monitoring results of multiple bridges are displayed in a visual form.

[0012] The present invention also provides a bridge data access and processing device, comprising: An access module is used to obtain target data corresponding to multiple bridges based on a data bus, wherein the target data includes temperature, humidity and pressure data collected by different data sensors on a single bridge; A processing module is used to perform data stream conversion on the target data corresponding to the multiple bridges, and to perform data cleaning and preprocessing on the target data corresponding to the multiple bridges after the data stream conversion, so as to obtain a target data set; A determination module is used to write the target data set into a monitoring database and determine the monitoring results of multiple bridges based on the monitoring database, wherein the monitoring database determines the monitoring results of multiple bridges based on edge computing.

[0013] The present invention also provides an electronic device, including a memory and a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the bridge data access and processing method described above is implemented.

[0014] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the bridge data access and processing method described above is implemented.

[0015] The beneficial effects of the present invention are as follows: the bridge data access and processing method, device, electronic device and storage medium provided by the present invention first obtain data collected by different data sensors on multiple bridges through a data bus to improve data processing efficiency, then reduce the complexity of the data processing process by performing data stream conversion, data cleaning and preprocessing on the target data to obtain a target data set, and finally write the target data set into a monitoring database, and obtain monitoring results of multiple bridges based on the monitoring database for display. While realizing data access and processing of multiple bridges, the present invention reduces the complexity of data processing, improves data processing efficiency, and thereby improves the real-time nature of data display. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a flow chart of an embodiment of a method for accessing and processing bridge data provided by the present invention; Figure 2 A flowchart of an embodiment of the Apache Flink framework application process provided by the present invention; Figure 3 A schematic diagram of a flow chart of an embodiment of a real-time display process of bridge data provided by the present invention; Figure 4 A schematic diagram of the structure of an embodiment of a bridge data access and processing device provided by the present invention; Figure 5 A schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0018] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0019] In the description of the present invention, reference to an "embodiment" means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the described embodiments may be combined with other embodiments.

[0020] As the number of bridges connected to the platform gradually increases, the amount of access data and data processing requirements of the existing bridge group platform health monitoring platform are growing exponentially compared to a single bridge. The existing technology of obtaining data statistics from the data warehouse, then storing it locally and performing data batch processing can no longer meet the current bridge data processing needs.

[0021] At present, the single bridge realizes regular batch transmission of data by scheduling and coordinating various batch computing tools and sending them to the target regularly. This solution can only process data sets of limited size. The system data processing cycle is long, which increases the time loss and hardware storage in the data calculation process, resulting in problems such as feedback lag and high latency in the real-time display of data.

[0022] In order to solve the above problems, the present invention provides a method for accessing and processing bridge data.

[0023] The specific embodiments are described in detail below: A specific embodiment of the present invention discloses a method for accessing and processing bridge data, combining Figure 1 Come and see, Figure 1 A schematic flow chart of an embodiment of a method for accessing and processing bridge data provided by the present invention includes steps S101 to S103, wherein: In step S101, based on the data bus, target data corresponding to multiple bridges are obtained, wherein the target data includes temperature, humidity and pressure data collected by different data sensors on a single bridge; In step S102, data stream conversion is performed on the target data corresponding to the plurality of bridges, and data cleaning and preprocessing are performed on the target data corresponding to the plurality of bridges after the data stream conversion to obtain a target data set; In step S103, the target data set is written into a monitoring database, and monitoring results of multiple bridges are determined based on the monitoring database, where the monitoring database determines the monitoring results of multiple bridges based on edge computing.

[0024] During implementation, the data collected by different data sensors on multiple bridges can be obtained through the data bus, such as the data collected by temperature and humidity meters, infrared thermometers, pressure gauges and other data sensors on the bridges. By accessing the target data corresponding to a single bridge through the data bus, the networking of multiple bridges can be completed quickly, improving data processing efficiency.

[0025] After completing the acquisition of the target data corresponding to the multiple bridges, the target data corresponding to the multiple bridges can be first converted into data streams, and then the target data corresponding to the multiple bridges after the data stream conversion can be cleaned and preprocessed to obtain the target data set, thereby reducing the complexity of the data processing process.

[0026] Finally, the target data set can be written into the monitoring database through the data writing operation, and the monitoring results of multiple bridges can be determined according to the monitoring database, which is convenient for the display of bridge monitoring results. The monitoring database can determine the monitoring results of multiple bridges through edge computing technology to improve the efficiency of data processing.

[0027] The bridge data access and processing method provided by the present invention can be applied to data access and processing scenarios of multiple bridges, and can also be applied to access and processing scenarios of other multi-source heterogeneous data, and the present invention does not make specific limitations on this.

[0028] Compared with the prior art, the bridge data access and processing method provided in this embodiment first obtains data collected by different data sensors on multiple bridges through a data bus to improve data processing efficiency, then reduces the complexity of the data processing process by performing data stream conversion, data cleaning and preprocessing on the target data to obtain a target data set, and finally writes the target data set into a monitoring database. Based on the monitoring database, monitoring results of multiple bridges are obtained for display. While realizing data access and processing of multiple bridges, the present invention reduces the complexity of data processing, improves data processing efficiency, and thereby improves the real-time performance of data display.

[0029] Exemplarily, the data stream conversion of the target data corresponding to the plurality of bridges includes: The offline data in the target data corresponding to the multiple bridges are converted into a bounded data stream, and the real-time data in the target data corresponding to the multiple bridges are converted into an unbounded data stream.

[0030] Specifically, when performing data stream conversion on target data corresponding to multiple bridges, offline data can be converted into bounded data streams, and real-time data can be converted into unbounded data streams, thereby further improving the efficiency of data processing.

[0031] Exemplarily, in a possible implementation, the bounded data stream is bounded by the collection time of the offline data.

[0032] Specifically, when converting offline data into a bounded data stream, the collection time of the offline data can be used as a boundary to convert the offline data into a bounded data stream, thereby improving the processing efficiency of the offline data.

[0033] Exemplarily, the data cleaning of the target data corresponding to the plurality of bridges after the data stream conversion includes: The missing values ​​and blank values ​​in the target data corresponding to the multiple bridges after the data stream conversion are eliminated, and data validity and data integrity tests are performed.

[0034] Specifically, when performing data cleaning on the target data corresponding to multiple bridges after data stream conversion, the missing values ​​and blank values ​​in the target data can be removed first to avoid the increase in calculation complexity caused by missing values ​​and blank values. Then, data validity detection and data integrity detection can be performed to filter out the noise in the data, thereby further reducing the complexity of data processing.

[0035] Exemplarily, the preprocessing of the target data corresponding to the plurality of bridges after the data stream conversion includes: The target data corresponding to the multiple bridges after data stream conversion are normalized.

[0036] Specifically, in order to further improve the data processing efficiency, when pre-processing the target data corresponding to the multiple bridges after the data stream conversion, the target data corresponding to the multiple bridges after the data stream conversion can be normalized, such as maximum and minimum normalization, logarithmic transformation normalization, square root normalization, etc.

[0037] Exemplarily, writing the target data set into a monitoring database includes: The target data set is written into the monitoring database based on the Sink operation in Apache Flink.

[0038] Specifically, when writing the target data set into the monitoring database, the target data set can be written into the monitoring database through the Sink operation in Apache Flink, and the monitoring database can be set in the bridge data monitoring platform or the bridge data monitoring workstation.

[0039] Exemplarily, determining the monitoring results of the plurality of bridges based on the monitoring database includes: Data is classified on the monitoring database to obtain monitoring results of multiple bridges, and the monitoring results of multiple bridges are displayed in a visual form.

[0040] Specifically, when determining the monitoring results of multiple bridges according to the monitoring database, the data in the monitoring database can be classified, for example, according to the data type and the label, to obtain the monitoring results of the multiple bridges. After obtaining the monitoring results of the multiple bridges, the monitoring results of the multiple bridges can also be displayed in a visual form.

[0041] The technical solution of the present invention is better described below with reference to a specific embodiment: The present invention provides a bridge data access and processing method based on Apache Flink. The Apache Flink framework supports unified batch stream processing, simplifies the data processing process, enables data stream processing to reach the millisecond level, and supports horizontal and vertical expansion, so as to achieve the purpose of flexibly adjusting resource allocation according to business needs.

[0042] Apache Flink regards all data as streams. Offline data is a bounded stream, and real-time data is an unbounded stream. It uses the same application programming interface (API) to implement data task processing. The powerful real-time processing capability greatly improves the efficiency of data stream processing and analysis.

[0043] Combination Figure 2 Come and see, Figure 2 This is a flow chart of an embodiment of the Apache Flink framework application process provided by the present invention, and the process includes the following steps: 1. Single bridge access: single bridge equipment is connected to the system through the bus, which can realize single bridge fast and system networking.

[0044] 2. Data access: The data gateway receives multiple types of sensor data from different sources and formats in real time, schedules Apache Flink's data source API for unified batch and stream processing, and performs filter conversion operations to achieve bridge data cleaning and preprocessing.

[0045] 3. Data processing and analysis: Based on the stream processing characteristics of Apache Flink, event processing and state management functions are used to achieve more complex real-time data analysis tasks, such as real-time monitoring of bridges and real-time log analysis.

[0046] 4. Data output: Use Apache Flink's Sink operation to write data to the bridge group platform, display the result data to users, or view the real-time status and results of bridge data processing through monitoring tools.

[0047] Combination Figure 3 Come and see, Figure 3The present invention provides a flowchart of an embodiment of a real-time display process of bridge data. The process specifically includes: receiving heterogeneous data transmitted from the front end, using the data source API provided by Apache Flink to implement unified batch stream processing, and implementing conversion operations through filters, and then completing the real-time bridge data analysis task according to the stream processing characteristics. Finally, the data is written into the bridge group platform through the Sink operation, and the bridge data information is displayed in a three-dimensional visualization form, which is convenient for users to view at any time.

[0048] The embodiment of the present invention also provides a bridge data access and processing device, combined with Figure 4 Come and see, Figure 4 This is a schematic diagram of the structure of an embodiment of a bridge data access and processing device provided by the present invention. The bridge data access and processing device 400 includes: The access module 401 is used to obtain target data corresponding to multiple bridges based on the data bus, wherein the target data includes temperature, humidity and pressure data collected by different data sensors on a single bridge; The processing module 402 is used to perform data stream conversion on the target data corresponding to the plurality of bridges, and perform data cleaning and preprocessing on the target data corresponding to the plurality of bridges after the data stream conversion to obtain a target data set; The determination module 403 is used to write the target data set into a monitoring database and determine the monitoring results of multiple bridges based on the monitoring database, wherein the monitoring database determines the monitoring results of multiple bridges based on edge computing.

[0049] The specific implementation of each module of the bridge data access and processing device can refer to the description of the above-mentioned bridge data access and processing method, and has similar beneficial effects, which will not be repeated here.

[0050] The embodiment of the present invention further provides an electronic device, Figure 5 Come and see, Figure 5 This is a structural diagram of an embodiment of an electronic device provided by the present invention. The electronic device 500 includes a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the processor 501 executes the program, the access and processing method of the bridge data as described above is implemented.

[0051] As a preferred embodiment, the electronic device 500 further includes a display 503 for displaying the processor 501 executing the bridge data access and processing method as described above.

[0052] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the memory 502, and are executed by the processor 501 to complete the present invention. One or more modules / units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 500. For example, the computer program may be divided into the access module 401, the processing module 402, and the determination module 403 in the above embodiment. The specific functions of each module are as described above, and are not described one by one here.

[0053] The electronic device 500 may be a desktop computer, a notebook, a PDA or a smart phone with an adjustable camera module.

[0054] Among them, the processor 501 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor 501 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0055] The memory 502 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 502 is used to store programs, and the processor 501 executes the program after receiving an execution instruction. The process definition method disclosed in any of the above embodiments of the present invention may be applied to the processor 501, or implemented by the processor 501.

[0056] The display 503 may be an LCD display screen or an LED display screen, for example, a display screen on a mobile phone.

[0057] Understandably, Figure 5 The structure shown is only a schematic diagram of the structure of the electronic device 500. The electronic device 500 may also include Figure 5 More or fewer components as shown. Figure 5 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0058] The electronic device provided according to the above embodiment of the present invention can be implemented with reference to the specific description of the method for accessing and processing bridge data as described above according to the present invention, and has similar beneficial effects as the method for accessing and processing bridge data as described above, which will not be repeated here.

[0059] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned bridge data access and processing method is implemented.

[0060] Generally speaking, the computer instructions for implementing the method of the present invention can be carried by any combination of one or more computer-readable storage media. Non-transitory computer-readable storage media can include any computer-readable media except for the signal itself that is temporarily propagating.

[0061] Computer-readable storage media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0062] Computer program codes for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages, in particular, Python suitable for neural network computing and platform frameworks based on TensorFlow, PyTorch, etc. may be used. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0063] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0064] The present invention discloses a method, device, electronic device and storage medium for accessing and processing bridge data. Firstly, data collected by different data sensors on multiple bridges are acquired through a data bus to improve data processing efficiency. Then, the complexity of the data processing process is reduced by performing data stream conversion, data cleaning and preprocessing on the target data to obtain a target data set. Finally, the target data set is written into a monitoring database. Based on the monitoring database, monitoring results of multiple bridges are obtained for display. While realizing data access and processing of multiple bridges, the present invention reduces the complexity of data processing, improves data processing efficiency, and thereby improves the real-time performance of data display.

[0065] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for accessing and processing bridge data, characterized in that: include: Based on the data bus, target data corresponding to the multiple bridges are obtained, wherein the target data includes temperature, humidity and pressure data collected by different data sensors on a single bridge; Performing data stream conversion on the target data corresponding to the multiple bridges, and performing data cleaning and preprocessing on the target data corresponding to the multiple bridges after the data stream conversion to obtain a target data set; The target data set is written into a monitoring database, and monitoring results of multiple bridges are determined based on the monitoring database, wherein the monitoring database determines the monitoring results of multiple bridges based on edge computing.

2. The bridge data access and processing method according to claim 1 is characterized in that: The data stream conversion of the target data corresponding to the plurality of bridges includes: The offline data in the target data corresponding to the multiple bridges are converted into a bounded data stream, and the real-time data in the target data corresponding to the multiple bridges are converted into an unbounded data stream.

3. The bridge data access and processing method according to claim 2 is characterized in that: The bounded data stream is bounded by the collection time of the offline data.

4. The bridge data access and processing method according to claim 1 is characterized in that: The data cleaning of the target data corresponding to the plurality of bridges after the data stream conversion includes: The missing values ​​and blank values ​​in the target data corresponding to the multiple bridges after the data stream conversion are eliminated, and data validity and data integrity tests are performed.

5. The bridge data access and processing method according to claim 1 is characterized in that: The preprocessing of the target data corresponding to the plurality of bridges after the data stream conversion comprises: The target data corresponding to the multiple bridges after data stream conversion are normalized.

6. The bridge data access and processing method according to claim 1 is characterized in that: Writing the target data set into a monitoring database comprises: The target data set is written into the monitoring database based on the Sink operation in Apache Flink.

7. The bridge data access and processing method according to claim 1 is characterized in that: Determining the monitoring results of the plurality of bridges based on the monitoring database includes: Data is classified on the monitoring database to obtain monitoring results of multiple bridges, and the monitoring results of multiple bridges are displayed in a visual form.

8. A bridge data access and processing device, characterized in that: include: An access module is used to obtain target data corresponding to multiple bridges based on a data bus, wherein the target data includes temperature, humidity and pressure data collected by different data sensors on a single bridge; A processing module is used to perform data stream conversion on the target data corresponding to the multiple bridges, and to perform data cleaning and preprocessing on the target data corresponding to the multiple bridges after the data stream conversion, so as to obtain a target data set; A determination module is used to write the target data set into a monitoring database and determine the monitoring results of multiple bridges based on the monitoring database, wherein the monitoring database determines the monitoring results of multiple bridges based on edge computing.

9. An electronic device, characterized in that: It comprises a memory and a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for accessing and processing bridge data according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method for accessing and processing bridge data as described in any one of claims 1 to 7 is implemented.

Images