Data analysis system based on bird watching camera, electronic equipment and storage medium

Abstract

The invention discloses a data analysis system based on a bird watching camera, electronic equipment and a storage medium. The data analysis system comprises a data acquisition module used for acquiring a plurality of log data generated by a plurality of business databases; the analysis module is used for cleaning, fusing and calculating the plurality of log data; the scheduling module is used for adjusting the execution sequence of the scripts corresponding to the log data; the background module is used for providing analysis result query of the log data; and the front-end module is used for analyzing the behavior data of the bird watching crowd based on the plurality of log data and displaying a corresponding analysis result. All the modules of the data analysis system can work cooperatively, the efficiency and effect of the data analysis system are comprehensively improved, and then the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a data analysis system, electronic device and storage medium based on a bird watching camera. Background Art

[0002] Data, as a new production factor of enterprises, is integrating with other production factors to jointly create a series of new values. Data analysis systems, as an important means of processing data, have become indispensable in business operations.

[0003] The current data analysis system mostly has one system for each product, and operators need to log in to each system to enter the relevant system for corresponding data analysis. For business systems with multiple sets of products, the data analysis method of the current data analysis system greatly reduces the efficiency of data analysis, thus affecting the user experience. Summary of the invention

[0004] The present application proposes a data analysis system, electronic device and storage medium based on a bird watching camera to improve the above-mentioned problems.

[0005] In a first aspect, an embodiment of the present application provides a data analysis system based on a bird watching camera, the data analysis system comprising: a data acquisition module for collecting multiple log data generated by multiple business databases; an analysis module for cleaning, fusing, and calculating the multiple log data; a scheduling module for adjusting the execution order of the scripts corresponding to the multiple log data; a background module for providing analysis result queries for the multiple log data; and a front-end module for analyzing the behavioral data of bird watchers based on the multiple log data, and displaying the corresponding analysis results.

[0006] In some embodiments, the analysis module includes a source layer, a detail layer, an aggregation layer and an application layer: the source layer is used to store multiple log data generated by the multiple business databases; the detail layer is used to store the multiple log data after cleaning; the aggregation layer is used to perform fine-grained aggregation of target analysis indicators; the application layer is used to store the analysis results of the analysis module.

[0007] In some embodiments, the data collection module is further used to convert the plurality of log data into preset format types respectively and store them in a designated database.

[0008] In some embodiments, the background module includes an overview interface, various module interfaces and a current country equipment interface: the overview interface is used to determine the current data browsing range according to the product permissions corresponding to the product owned by the user and the parameters submitted by the front-end module, and return the corresponding data; the various module interfaces are used to query the specified database according to the instructions submitted by the front-end module, and return the query results to the front-end module; the current country equipment interface is used to query the results of the analysis module according to the instructions submitted by the front-end module, and return the query results to the front-end module for display.

[0009] In some embodiments, components used by the data collection module in the process of collecting log data include canal components, flinkcdc components or maxwell components.

[0010] In some embodiments, the platforms on which the data acquisition module runs tasks include the Yarn platform and the K8S platform.

[0011] In some embodiments, the multiple business databases need to enable database execution logs, configure corresponding log formats and log retention times, and configure corresponding account permissions. Each business database needs to configure corresponding connection parameters, filtering rules, and corresponding table name generation rules, wherein there is a corresponding relationship between the column data types of the tables corresponding to the multiple business databases, and there is a corresponding relationship between the corresponding table names.

[0012] In some embodiments, the scripts corresponding to the plurality of log data support deployment on a resource management platform, support configuration of connection modes for various business databases, and support flexible configuration of resources used by tasks.

[0013] In a second aspect, the present application provides an electronic device comprising one or more processors and a memory; one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the system of the first aspect above.

[0014] In a third aspect, the present application provides a computer-readable storage medium having program code stored therein, wherein the system of the first aspect is executed when the program code is run.

[0015] The present application provides a data analysis system, electronic device and storage medium based on a bird watching camera. The data analysis system includes: a data acquisition module for collecting multiple log data generated by multiple business databases; an analysis module for cleaning, fusing and calculating the multiple log data; a scheduling module for adjusting the execution order of the scripts corresponding to the multiple log data; a background module for providing analysis result query of the multiple log data; and a front-end module for analyzing the behavior data of the bird watching crowd based on the multiple log data and displaying the corresponding analysis results. The various modules of the data analysis system can work together to comprehensively improve the efficiency and effect of the data analysis system, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A structural block diagram of a data analysis system based on a bird watching camera provided in an embodiment of the present application is shown.

[0018] Figure 2 Shows Figure 1 The specific structural block diagram of the analysis module in .

[0019] Figure 3 A structural block diagram of an electronic device provided in an embodiment of the present application is shown.

[0020] Figure 4 A storage unit for storing or carrying program codes for implementing a data analysis system based on a bird watching camera according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0022] Data, as a new production factor of enterprises, is integrating with other production factors to jointly create a series of new values. Data analysis systems, as an important means of processing data, have become indispensable in business operations.

[0023] However, the inventors found that the current data analysis system mostly has one system for each product, and operators need to log in to each system to enter the relevant system for corresponding data analysis. For business systems with multiple sets of products, the data analysis method of the current data analysis system greatly reduces the data analysis efficiency, thereby affecting the user experience.

[0024] After long-term research, the inventors found that the various modules of the data analysis system can work together to comprehensively improve the efficiency and effectiveness of the data analysis system, thereby improving the user experience. The data analysis system includes: a data acquisition module for collecting multiple log data generated by multiple business databases; an analysis module for cleaning, fusing, and calculating the multiple log data; a scheduling module for adjusting the execution order of the scripts corresponding to the multiple log data; a background module for providing analysis result queries for the multiple log data; and a front-end module for analyzing the behavior data of bird watchers based on the multiple log data and displaying the corresponding analysis results.

[0025] Therefore, in order to improve the above-mentioned problems, the inventor proposed the data analysis system, electronic device and computer-readable storage medium based on the bird watching camera provided in the present application, which can improve the efficiency and effect of the data analysis system and thus improve the user experience.

[0026] In order to facilitate a better understanding of the solution described in the embodiments of the present application, the following briefly describes the terms involved in the description of the embodiments of the present application:

[0027] Canal component: Canal is an incremental subscription & consumption component based on MySQL database binlog.

[0028] Flink CDC component: Replace the Debezium+Kafka data collection module with the Flink CDC Connector to achieve the integration of Flink Sql collection + computing + transmission (ETL).

[0029] Maxwell component: binlog parser, reads MySQL binlog, and synchronizes data to Kafka.

[0030] Yarn platform: Apache Hadoop YARN (Yet Another Resource Negotiator) is a new Hadoop resource manager. It is a general resource management system that can provide unified resource management and scheduling for upper-level applications. Its introduction has brought huge benefits to the cluster in terms of utilization, unified resource management and data sharing.

[0031] K8S platform: kubernetes, or K8s for short, is an abbreviation that uses 8 to replace the eight characters in the middle of the name "ubernete". It is an open source containerized application used to manage multiple hosts in a cloud platform.

[0032] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0033] See also Figure 1 , shows a structural block diagram of a data analysis system based on a bird watching camera provided by an embodiment of the present application. This embodiment provides a data analysis system based on a bird watching camera, which can be run on an electronic device, which can be a mobile phone, a computer, a tablet or a wearable electronic device. The type of specific electronic device may not be limited. Figure 1 As shown, the data analysis system may include a data acquisition module, an analysis module, a scheduling module, a background module and a front-end module, wherein the data acquisition module 01 is connected to the analysis module 02 in communication, the scheduling module 05 is connected to the analysis module 02 in communication, and the background module 03 and the front-end module 04 are both connected to the analysis module 02 in communication.

[0034] The data collection module is used to collect multiple log data generated by multiple business databases. Specifically, it can be used to collect the execution logs of each business database in the Flink CDC task mode.

[0035] The analysis module is used to clean, fuse and calculate the multiple log data.

[0036] The scheduling module is used to adjust the execution order of the scripts corresponding to the multiple log data.

[0037] The backend module is used to provide analysis result query of the plurality of log data. Specifically, it can be used to provide overview data query and various dimension analysis data query.

[0038] The front-end module is used to analyze the behavior data of the bird-watching crowd based on the multiple log data and display the corresponding analysis results.

[0039] As an implementation method, the execution log of the database can be enabled in advance in each business database. In this way, the data acquisition module can be used to parse the above execution log using a database change capture tool, and add two columns to the parsed execution log data, namely, data_ms (which can be understood as the time when the data was generated) for recording data generation, and a column in which data may be deleted (delete_enable).

[0040] Furthermore, after the execution log data is parsed, the data acquisition module can be used to convert the multiple log data into preset format types respectively, and store them in a designated database. Among them, the preset format type can be the format type required by the user, the specific type can be not limited, and the preset format type is a data type that is compatible with the designated database. Exemplarily, the data acquisition module can be used to convert the multiple log data into preset format types respectively, and then update the log data to the analytical database (i.e., the designated database) through the connection protocol. Specifically, the log data can be saved to the analytical database by updating if new log data exists and inserting if it does not exist.

[0041] In one embodiment, Figure 1 As shown, the data collection module 01 can collect the execution logs generated by each node of each business database in a plug-in manner, so as to read data from each business database without affecting normal business, and realize that changes in the structure of the source database have no effect on the analysis module. The entire data collection work only needs to set the corresponding configuration in each database. Based on this, the analysis module 02 can track the data changes of the source database in real time and save the changes in the analytical database.

[0042] Optionally, the data collection module can also support real-time synchronization of data and source table structures, and can reasonably set reasonable resource usage based on the size of the synchronization task.

[0043] Optionally, the components used by the data acquisition module in the process of collecting log data may include canal components, flinkcdc components or maxwell components. The data acquisition module can achieve flexibility in data acquisition by extending the plug-in that supports capturing data changes. Exemplarily, the data acquisition module returns the collected log data through Flink CDC to achieve real-time data synchronization. Compared with the traditional mode of using kafka() for data transmission, it greatly simplifies the process, avoids waste of resources, and thus improves the flexibility of data acquisition.

[0044] In a specific application scenario, during the log data collection process, the Flink CDC plug-in uses FlinkCDC, and the underlying Maxwell is used to parse the binlog logs generated by MySQL. The parsed logs include database structure change logs and data change logs. Then, the logs are parsed separately to convert the log data type of the original database into a data type compatible with the specified database, so that real-time data collection can be achieved.

[0045] Among them, by adopting Flink CDC, it can be deployed using resource management platforms such as yarn and kubernetes, and resource isolation is achieved between tasks when a large number of tasks are executed. Therefore, even if one task has an exception, other tasks can be guaranteed to run normally.

[0046] In addition, by adopting Flink CDC, the whole database can be synchronized. Specifically, first, compared with SQOOP and Flink SQL, each table must establish a synchronization task, which greatly reduces the amount of synchronization code to be written; second, FlinkCDC also supports initializing data before synchronizing data, which simplifies the data synchronization process compared to traditional methods; third, in the past, if data synchronization was achieved, it was necessary to maintain the data structure and reinitialize the data every time the data structure changed. However, with Flink CDC, there is no need to reinitialize the data, because when the data structure changes, Flink CDC can automatically suspend data synchronization, modify the specified database structure first, and then automatically synchronize the data. The entire process does not require manual intervention.

[0047] Optionally, in a specific implementation, taking the data type of the collected log data as Mysqlbinlog log as an example, after the data collection module collects the log data, the Mysql binlog log can be converted into the Apache doris data type, and the database-related interface can be used to save the data in batches. The background module can directly read the Apache doris data using springboot+Mybatis plus.

[0048] Optionally, the platform on which the data collection module runs the task may include a Yarn platform and a K8S platform.

[0049] In the implementation manner of the present application, the multiple business databases need to enable the execution log of the database, configure the corresponding log format and log retention time, and configure the corresponding account permissions. Each business database needs to configure the corresponding connection parameters, filtering rules and corresponding table name generation rules, wherein the column data types of the tables corresponding to the multiple business databases correspond to each other, and the corresponding table names correspond to each other.

[0050] Optionally, the scripts corresponding to the multiple log data support deployment on a resource management platform, support configuration of connection methods for various business databases, and support flexible configuration of resources used by tasks.

[0051] This implementation method can effectively eliminate the impact of synchronized log data on the business database by only reading the execution log of the database instead of directly reading the data. At the same time, by synchronizing data to the analysis module through Flink CDC, the development workload of most data synchronization scripts can be reduced. Secondly, Flink CDC supports automatic synchronization of table structures, eliminating the maintenance cost of synchronization scripts. Finally, because Flink CDC uses the Yarn platform for resource scheduling, resource utilization is improved.

[0052] In the implementation manner of the present application, the analysis module is used to store and calculate all data, and is also used to provide it to the background module for query.

[0053] In one implementation, see Figure 2 The analysis module may specifically include a source layer, a detail layer, an aggregation layer and an application layer, wherein the source layer is used to store the multiple log data generated by the multiple business databases; the detail layer is used to store the multiple log data after cleaning; the aggregation layer is used to perform fine-grained aggregation of target analysis indicators; and the application layer is used to store the analysis results of the analysis module.

[0054] The source layer adds two system fields to the source database: one is the data time field, and the other is the logical deletion field. The detail layer can merge the same type of data from different products during data cleaning. The aggregation layer can aggregate data by dimension. The application layer can merge or calculate the data in the aggregation layer to obtain the corresponding results according to the indicators required by the front end.

[0055] Optionally, the data storage of the analysis module can use the Apache Doris database, which can support response to hundreds of millions of data in seconds. Using Apache Doris as a data storage tool can achieve efficient data reading and writing operations. It can also support more users to access the data analysis system at the same time, thereby improving the efficiency of data analysis.

[0056] Optionally, when the data storage of the analysis module uses the Apache Doris database, if a miniaturized analysis module is required, the MySQL database may be used instead of the Apache Doris database.

[0057] The data layering method is used for data analysis, which improves data utilization and reduces development time.

[0058] In the implementation mode of the present application, the backend module is used to receive the frontend module instruction, and then convert the frontend module instruction into a specific query statement, transmit it to the analysis module, and then execute the corresponding statement to query. Further, the backend module can obtain the execution result of the analysis module statement and return it to the frontend module.

[0059] As an implementation method, the background module may include an overview interface, various module interfaces and a current country equipment interface, wherein the overview interface is used to determine the current data browsing range based on the product permissions corresponding to the product owned by the user and the parameters submitted by the front-end module, and return the corresponding data; the various module interfaces are used to query the specified database according to the instructions submitted by the front-end module, and return the query results to the front-end module; the current country equipment interface is used to query the results of the analysis module according to the instructions submitted by the front-end module, and return the query results to the front-end module for display.

[0060] In an implementation manner of the present application, data analysis may include analyzing overview data and dimensional analysis data. In this manner, the backend module may include a global overview data interface for receiving instructions sent by the frontend module and relevant permissions configured by the backend module, connecting to the analytical database (i.e., the aforementioned specified database) through MySQL to obtain corresponding results, and returning them to the frontend module for corresponding visual display.

[0061] Specifically, the backend module can be developed using springboot+mybatis plus, and the frontend module adopts a front-end and back-end separation architecture, and is connected to the frontend module through an overview interface and a dimensional analysis interface, wherein the above-mentioned dimensional analysis interface includes but is not limited to query conditions such as device, image, country, product, and data generation time, and includes but is not limited to keywords and filtering conditions based on the provided filtering options, and the filtering conditions include at least one of the user's operating permissions, specific operating behaviors, and whether sensitive operating permissions are involved.

[0062] In the implementation mode of the present application, the front-end module can interact with the back-end module based on the screening conditions submitted by the user, obtain the back-end module data, and perform corresponding visual display. Specifically, the front-end module is used to receive the data provided by the back-end module, and then encapsulate the data into bar charts and pie charts for visual display so that the user can view it in real time.

[0063] For example, in a specific application scenario, if a user's query request is received, the front-end module will read the log data according to the user's cached permission field and merge it with the filtering conditions, and then submit it to the back-end module for corresponding query operations. Then, based on the query results of the back-end module, the corresponding log data is rendered, and the rendered log data is displayed as pie charts, bar charts, doughnut charts, etc., so that users can intuitively compare the differences in data in each dimension and discover the value of the data.

[0064] Optionally, the front-end module can be developed using Vue+ant using Vscode to provide users with interfaces such as overview interface, dimension analysis interface, and each device information display interface. Optionally, the front-end module can be set as a web display front end, and can be deployed in a separate mode or in docker integrated deployment mode according to needs.

[0065] As an implementation method, the order and time of script execution of each business database in the analysis module can be controlled by the scheduling module. The specific adjustment logic can be set according to actual needs and is not limited here.

[0066] In this implementation, the front-end module is used to visualize the data, and the back-end module supports flexible configuration of query conditions in multiple dimensions. The two modules work together to lower the threshold for operators to use data. At the same time, the front-end module and the back-end module are separated into a separate architecture mode, so that the back-end module only needs to provide callable services to the front-end module, which can effectively reduce the coupling between the front-end and the back-end.

[0067] In addition, by integrating the data of various product lines (i.e., various business databases) through the data analysis system, the data analysis perspective can be made more comprehensive, thereby comprehensively improving the efficiency and effectiveness of the data analysis system, and further improving the user experience.

[0068] See also Figure 3 Based on the above-mentioned data analysis system based on the bird watching camera, the embodiment of the present application further provides an electronic device 100 that can execute the above-mentioned data analysis system based on the bird watching camera. The electronic device 100 includes a memory 102 and one or more (only one is shown in the figure) processors 104 coupled to each other, and the memory 102 and the processor 104 are connected by a communication line. The memory 102 stores a program that can execute the content of the above-mentioned embodiment, and the processor 104 can execute the program stored in the memory 102.

[0069] Among them, the processor 104 may include one or more processing cores. The processor 104 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 102, and calling data stored in the memory 102. Optionally, the processor 104 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 104 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 104, but may be implemented separately through a communication chip.

[0070] The memory 102 may include a random access memory (RAM) or a read-only memory (ROM). The memory 102 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 102 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the aforementioned embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 100 during use.

[0071] Please refer to Figure 4 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 400 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0072] The computer readable storage medium 400 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 400 includes a non-transitory computer-readable storage medium. The computer readable storage medium 400 has storage space for program code 410 that performs any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 410 can be compressed, for example, in an appropriate form.

[0073] In summary, the embodiment of the present application provides a data analysis system, electronic device and storage medium based on a bird watching camera, and the data analysis system includes: a data acquisition module for collecting multiple log data generated by multiple business databases; an analysis module for cleaning, fusing and calculating the multiple log data; a scheduling module for adjusting the execution order of the scripts corresponding to the multiple log data; a background module for providing analysis result query of the multiple log data; a front-end module for analyzing the behavior data of the bird watching crowd based on the multiple log data and displaying the corresponding analysis results. The various modules of the data analysis system can work together to comprehensively improve the efficiency and effect of the data analysis system, thereby improving the user experience.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A data analysis system based on a bird watching camera, characterized in that: The data analysis system comprises: A data collection module is used to collect multiple log data generated by multiple business databases; An analysis module, used for cleaning, fusing and calculating the plurality of log data; A scheduling module, used to adjust the execution order of the scripts corresponding to the plurality of log data; A backend module, used for providing analysis result query of the plurality of log data; The front-end module is used to analyze the behavior data of the bird-watching crowd based on the multiple log data and display the corresponding analysis results.

2. The data analysis system according to claim 1, characterized in that: The analysis module includes a source layer, a detail layer, an aggregation layer and an application layer: The access layer is used to store multiple log data generated by the multiple business databases; The detail layer is used to store multiple log data after cleaning; The aggregation layer is used to perform fine-grained aggregation of target analysis indicators; The application layer is used to store the analysis results of the analysis module.

3. The data analysis system according to claim 1, characterized in that: The data collection module is also used to convert the multiple log data into preset format types respectively and store them in a designated database.

4. The data analysis system according to claim 3, characterized in that: The backend module includes an overview interface, various module interfaces, and the current country device interface: The overview interface is used to determine the current data browsing range according to the product permissions corresponding to the product owned by the user and the parameters submitted by the front-end module, and return the corresponding data; Each module interface is used to query the specified database according to the instruction submitted by the front-end module and return the query result to the front-end module; The current country device interface is used to query the results of the analysis module according to the instructions submitted by the front-end module, and return the query results to the front-end module for display.

5. The data analysis system according to claim 1, characterized in that: The components used by the data acquisition module in the process of collecting log data include canal components, flinkcdc components or maxwell components.

6. The data analysis system according to claim 1, characterized in that: The platforms on which the data acquisition module runs tasks include the Yarn platform and the K8S platform.

7. The data analysis system according to claim 1, characterized in that: The multiple business databases need to enable database execution logs, configure corresponding log formats and log retention times, and configure corresponding account permissions. Each business database needs to configure corresponding connection parameters, filtering rules, and corresponding table name generation rules, wherein there is a corresponding relationship between the column data types of the tables corresponding to the multiple business databases, and there is a corresponding relationship between the corresponding table names.

8. The data analysis system according to claim 1, characterized in that: The scripts corresponding to the multiple log data support deployment on the resource management platform, support configuration of connection methods for various business databases, and support flexible configuration of resources used by tasks.

9. An electronic device, characterized in that: including one or more processors and memory; One or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the system according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, wherein when the program code is executed by a processor, the system according to any one of claims 1 to 8 is executed.

Images