Big data analysis system, method and equipment for translation industry and medium

Abstract

The invention relates to the technical field of big data, and provides a big data analysis system, method, equipment and medium for the translation industry, and the system comprises a data scheduling tool which is used for creating a data collection task, collecting data from each data source, and preprocessing the data to obtain preprocessed data; the data analysis platform is used for analyzing the preprocessed data based on an SQL (Structured Query Language) interface of an analytical database to obtain an analysis result; and the business intelligence BI report system is used for creating a real-time data report and calling a service interface to visually display the analysis result. According to the big data analysis system, method, equipment and medium for the translation industry, various data in the translation industry are efficiently collected, processed, analyzed and displayed through the big data technology, and therefore data analysis and enterprise decision making are facilitated. The system can provide accurate data analysis service for the translation industry and help to improve the translation quality and efficiency.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a big data analysis system, method, device and medium for the translation industry. Background Art

[0002] The translation industry has to deal with a huge amount of text data processing tasks on a daily basis, covering the original text and corresponding translation of the translation project, the revision history in the translation process, and the workload statistics and efficiency evaluation of translators. Traditional manual statistics and analysis methods are inefficient and can no longer meet the stringent requirements of the modern translation industry for data analysis efficiency and accuracy.

[0003] Based on this, it is particularly urgent to develop an analysis system based on big data technology to effectively respond to this challenge. Summary of the invention

[0004] The present invention provides a big data analysis system, method, device and medium for the translation industry, so as to solve the defect of low efficiency of data analysis means in the translation industry in the prior art.

[0005] The present invention provides a big data analysis system for the translation industry, comprising: A data scheduling tool is used to create a data collection task to collect data from various data sources, and pre-process the data to obtain pre-processed data; A data analysis platform, used to analyze the pre-processed data based on the SQL interface of the analytical database to obtain analysis results; The business intelligence BI reporting system is used to create real-time data reports and call service interfaces to visualize the analysis results.

[0006] According to the big data analysis system for the translation industry provided by the present invention, the data scheduling tool is specifically used for: The pre-processed data is stored in the Hadoop distributed file system HDFS component of the CDH cluster.

[0007] According to the big data analysis system for the translation industry provided by the present invention, the BI reporting system is specifically used for: Receive user input; Based on the user input, the analysis result is updated, and the updated analysis result is visually displayed.

[0008] According to the big data analysis system for the translation industry provided by the present invention, the data scheduling tool is a distributed task scheduling system DolphinScheduler.

[0009] According to the big data analysis system for the translation industry provided by the present invention, the analytical database is a Hive database based on Hadoop.

[0010] According to the big data analysis system for the translation industry provided by the present invention, the BI reporting system is DataEase.

[0011] According to the big data analysis system for the translation industry provided by the present invention, the data sources include at least one of a translation project management system, a translation corpus, a revision record, and a translator record; The analysis result includes at least one of the translation project progress, translation quality score and translator work efficiency.

[0012] The present invention also provides a big data analysis method for the translation industry, comprising: Creating a data collection task, collecting data from various data sources, and preprocessing the data to obtain preprocessed data; Analyze the pre-processed data based on the SQL interface of the analytical database to obtain analysis results; Create real-time data reports and call service interfaces to visualize the analysis results.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a big data analysis method for the translation industry as described in any one of the above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described big data analysis methods for the translation industry.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned big data analysis methods for the translation industry.

[0016] The big data analysis system, method, device and medium for the translation industry provided by the present invention efficiently collect, process, analyze and display various data in the translation industry through big data technology, which is conducive to data analysis and enterprise decision-making. The system consists of three parts: a data scheduling tool, a data analysis platform and a BI reporting system. Among them, the data scheduling tool is used to create a data collection task to collect data from various data sources, and pre-process the data to obtain pre-processed data; the data analysis platform is used to analyze the pre-processed data based on the SQL interface of the analytical database to obtain the analysis results; the business intelligence BI reporting system is used to create real-time data reports and call the service interface to visualize the analysis results. The system can provide accurate data analysis services for the translation industry and help improve translation quality and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a structural schematic diagram of a big data analysis system for the translation industry provided by the present invention.

[0019] Figure 2 This is one of the flow charts of the big data analysis method for the translation industry provided by the present invention.

[0020] Figure 3 This is the second flow chart of the big data analysis method for the translation industry provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The translation industry needs to process a large amount of text data, including but not limited to the original text and translation of the translation project, revision records during the translation process, translators' workload and efficiency, etc. Traditional manual statistics and analysis methods are inefficient and cannot meet the modern translation industry's requirements for high efficiency and accuracy of data analysis.

[0024] In response to this problem, an embodiment of the present invention provides a big data analysis system for the translation industry, which aims to efficiently collect, process, analyze and display various data in the translation industry through big data technology, thereby facilitating data analysis and corporate decision-making. The system consists of three parts: a data scheduling tool, a data analysis platform and a BI reporting system. The data scheduling tool is used to create data collection tasks to collect data from various data sources, and pre-process the data to obtain pre-processed data; the data analysis platform is used to analyze the pre-processed data based on the SQL interface of the analytical database to obtain analysis results; the business intelligence BI reporting system is used to create real-time data reports and call the service interface to visualize the analysis results. The system can provide accurate data analysis services for the translation industry and help improve translation quality and efficiency.

[0025] Figure 1 is a schematic diagram of the structure of the big data analysis system for the translation industry provided by the present invention. Figure 1 As shown, the system includes: The data scheduling tool 110 is used to create a data collection task to collect data from various data sources and pre-process the data to obtain pre-processed data; The data analysis platform 120 is used to analyze the pre-processed data based on the SQL interface of the analytical database to obtain analysis results; The business intelligence BI reporting system 130 is used to create real-time data reports and call service interfaces to visualize the analysis results.

[0026] Specifically, first configure the data source in the data scheduling tool, including databases (such as MySQL, PostgreSQL, etc.), file storage (such as HDFS, S3, etc.), and API interfaces, etc. The configuration process includes entering the connection information and authentication information of the data source.

[0027] Design data collection task templates based on business needs. The templates include the selection of data sources, collection frequency (such as real-time collection, scheduled collection, etc.), collected data fields, etc. Use the scheduling engine of the data scheduling tool to create specific collection tasks based on the templates and set the execution time and priority of the tasks.

[0028] Then, the collected raw data is preprocessed, which includes data cleaning, such as removing duplicate data, processing missing values, converting data types, etc. Preprocessing can also transform the data according to business needs, such as field mapping, data aggregation, data splitting, etc.

[0029] Store the preprocessed data in a middle-tier database or data warehouse in preparation for subsequent analysis.

[0030] In the data analysis platform, you can select a suitable analytical database (such as Hive, Amazon Redshift, Google BigQuery, Snowflake, etc.) and configure it according to business needs. Configure the SQL interface to ensure that the data analysis platform can connect to the analytical database and execute SQL queries.

[0031] Load the preprocessed data into the analytical database. Write SQL query statements based on business needs to analyze the loaded data. The analysis process may include data aggregation, grouping, sorting, filtering, etc. Store the analysis results in the analytical database or export them to other storage media (such as CSV files, Excel files, etc.).

[0032] For the creation of real-time data reports, you can design report templates according to business needs, including report layout, style, data fields, etc. Configure the data source of the report, and select the storage location of the analytical database or pre-processed data. Bind the fields in the report template with the fields in the data source to ensure that the report can display the data correctly.

[0033] According to the characteristics of the analysis results, select the appropriate chart type (such as bar chart, line chart, pie chart, etc.) for display. Use the service interface provided by the BI reporting system to display the analysis results in the form of charts in the report. This may require writing some custom scripts or plug-ins to interact with the BI reporting system.

[0034] In addition, the created reports can be published to the BI reporting system platform and access permissions can be set so that team members can view and share analysis results.

[0035] The system provided by the embodiment of the present invention consists of three parts: data scheduling tool, data analysis platform and BI report system. It uses big data technology to efficiently collect, process, analyze and display various data in the translation industry, which is beneficial to data analysis and corporate decision-making. The system can provide accurate data analysis services for the translation industry and help improve translation quality and efficiency.

[0036] In some embodiments, after the pre-processed data is obtained, the pre-processed data may be stored in a Hadoop distributed file system HDFS component of a CDH cluster.

[0037] CDH (Cloudera's Distribution Including Apache Hadoop) is a big data product launched by Cloudera. It is built on a stable version of Apache Hadoop, provides Hadoop's core scalable storage and distributed computing, and provides a user interface based on the Web. In addition, CDH also includes multiple components and tools in the Hadoop ecosystem, such as HDFS, MapReduce, HBase, Hive, Spark, etc., providing users with a complete big data platform solution.

[0038] Hadoop Distributed File System HDFS is one of the core components of the Hadoop project, used to store and manage files in large data sets. HDFS is designed to run on a large number of commodity hardware, provide high-throughput data access, and is very suitable for distributed computing on large-scale data sets.

[0039] You can use the command line tools provided by Hadoop (such as hdfs dfs -put) to upload preprocessed data to HDFS. You can also use the API provided by HDFS. This usually involves writing code in languages ​​such as Java or Python to interact with HDFS. Through the API, you can create HDFS files, write data, close files, and other operations. You can also use some third-party tools (such as Apache Sqoop, Apache Flume, etc.) to import preprocessed data into HDFS.

[0040] During this process, you can also use the monitoring tools provided by the CDH cluster (such as Cloudera Manager) to monitor the status and performance of HDFS. Pay attention to indicators such as HDFS storage capacity, node status, and data block distribution to ensure that HDFS is in a healthy state.

[0041] Based on any of the above embodiments, the BI reporting system is specifically used for: Receive user input; Based on user input, the analysis results are updated and the updated analysis results are displayed visually.

[0042] Specifically, the BI reporting system supports dynamic refresh and interaction, and users can customize query and display content according to their needs.

[0043] Here, user input refers to information or instructions provided by a user when interacting with the system through a certain interface or device. User input can be text / value / image / voice / gesture input, or click operation, file upload, etc., which is not specifically limited in the embodiments of the present invention.

[0044] Based on user input, load the required data from the data source (such as database, file system, API, etc.). If the data has been loaded into memory or cache, it can be used directly. Alternatively, adjust the parameters or conditions of the analysis logic based on user input to generate updated analysis results and visualize the updated analysis results.

[0045] In some embodiments, the data scheduling tool is a distributed task scheduling system DolphinScheduler.

[0046] DolphinScheduler is an open source, distributed, and easily extensible task scheduling system, which is mainly used to manage and schedule the execution of large-scale tasks to achieve the scheduling and monitoring of automated tasks such as data processing, ETL (extraction, transformation, loading) workflows, and data analysis.

[0047] In some embodiments, the analytical database is a Hive database based on Hadoop.

[0048] Hive's data operations mainly revolve around data query, insertion, update, and deletion. Users can perform various data operations through Hive's SQL interface without having to worry about the complex details of the underlying Hadoop.

[0049] Hive not only supports large-scale data storage, but also provides rich data analysis functions. Common Hive data analysis scenarios include: Simple query analysis: Use aggregate functions (such as count()) to calculate the row data in the table, or add the distinct keyword to find the number of unique data items.

[0050] Conditional query analysis: Filter and analyze data based on specific conditions (such as date range, location, user behavior, etc.).

[0051] Statistical analysis: Perform statistical analysis on data, such as counting the top 10 videos viewed, counting the popularity of video categories, etc. These analyses can help users understand the distribution and trends of data and make more informed decisions.

[0052] User behavior analysis: Analyze based on user behavior data, such as querying the purchase ratio or browsing ratio of a product on a certain day, or querying the click behavior of a certain user on a certain day, etc. These analyses help to understand user behavior patterns and preferences, thereby optimizing products and services.

[0053] In some embodiments, the BI reporting system is DataEase.

[0054] Specifically, DataEase is an open source data visualization analysis tool designed to help users quickly analyze data and gain insights into business trends, thereby improving and optimizing their business. It supports a wide range of data source connections, can quickly create charts by dragging and dropping, and can be easily shared with others.

[0055] In some embodiments, each data source includes at least one of a translation project management system, a translation corpus, a revision record, and a translator record; The analysis result includes at least one of a translation project progress, a translation quality score, and a translator's work efficiency.

[0056] In some embodiments, a big data analysis system for the translation industry is provided, which mainly includes the following parts: 1. Data scheduling tool: DolphinScheduler is used as an ETL scheduling tool to collect data from various data sources and perform preliminary processing and cleaning. DolphinScheduler's task scheduling function can ensure the efficiency and stability of the data collection process.

[0057] 2. Data analysis platform: CDH (Cloudera Distribution Hadoop) cluster is used as the data analysis base to provide efficient big data storage and computing capabilities. Hive is used for data processing and analysis. Hive's SQL query capabilities can meet complex query requirements and support the processing of large-scale data sets.

[0058] 3. BI Report System: Use DataEase as the BI (Business Intelligence) report system to visualize data. DataEase's flexibility and powerful charting capabilities can intuitively display analysis results to users, helping management make data-driven decisions.

[0059] The big data analysis system for the translation industry provided by the embodiment of the present invention can meet the data analysis requirements required by the translation industry, and has high performance indicators, can analyze large quantities of data in the translation industry, and precipitate and summarize the corpus and data used in common translations, which can significantly improve the efficiency of translation. At the same time, the system architecture introduced in the present invention uses open source big data applications and has a high community activity, which makes the cost of using this system low and is also convenient for developers and operation and maintenance personnel. Work efficiency.

[0060] After successfully building the big data analysis system for the translation industry as described above, the entire big data framework is built, and the full-link analysis from the underlying data to the upper-level application can be completed. Through this system, the data set of the translation company can be collected, and through data analysis, terminology and high-frequency corpus can be refined, which can significantly improve the efficiency and quality of translation. The quality and cost changes of translation can also be monitored through data analysis, which is convenient for changes and decisions in the translation process, and can also predict the translation market demand.

[0061] Based on any of the above embodiments, Figure 2 This is one of the flow charts of the big data analysis method for the translation industry provided by the present invention. Figure 3 This is a second flow chart of a big data analysis method for the translation industry provided by the present invention. As shown in the figure, the method includes: Step 210, creating a data collection task, collecting data from various data sources, and preprocessing the data to obtain preprocessed data; Step 220, analyzing the preprocessed data based on the SQL interface of the analytical database to obtain analysis results; Step 230, create a real-time data report and call the service interface to visualize the analysis results.

[0062] Specifically, the data sources here include translation project management systems, translation corpora, revision records, translator records and other databases. Use DolphinScheduler to create data collection tasks and extract data from various data sources to HDFS on a regular basis. Clean and convert the collected data to ensure data consistency and accuracy.

[0063] Store the cleaned data in the HDFS component of the CDH cluster. Use Hive to classify and stratify the data and perform analysis and processing, including translation efficiency analysis, quality assessment, workload statistics, etc. Hive SQL query optimization improves query speed and efficiency.

[0064] Use DataEase to create reports and dashboards to visualize analysis results. Use DataEase to create real-time data reports, call web service interfaces or MySQL services, provide real-time data, and facilitate real-time decision-making. Data display includes translation project progress, translation quality ratings, translator work efficiency, etc. Reports support dynamic refresh and interaction, and users can customize query and display content according to their needs.

[0065] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communications interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a big data analysis method for the translation industry, the method comprising: creating a data collection task, collecting data from various data sources, and preprocessing the data to obtain preprocessed data; analyzing the preprocessed data based on the SQL interface of the analytical database to obtain analysis results; creating a real-time data report, and calling the service interface to visualize the analysis results.

[0066] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0067] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the big data analysis method for the translation industry provided by the above methods. The method includes: creating a data collection task, collecting data from various data sources, and preprocessing the data to obtain preprocessed data; analyzing the preprocessed data based on the SQL interface of the analytical database to obtain analysis results; creating real-time data reports, and calling the service interface to visualize the analysis results.

[0068] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the big data analysis method for the translation industry provided by the above methods. The method includes: creating a data collection task, collecting data from various data sources, and preprocessing the data to obtain preprocessed data; analyzing the preprocessed data based on the SQL interface of the analytical database to obtain analysis results; creating a real-time data report, and calling a service interface to visualize the analysis results.

[0069] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0070] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A big data analysis system for the translation industry, characterized in that: include: A data scheduling tool is used to create a data collection task to collect data from various data sources, and pre-process the data to obtain pre-processed data; A data analysis platform, used to analyze the pre-processed data based on the SQL interface of the analytical database to obtain analysis results; The business intelligence BI reporting system is used to create real-time data reports and call service interfaces to visualize the analysis results.

2. The big data analysis system for the translation industry according to claim 1, characterized in that: The data scheduling tool is specifically used for: The pre-processed data is stored in the Hadoop distributed file system HDFS component of the CDH cluster.

3. The big data analysis system for the translation industry according to claim 2, characterized in that: The BI reporting system is specifically used for: Receive user input; Based on the user input, the analysis result is updated, and the updated analysis result is visually displayed.

4. The big data analysis system for the translation industry according to any one of claims 1 to 3, characterized in that: The data scheduling tool is the distributed task scheduling system DolphinScheduler.

5. The big data analysis system for the translation industry according to any one of claims 1 to 3, characterized in that: The analytical database is a Hive database based on Hadoop.

6. The big data analysis system for the translation industry according to any one of claims 1 to 3, characterized in that: The BI reporting system is DataEase.

7. The big data analysis system for the translation industry according to any one of claims 1 to 3, characterized in that: The data sources include at least one of a translation project management system, a translation corpus, a revision record, and a translator record; The analysis result includes at least one of the translation project progress, translation quality score and translator work efficiency.

8. A big data analysis method for the translation industry, characterized in that: include: Creating a data collection task, collecting data from various data sources, and preprocessing the data to obtain preprocessed data; Analyze the pre-processed data based on the SQL interface of the analytical database to obtain analysis results; Create real-time data reports and call service interfaces to visualize the analysis results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the big data analysis method for the translation industry as claimed in claim 8 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the big data analysis method for the translation industry as claimed in claim 8 is implemented.

Images