Python-based animation platform data analysis method and system

Abstract

The invention particularly relates to a python-based animation platform data analysis method and system. According to the python-based animation platform data analysis method and system, a python development environment is established, a Scrapy acquisition framework is configured, and data is acquired by utilizing a crawler technology and is stored in an HDFS (Hadoop Distributed File System); the method comprises the following steps of: cleaning and sorting data by adopting a big data technology SparkCore component or a SparkSQL component, removing dirty data, and extracting valuable vocabularies by using jieba word segmentation custom; and drawing a histogram, a line chart or a pie chart by using a python data visualization library, visually displaying hot works, score distribution and user preference information of the animation, and analyzing and summarizing the obtained visualization result. According to the python-based cartoon platform data analysis method and system, the development trend and the user preference of the cartoon industry are known by using a visual analysis technology, valuable reference and guidance can be provided for practitioners and enthusiasts in the cartoon industry, and the practitioners and the enthusiasts are helped to better know the market requirements and the user preference, so that a more intelligent decision is made.

Description

Technical Field

[0001] The present invention relates to the field of data governance technology, and in particular to a Python-based animation platform data analysis method and system. Background Art

[0002] With the rapid development of the economy and society, the film and television industry has ushered in an opportunity for vigorous development, and the animation industry in particular has seen significant growth. In order to better understand audience needs, guide content creation, optimize marketing strategies, promote industry innovation, support academic research, and promote the sustainable and healthy development of the industry, in-depth exploration and data analysis of animation platforms are necessary.

[0003] However, due to the diversity and complexity of the animation industry, it is necessary to comprehensively consider multiple factors and indicators for analysis and evaluation. Based on the above situation, the present invention proposes a Python-based animation platform data analysis method and system. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a simple and efficient Python-based animation platform data analysis method and system.

[0005] The present invention is achieved through the following technical solutions:

[0006] A Python-based animation platform data analysis method, characterized by comprising the following steps:

[0007] Step S1: Data collection

[0008] First, we set up a Python development environment, configured the Scrapy collection framework, used crawler technology to collect Chinese animation data from the animation platform, and stored the collected data in the HDFS (Hadoop Distributed File System) distributed file system;

[0009] In step S1, the collection requirements for the animation platform's domestic animation data are formulated, Hadoop big data related components and Python development environment are built, and the Scrapy collection framework is configured to collect the name, rating, playback volume and serialization data information of the domestic animation, as well as user comments and evaluations.

[0010] In the step S1, the demand for collecting domestic animation data of Bilibili, an animation platform, is formulated.

[0011] Step S2: Data cleaning

[0012] Use big data technology SparkCore or SparkSQL to clean and organize the collected data, remove duplicate and invalid dirty data, and use Jieba word segmentation to extract valuable words to ensure the accuracy and reliability of the analysis;

[0013] In step S2, the SparkCore component or the SparkSQL component processes the collected data offline, specifically including the following steps:

[0014] Step S2.1: Remove duplicate comments. For the same animation work, only one comment is kept. For the same text, only one comment is kept randomly.

[0015] Step S2.2: Before word segmentation, remove special symbols containing letters and emoticons from the data to clean the data.

[0016] In step S2, cluster resources are adjusted based on the amount of data, and the interface posseg.cut used by Jieba word segmentation is used to return the word segmentation and part of speech, and then a word cloud is drawn to view the word segmentation effect;

[0017] First, perform word frequency statistics on the returned word segments, sort the word frequencies in descending order, select the word segments with word frequencies in the top 25% to 30%, use WordCloud in the wordcloud module to draw the word cloud, and visually highlight the keywords with word frequencies in the top 5% to 8%.

[0018] Step S3: Data display

[0019] Use the Python data visualization library to draw bar charts, line charts, or pie charts to intuitively display popular anime works, rating distribution, and user preference information, and analyze and summarize the obtained visualization results.

[0020] Python has many libraries dedicated to data visualization, the most famous of which are Matplotlib and Seaborn. As the oldest data visualization library in Python, Matplotlib provides a rich set of plotting functions and flexible configuration options, meeting most basic plotting needs. This article uses Matplotlib to create the charts.

[0021] In step S3, a pie chart is drawn using the pie function in the matplotlib package of the Python data visualization library;

[0022] Use the plt function in the matplotlib package of the Python data visualization library to draw a histogram;

[0023] Use the plot function in the matplotlib package of the Python data visualization library to draw a line chart.

[0024] The Python-based animation platform data analysis system includes:

[0025] The data collection module is responsible for setting up the Python development environment, configuring the Scrapy collection framework, using crawler technology to collect Chinese animation data from the animation platform, and storing the collected data in the HDFS distributed file system;

[0026] The data cleaning module is responsible for cleaning and organizing the collected data using the big data technology SparkCore component or SparkSQL component, removing duplicate and invalid dirty data, and using Jieba word segmentation to extract valuable words to ensure the accuracy and reliability of the analysis;

[0027] The data display module is responsible for using the Python data visualization library to draw bar charts, line charts, or pie charts to intuitively display popular anime works, rating distribution, and user preference information, and analyze and summarize the obtained visualization results.

[0028] A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the above methods.

[0029] A computing device comprising:

[0030] One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the above methods.

[0031] The beneficial effects of the present invention are as follows: the Python-based animation platform data analysis method and system use visual analysis technology to understand the development trends of the animation industry and user preferences, and can provide valuable reference and guidance for animation industry practitioners and enthusiasts, helping them to better understand market demand and user preferences, and thus make more informed decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Attachment Figure 1 This is a schematic diagram of the Python-based animation platform data analysis method of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention.

[0035] The Python-based animation platform data analysis method includes the following steps:

[0036] Step S1: Data collection

[0037] First, we set up a Python development environment, configured the Scrapy collection framework, used crawler technology to collect Chinese animation data from the animation platform, and stored the collected data in the HDFS (Hadoop Distributed File System) distributed file system;

[0038] In step S1, the collection requirements for the animation platform's domestic animation data are formulated, Hadoop big data related components and Python development environment are built, and the Scrapy collection framework is configured to collect the name, rating, playback volume and serialization data information of the domestic animation, as well as user comments and evaluations.

[0039] In the step S1, the demand for collecting domestic animation data of Bilibili, an animation platform, is formulated.

[0040] Step S2: Data cleaning

[0041] Use big data technology SparkCore or SparkSQL to clean and organize the collected data, remove duplicate and invalid dirty data, and use Jieba word segmentation to extract valuable words to ensure the accuracy and reliability of the analysis;

[0042] In step S2, the SparkCore component or the SparkSQL component processes the collected data offline, specifically including the following steps:

[0043] Step S2.1: For cases where the collected data contains multiple user comments on the same animation work, or where different users copy the same comment text, duplicate comments are removed to ensure data accuracy. Only one user comment on the same animation work is retained, and only one comment with the same text is randomly retained.

[0044] Step S2.2: When manually reviewing the data, it was found that the data contained special symbols such as letters and emoticons. For the mining target of this case, this type of data itself is of no substantial help and has no effect on the analysis target. Therefore, before word segmentation, the special symbols containing letters and emoticons in the data are removed and the data is cleaned.

[0045] In step S2, cluster resources are adjusted based on the amount of data, and the interface posseg.cut used by Jieba word segmentation is used to return the word segmentation and part of speech, and then a word cloud is drawn to view the word segmentation effect;

[0046] First, perform word frequency statistics on the returned word segments, sort the word frequencies in descending order, select the word segments with word frequencies in the top 25% to 30%, use WordCloud in the wordcloud module to draw the word cloud, and visually highlight the keywords with word frequencies in the top 5% to 8%.

[0047] Step S3: Data display

[0048] Use the Python data visualization library to draw bar charts, line charts, or pie charts to intuitively display popular anime works, rating distribution, and user preference information, and analyze and summarize the obtained visualization results.

[0049] Python has many libraries dedicated to data visualization, the most famous of which are Matplotlib and Seaborn. As the oldest data visualization library in Python, Matplotlib provides a rich set of plotting functions and flexible configuration options, meeting most basic plotting needs. This article uses Matplotlib to create the charts.

[0050] In step S3, a pie chart is drawn using the pie function in the matplotlib package of the Python data visualization library;

[0051] Use the plt function in the matplotlib package of the Python data visualization library to draw a histogram;

[0052] Use the plot function in the matplotlib package of the Python data visualization library to draw a line chart.

[0053] The Python-based animation platform data analysis system includes:

[0054] The data collection module is responsible for setting up the Python development environment, configuring the Scrapy collection framework, using crawler technology to collect Chinese animation data from the animation platform, and storing the collected data in the HDFS distributed file system;

[0055] The data cleaning module is responsible for cleaning and organizing the collected data using the big data technology SparkCore component or SparkSQL component, removing duplicate and invalid dirty data, and using Jieba word segmentation to extract valuable words to ensure the accuracy and reliability of the analysis;

[0056] The data display module is responsible for using the Python data visualization library to draw bar charts, line charts, or pie charts to intuitively display popular anime works, rating distribution, and user preference information, and analyze and summarize the obtained visualization results.

[0057] By observing and comparing the data, we can draw conclusions, such as rating trends for popular anime works and user preferences for different types of anime. These conclusions can provide valuable reference and guidance for anime industry practitioners and enthusiasts, helping them better understand market demand and user preferences, and thus make more informed decisions.

[0058] The computer-readable storage medium stores one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by a computing device, the computing device is caused to perform any one of the above methods.

[0059] The computing device comprises:

[0060] One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the above methods.

[0061] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A Python-based animation platform data analysis method, characterized by: The following steps are involved: Step S1: Data collection First, build a Python development environment, configure the Scrapy collection framework, use crawler technology to collect Chinese animation data from the animation platform, and store the collected data in the HDFS distributed file system; Step S2: Data cleaning Use big data technology SparkCore component or SparkSQL component to clean and organize the collected data, remove duplicate and invalid dirty data, and use Jieba word segmentation to extract valuable words to ensure the accuracy and reliability of the analysis; Step S3: Data display Use the Python data visualization library to draw bar charts, line charts, or pie charts to intuitively display the popular works of animation, rating distribution, and user preference information, and analyze and summarize the obtained visualization results.

2. The Python-based animation platform data analysis method according to claim 1 is characterized in that: In step S1, the collection requirements for the domestic animation data of the animation platform are formulated, Hadoop big data related components and Python development environment are built, and the Scrapy collection framework is configured to collect the name, rating, playback volume and serialization data information of the domestic animation, as well as user comments and evaluations.

3. The Python-based animation platform data analysis method according to claim 1 or 2, characterized in that: In the step S1, the demand for collecting the domestic animation data of Bilibili, an animation platform, is formulated.

4. The Python-based animation platform data analysis method according to claim 1 is characterized in that: In step S2, the SparkCore component or the SparkSQL component processes the collected data offline, specifically including the following steps: Step S2.1, remove duplicate comments, keep only one comment from the user of the same animation work, and keep only one comment with the same text at random; Step S2.2: Before word segmentation, remove special symbols containing letters and emoticons in the data to clean the data.

5. The Python-based animation platform data analysis method according to claim 1 is characterized in that: In step S2, cluster resources are adjusted based on the amount of data, the interface posseg.cut used by Jieba word segmentation is used to return the word segmentation and part of speech, and then a word cloud is drawn to view the word segmentation effect; First, we count the frequency of the returned words, sort them in descending order, select the words with the top 25% to 30% frequency, use WordCloud in the wordcloud module to draw the word cloud, and visually highlight the keywords with the top 5% to 8% frequency.

6. The Python-based animation platform data analysis method according to claim 1 is characterized in that: In step S3, a pie chart is drawn using the pie function in the matplotlib package of the python data visualization library; Use the plt function in the matplotlib package of the Python data visualization library to draw a bar chart; Use the plot function in the matplotlib package of the Python data visualization library to draw a line chart.

7. A Python-based animation platform data analysis system, characterized by: include: The data collection module is responsible for building the Python development environment, configuring the Scrapy collection framework, using crawler technology to collect the Chinese animation data of the animation platform, and storing the collected data in the HDFS distributed file system; The data cleaning module is responsible for using the big data technology SparkCore component or SparkSQL component to clean and organize the collected data, remove duplicate and invalid dirty data, and use Jieba word segmentation to extract valuable words to ensure the accuracy and reliability of the analysis; The data display module is responsible for using the Python data visualization library to draw bar charts, line charts or pie charts, intuitively displaying popular animation works, rating distribution and user preference information, and analyzing and summarizing the obtained visualization results.

8. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions which, when executed by a computing device, cause the computing device to perform any one of the methods according to claims 1 to 6.

9. A computing device, characterized in that include: One or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the methods according to claims 1 to 6.

Images