Marine fishery resource e-commerce data processing method

Abstract

The invention discloses a marine fishery resource e-commerce data processing method, and relates to the technical field of e-commerce data processing, and the method comprises the following steps: data collection: employing a satellite remote sensing method to obtain local marine fishery resource information, and carrying out the centralized collection and data processing of the obtained local marine fishery information, filtering, denoising and cleaning the collected data, and converting and standardizing the cleaned data, the marine fishery resource e-commerce data processing method can efficiently process mass data and improve the analysis speed, statistical analysis and clustering analysis can reveal a hidden mode and an association relationship in the data, and the data processing efficiency is improved. The method is advantaged in that support is provided for decision making, data visualization enables an analysis result to be visual and easy to understand and convenient to understand and apply, through establishment of a distributed calculation cluster, calculation capability can be flexibly expanded according to hardware resources and business demands, different analysis demands are satisfied, and fishery e-commerce platform operation and scientization and precision of decision making are jointly promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of e-commerce data processing, and in particular to a method for processing e-commerce data of marine fishery resources. Background Art

[0002] With the increasing importance of marine fishery resources and the rise of e-commerce platforms, efficient and accurate processing of marine fishery resource data has become the key. This technology proposes a comprehensive marine fishery resource e-commerce data processing method, covering data collection, processing, analysis, storage and big data analysis.

[0003] By obtaining resource information through satellite remote sensing technology, combined with distributed computing clusters and statistical analysis methods, the hidden patterns and associations in the data are deeply mined to provide strong support for the operation and decision-making of the fishery e-commerce platform. At the same time, through data storage and classification management, the security and accessibility of the data are ensured, providing data support for the sustainable development of marine fishery resources. To this end, a method for processing marine fishery resources e-commerce data is proposed. Summary of the invention

[0004] The purpose of the present invention is to provide a method for processing marine fishery resources e-commerce data to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for processing marine fishery resources e-commerce data, comprising the following steps:

[0006] S1: Data collection, using satellite remote sensing to obtain local marine fishery resource information, and centrally collect the local marine fishery information obtained;

[0007] S2: Data processing: filtering, removing noise, and cleaning the collected data, and converting and standardizing the cleaned data;

[0008] S3: Data analysis: statistical analysis of the processed data, establishment of a prediction model, prediction of future catch trends and possible influencing factors, and conversion of data into charts and graphs, which are then displayed on the monitor screen;

[0009] S4: Data storage, storing the analyzed data, classifying the cultured and caught marine fishery resources and creating folders, and setting up subfolders inside the folders to store different types of fishery resources;

[0010] S5: Big data analysis: Based on data storage, use big data technology to conduct deeper mining and analysis of data, use distributed computing clusters to process and analyze massive data, and use statistical analysis methods to discover hidden patterns and correlations in the data, providing support for the operation and decision-making of the fishery e-commerce platform.

[0011] Preferably, the step S1 specifically includes using MODIS satellite remote sensing data to obtain local marine fishery data, removing clouds, performing atmospheric correction, and geometric correction on the obtained data, centrally collecting the pre-processed satellite remote sensing data, and establishing a database to centrally manage the obtained data.

[0012] Preferably, the filtering and noise removal in step S2 uses median filtering to process abnormal values ​​and noise in the data.

[0013] Preferably, the cleaning step in step S2 comprises:

[0014] A1: Deduplication: Use hash algorithm to remove duplicate data.

[0015] A2: Missing value processing, using the median algorithm to fill in missing values;

[0016] A3: Outlier processing: using the Z-score method to identify and process outliers based on the distribution characteristics of the data;

[0017] A4: Data format conversion: converting unstructured or semi-structured data into structured data.

[0018] Preferably, the statistical analysis in step S3 includes calculating the mean, median, mode, standard deviation, and variance statistics of the data, and analyzing the central tendency, dispersion, and distribution of the data.

[0019] Preferably, the establishment of the prediction model in step S3 comprises the following steps:

[0020] B1: Data preparation, selecting independent variables of water temperature, salinity, wind direction, wind speed and dependent variable of catch, and dividing into training set and test set;

[0021] B2: Model selection, select the logistic regression prediction model;

[0022] B3: Model training: bring the training data into the logistic regression prediction model to train the logistic regression prediction model and adjust the parameters of the model until the optimal prediction result is achieved;

[0023] B4: Model evaluation: Use the test set data to evaluate the model, calculate the model's prediction accuracy, recall rate, and F1 value, and evaluate the model's performance.

[0024] Preferably, converting the data in step S3 into a chart form includes selecting a chart type, selecting a bar chart format, calculating the total catch of each time period, calculating the weight of each influencing factor, and converting the calculated data into a bar chart format, using the Seaborn data visualization tool to draw a chart, and projecting the drawn chart onto the screen of the display.

[0025] Preferably, the step S4 specifically includes selecting a local server as a storage medium, selecting an Excel storage format, creating a main folder for marine fishery resources, and creating two subfolders based on breeding and fishing, and further refining and distinguishing the two subfolders based on the types of economic crops in the fishery.

[0026] Preferably, the step S5 specifically includes integrating the stored data, processing the data, converting the integrated data into Excel, building a Spark distributed computing cluster according to hardware resources and business needs, loading the prepared data into the Spark distributed computing cluster, using statistical analysis methods to perform descriptive analysis on the data, using cluster analysis methods to discover hidden patterns and associations in the data, establishing a decision support model based on the mined patterns and associations, and visually displaying the analysis results in the form of charts, images, etc., so as to more intuitively understand the patterns and associations in the data.

[0027] Preferably, in step S5, the prepared data is loaded into the Spark distributed computing cluster, and the prepared data needs to be compressed.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention utilizes this method to efficiently process massive amounts of data and improve analysis speed. Statistical analysis and cluster analysis can reveal hidden patterns and correlations in the data to provide support for decision-making. Data visualization makes the analysis results intuitive and easy to understand and apply. By building a distributed computing cluster, computing power can be flexibly expanded according to hardware resources and business needs to meet different analysis needs, thereby jointly promoting the scientific and precise operation and decision-making of the fishery e-commerce platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of the e-commerce data processing method of the present invention;

[0031] Figure 2 A flowchart of cleaning in data processing of the present invention;

[0032] Figure 3 A flow chart for establishing a prediction model for the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Example

[0035] See also Figure 1-3 The present invention provides a technical solution: a method for processing marine fishery resources e-commerce data. The present invention makes corresponding improvements to the technical problems mentioned in the background technology, including the following steps:

[0036] S1: Data collection, using satellite remote sensing to obtain local marine fishery resource information, and centrally collect the local marine fishery information obtained;

[0037] S2: Data processing: filtering, removing noise, and cleaning the collected data, and converting and standardizing the cleaned data;

[0038] S3: Data analysis: statistical analysis of the processed data, establishment of a prediction model, prediction of future catch trends and possible influencing factors, and conversion of data into charts and graphs, which are then displayed on the monitor screen;

[0039] S4: Data storage, storing the analyzed data, classifying the cultured and caught marine fishery resources and creating folders, and setting up subfolders inside the folders to store different types of fishery resources;

[0040] S5: Big data analysis: Based on data storage, use big data technology to conduct deeper data mining and analysis, use distributed computing clusters to process and analyze massive data, and use statistical analysis methods to discover hidden patterns and correlations in the data to provide support for the operation and decision-making of the fishery e-commerce platform;

[0041] Step S1 specifically includes: using MODIS satellite remote sensing data to obtain local marine fishery data, removing clouds, performing atmospheric correction, and geometric correction on the obtained data, centrally collecting the pre-processed satellite remote sensing data, and establishing a database to centrally manage the obtained data;

[0042] The filtering in step S2, removing noise, uses median filtering to process the outliers and noise of the data;

[0043] The cleaning step in step S2 includes:

[0044] A1: Deduplication: Use hash algorithm to remove duplicate data.

[0045] A2: Missing value processing, using the median algorithm to fill in missing values;

[0046] A3: Outlier processing: using the Z-score method to identify and process outliers based on the distribution characteristics of the data;

[0047] A4: Data format conversion: converting unstructured or semi-structured data into structured data;

[0048] The statistical analysis in step S3 includes calculating the mean, median, mode, standard deviation, and variance statistics of the data, and analyzing the central tendency, dispersion, and distribution of the data;

[0049] The establishment of the prediction model in step S3 includes the following steps:

[0050] B1: Data preparation, selecting independent variables of water temperature, salinity, wind direction, wind speed and dependent variable of catch, and dividing into training set and test set;

[0051] B2: Model selection, select the logistic regression prediction model;

[0052] B3: Model training: bring the training data into the logistic regression prediction model to train the logistic regression prediction model and adjust the parameters of the model until the optimal prediction result is achieved;

[0053] B4: Model evaluation: Use the test set data to evaluate the model, calculate the model's prediction accuracy, recall rate, and F1 value, and evaluate the model's performance;

[0054] The data in step S3 is converted into a chart form, including selecting a chart type, selecting a bar chart format, calculating the total catch of each time period, calculating the weight of each influencing factor, and converting the calculated data into a bar chart format, using the Seaborn data visualization tool to draw a chart, and projecting the drawn chart onto the screen of the display;

[0055] Step S4 specifically includes selecting a local server as a storage medium, selecting an Excel storage format, creating a main folder for marine fishery resources, and creating two subfolders according to aquaculture and fishing, and further refining and distinguishing the two subfolders according to the types of economic crops in the fishery;

[0056] Step S5 specifically includes: integrating the stored data, processing the data, converting the integrated data into Excel, building a Spark distributed computing cluster according to hardware resources and business needs, loading the prepared data into the Spark distributed computing cluster, using statistical analysis methods to perform descriptive analysis on the data, using cluster analysis methods to discover hidden patterns and associations in the data, establishing a decision support model based on the mined patterns and associations, and visually displaying the analysis results in the form of charts, images, etc., so as to more intuitively understand the patterns and associations in the data;

[0057] In step S5, the prepared data is loaded into the Spark distributed computing cluster, and the prepared data needs to be compressed.

[0058] The specific implementation of this embodiment is as follows:

[0059] Data collection: In the process of collecting data on marine fishery resources, MODIS satellite remote sensing data is used to obtain resource information of local marine fisheries. These data contain rich marine fishery environmental data, such as ocean temperature, salinity, and plankton distribution. The acquired data will first undergo a series of preprocessing steps, including cloud removal to eliminate the impact of clouds on data quality, atmospheric correction to correct the impact of the atmosphere on radiation, and geometric correction to ensure the spatial accuracy of the data. The preprocessed data will be collected centrally and a special database will be established to store and manage these data. This database can provide efficient data access and analysis support, which is helpful for subsequent fishery resource assessment and decision-making;

[0060] Data processing: In the process of data processing, the collected marine fishery data will first be filtered and de-noised to ensure the accuracy and reliability of the data. The median filtering method can effectively remove outliers and noise in the data. Subsequently, the data cleaning step is crucial, which includes deduplication, missing value processing, outlier processing and data format conversion. Deduplication uses the hash algorithm to identify and remove duplicate data, and missing value processing uses the median algorithm to fill in to maintain data integrity. Outlier processing is identified and processed by the Z-score method to avoid its impact on the overall data. Finally, data format conversion ensures that unstructured or semi-structured data is converted into structured data for subsequent data analysis and application. These steps together ensure the quality and analyzability of the data.

[0061] Data analysis: In the data analysis stage, it is crucial to conduct statistical analysis on the processed marine fishery data and build a prediction model. First, by calculating the mean, median, mode, standard deviation and variance statistics, the central tendency, dispersion and distribution of the data are deeply analyzed. These statistics provide an overall view of the data and help understand the inherent laws and characteristics of the data.

[0062] Next, in order to predict the changing trend of future catches and possible influencing factors, a prediction model was established according to the following steps:

[0063] B1: Data preparation: Select key factors such as water temperature, salinity, wind direction and wind speed as independent variables, and fish catch as dependent variable. Then, divide the data set into training set and test set to ensure that the model can make effective predictions on unseen data;

[0064] B2: Model selection, select the logistic regression prediction model, because it can handle the situation where the dependent variable is a continuous variable and explain the influence of the independent variable on the dependent variable;

[0065] B3: Model training: the training set data is input into the logistic regression model. By continuously adjusting the model parameters, the model can accurately fit the training data. The goal of this process is to find the best model parameters to obtain the best prediction effect.

[0066] B4: Model evaluation: Use the test set data to evaluate the trained model and calculate the model's prediction accuracy, recall rate, F1 value and other indicators to comprehensively evaluate the model's performance on unseen data;

[0067] Finally, in order to more intuitively display the results of data analysis, a bar chart was selected as the chart type. By calculating the total catch of each time period and the weight of each influencing factor, these data were converted into a bar chart format. Using data visualization tools such as Seaborn, clear and easy-to-understand charts were drawn and displayed on the monitor screen for reference and analysis by relevant personnel. In this way, decision makers can quickly understand the meaning behind the data and provide strong support for the future management and utilization of marine fishery resources.

[0068] Data storage. In the data storage stage, in order to effectively manage the analyzed marine fishery resource data, a main folder will be created first to store the data. Then, according to the source and nature of the data, the data will be divided into two categories: breeding and fishing. Subfolders will be created for each of these two types of data. Within each subfolder, further subfolders will be set up to store different types of fishery resource data. The local server will be selected as the storage medium to ensure the security and accessibility of the data. At the same time, Excel will be selected as the data storage format because it has powerful data processing and visualization capabilities, which facilitates subsequent data analysis and utilization. Through such classification and storage methods, marine fishery resource data can be efficiently managed, providing strong support for subsequent data analysis and decision-making;

[0069] In the process of big data analysis, it is crucial to conduct in-depth mining and analysis of the stored marine fishery resource data. Based on the existing data storage, big data technology, especially distributed computing clusters, will be used to efficiently process and analyze massive data. First, Spark distributed computing clusters will be built according to hardware resources and business needs. Spark has become the preferred tool for big data analysis with its efficient data processing and powerful data analysis capabilities. Subsequently, the prepared data will be loaded into the Spark cluster, but before that, in order to reduce the cluster processing burden and improve transmission efficiency, the data will be compressed to a suitable size. After the data is loaded into the Spark cluster, statistical analysis methods will be used to descriptively analyze the data. Analysis, such as calculating statistics such as mean and standard deviation, is used to understand the basic characteristics of the data. Then, cluster analysis methods are used to further explore the hidden patterns and associations in the data. Through clustering, the data can be divided into different groups to reveal the inherent connections and potential laws between the data. Based on the mined patterns and associations, a decision support model will be established to provide a scientific basis for the operation and decision-making of the fishery e-commerce platform. These models can help the platform optimize supply chain management, predict market trends, formulate sales strategies, etc. Finally, in order to make the analysis results more intuitive and easy to understand, they will be visualized in the form of charts, images, etc. Through data visualization, the complex relationships and patterns in the data can be presented to users in an intuitive way, helping them better understand the connotation and value of the data.

[0070] In summary, big data analysis can not only help to deeply understand marine fishery resource data, but also provide strong support for the operation and decision-making of fishery e-commerce platforms, and promote the sustainable development and innovation of the platforms.

[0071] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0072] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for processing marine fishery resources e-commerce data, characterized by: The following steps are involved: S1: Data collection, using satellite remote sensing to obtain local marine fishery resource information, and centrally collect the local marine fishery information obtained; S2: Data processing: filtering, removing noise, and cleaning the collected data, and converting and standardizing the cleaned data; S3: Data analysis: statistical analysis of the processed data, establishment of a prediction model, prediction of future catch trends and possible influencing factors, and conversion of data into charts and graphs, which are then displayed on the monitor screen; S4: Data storage, storing the analyzed data, classifying the cultured and caught marine fishery resources and creating folders, and setting up subfolders inside the folders to store different types of fishery resources; S5: Big data analysis: Based on data storage, use big data technology to conduct deeper mining and analysis of data, use distributed computing clusters to process and analyze massive data, and use statistical analysis methods to discover hidden patterns and correlations in the data, providing support for the operation and decision-making of the fishery e-commerce platform.

2. A method for processing marine fishery resources e-commerce data according to claim 1, characterized in that: The step S1 specifically includes using MODIS satellite remote sensing data to obtain local marine fishery data, removing clouds, performing atmospheric correction, and geometric correction on the obtained data, centrally collecting the pre-processed satellite remote sensing data, and establishing a database to centrally manage the obtained data.

3. A method for processing marine fishery resources e-commerce data according to claim 2, characterized in that: The filtering and noise removal in step S2 uses median filtering to process abnormal values ​​and noise in the data.

4. A method for processing marine fishery resources e-commerce data according to claim 1, characterized in that: The cleaning step in step S2 comprises: A1: Deduplication: Use hash algorithm to remove duplicate data. A2: Missing value processing, using the median algorithm to fill in missing values; A3: Outlier processing: using the Z-score method to identify and process outliers based on the distribution characteristics of the data; A4: Data format conversion: converting unstructured or semi-structured data into structured data.

5. A method for processing marine fishery resources e-commerce data according to claim 1, characterized in that: The statistical analysis in step S3 includes calculating the mean, median, mode, standard deviation, and variance statistics of the data, and analyzing the central tendency, dispersion, and distribution of the data.

6. A method for processing marine fishery resources e-commerce data according to claim 1, characterized in that: The establishment of the prediction model in step S3 comprises the following steps: B1: Data preparation, selecting independent variables of water temperature, salinity, wind direction, wind speed and dependent variable of catch, and dividing into training set and test set; B2: Model selection, select the logistic regression prediction model; B3: Model training: bring the training data into the logistic regression prediction model to train the logistic regression prediction model and adjust the parameters of the model until the optimal prediction result is achieved; B4: Model evaluation: Use the test set data to evaluate the model, calculate the model's prediction accuracy, recall rate, and F1 value, and evaluate the model's performance.

7. A method for processing marine fishery resources e-commerce data according to claim 1, characterized in that: The data in step S3 is converted into a chart form, including selecting a chart type, selecting a bar chart format, calculating the total catch of each time period, calculating the weight of each influencing factor, and converting the calculated data into a bar chart format, using the Seaborn data visualization tool to draw a chart, and projecting the drawn chart onto the screen of the display.

8. A method for processing marine fishery resources e-commerce data according to claim 1, characterized in that: The step S4 specifically includes selecting a local server as a storage medium, selecting an Excel storage format, creating a main folder for marine fishery resources, and creating two subfolders according to breeding and fishing, and further refining and distinguishing the two subfolders according to the types of economic crops in the fishery.

9. A method for processing marine fishery resources e-commerce data according to claim 1, characterized in that: The step S5 specifically includes integrating the stored data, processing the data, converting the integrated data into Excel, building a Spark distributed computing cluster according to hardware resources and business needs, loading the prepared data into the Spark distributed computing cluster, using statistical analysis methods to perform descriptive analysis on the data, using cluster analysis methods to discover hidden patterns and associations in the data, establishing a decision support model based on the mined patterns and associations, and visually displaying the analysis results in the form of charts, images, etc., so as to more intuitively understand the patterns and associations in the data.

10. A method for processing marine fishery resources e-commerce data according to claim 9, characterized in that: In step S5, the prepared data is loaded into the Spark distributed computing cluster, and the prepared data needs to be compressed.

Images