Feature derivation method and system for animal science original data

Abstract

The invention relates to the technical field of animal science data processing, and particularly discloses a feature derivation method and system for animal science original data. The method comprises the following steps: acquiring animal science original data, and constructing and storing a table structure in combination with data types, field meanings and user requirements; the method comprises the following steps: analyzing single-table column features and multi-table association relationships through data exploration, carrying out feature derivation based on an exploration conclusion, deeply mining data implicit association and internal laws, and converting the data implicit association and internal laws into quantifiable and explainable feature indexes. According to the scheme, the data quality is improved in a structured storage structured data form, the potential value of the data is released through a systematic feature derivation strategy, and the data information density and the analysis efficiency are enhanced. The technology breaks through the limitation of traditional data analysis, provides accurate and comprehensive data support for animal scientific research, assists in revealing essential rules and core mechanisms behind data, and lays a solid foundation for field theoretical innovation and practical application; the method is suitable for multiple directions in the animal science field and is wide in application range.

Description

Technical Field

[0001] This invention relates to the field of animal science data processing technology, specifically to a method and system for feature derivation of raw animal science data. Background Technology

[0002] With the continuous expansion of animal science research and the increasing sophistication of technological means, data has become a core element driving innovation in this field, broadly encompassing multiple dimensions such as animal genetic information, growth and development data, disease monitoring records, and behavioral observation data. However, raw data often fails to directly meet the needs of in-depth research and precise decision-making after collection, and prominent problems such as non-standard storage, poor data quality, and insufficient feature mining are prevalent, severely restricting the realization of data value.

[0003] In terms of data storage, the sources of raw data in animal science are complex and diverse, including laboratory test reports, field observation records, and data from breeding management systems, with significant differences in data formats across different data sources. Furthermore, the meanings of data fields vary considerably depending on research objectives and recording habits. Failure to accurately grasp the data types and core meanings of fields can easily lead to unreasonable table structure design.

[0004] Regarding data quality, various problems occur frequently and have far-reaching impacts. Duplicate records are common, such as identical data appearing in multiple animal weight measurements, which not only causes redundant storage space but also interferes with the accuracy of analysis results. The data also contains a large number of missing values, outliers, and erroneous values, such as missing symptom fields in animal disease monitoring data, incorrect descriptions that do not conform to the logic of disease occurrence, and abnormal fluctuations in indicators such as weight and body length in animal growth data. In addition, the data in the same column are often inconsistent in type, format, and unit of measurement. For example, animal weight records contain both kilograms and grams, which brings great difficulties to data integration and comparative analysis.

[0005] In terms of feature mining, the features of raw data are often superficial and difficult to support the needs of complex scientific research. For example, in animal behavior research, simply recording the time and area of ​​activity cannot provide in-depth insights into the animal's behavioral patterns and habits. Traditional feature derivation methods are simple in process and have limited functionality, making it difficult to fully explore the potential value of the data. For instance, in animal genetic breeding research, relying solely on basic genetic marker data makes it difficult to accurately assess the genetic potential of breeding stock.

[0006] Therefore, existing technologies are clearly insufficient in addressing the above problems, making it difficult to efficiently and accurately extract valuable information from raw data, resulting in insufficient efficiency and accuracy in data analysis. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a feature derivation method and system for raw animal science data, so as to solve the problems of disordered storage, poor quality and insufficient feature mining of raw animal science data in the prior art.

[0008] According to a first aspect of the present invention, a method for feature derivation of raw animal science data is provided, comprising: Obtain raw animal science data, analyze it to determine its type, and interpret the meaning of its fields; based on user needs, construct a table structure according to the type and meaning of the raw animal science data; and store the raw animal science data in tabular form according to the table structure. Analyze the data in each column of a single stored table and the relationships between multiple tables to obtain data exploration results; Based on the data exploration results, aggregate one or more rows of data in the stored table to obtain a new feature index; based on the data exploration results, transform one or more columns of data in the stored table to output one or more new potential feature columns.

[0009] Preferably, after storing the raw animal science data in tabular form, the method further includes: Data cleaning is performed on the raw animal science data stored in tabular form, including: removal of duplicate and redundant data, handling of missing values, identification and removal of outliers, and correction of spelling errors.

[0010] Preferably, the types of raw animal science data obtained from the analysis include: The types of raw animal science data include numeric, character, date / time, and Boolean types.

[0011] Preferably, after obtaining the data exploration results, it also includes: Identify and delete meaningless character data in the table; Identify meaningful character data in the table and convert it into corresponding numerical data according to preset rules.

[0012] Preferably, the operators for the aggregation operation include, but are not limited to: Calculate the maximum value, minimum value, mean, sum, median, standard deviation, count, extract the nth largest value after sorting, extract the nth smallest value after sorting, determine the monotonicity of data, calculate kurtosis, calculate skewness, calculate variance, or calculate the number of values ​​greater than the mean.

[0013] Preferably, the operators of the transformation operation include, but are not limited to: Square, root, logarithmic operations, trigonometric function operations, cumulative summation, cumulative counting, cumulative mean, four arithmetic operations, calculation of percentiles, data offset, rolling count, rolling mean, rolling summation, or polynomial feature generation.

[0014] Preferably, the method further includes: Identify special character data from stored tables; Feature derivation of the special character data includes, but is not limited to: If the special character data is a species name, then identify the genus name and species name corresponding to the species name; If the special character data is a DNA sequence, the sequence length, number of specific base pairs, and GC content of the DNA sequence are calculated. If the special character data is an animal ID with encoding rules, then the sex and breed are extracted from the animal ID according to the encoding rules.

[0015] According to a second aspect of the present invention, a feature derivation system for raw animal science data is provided, comprising: The storage module is used to acquire raw animal science data, analyze and determine the type of the raw animal science data, and interpret the meaning of the fields in the raw animal science data; based on user needs, it constructs a table structure according to the type and meaning of the fields in the raw animal science data; and stores the raw animal science data in tabular form according to the table structure. The derivation module is used to analyze the data in each column of a stored single table and to analyze the relationships between multiple tables to obtain data exploration results. Based on the data exploration results, it performs aggregation operations on one or more rows of data in the stored table to obtain a new feature index. Based on the data exploration results, it performs transformation operations on one or more columns of data in the stored table to output one or more new potential feature columns.

[0016] Preferably, the feature derivation system for the raw animal science data further includes: The cleaning module is used to clean the raw animal science data stored in tabular form, including: removing duplicate and redundant data, handling missing values, identifying and removing outliers, and correcting spelling errors. The derived module is also used to identify meaningless character data in the table and delete it; identify meaningful character data in the table and convert the meaningful character data into corresponding numerical data according to preset rules.

[0017] Preferably, the feature derivation system for the raw animal science data further includes: A special derivation module is used to identify special character data from a stored table; and to perform feature derivation on the special character data, including: if the special character data is a species name, then identifying the genus name and species name corresponding to the species name; if the special character data is a DNA sequence, then calculating the sequence length, number of specific base pairs and GC content of the DNA sequence; if the special character data is an animal number with encoding rules, then extracting the sex and strain from the animal number according to the encoding rules.

[0018] The technical solution provided by this invention may include the following beneficial effects: It is understood that the technical solution presented in this invention can acquire raw animal science data, construct and store table structures based on the data type and field meaning, combined with user needs; conduct data exploration, analyze the data in each column of a single stored table, and analyze the relationships between multiple tables; and perform feature derivation based on the data exploration results to deeply mine the value of the data. It is understood that this technical solution improves data quality through structured data storage; utilizes feature derivation to mine potential features, thereby enhancing the analytical value of the data; and is applicable to multiple directions in the field of animal science, with a wide range of applications.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0021] Figure 1 This is a schematic diagram illustrating the steps of a method for deriving features from raw animal science data according to an exemplary embodiment. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0023] In one embodiment, Figure 1 This is a schematic diagram illustrating the steps of a method for feature derivation from raw animal science data according to an exemplary embodiment. See also: Figure 1 This provides a method for feature derivation from raw animal science data, including: Step S11: Obtain raw animal science data, analyze and determine the type of raw animal science data, and interpret the meaning of the fields in the raw animal science data; based on user needs, construct a table structure according to the type and meaning of the fields in the raw animal science data; and store the raw animal science data in tabular form according to the table structure.

[0024] In practice, this step involves structured data storage. During the structured data storage phase, the first aspect is to systematically clarify the data types of raw data in the field of animal science.

[0025] Preferably, the types of the raw animal science data include numeric, character, date / time, and Boolean types.

[0026] Numerical data is mainly used to quantify various measurable indicators of animals, such as weight, body length, and body width reflecting growth status, and milk production and egg production reflecting production performance. Character data focuses on carrying qualitative descriptive information, including species name, disease name, feed type, and breeding site. Date and time data is used to accurately record key time-related nodes, such as the animal's birth time, vaccination time, experimental observation time, and sampling time. Boolean data is used to represent binary states, such as whether the animal is sick, whether it has completed quarantine, and whether it has reached the slaughter standard.

[0027] In the data structure storage stage, on the other hand, it is necessary to confirm the meaning of each data field one by one to ensure that the understanding of the field meaning is accurate and complete, and to avoid logical loopholes in subsequent data processing due to misinterpretation of the field.

[0028] After obtaining the data types and field meanings of the raw animal science data, it is necessary to construct the table structure based on user needs. For example, considering the specific research needs of animal genetics and breeding, disease prevention and control, and aquaculture management, a scientifically sound and highly adaptable database table structure should be designed to ensure that the data can be accurately parsed and stored in a standardized manner. Taking animal growth data storage as an example, indicators that need to participate in statistical calculations, such as weight and body length, should be clearly defined as numerical types; fields recording classification attributes such as animal breed and lineage should be set as character types; and time-related information such as birth date should be standardized as date-time type, ensuring that quantitative indicators can support subsequent calculations, classification information is accurately described, and time nodes are recorded in a standardized manner.

[0029] In one exemplary embodiment, original record data related to pig farming is obtained, specifying that "weight" is a numerical type, representing the weight of the pig at a specific point in time; "date of birth" is a date-time type, recording the time of the pig's birth; and "breed" is a character type. A thorough clarification of the core meaning of each field lays the foundation for subsequent processing steps.

[0030] Based on data types and field meanings, and considering both practical and research needs in livestock management, a suitable table structure was designed. In addition to the basic fields mentioned above, necessary fields such as "Pig Number" (used to uniquely identify each pig) and "Batch Number" (records the batch information to which the pig belongs) were added to ensure that the raw data can be accurately parsed and stored in a standardized manner.

[0031] Finally, following the designed table structure, the original data was accurately entered into the database, strictly adhering to the table structure definition to ensure data integrity and consistency. When storing "weight," it was necessary to ensure that its format conformed to the requirements of a numeric field, and the date and time format of "date of birth" was validated to ensure data accuracy and reliability.

[0032] In a preferred embodiment, after performing step S11, data cleaning can be performed on the stored data. Data cleaning of the raw animal science data stored in tabular form includes: removal of duplicate and redundant data, handling of missing values, identification and removal of outliers, and correction of spelling errors.

[0033] Redundant data removal can be performed by processing rows and columns separately. Row processing identifies and removes completely duplicate rows to avoid data redundancy; column processing standardizes column names across different data sources and removes duplicate columns to make the data structure clearer.

[0034] For example, in pig farming records, duplicate records may occur due to human error or system malfunction. Therefore, row processing can effectively avoid data redundancy by deduplicating data, thus improving data processing efficiency and the accuracy of analysis results. Column processing can standardize column naming, solving the problem of inconsistent names for the same field in different data sources (e.g., unifying "gross weight" and "actual weight" as "weight"); it can also check and delete duplicate columns, avoiding data redundancy and confusion, and making the data structure more concise and efficient.

[0035] For missing values, appropriate methods such as imputation and deletion are used: for missing values ​​in important fields, imputation can be performed based on other relevant data; for records with a high proportion of missing values ​​and minimal impact on the analysis, deletion can be selected. Outliers and invalid values ​​are identified and removed. For example, when processing animal reproduction data, if a litter size is negative or far exceeds the normal physiological range, it must be verified before processing. Values ​​with spelling errors are corrected, and the data type, format, and unit of measurement for the same column are standardized (e.g., animal weight is standardized to the same unit) to ensure data consistency and standardization.

[0036] In pig farming records, for missing values ​​in numerical fields such as "weight," if the data is evenly distributed, the mean should be used for filling; if the data is skewed, the median should be considered. Missing values ​​in character fields can be retained or marked as "unknown" if they do not affect the analysis. Identify and handle outliers in fields such as "weight" (e.g., negative numbers, values ​​far exceeding the normal growth range of pigs), verifying and correcting them based on actual farming conditions. Check and correct values ​​with spelling errors, and standardize the "date of birth" field to the "YYYY / MM / DD" format to ensure data consistency and standardization, facilitating subsequent analysis and calculations.

[0037] Step S12: Analyze the data in each column of a single stored table and analyze the relationships between multiple tables to obtain data exploration results.

[0038] Steps S12 and S13 are the core of this embodiment and are the feature derivation stage.

[0039] The first step is the data exploration phase, S12. This phase requires in-depth analysis of the data types (such as numeric, character, and date) of each column in a single data table, and data exploration based on the column data. It also involves identifying the relationships between tables through primary keys, foreign keys, and other related fields, and, combined with professional knowledge in animal science, accurately grasping the inherent logic and relational structure of the data. For example, in animal genetic research, the focus is on analyzing the potential relationships between gene sequence data and phenotypic data such as animal growth traits, disease resistance traits, and reproductive traits, providing a clear direction for subsequent feature derivation.

[0040] Taking pig farming records as an example, data exploration can delve into the "weight" field, obtaining statistical information such as mean, median, maximum, and minimum values ​​through statistical analysis to understand the central tendency and dispersion of pig weight. It can also study the relationship between "pig numbers" in the "pig information table" and "growth record table," obtaining basic information about the pigs and combining this with growth records for more in-depth analysis, thereby deriving data exploration results.

[0041] In a preferred embodiment, after step S12, the method further includes data type conversion: identifying meaningless character data in the table and deleting it; identifying meaningful character data in the table and converting the meaningful character data into corresponding numerical data according to a preset rule.

[0042] In the data type conversion stage, character-based data without practical analytical value needs to be removed, and character-based data with analytical significance needs to be converted into numerical data. For example, in the animal sex field, "male" and "female" are mapped to 0 and 1 respectively to facilitate subsequent quantitative calculations and analysis. For numerical data, the original data type remains unchanged; for date-based data, the original type is also retained, but targeted information extraction processing is required. For example, key time information such as year, season, and age in months can be extracted from date-based data to provide data support for analyzing the impact of different seasons and age groups on animal growth.

[0043] Taking pig farming records as an example, the "breed" is coded, with "Large White" coded as 1 and "Landrace" coded as 2, etc., to facilitate the calculation of the differences in growth performance between different breeds of pigs.

[0044] Step S13: Based on the data exploration results, perform aggregation operations on one or more rows of data in the stored table to obtain a new feature index; based on the data exploration results, perform transformation operations on one or more columns of data in the stored table to output one or more new potential feature columns.

[0045] This step is the feature derivation operation stage, implemented through two core methods: aggregation and transformation. Aggregation takes one or more rows of similar data as input, performs standardization calculations, and outputs a comprehensive value that provides a summary. For example, it can calculate the average daily weight gain, maximum milk production, total weight gain, and median weight of animals in a specific period at a farm. This allows for the rapid extraction of core statistical features and key information from massive amounts of discrete data, intuitively reflecting the overall patterns of the data.

[0046] Taking pig farming records as an example, aggregation operations can be used to calculate the average daily weight gain, maximum weekly weight gain, or total weight gain of a pig over a period of time. For example, to calculate the average daily weight gain of a pig over 30 days, divide the weight gain over 30 days by 30 to understand its growth rate.

[0047] The transformation operation takes one or more columns of related data as input and generates one or more new columns of data through preset operation rules. For example, it can perform square operations on the body size data of animals, such as body length and body width, to analyze body development characteristics, calculate growth age and growth month age based on the animal's birth time and observation time, or process data through arithmetic operations, logarithmic operations, etc., to deeply explore the potential correlations and implicit features behind the data and enrich the analysis dimensions of the data.

[0048] Taking pig farming records as an example, the transformation operation can be to calculate the square of the pig's weight, squaring the "weight" in each pig record to generate a "weight squared" field, and analyzing the impact of weight changes on other growth indicators. Quarterly information can be extracted based on the "date of birth" to generate a "birth quarter" field, and the differences in fattening effects of pigs in different birth quarters can be analyzed.

[0049] In a preferred embodiment, the operators of the aggregation operation include, but are not limited to: calculating the maximum value, calculating the minimum value, calculating the mean, calculating the sum, calculating the median, calculating the standard deviation, counting, extracting the nth largest value after sorting, extracting the nth smallest value after sorting, determining the monotonicity of the data, calculating kurtosis, calculating skewness, calculating variance, or calculating the number of values ​​greater than the mean.

[0050] In a preferred embodiment, the operator of the transformation operation includes, but is not limited to: Square, root, logarithmic operations, trigonometric function operations, cumulative summation, cumulative counting, cumulative mean, four arithmetic operations, calculation of percentiles, data offset, rolling count, rolling mean, rolling summation, or polynomial feature generation.

[0051] Aggregation operations or transformation operations are simply referred to as operators. The operators listed here are only a subset, not all, of the operators. In actual feature derivation processes, many other operators exist, which can be existing mathematical / statistical methods or custom-defined ones. For aggregation operators, any operation that aggregates data from one or more rows is an aggregation operator; for transformation operators, any operation that transforms data from one or more columns is a transformation operator.

[0052] In a preferred embodiment, the method further includes: identifying special character data from a stored table; performing feature derivation on the special character data, including: if the special character data is a species name, identifying the genus name and species name corresponding to the species name; if the special character data is a DNA sequence, calculating the sequence length, number of specific base pairs, and GC content of the DNA sequence; if the special character data is an animal number with encoding rules, extracting the sex and strain from the animal number according to the encoding rules.

[0053] This embodiment specifically designs a feature derivation strategy for special character-based data such as species names, DNA sequences, and animal numbers with special coding rules, to ensure that the value of this type of data is fully explored.

[0054] In practice, character segmentation technology can be used to break down the core structure of species names into "genus name-species name" to accurately extract key information such as genus name and species name. This allows for classification and statistical analysis, providing data support for studying the distribution density and population structure of different species in specific regions.

[0055] For DNA sequences, the focus is on calculating core genetic indicators such as sequence length, number of specific base pairs (e.g., AT, CG), and GC content. These quantitative characteristics provide crucial evidence for studies such as genetic diversity assessment, kinship identification, and gene function analysis.

[0056] For animal numbers with coding rules, by parsing their preset coding logic, key information such as sex, breed, birth batch, and breeding unit can be accurately extracted, providing efficient data support for genetic pedigree construction, breeding stock selection, and breeding traceability.

[0057] The aforementioned specialized derivation methods for special character-type data effectively enrich the feature dimensions of the data, breaking through the limitations of traditional data processing that makes it difficult to directly analyze such data. This provides more comprehensive and accurate information support for diverse directions in animal science, such as species surveys, genetic research, and breeding planning.

[0058] The technical solution presented in this invention effectively addresses the problems of unreasonable and poor-quality data storage in animal science through data structuring and cleaning steps, making the data more standardized, accurate, and complete, thus laying a solid foundation for subsequent research. By leveraging the aggregation and transformation operations in the feature derivation step and the dedicated derivation strategy for special character-type data, the potential features of the original data are deeply mined, helping researchers understand the core laws of animal growth, genetics, and diseases. Furthermore, this method is applicable to multiple animal science fields such as genetic breeding, disease prevention and control, aquaculture management, and behavioral research, meeting the data analysis needs of different scenarios and providing strong support for researchers and aquaculture enterprises. The processed and derived data can be directly used for in-depth analysis, reducing data processing time and costs, and significantly improving the efficiency and accuracy of data analysis. Simultaneously, this method supports implementation in multiple programming languages ​​such as SQL, Python, R, and Java, allowing different users to flexibly choose according to their own technology stack, greatly enhancing the practicality and scalability of the invention.

[0059] In another embodiment, a feature derivation system for raw animal science data is provided, comprising: The storage module is used to acquire raw animal science data, analyze and determine the type of the raw animal science data, and interpret the meaning of the fields in the raw animal science data; based on user needs, it constructs a table structure according to the type and meaning of the fields in the raw animal science data; and stores the raw animal science data in tabular form according to the table structure. The derivation module is used to analyze the data in each column of a stored single table and to analyze the relationships between multiple tables to obtain data exploration results. Based on the data exploration results, it performs aggregation operations on one or more rows of data in the stored table to obtain a new feature index. Based on the data exploration results, it performs transformation operations on one or more columns of data in the stored table to output one or more new potential feature columns.

[0060] In the derivation module, the stored original data table contains a large amount of raw data in row / column dimensions. This data can be used to generate a large number of rich new features through various aggregation or transformation operations (different operators). The specific derivation process can be implemented through the following steps: Derivative analysis: Exploratory analysis is conducted on the data characteristics of each column in a single data table and the relationships between multiple data tables, and the results of the data exploration are output. Aggregation-generated features: Based on the data exploration results, perform aggregation operations on one or more rows of data in the data table to generate new feature indicators; Transformation-generated features: Based on the results of data exploration, perform transformation operations on one or more columns of data in the data table to generate one or more potential new feature columns.

[0061] Preferably, the feature derivation system for the raw animal science data further includes: The cleaning module is used to clean the raw animal science data stored in tabular form, including: removing duplicate and redundant data, handling missing values, identifying and removing outliers, and correcting spelling errors. The derived module is also used to identify meaningless character data in the table and delete it; identify meaningful character data in the table and convert the meaningful character data into corresponding numerical data according to preset rules.

[0062] Preferably, the feature derivation system for the raw animal science data further includes: A special derivation module is used to identify special character data from a stored table; and to perform feature derivation on the special character data, including: if the special character data is a species name, then identifying the genus name and species name corresponding to the species name; if the special character data is a DNA sequence, then calculating the sequence length, number of specific base pairs and GC content of the DNA sequence; if the special character data is an animal number with encoding rules, then extracting the sex and strain from the animal number according to the encoding rules.

[0063] Special derivative modules include, but are not limited to, the methods mentioned above. Other special derivative methods depend on the specific circumstances encountered, and sometimes it is necessary to customize special derivative methods.

[0064] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0065] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0066] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0067] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0068] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0069] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0070] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0071] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0072] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A feature derivation method for animal science raw data, characterized in that, The method comprises the following steps: acquiring animal science raw data, analyzing the type of the animal science raw data, and interpreting the field meaning of the animal science raw data; constructing a table structure according to the type of the animal science raw data and the field meaning in combination with user demand; storing the animal science raw data in the form of a table according to the table structure; analyzing the data in each column of a single table and the association between multiple tables to obtain a data exploration result; performing an aggregation operation on one or more rows of data in the stored table according to the data exploration result to obtain a new characteristic index, and performing a conversion operation on one or more columns of data in the stored table according to the data exploration result to output a new column or multiple new columns of potential characteristic columns.

2. The method of claim 1, wherein, After storing the animal science raw data in the form of a table, the method further comprises the following steps: performing data cleaning on the animal science raw data stored in the form of a table, including removing redundant data, processing missing values, identifying and removing abnormal values, and correcting spelling errors.

3. The method of claim 1, wherein, analyzing the type of the animal science raw data comprises the following steps: the type of the animal science raw data includes numerical type, character type, date and time type, and Boolean type.

4. The method of claim 3, wherein, After obtaining the data exploration result, the method further comprises the following steps: identifying meaningless character type data in the table and deleting the meaningless character type data; identifying meaningful character type data in the table, and converting the meaningful character type data into corresponding numerical type data according to a preset rule.

5. The method of claim 1, wherein, The operators of the aggregation operation at least include: calculating the maximum value, calculating the minimum value, calculating the average value, calculating the total sum, calculating the median, calculating the standard deviation, counting, extracting the nth largest value after arrangement, extracting the nth smallest value after arrangement, judging the monotonicity of data, calculating the kurtosis, calculating the skewness, calculating the variance, or calculating the number of values greater than the average value.

6. The method of claim 1, wherein, The operators of the conversion operation at least include: squaring, square root, logarithmic operation, trigonometric function operation, cumulative summation, cumulative count, cumulative average, four arithmetic operations, calculating the percentile, data offset, rolling count, rolling average, rolling summation, or polynomial feature generation.

7. The method of claim 1, wherein, The method further comprises the following steps: identifying special character type data from the stored table; performing feature derivation on the special character type data, including: if the special character type data is a scientific name of a species, identifying the genus name and the species name corresponding to the scientific name of the species; if the special character type data is a DNA sequence, calculating the sequence length, the number of specific base pairs, and the GC content of the DNA sequence; if the special character type data is an animal number with a coding rule, extracting the gender and strain from the animal number according to the coding rule.

8. A feature derivation system for animal science raw data, characterized by, The method comprises the following steps: a storage module is configured to acquire animal science raw data, analyze the type of the animal science raw data, and interpret the field meaning of the animal science raw data; a table structure is constructed according to the type of the animal science raw data and the field meaning in combination with user demand; the animal science raw data is stored in the form of a table according to the table structure; The derivative module is configured to analyze data of each column of the stored single table and analyze correlations between multiple tables to obtain a data exploration result; perform an aggregation operation on one or more rows of data in the stored table according to the data exploration result to obtain a new characteristic index; and perform a conversion operation on one or more columns of data in the stored table according to the data exploration result to output one or more new potential characteristic columns.

9. The animal science raw data feature derivation system of claim 8, wherein, Further comprising: The cleaning module is configured to perform data cleaning on the animal science raw data stored in the form of a table, including: removing repeated redundant data, processing missing values, identifying and removing abnormal values, and correcting spelling errors; The derivative module is further configured to identify meaningless character type data in the table and delete the meaningless character type data; and identify meaningful character type data in the table, and convert the meaningful character type data into corresponding numerical type data according to a preset regulation.

10. The animal science raw data feature derivation system of claim 9, wherein, Further comprising: The special derivative module is configured to identify special character type data from the stored table; The special character type data is subjected to feature derivation, including: if the special character type data is a scientific name of a species, identifying a genus name and a species name corresponding to the scientific name of the species; If the special character type data is a DNA sequence, the sequence length, the number of specific base pairs, and the GC content of the DNA sequence are calculated; and if the special character type data is an animal number with a coding rule, the gender and strain are extracted from the animal number according to the coding rule.

Images