Feature extraction method and device, electronic equipment and storage medium

Abstract

The embodiment of the application discloses a feature extraction method and device, electronic equipment and storage medium, relates to the computer technical field, and can greatly reduce the difficulty and risk of development test of an artificial intelligence model.The method comprises the following steps: determining the data attribute of target data to be subjected to feature extraction; in the case that the data attribute is offline data, performing a preset offline feature extraction plan on the target data, or in the case that the data attribute is online data, performing a preset online feature extraction plan on the target data; the offline feature extraction plan comprises a first operation flow of performing feature extraction on offline data according to a preset feature extraction algorithm, and the online feature extraction plan comprises a second operation flow of performing feature extraction on online data according to the feature extraction algorithm; each operation node constituting the first operation flow and each operation node constituting the second operation flow have a consistency relationship.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a feature extraction method and apparatus, electronic device, and storage medium. Background Technology

[0002] In machine learning scenarios, feature extraction, feature signing, training, and the generation of an AI model need to be performed offline first. Then, feature extraction and feature signing are performed online, and the results are calculated by combining the offline AI model. In most engineering practices, it is required that the online feature extraction calculation be consistent with the offline feature extraction calculation; otherwise, the online prediction results will deviate from the model's expected performance.

[0003] Because offline and online feature extraction have different data volume and latency requirements, related technologies typically perform feature extraction in two separate execution environments, one offline and one online, with developers maintaining computational consistency between the two. This includes maintaining consistent computational logic, data boundary handling logic, function exception handling mechanisms, and data precision. However, this requires significant time and effort, greatly increasing the difficulty and risk of developing and testing AI models, and significantly reducing development and testing efficiency. Summary of the Invention

[0004] In view of this, one or more embodiments of the present invention provide a feature extraction method and apparatus, electronic device, and storage medium, which can greatly reduce the difficulty and risk of developing and testing artificial intelligence models, and also effectively improve the efficiency of development and testing.

[0005] In a first aspect, embodiments of the present invention provide a feature extraction method, comprising: determining data attributes of target data to be feature extracted, the data attributes including offline data or online data; executing a preset offline feature extraction plan on the target data when the data attribute is offline data, or executing a preset online feature extraction plan on the target data when the data attribute is online data; wherein the offline feature extraction plan includes a first operation flow of extracting features from offline data according to a preset feature extraction algorithm, and the online feature extraction plan includes a second operation flow of extracting features from online data according to the feature extraction algorithm; and each operation node constituting the first operation flow and each operation node constituting the second operation flow have a consistency relationship.

[0006] Optionally, the consistency relationship between the operation nodes constituting the first operation flow and the operation nodes constituting the second operation flow includes: the operation nodes in the first operation flow and the operation nodes in the second operation flow have the same type, and the feature extraction operator used in the first operation flow is the same as the feature extraction operator used in the second operation flow; or, the operation semantics corresponding to the operation nodes in the first operation flow are consistent with the operation semantics corresponding to the operation nodes in the second operation flow, and the feature extraction operator used in the first operation flow is the same as the feature extraction operator used in the second operation flow.

[0007] Optionally, the operation semantics corresponding to the operation nodes in the first operation process being consistent with the operation semantics corresponding to the operation nodes in the second operation process includes: the operation semantics corresponding to one operation node in the first operation process being equal to the operation semantics corresponding to one or more operation nodes in the second operation process; or, the operation semantics corresponding to one or more operation nodes in the first operation process being equal to the operation semantics corresponding to one operation node in the second operation process.

[0008] Optionally, the feature extraction operator includes at least one of the following: expression operator, non-aggregational function operator, and aggregation function operator.

[0009] Optionally, before executing a preset offline feature extraction plan on the target data, or before executing a preset online feature extraction plan on the target data, the method further includes: generating the offline feature extraction plan and the online feature extraction plan respectively according to the feature extraction algorithm recorded in the same preset feature extraction script, and ensuring that the operation nodes in the first operation process and the operation nodes in the second operation process have the consistency relationship.

[0010] Optionally, after executing a preset offline feature extraction plan on the target data, or executing a preset online feature extraction plan on the target data, the method further includes: obtaining feature extraction results of the target data based on the execution results of the offline feature extraction plan or the online feature extraction plan, so as to use the feature extraction results for offline model training or online interactive services.

[0011] Optionally, the offline feature extraction plan is based on a batch data processing method, while the online feature extraction plan is based on a request-response data processing method.

[0012] Secondly, embodiments of the present invention also provide a feature extraction apparatus, comprising: a determining unit, configured to determine data attributes of target data to be feature extracted, the data attributes including offline data or online data; and an execution unit, configured to execute a preset offline feature extraction plan on the target data when the data attribute determined by the determining unit is offline data, or to execute a preset online feature extraction plan on the target data when the data attribute determined by the determining unit is online data; wherein the offline feature extraction plan includes a first operation flow of extracting features from offline data according to a preset feature extraction algorithm, and the online feature extraction plan includes a second operation flow of extracting features from online data according to the feature extraction algorithm; and each operation node constituting the first operation flow and each operation node constituting the second operation flow have a consistency relationship.

[0013] Optionally, the consistency relationship between the operation nodes constituting the first operation flow and the operation nodes constituting the second operation flow includes: the operation nodes in the first operation flow and the operation nodes in the second operation flow have the same type, and the feature extraction operator used in the first operation flow is the same as the feature extraction operator used in the second operation flow; or, the operation semantics corresponding to the operation nodes in the first operation flow are consistent with the operation semantics corresponding to the operation nodes in the second operation flow, and the feature extraction operator used in the first operation flow is the same as the feature extraction operator used in the second operation flow.

[0014] Optionally, the operation semantics corresponding to the operation nodes in the first operation process being consistent with the operation semantics corresponding to the operation nodes in the second operation process includes: the operation semantics corresponding to one operation node in the first operation process being equal to the operation semantics corresponding to one or more operation nodes in the second operation process; or, the operation semantics corresponding to one or more operation nodes in the first operation process being equal to the operation semantics corresponding to one operation node in the second operation process.

[0015] Optionally, the feature extraction operator includes at least one of the following: expression operator, non-aggregational function operator, and aggregation function operator.

[0016] Optionally, the apparatus further includes: a generation unit, configured to generate the offline feature extraction plan and the online feature extraction plan respectively according to the feature extraction algorithm recorded in the same preset feature extraction script before executing a preset offline feature extraction plan on the target data, or before executing a preset online feature extraction plan on the target data, and to ensure that the operation nodes in the first operation process and the operation nodes in the second operation process have the consistency relationship.

[0017] Optionally, the device further includes: a result acquisition unit, configured to obtain feature extraction results of the target data based on the execution results of the offline feature extraction plan or the online feature extraction plan after executing a preset offline feature extraction plan on the target data, or executing a preset online feature extraction plan on the target data, so as to use the feature extraction results for offline model training or online interactive services.

[0018] Optionally, the offline feature extraction plan is based on a batch data processing method, while the online feature extraction plan is based on a request-response data processing method.

[0019] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising: a housing, a processor, a memory, a circuit board, and a power supply circuit, wherein the circuit board is disposed within the space enclosed by the housing, and the processor and the memory are disposed on the circuit board; the power supply circuit is used to supply power to various circuits or devices of the above-mentioned electronic device; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, for executing any feature extraction method provided by embodiments of the present invention.

[0020] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement any of the feature extraction methods provided in the embodiments of the present invention.

[0021] The feature extraction method, apparatus, electronic device, and storage medium provided in the embodiments of the present invention can select an offline feature extraction plan or an online feature extraction plan to extract features from the target data based on the data attributes of the target data to be extracted. Although the offline feature extraction plan and the online feature extraction plan are different, since the first operation process corresponding to the offline feature extraction plan and the second operation process corresponding to the online feature extraction plan are both based on the same feature extraction algorithm, and there is a consistency relationship between the operation nodes that make up the first operation process and the operation nodes that make up the second operation process, it can automatically ensure that the feature extraction of the target data in the offline feature extraction plan and the online feature extraction plan are also consistent. Therefore, developers do not need to spend extra time and effort to maintain the computational consistency between offline feature extraction and online feature extraction, which greatly reduces the difficulty and risk of developing and testing artificial intelligence models, and also effectively improves the efficiency of development and testing. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart of a feature extraction method provided in an embodiment of the present invention;

[0024] Figure 2 A detailed flowchart of a feature extraction method provided for embodiments of the present invention;

[0025] Figure 3 for Figure 2 The illustrated embodiment is a schematic diagram of the first operation flow corresponding to the offline feature extraction plan and the second operation flow corresponding to the online feature extraction plan;

[0026] Figure 4 A schematic diagram of a feature extraction device provided in an embodiment of the present invention;

[0027] Figure 5 A schematic diagram of an electronic device provided as an embodiment of the present invention. Detailed Implementation

[0028] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0030] Firstly, embodiments of the present invention provide a feature extraction method that can greatly reduce the difficulty and risk of developing and testing artificial intelligence models, and also effectively improve the efficiency of development and testing.

[0031] Feature extraction is a crucial step in the development and application of machine learning and artificial intelligence models, directly impacting their accuracy. These AI models can be widely applied in various aspects of industrial production and daily life, such as predicting image categories, text categories, voice sentiment, fraudulent transactions, and ad click-through rates. Accordingly, the feature extraction method in this embodiment can also be applied to these areas. Specifically, the scenarios in which the feature extraction method in this embodiment can be used may include the following:

[0032] Image processing scenarios include: Optical Character Recognition (OCR), face recognition, object recognition, and image classification. More specifically, OCR can be applied to document (such as invoice) recognition, handwritten character recognition, etc.; face recognition can be applied to security and other fields; object recognition can be applied to traffic sign recognition in autonomous driving scenarios; and image classification can be applied to "photo shopping" and "finding similar items" on e-commerce platforms.

[0033] Voice recognition scenarios include products that enable human-computer interaction via voice, such as mobile phone voice assistants (like Apple's Siri) and smart speakers.

[0034] Natural language processing scenarios include: text review (such as contracts, legal documents, and customer service records), spam content identification (such as spam SMS identification), and text classification (sentiment, intent, and topic).

[0035] Automatic control scenarios include: prediction of mine group regulation operations, prediction of wind turbine generator group regulation operations, and prediction of air conditioning system regulation operations; specifically, for mine groups, a set of regulation operations with high mining efficiency can be predicted; for wind turbine generator groups, a set of regulation operations with high power generation efficiency can be predicted; and for air conditioning systems, a set of regulation operations that meet demand while saving energy can be predicted.

[0036] Intelligent question-and-answer scenarios include: chatbots and intelligent customer service.

[0037] Business decision-making scenarios include those in the fintech, healthcare, and municipal sectors. Specifically, the fintech sector includes marketing (such as coupon usage prediction, ad click behavior prediction, and user profiling) and customer acquisition, abnormal transaction identification, underwriting and credit scoring, and commodity price prediction. The healthcare sector includes disease screening and prevention, personalized health management, and assisted diagnosis. The municipal sector includes social governance and regulatory enforcement, resource, environmental, and facility management, industrial development and economic analysis, public services and livelihood security, and smart cities (allocation and management of various urban resources such as public transportation, ride-hailing services, and shared bicycles).

[0038] Recommended business scenarios include: news, advertisements, music, information, videos, and financial products (such as wealth management and insurance).

[0039] Search scenarios include: web search, image search, text search, video search, etc.

[0040] Abnormal behavior detection scenarios include: detection of abnormal electricity consumption behavior of State Grid customers, detection of malicious network traffic, and detection of abnormal behavior in operation logs.

[0041] like Figure 1 As shown, the feature extraction method provided in the embodiments of the present invention may include:

[0042] S11, determine the data attributes of the target data to be feature extracted, the data attributes include offline data or online data;

[0043] The target data can be any data for feature extraction. It can come from offline distributed file systems (such as HDFS), network requests, or online in-memory databases. Depending on the feature extraction requirements, the target data can be categorized into different data attributes. For example, if the target data volume is large but allows for a large response time latency, its data attribute can be offline data; conversely, if the target data volume is small but requires a small response time latency, its data attribute can be online data. Generally, offline sample data can come from offline distributed file systems (such as HDFS), while online data can come from network requests and online in-memory databases.

[0044] S12, if the data attribute is offline data, execute a preset offline feature extraction plan on the target data; or, if the data attribute is online data, execute a preset online feature extraction plan on the target data; wherein, the offline feature extraction plan includes a first operation flow of extracting features from offline data according to a preset feature extraction algorithm, and the online feature extraction plan includes a second operation flow of extracting features from online data according to the feature extraction algorithm; each operation node constituting the first operation flow and each operation node constituting the second operation flow have a consistency relationship.

[0045] In this step, different feature extraction plans can be executed for the target data based on its different data attributes. For example, if the data attribute is offline data, a preset offline feature extraction plan can be executed for the target data; if the data attribute is online data, a preset online feature extraction plan can be executed for the target data. In embodiments of the present invention, the offline feature extraction plan and the online feature extraction plan can specifically refer to an operation flow formed by a series of interconnected operation nodes, where each operation node can represent a relatively independent function or step in the operation flow. Different plans will result in different corresponding operation flows. For example, the offline feature extraction plan can correspond to the first operation flow, which is beneficial for handling large-scale feature extraction operations, while the online feature extraction plan can correspond to the second operation flow, which is beneficial for handling feature extraction operations with low latency.

[0046] In the embodiments of the present invention, since offline feature extraction and online feature extraction have different requirements for data volume and response speed, the first operation process corresponding to the offline feature extraction plan and the second operation process corresponding to the online feature extraction plan may also have different performance focuses. Specifically, the number of operation nodes in the first operation process and the number of operation nodes in the second operation process may be the same or different, and the relationship between each operation node in the first operation process and the relationship between each operation node in the second operation process may also be the same or different.

[0047] To ensure consistency between offline and online feature extraction, in embodiments of the present invention, the operation nodes constituting the first operation flow and the operation nodes constituting the second operation flow can be designed to maintain consistency. That is, although the specific steps of the first and second operation flows may differ, it does not affect the feature extraction result; that is, the feature extraction result obtained after performing feature extraction on the same target data according to either the first or second operation flow is the same.

[0048] The feature extraction method provided by the embodiments of the present invention can select an offline feature extraction plan or an online feature extraction plan to extract features from the target data based on the data attributes of the target data to be extracted. Although the offline feature extraction plan and the online feature extraction plan are different, since the first operation process corresponding to the offline feature extraction plan and the second operation process corresponding to the online feature extraction plan are both based on the same feature extraction algorithm, and there is a consistency relationship between the operation nodes that make up the first operation process and the operation nodes that make up the second operation process, it can automatically ensure that the feature extraction of the target data in the offline feature extraction plan and the online feature extraction plan are also consistent. Therefore, developers do not need to spend extra time and effort to maintain the computational consistency between offline feature extraction and online feature extraction, which greatly reduces the difficulty and risk of developing and testing artificial intelligence models, and also effectively improves the efficiency of development and testing.

[0049] Optionally, in embodiments of the present invention, the consistency relationship between the operation nodes constituting the first operation flow and the operation nodes constituting the second operation flow can take different forms. For example, in one embodiment of the present invention, the consistency relationship between the operation nodes constituting the first operation flow and the operation nodes constituting the second operation flow can be: the operation nodes in the first operation flow and the operation nodes in the second operation flow are of the same type, and the feature extraction operator used in the first operation flow is the same as the feature extraction operator used in the second operation flow.

[0050] Specifically, in embodiments of the present invention, operation nodes can be classified into different types based on the specific operations performed by each operation node. Operation nodes performing the same operation have the same type, while operation nodes performing different operations have different types. For example, if operation node A performs an aggregation operation, operation node B performs a concatenation operation, and operation node C performs an aggregation operation, then operation node A and operation node C have the same type, while operation node A and operation node B have different types. Thus, by ensuring that the operation nodes in the first operation flow and the operation nodes in the second operation flow have the same type, it can be guaranteed that the feature extraction operations performed on the target data by the offline feature extraction plan and the online feature extraction plan are the same.

[0051] To further ensure consistency between the operation nodes constituting the first operation flow and the operation nodes constituting the second operation flow, in addition to ensuring that the operation nodes in the first and second operation flows are of the same type, it is also possible to further ensure that the feature extraction operators used in the first and second operation flows are the same. Here, a feature extraction operator can refer to a function or formula used in the feature extraction process. For example, in one embodiment of the present invention, the feature extraction operator may include one or more of the following: expression operators, non-aggregate function operators, and aggregate function operators. Aggregate functions are basic functions of SQL (Structured Query Language). Aggregate functions perform calculations on a set of values ​​and return a single value; they are also called group functions. Functions other than aggregate functions are non-aggregate functions. In this way, since the feature extraction operators used in the first and second operation processes are the same, it can be guaranteed that the offline feature extraction plan and the online feature extraction plan use the same computational logic, the same data boundary processing logic, the same function exception handling mechanism, and the same data precision. Thus, when performing feature extraction on each feature of each target data, the computational consistency of the two feature extraction plans can be guaranteed at the single feature granularity.

[0052] Optionally, in another embodiment of the present invention, the consistency relationship between each operation node constituting the first operation flow and each operation node constituting the second operation flow can be: the operation semantics corresponding to the operation node in the first operation flow are consistent with the operation semantics corresponding to the operation node in the second operation flow, and the feature extraction operator used in the first operation flow is the same as the feature extraction operator used in the second operation flow.

[0053] Semantics refers to the meaning inherent in language. Simply put, semantics can be seen as the meaning of the concepts represented by things in the real world corresponding to data, and the relationships between these meanings; it is the interpretation and logical representation of data in a certain domain. The operational semantics corresponding to the operational nodes in the first operational flow are consistent with the operational semantics corresponding to the operational nodes in the second operational flow. That is to say, for the operational nodes in the first operational flow, the language expression of the specific operation they perform may differ from that of the operational nodes in the second operational flow, but the operational actions performed by the operational nodes in the first and second operational flows are essentially the same. In other words, the language expressions may differ, but different language expressions have a corresponding relationship.

[0054] Optionally, in embodiments of the present invention, the consistency between the operation semantics corresponding to the operation nodes in the first operation flow and the operation semantics corresponding to the operation nodes in the second operation flow can have multiple forms. For example, the operation semantics corresponding to one operation node in the first operation flow may be equal to the operation semantics corresponding to one or more operation nodes in the second operation flow, or the operation semantics corresponding to one or more operation nodes in the first operation flow may be equal to the operation semantics corresponding to one operation node in the second operation flow. In short, as long as the feature extraction operations performed by the two flows have a corresponding relationship when viewed as a whole from the perspective of the first and second operation flows, it is acceptable.

[0055] To enable feature extraction from target data using corresponding offline or online feature extraction plans based on different data attributes, the feature extraction method provided in this invention can pre-generate offline and online feature extraction plans based on the same feature extraction algorithm. The feature extraction algorithm can be recorded in a script file of a preset type, such as an SQL script file. For example, in one embodiment of this invention, before executing the preset offline or online feature extraction plan on the target data in step S12, the feature extraction method provided in this invention may further include: generating the offline and online feature extraction plans based on the feature extraction algorithm recorded in the same preset feature extraction script, and ensuring that the operation nodes in the first and second operation flows have the aforementioned consistency relationship. Thus, after obtaining the offline and online feature extraction plans based on the same feature extraction algorithm, the corresponding offline or online feature extraction plan can be executed on the target data according to its data attributes.

[0056] Further, after executing a preset offline feature extraction plan on the target data in step S12, or executing a preset online feature extraction plan on the target data, the feature extraction method provided by the embodiments of the present invention may further include: obtaining feature extraction results of the target data based on the execution results of the offline feature extraction plan or the online feature extraction plan, so as to use the feature extraction results for offline model training or online interactive services. For offline model training, a large amount of training data can be obtained by extracting features from a large amount of offline data, and then this training data can be used for model training. For online interactive services, features can be quickly extracted from the request data, and the extracted features can be input into the aforementioned trained model, thereby enabling timely responses to requests.

[0057] Optionally, in one embodiment of the present invention, if the data attribute of the target data is offline data, then in order to improve the model training efficiency, the offline feature extraction plan executed on the target data can be based on a batch data processing method; if the data attribute of the target data is online data, then in order to improve the response speed to online requests, the online feature extraction plan executed on the target data can be based on a request-response data processing method.

[0058] The feature extraction method provided by the present invention will be described in detail below through a specific embodiment.

[0059] like Figure 2 As shown, the feature extraction method provided in the embodiments of the present invention may include:

[0060] S201. Based on the feature extraction algorithm recorded in the preset feature extraction script SQLA, generate offline feature extraction plan plan1 and online feature extraction plan2 respectively, and ensure that the operation nodes in the first operation process corresponding to plan1 and the operation nodes in the second operation process corresponding to plan2 have a consistent relationship.

[0061] The feature extraction script SQLA can be, for example:

[0062]

[0063]

[0064] The SQL script above defines two sets of features: the first set of features is calculated on window w1; the second set of features is calculated on window w2. The two sets of features are concatenated together according to specified concatenation conditions. Specifically, first, window w1 is defined, and three features col1, w1_sum_col3, and w1_avg_col3 are constructed and saved to a temporary table out0. The specific information of features col1, w1_sum_col3, and w1_avg_col3 can be seen in Table 1.

[0065] Table 1

[0066]

[0067] Next, a window w2 is defined, and three features col1, w2_max_col3, and w2_min_col3 are constructed on w2 and saved to a temporary table out1. The specific information of features col1, w2_max_col3, and w2_min_col3 is shown in Table 2.

[0068] Table 2

[0069]

[0070]

[0071] Finally, the two temporary feature tables are concatenated together using col1 to form the final wide feature table.

[0072] The schematic diagrams of the first operation flow corresponding to the offline feature extraction plan 1 established based on SQLA, and the second operation flow corresponding to the online feature extraction plan 2, can be seen as follows: Figure 3 As shown. Figure 3 The correspondence between the operation nodes involved in each operation process is shown in Table 3. Combined with... Figure 3 As shown in Table 3, the consistency between the operation nodes in the first operation process and the operation nodes in the second operation process can be observed.

[0073] Table 3

[0074]

[0075] Specifically, such as Figure 3As shown, the feature extraction method provided by the embodiments of the present invention generates an offline batch plan and an online request plan for offline application environments and online application environments, respectively. The specific information of each node in the offline plan can be shown in Table 4, and the specific information of each node in the online plan can be shown in Table 5. The consistency relationship of each operation node in Table 4 and Table 5 can be shown in Table 6.

[0076] Table 4

[0077]

[0078] Table 5

[0079] Online planning nodes Online computing node description ReqJoin Request Join, appending request rows to the right table. Agg Aggregation, table aggregation functions Rename Rename Node Select The column filter node filters out several columns from the table for output.

[0080] Table 6

[0081]

[0082]

[0083] As shown in Table 6, the feature extraction method provided by the embodiments of the present invention can generate corresponding plan nodes both offline and online for each node in the preset feature extraction script SQLA, and ensure that these nodes are logically consistent. Meanwhile, the feature extraction method provided by the embodiments of the present invention uses the same set of feature extraction operators for both offline and online computations. For example, the same aggregation functions are used offline and online: SUM(), AVG(), MAX(), MIN(), thereby ensuring computational consistency between the two environments at the single feature granularity. Furthermore, since the same set of feature extraction operators is used, issues related to data boundary handling, function exception handling, and data precision consistency can also be avoided.

[0084] S202. Obtain the target data for feature extraction;

[0085] S203. Determine the data attributes of the target data. The data attributes include offline data or online data. If the data attributes of the target data are offline data, then proceed to step S204. If the data attributes of the target data are online data, then proceed to step S206.

[0086] S204. Perform a preset offline feature extraction plan on the target data to obtain offline feature extraction results;

[0087] S205. Train the model based on the offline feature extraction results to obtain the target model, and then the process ends.

[0088] S206. Execute a preset online feature extraction plan on the target data to obtain online feature extraction results;

[0089] S207. Input the online feature extraction results into the target model obtained through training;

[0090] S208. Based on the prediction results of the target model, perform an online response to the target data, and then end.

[0091] Secondly, embodiments of the present invention provide a feature extraction device that can greatly reduce the difficulty and risk of developing and testing artificial intelligence models, and also effectively improve the efficiency of development and testing.

[0092] like Figure 4 As shown, the feature extraction apparatus provided in the embodiments of the present invention may include:

[0093] The determining unit 31 is used to determine the data attributes of the target data to be extracted, the data attributes including offline data or online data;

[0094] The execution unit 32 is configured to execute a preset offline feature extraction plan on the target data when the data attribute determined by the determining unit is the offline data, or to execute a preset online feature extraction plan on the target data when the data attribute determined by the determining unit is the online data; wherein the offline feature extraction plan includes a first operation flow of extracting features from the offline data according to a preset feature extraction algorithm, and the online feature extraction plan includes a second operation flow of extracting features from the online data according to the feature extraction algorithm; there is a consistency relationship between each operation node constituting the first operation flow and each operation node constituting the second operation flow.

[0095] The feature extraction apparatus provided in the embodiments of the present invention can select an offline feature extraction plan or an online feature extraction plan to extract features from the target data based on the data attributes of the target data to be extracted. Although the offline feature extraction plan and the online feature extraction plan are different, since the first operation process corresponding to the offline feature extraction plan and the second operation process corresponding to the online feature extraction plan are both based on the same feature extraction algorithm, and there is a consistency relationship between the operation nodes that make up the first operation process and the operation nodes that make up the second operation process, it can automatically ensure that the feature extraction of the target data in the offline feature extraction plan and the online feature extraction plan are also consistent. Therefore, developers do not need to spend extra time and effort to maintain the computational consistency between offline feature extraction and online feature extraction, which greatly reduces the difficulty and risk of developing and testing artificial intelligence models, and also effectively improves the efficiency of development and testing.

[0096] In one embodiment of the present invention, the consistency relationship between the operation nodes constituting the first operation flow and the operation nodes constituting the second operation flow may include: the operation nodes in the first operation flow and the operation nodes in the second operation flow having the same type, and the feature extraction operator used in the first operation flow being the same as the feature extraction operator used in the second operation flow; or, the operation semantics corresponding to the operation nodes in the first operation flow being consistent with the operation semantics corresponding to the operation nodes in the second operation flow, and the feature extraction operator used in the first operation flow being the same as the feature extraction operator used in the second operation flow.

[0097] In one embodiment of the present invention, the consistency between the operation semantics corresponding to the operation node in the first operation flow and the operation semantics corresponding to the operation node in the second operation flow may include: the operation semantics corresponding to one operation node in the first operation flow is equal to the operation semantics corresponding to one or more operation nodes in the second operation flow; or, the operation semantics corresponding to one or more operation nodes in the first operation flow is equal to the operation semantics corresponding to one operation node in the second operation flow.

[0098] In one embodiment of the present invention, the feature extraction operator may include one or more of the following: expression operator, non-aggregation function operator, and aggregation function operator.

[0099] In one embodiment of the present invention, the feature extraction apparatus provided by the embodiment of the present invention may further include:

[0100] The generation unit is configured to generate the offline feature extraction plan and the online feature extraction plan respectively according to the feature extraction algorithm recorded in the same preset feature extraction script before executing the preset offline feature extraction plan on the target data or executing the preset online feature extraction plan on the target data, and to ensure that the operation nodes in the first operation process and the operation nodes in the second operation process have the consistency relationship.

[0101] In one embodiment of the present invention, the feature extraction apparatus provided by the embodiment of the present invention may further include: a result acquisition unit, configured to obtain the feature extraction result of the target data according to the execution result of the offline feature extraction plan or the online feature extraction plan after executing a preset offline feature extraction plan on the target data, or executing a preset online feature extraction plan on the target data, so as to use the feature extraction result for offline model training or online interactive services.

[0102] In one embodiment of the present invention, the offline feature extraction plan is based on a batch data processing method, and the online feature extraction plan is based on a request-response data processing method.

[0103] above Figure 1 The specific operations shown can be performed by Figure 4 The various units within the device will execute the commands; specific operational details will not be elaborated upon here.

[0104] Thirdly, correspondingly, the embodiments of the present invention provide an electronic device that can greatly reduce the difficulty and risk of developing and testing artificial intelligence models, and also effectively improve the efficiency of development and testing.

[0105] like Figure 5 As shown, an embodiment of the present invention provides an electronic device that may include: a housing 41, a processor 42, a memory 43, a circuit board 44, and a power supply circuit 45. The circuit board 44 is disposed inside the space enclosed by the housing 41, and the processor 42 and the memory 43 are disposed on the circuit board 44. The power supply circuit 45 is used to supply power to various circuits or devices of the above-mentioned electronic device. The memory 43 is used to store executable program code. The processor 42 runs a program corresponding to the executable program code by reading the executable program code stored in the memory 43, for executing the feature extraction method described in any of the foregoing embodiments.

[0106] The specific execution process of the above steps by the processor 42, as well as the steps further executed by the processor 42 by running executable program code, can be found in the description of the foregoing embodiments, and will not be repeated here.

[0107] The electronic device exists in various forms and may have a standalone or distributed computing structure; the present invention does not limit this.

[0108] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement any of the feature extraction methods provided in the foregoing embodiments, thus achieving the corresponding technical effects. This has been described in detail above and will not be repeated here.

[0109] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0110] The various embodiments in this specification are described in a related manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0111] In particular, the device embodiment is basically similar to the method embodiment, so the description is relatively simple. For relevant details, please refer to the description of the method embodiment.

[0112] For ease of description, the above apparatus is described by dividing it into various functional units / modules. Of course, in implementing this invention, the functions of each unit / module can be implemented in one or more software and / or hardware.

[0113] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0114] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A feature extraction method characterized by, The method comprises the following steps: determining a data attribute of target data to be subjected to feature extraction, the data attribute comprising offline data or online data; in a case where the data attribute is the offline data, performing a preset offline feature extraction plan on the target data, or in a case where the data attribute is the online data, performing a preset online feature extraction plan on the target data; wherein the offline feature extraction plan comprises a first operation flow of performing feature extraction on offline data according to a preset feature extraction algorithm, and the online feature extraction plan comprises a second operation flow of performing feature extraction on online data according to the feature extraction algorithm; each operation node constituting the first operation flow and each operation node constituting the second operation flow have a consistency relationship; the feature extraction method is used in any of the following scenarios: an image processing scenario, a speech recognition scenario, a natural language processing scenario, an automatic control scenario, an intelligent question and answer scenario, a business decision-making scenario, a recommended business scenario, a search scenario, and an abnormal behavior detection scenario; the offline data is derived from an offline distributed file system, and the online data is derived from network requests and an online in-memory database; before performing the preset offline feature extraction plan on the target data or performing the preset online feature extraction plan on the target data, the method further comprises: generating the offline feature extraction plan and the online feature extraction plan respectively according to the feature extraction algorithm recorded in the same preset feature extraction script, and ensuring that the operation nodes in the first operation flow and the operation nodes in the second operation flow have the consistency relationship.

2. The method of claim 1, wherein, the consistency relationship between the operation nodes constituting the first operation flow and the operation nodes constituting the second operation flow comprises: the operation nodes in the first operation flow and the operation nodes in the second operation flow are of the same type, and the feature extraction operators used in the first operation flow and the feature extraction operators used in the second operation flow are the same; or the operation semantics corresponding to the operation nodes in the first operation flow are consistent with the operation semantics corresponding to the operation nodes in the second operation flow, and the feature extraction operators used in the first operation flow and the feature extraction operators used in the second operation flow are the same.

3. The method of claim 2, wherein, the consistency of the operation semantics corresponding to the operation nodes in the first operation flow and the operation semantics corresponding to the operation nodes in the second operation flow comprises: the operation semantics corresponding to one operation node in the first operation flow are equal to the operation semantics corresponding to one or more operation nodes in the second operation flow; or the operation semantics corresponding to one or more operation nodes in the first operation flow are equal to the operation semantics corresponding to one operation node in the second operation flow.

4. The method of claim 2, wherein, the feature extraction operators comprise at least one of the following: an expression operator, a non-aggregation function operator, and an aggregation function operator.

5. The method according to any one of claims 1 to 4, characterized in that, after performing the preset offline feature extraction plan on the target data or performing the preset online feature extraction plan on the target data, the method further comprises: According to an execution result of the offline feature extraction plan or the online feature extraction plan, a feature extraction result of the target data is obtained, so as to perform offline model training or online interactive service by using the feature extraction result.

6. The method according to any one of claims 1 to 4, characterized in that, The offline feature extraction plan is based on a batch data processing mode, and the online feature extraction plan is based on a request-response data processing mode.

7. A feature extraction apparatus characterized by comprising: Comprise: A determination unit is configured to determine a data attribute of target data to be subjected to feature extraction, wherein the data attribute comprises offline data or online data; An execution unit is configured to execute a preset offline feature extraction plan on the target data when the data attribute determined by the determination unit is the offline data, or execute a preset online feature extraction plan on the target data when the data attribute determined by the determination unit is the online data; wherein the offline feature extraction plan comprises a first operation flow of performing feature extraction on offline data according to a preset feature extraction algorithm, and the online feature extraction plan comprises a second operation flow of performing feature extraction on online data according to the feature extraction algorithm; each operation node constituting the first operation flow and each operation node constituting the second operation flow have a consistency relationship; A generation unit is configured to generate the offline feature extraction plan and the online feature extraction plan respectively according to the feature extraction algorithm recorded in a same preset feature extraction script before executing the preset offline feature extraction plan on the target data or executing the preset online feature extraction plan on the target data, and ensure that the operation nodes in the first operation flow and the operation nodes in the second operation flow have the consistency relationship. The feature extraction device is used in any of the following scenarios: image processing scenarios, speech recognition scenarios, natural language processing scenarios, automatic control scenarios, intelligent question and answer scenarios, business decision-making scenarios, recommended business scenarios, search scenarios, and abnormal behavior detection scenarios. The offline data is derived from an offline distributed file system, and the online data is derived from network requests and online in-memory databases.

8. The apparatus of claim 7, wherein, The consistency relationship between each operation node constituting the first operation flow and each operation node constituting the second operation flow comprises: The operation nodes in the first operation flow and the operation nodes in the second operation flow are of the same type, and the feature extraction operators used in the first operation flow and the feature extraction operators used in the second operation flow are the same; Or The operation semantics corresponding to the operation nodes in the first operation flow are consistent with the operation semantics corresponding to the operation nodes in the second operation flow, and the feature extraction operators used in the first operation flow and the feature extraction operators used in the second operation flow are the same.

9. The apparatus of claim 8, wherein, The operation semantics corresponding to the operation nodes in the first operation flow are consistent with the operation semantics corresponding to the operation nodes in the second operation flow comprises: The operation semantics corresponding to one operation node in the first operation flow are equal to the operation semantics corresponding to one or more operation nodes in the second operation flow; Or, The operation semantics corresponding to one or more operation nodes in the first operation flow are equal to the operation semantics corresponding to one operation node in the second operation flow.

10. The apparatus of claim 8, wherein, The feature extraction operator comprises at least one of the following: an expression operator, a non-aggregation function operator, and an aggregation function operator.

11. The apparatus of any one of claims 7 to 10, wherein, Further comprising: A result obtaining unit is configured to obtain a feature extraction result of the target data according to an execution result of the offline feature extraction plan or the online feature extraction plan after performing the offline feature extraction plan on the target data or performing the online feature extraction plan on the target data, so as to perform offline model training or online interactive service by using the feature extraction result.

12. The apparatus of any one of claims 7-10, wherein, The offline feature extraction plan is based on a batch data processing mode, and the online feature extraction plan is based on a request-response data processing mode.

13. An electronic device, comprising: The electronic device comprises a shell, a processor, a memory, a circuit board and a power supply circuit, wherein the circuit board is arranged inside a space enclosed by the shell, and the processor and the memory are arranged on the circuit board; the power supply circuit is configured to supply power to each circuit or device of the electronic device; the memory is configured to store executable program codes; and the processor is configured to run programs corresponding to the executable program codes by reading the executable program codes stored in the memory, and execute the feature extraction method according to any one of the preceding claims 1 to 6.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores one or more programs, which can be executed by one or more processors to implement the feature extraction method according to any one of the preceding claims 1 to 6. The computer readable storage medium stores one or more programs, which can be executed by one or more processors to implement the feature extraction method according to any one of the preceding claims 1 to 6.

Images