Data processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a data processing method and device, electronic equipment and storage medium. The data method comprises: obtaining to-be-processed data and a target operator type; determining a target structured data configured in the client according to the target operator type; wherein the target structured data is encapsulated by a plurality of configuration items related to the target operator type; determining a target operator implementation rule corresponding to the target structured data according to each configuration item in the target structured data; and processing the to-be-processed data according to the target operator implementation rule to obtain a target feature. Through the data processing method of the embodiment, the client can realize the operator ability configured flexibly according to its own needs, that is, realize the customized operator ability, and no longer depend on the third-party SDK, thereby avoiding the limitation caused by the third party.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more particularly to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of the mobile internet, mobile devices have become the main platform for human-computer interaction, and client applications on mobile devices are becoming increasingly widespread. To make clients more intelligent, "edge intelligence" is gradually entering people's lives. In the application of edge intelligence, both the model training process and the edge-side inference process utilize the capabilities of operators to convert raw data into a specified input format.

[0003] Currently, in related technologies, for a client to implement operator capabilities, it can only do so by calling a third-party SDK to provide operator capabilities and integrating the corresponding third-party operator functions into the client to achieve data transformation. However, the solution of implementing client-side operator capabilities by calling a third-party SDK depends on the functionality of the third party, can only implement the operator capabilities provided by the third party, cannot implement customized operator capabilities, and does not support flexible configuration of operator capabilities according to requirements. Summary of the Invention

[0004] This invention provides a data processing method, apparatus, electronic device, and storage medium, enabling clients to no longer rely on third parties and achieve flexible configuration of operator capabilities according to their needs.

[0005] The first aspect of this invention provides a data processing method applied to a client, the data processing method comprising:

[0006] Obtain the data to be processed and the target operator type;

[0007] The target structured data configured in the client is determined based on the target operator type; wherein the target structured data is encapsulated by multiple configuration items related to the target operator type;

[0008] Based on each configuration item in the target structured data, determine the target operator implementation rules corresponding to the target structured data;

[0009] The data to be processed is processed according to the target operator implementation rules to obtain the target features.

[0010] Optionally, the client deploys a feature center, which includes multiple pre-configured structured data. Each structured data corresponds to an operator type and an operator implementation rule. Each structured data is encapsulated by multiple configuration items related to their respective operator types. The operator implementation rules include: custom calculation rules or mathematical functions.

[0011] The step of determining the target structured data configured in the client according to the target operator type includes:

[0012] Based on the target operator type, target structured data related to the target operator type is determined from the pre-configured multiple structured data.

[0013] Optionally, the feature center is configured with unified parsing rules for structured data, which can parse all structured data.

[0014] The step of determining the target operator implementation rule corresponding to the target structured data based on each configuration item in the target structured data includes:

[0015] The configuration items in the target structured data are parsed according to the parsing rules to determine the target operator implementation rules corresponding to the target structured data.

[0016] Optionally, the pre-configured structured data included in the feature center is configured by the server and then sent to the client. The data processing method further includes:

[0017] Receive the newly configured structured data sent by the server;

[0018] When the new structured data includes target structured data corresponding to the target operator type, the step of parsing each configuration item in the target structured data according to the parsing rules to determine the target operator implementation rules corresponding to the target structured data includes:

[0019] The configuration items of the target structured data in the new structured data are parsed according to the parsing rules to determine the target operator implementation rules corresponding to the target structured data.

[0020] Optionally, the data to be processed is data generated by the terminal where the client is located;

[0021] The data processing method further includes:

[0022] The target features are input into the model invoked by the client;

[0023] And / or, model training is performed on the client based on the target features.

[0024] A second aspect of the present invention provides a data processing apparatus applied to a client, the data processing apparatus comprising:

[0025] The acquisition module is used to acquire the data to be processed and the target operator type;

[0026] The structured data determination module is used to determine the target structured data configured in the client according to the target operator type; wherein the target structured data is encapsulated by multiple configuration items related to the target operator type;

[0027] The operator implementation rule determination module is used to determine the target operator implementation rule corresponding to the target structured data based on each configuration item in the target structured data.

[0028] The processing module is used to process the data to be processed according to the implementation rules of the target operator to obtain the target features.

[0029] Optionally, the client deploys a feature center, which includes multiple pre-configured structured data. Each structured data corresponds to an operator type and an operator implementation rule. Each structured data is encapsulated by multiple configuration items related to their respective operator types. The operator implementation rules include: custom calculation rules or mathematical functions.

[0030] The structured data determination module includes:

[0031] The structured data determination submodule is used to determine the target structured data related to the target operator type from the pre-configured multiple structured data based on the target operator type.

[0032] Optionally, the feature center is configured with unified parsing rules for structured data, which can parse all structured data.

[0033] The operator implementation rule determination module includes:

[0034] The first parsing module is used to parse each configuration item in the target structured data according to the parsing rules, and determine the target operator implementation rules corresponding to the target structured data.

[0035] Optionally, the pre-configured structured data included in the feature center is configured by the server and then sent to the client. The data processing device further includes:

[0036] The receiving module is used to receive the newly configured structured data sent by the server;

[0037] When the new structured data includes target structured data corresponding to the target operator type, the first parsing module includes:

[0038] The first parsing submodule is used to parse each configuration item of the target structured data in the new structured data according to the parsing rules, and determine the target operator implementation rules corresponding to the target structured data.

[0039] Optionally, the data to be processed is data generated by the terminal where the client is located;

[0040] The data processing device further includes:

[0041] The input module is used to input the target features into the model invoked by the client;

[0042] And / or, a training module for training a model on the client based on the target features.

[0043] A third aspect of the present invention provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the data processing method as described in the first aspect of the present invention.

[0044] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data processing method described in the first aspect of the present invention.

[0045] Using the data processing method provided in this embodiment of the invention, after the client obtains the data to be processed and the target operator type, it can determine the target structured data pre-configured on the client according to the target operator type, and then determine the target operator implementation rules according to the configuration items of the target structured data. Based on these rules, the client processes the data to be processed to obtain target features, thus realizing the operator capabilities on the client side. Since the client in this embodiment of the invention contains target structured data pre-configured according to requirements and encapsulated, the data processing method of this embodiment allows the client to flexibly configure operator capabilities according to its own needs, i.e., to achieve customized operator capabilities, no longer relying on third-party SDKs and avoiding limitations caused by third parties. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating a data processing method according to an embodiment of the present invention;

[0048] Figure 2 This is a flowchart illustrating a data processing method according to an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram illustrating a data processing method according to an embodiment of the present invention.

[0050] Figure 4 This is a structural block diagram of a data processing apparatus provided in an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] As mentioned earlier, with the rapid development of the mobile internet, mobile devices have become the primary platform for human-computer interaction. The advancement of AI technology is making clients increasingly intelligent, and small-scale, low-latency intelligent computing is well-suited for the edge. The "edge," as the final link in the intelligent system, primarily uses inference engines to perform model reasoning. The "edge intelligence" system requires many steps, such as data collection, feature engineering, model training, and edge inference. During edge intelligence applications, such as model training and edge inference, the capabilities of operators are utilized to transform raw data into a specified input format.

[0054] Edge intelligence refers to directly placing inference services in some form (such as an SDK) on the client side, allowing the client to directly call them for model inference. The corresponding solution is to deploy the model in the cloud or on a server, and then provide inference service APIs to clients.

[0055] An operator is a mapping from one function space to another; simply put, it is an operation performed on a function space.

[0056] In existing technologies, clients can only provide operator capabilities by calling third-party SDKs to integrate the corresponding third-party operator functions into the client, enabling the client to perform data transformation during model training and prediction, thereby realizing operator capabilities.

[0057] The above-mentioned solution of implementing client-side operator capabilities by calling a third-party SDK has the following drawbacks:

[0058] 1) It is not flexible enough. It requires the third-party SDK to be integrated into the client in advance, and it can only use the pre-integrated third-party operator capabilities. It does not support online configuration switching.

[0059] 2) The customization level is low, and only third-party SDKs integrated into the client can be used. The operator capabilities provided by the third-party SDK may not be the actual operator capabilities required by the client, and the client does not support configuring operator capabilities according to actual needs. Furthermore, integrating all the operator capabilities provided by the third-party SDK into the client means that some of the operators are unnecessary, wasting the client's storage space. In addition, if the local package of the required operator capabilities is too large, it will have a significant impact on the peer, resulting in high costs. In this case, it is not recommended to use operators from the third-party SDK.

[0060] 3) It relies on the functionality of third-party SDKs and can only implement the operator capabilities that the third party has, but cannot implement operator capabilities that the third party does not have.

[0061] To at least partially address one or more of the aforementioned problems and other potential issues, this invention proposes a data processing method. This method transforms data formats through feature engineering on the client side, processing the data into the required target format (i.e., target features), thus enabling the use of alternative operators on the client side. This solution enhances the flexibility of feature engineering by designing the relevant parameters of the operators as configuration items, allowing customization of the target structured data according to specific needs. Furthermore, by parsing the target structured data on the client side to obtain the target operator implementation rules and performing logical processing, the target features are obtained. This eliminates reliance on third-party SDKs and achieves customized operator capabilities.

[0062] Feature engineering involves performing a series of engineering processes on raw data to extract features, which are then used as input for algorithms and models. Essentially, feature engineering is a process of representing and displaying data. In practice, feature engineering aims to remove impurities and redundancy from raw data, designing more efficient features to characterize the relationship between the problem being solved and the prediction model. Feature engineering typically includes multiple aspects, such as feature schemes, feature acquisition schemes, feature processing, and feature monitoring. This embodiment focuses on feature processing, using feature engineering to process the raw data, completing the transformation from raw data to target data, and enabling the capabilities of operators.

[0063] Please refer to Figure 1 , Figure 1This is a flowchart illustrating a data processing method according to an embodiment of the present invention. The data processing method provided in this embodiment can be applied to a client. The client in this embodiment can be a program providing local services to a user. The client can implement the same software function in different program forms: for example, the client can be an application software (APP), a mini-program, a web browser, etc. Figure 1 As shown, the data processing method in this embodiment may include the following steps:

[0064] Step S11: Obtain the data to be processed and the target operator type.

[0065] In this embodiment, when data processing is performed on the client side, the client can obtain the data to be processed, and in response to the user's confirmation operation, the client can obtain the target operator type corresponding to the confirmation operation. The target operator type is the type of operator specified by the user for processing the data to be processed; for example, the operator type may include: numerical feature normalization, categorical features (such as ordinal encoding, which is typically used to process data with size relationships between categories), data binning, etc. These are just some examples, and this embodiment does not impose any specific limitations on the operator type. The user's confirmation operation can be a touch, selection, or input operation performed by the user on the client interface to specify the target operator type for the current data to be processed.

[0066] Step S12: Determine the target structured data configured in the client according to the target operator type; wherein the target structured data is encapsulated by multiple configuration items related to the target operator type.

[0067] In this embodiment, after obtaining the target operator type, the client can determine the target structured data related to the target operator type in the client. The target structured data is a pre-configured structured data entity object in the client, encapsulated by multiple configuration items related to the target operator type.

[0068] In this embodiment, users can configure various configuration items according to their needs (e.g., parameters related to the target operator). Each configuration item is related to the target operator type, and these multiple configuration items are encapsulated to obtain target structured data that matches the target operator type. Furthermore, this target structured data is stored in the client, allowing the client to directly determine the configured target structured data based on the target operator type after obtaining it.

[0069] Step S13: Determine the target operator implementation rules corresponding to the target structured data based on each configuration item in the target structured data.

[0070] In this embodiment, the target structured data in the client corresponds to the corresponding implementation of the target operator type, and the implementation content of the corresponding implementation is the target operator implementation rule. After the client determines the target structured data, it can determine the target operator implementation rule corresponding to the target structured data based on the various configuration items in the target structured data.

[0071] Step S14: Process the data to be processed according to the target operator implementation rules to obtain the target features.

[0072] In this embodiment, after the client determines the implementation rules of the target operator, it can process the data to be processed obtained by the client according to the implementation rules of the target operator, thereby obtaining the processed target features and realizing the operator capabilities on the client.

[0073] In this embodiment, the client contains pre-configured configuration items based on its own needs, and these configuration items are encapsulated into target structured data. When the client obtains the data to be processed and the target operator type, it can determine the pre-configured target structured data based on the target operator type, and then determine the target operator implementation rules based on the configuration items of the target structured data. Based on these rules, the client processes the data to be processed to obtain target features, thereby realizing the operator capabilities on the client side. This data processing method provides strong foundational support for edge intelligence. The client can flexibly configure operator capabilities according to its own needs, achieving customized operator capabilities without relying on third-party SDKs. This avoids capability limitations caused by third parties, enhances autonomy, and avoids increased local memory usage due to calling third-party SDKs.

[0074] In conjunction with the above embodiments, in one embodiment, the present invention also provides a data processing method. For example... Figure 2 As shown, Figure 2 This is a flowchart illustrating a data processing method according to an embodiment of the present invention. Specifically, in this method, step S12 specifically includes step S21:

[0075] Step S21: Based on the target operator type, determine the target structured data related to the target operator type from the pre-configured multiple structured data.

[0076] In this embodiment, a feature center is deployed in the client. The feature center includes a plurality of pre-configured structured data. Each structured data corresponds to an operator type and an operator implementation rule, and each structured data is encapsulated by a plurality of configuration items related to the corresponding operator type. Among them, the configuration items in the structured data at least include: the type of data to be processed, the operator type, the processing rule, and the target feature type; and the operator implementation rules corresponding to the plurality of structured data include: custom calculation rules and / or mathematical functions, and the operator implementation rule corresponding to each structured data is a custom calculation rule or a mathematical function. The custom calculation rule in this embodiment is a data processing rule that cannot be implemented by existing mathematical functions and needs to be formulated by oneself.

[0077] The client can determine the target structured data related to the target operator type from the plurality of pre-configured structured data stored in the feature center according to the target operator type. Specifically, since the structured data is encapsulated by a plurality of configuration items, and the operator type is included in the configuration items, after the client determines the target operator type, it can use the target operator type as a keyword field to search in the plurality of pre-configured structured data in the feature center, so as to determine the structured data whose operator type in the configuration item is the target operator type, and determine this structured data as the target structured data related to the target operator type.

[0078] For example, a pre-configured structured data A in this embodiment is as follows:

[0079]

[0080] Among them, data_type is the type of data to be processed, which is float32; raw_ops is the operator type, which is Bucketize (data bucketing), and classifies the data to be processed according to the boundary values; param is the parameter; boundaries are the data processing boundaries; desc (description) is the calculation rule; according to the above rules, if the input value is [0, 0.2, 0.5, 1], the processed output result is [0, 1, 2, 3]. That is to say, the configuration items in this structured data A are:

[0081] Type of data to be processed: float32; Operator type: Bucketize (data bucketing); Processing rule: data processing boundaries [0.1, 0.3, 0.8] and calculation rule ">=left<right"; Target feature type is: numerical value {0, 1, 2, 3}. When the client determines that the target operator type is "Bucketize (data bucketing)", the client can determine that the target structured data related to the target operator type in the plurality of pre-configured structured data in the feature center is structured data A.

[0082] In this embodiment, the feature center deployed on the client includes multiple structured data sets pre-configured according to requirements, each implementing different operator capabilities. When the client needs to process the data to be processed, it can directly determine the target structured data set related to the target operator type from the multiple structured data sets in the feature center, based on the target operator type specified by the user. Then, based on the target operator implementation rules corresponding to the target structured data set, the client processes the data to be processed to obtain the target features. In other words, in this embodiment, to provide more flexible and customized operator capabilities, the client's feature center stores multiple pre-customized structured data sets encapsulated with multiple operator parameters as configuration items, each capable of implementing different operator capabilities. This allows for more flexible implementation of operator capabilities, improves data transformation capabilities, and increases customization capabilities on the client side, enabling users to customize data processing and calculation methods according to their own needs, thus achieving customized operator capabilities.

[0083] In conjunction with the above embodiments, in one embodiment, the present invention also provides a data processing method. In this method, step S13 may specifically include step S31:

[0084] Step S31: Parse each configuration item in the target structured data according to the parsing rules to determine the target operator implementation rules corresponding to the target structured data.

[0085] In this embodiment, the feature center deployed on the client is configured with unified parsing rules for structured data. These unified parsing rules can parse all structured data on the client. Based on this, after determining the target structured data, the client can parse each configuration item in the target structured data according to the unified parsing rules, thereby determining the target operator implementation rules corresponding to the target structured data. The obtained target operator implementation rules are the specific processing logic that needs to be processed for the data to be processed.

[0086] In this embodiment, by configuring a unified structured parsing rule in the feature center, the unified parsing rule can parse all structured data configured by the user according to various needs. Thus, regardless of the operator capability corresponding to the target structured data, the target structured data can be parsed according to the parsing rule to obtain the target operator implementation rule, thereby realizing the processization and unification of data processing logic.

[0087] In conjunction with the above embodiments, in one embodiment, the present invention also provides a data processing method. In this method, the pre-configured multiple structured data sets included in the feature center are configured by the server and then sent to the client. In this method, the data processing method further includes step S41:

[0088] Step S41: Receive the newly configured structured data sent by the server.

[0089] In this embodiment, the user can configure structured data in advance on the backend server. After the structured data is configured, the backend server can send multiple configured structured data sets to the client. Specifically, the backend server can send a receiving instruction to the client based on the communication connection with the client. This receiving instruction includes sending a notification to inform the client to receive the structured data set sent by the server. In response to the receiving instruction, the client requests multiple structured data sets from the backend server, so that the backend server responds to the request and sends the configured structured data sets to the client. The client receives and stores the multiple structured data sets in the feature center. Alternatively, the backend server can directly send the configured structured data sets to the client, and the client receives the multiple structured data sets.

[0090] This embodiment can update the structured data in the client: based on the feature center of the client including multiple structured data, new structured data can be configured through the backend server, and the backend server will send the configured new structured data to the client, and the client will receive the configured new structured data sent by the server.

[0091] In the data processing method provided in this embodiment, when the new structured data includes target structured data corresponding to the target operator type, the above step S31 specifically includes step S42:

[0092] Step S42: Parse each configuration item of the target structured data in the new structured data according to the parsing rules, and determine the target operator implementation rules corresponding to the target structured data.

[0093] In this embodiment, the unified parsing rules configured in the feature center can parse new structured data sent by the server. When the target structured data corresponding to the target operator type specified by the user is new structured data sent by the server, the client can still parse the target structured data according to the unified parsing rules. Specifically, the client can parse each configuration item of the target structured data in the new structured data according to the parsing rules to determine the target operator implementation rules corresponding to the target structured data.

[0094] One application scenario in this embodiment is as follows: If none of the pre-configured structured data in the feature center meets the user's needs (i.e., none correspond to the target operator type specified by the user), the user can configure new structured data in the server according to the current needs. This new structured data corresponds to the target operator type that the user wants to specify, and the new structured data is sent to the client. The client receives and stores the new structured data, so that through the data processing method of this embodiment, the client can find the target structured data in the new structured data according to the target operator type and parse it. Then, according to the target operator implementation rules obtained from the parsing, the client can process the data to be processed, enabling the client to implement the operator capabilities specified by the user and obtain the target features.

[0095] Another application scenario in this embodiment is: users can update and expand the structured data in the feature center of the client according to their own needs. For example, they can periodically or irregularly expand and update the structured data in the feature center. In subsequent updates and expansions, users only need to set up each configuration item based on various needs, encapsulate it into new structured data, and send it to the client.

[0096] In this embodiment, a feature engineering mechanism is implemented on the client side. In subsequent updates and expansions, the server can send newly configured structured data to the client to expand the feature center deployed on the client. The structured data configuration includes data processing rules. By parsing the structured data, operators can be implemented, achieving excellent scalability. Customizing unified rules enables the streamlining and standardization of processing logic and provides better scalability. When expanding new capabilities, it is only necessary to configure based on the existing data format (i.e., determine the specific content of each configuration item), and then use the same parsing rules to parse the new structured data to obtain the operator implementation rules. Data transformation is then performed according to the operator implementation rules to implement the new operator capabilities.

[0097] In conjunction with any of the above embodiments, in one embodiment, the present invention also provides a data processing method. In this method, the data to be processed is data generated by the terminal where the client is located, and the data processing method further includes: step S51 and / or step S52:

[0098] Step S51: Input the target features into the model invoked by the client.

[0099] The ability to use machine learning on the client side is an application of edge intelligence. In this embodiment, after obtaining the target features, the client can input the target features into the model invoked by the client for related inference services. Since the data to be processed in this embodiment is data generated by the client's terminal, such as data collected by the client itself, the client obtains the data to be processed and processes it using the data processing methods in the above embodiments to obtain the required target features. Then, the target features are input into the model invoked by the client to realize the inference service. This entire process fully realizes the application of "edge intelligence".

[0100] Step S52: Train the model on the client based on the target features.

[0101] In this embodiment, after obtaining the target features, the client can perform relevant model training on the client to achieve model-based training. Since the data to be processed in this embodiment is generated by the terminal where the client is located, such as data collected by the client itself, the client obtains the data to be processed and processes it using the data processing methods in the above embodiments to obtain the required target features. Then, based on the target features, the client performs relevant model training on the client to achieve model-based training. This entire process fully realizes the application of "edge intelligence".

[0102] Furthermore, in this embodiment, after obtaining the first target feature for training, the model can be trained on the client based on the target feature. After obtaining the trained model, the second target feature for inference is obtained through the data processing method in this embodiment. The second target feature is then input into the trained model on the client to achieve model inference. It can be seen that the client can realize data collection, feature engineering, model training, and edge inference, thus constructing a corresponding complete "edge intelligence" system.

[0103] In this embodiment, after obtaining the target features, the client can perform corresponding model training and / or model inference applications on the edge based on the target features, thereby fully realizing "edge intelligence". The advantages are as follows: high real-time performance, edge processing can save network transmission time of data; resource saving, making full use of edge computing power and storage space; good privacy, from data generation to data consumption, all are completed on the edge, avoiding the risk of privacy leakage caused by transmission.

[0104] In conjunction with any of the above embodiments, in one embodiment, the present invention also provides a data processing method. In this method, after obtaining the data to be processed and the target operator type, the client can encapsulate the data to be processed according to a prescribed format, thereby achieving process standardization. The client only needs to input the data to be processed into the feature center according to the prescribed format, and the feature center can parse it according to the standard format, enabling the feature center to better process the encapsulated data to be processed accordingly.

[0105] The format specified in this embodiment includes at least: target operator type, feature name, and data to be processed; for example, the format may be as follows:

[0106] raw_ops:ops1

[0107] field:1001

[0108] input_data:datal

[0109] ...

[0110] Here, raw_ops represents the target operator type; input_data represents the data to be processed; and field represents the feature name. The feature name can be user-defined and is intended to label the data so that the type of data can be identified when outputting the target feature. For example, the feature name can be date, duration, etc.

[0111] In one embodiment, see Figure 3 , Figure 3 This is a schematic diagram illustrating a data processing method according to an embodiment of the present invention. Figure 3 As shown, after the client obtains the data to be processed and the target operator type, it encapsulates the data according to a specified format to obtain encapsulated data to be processed. This encapsulated data is also a type of structured data, and its structure includes at least: target operator type, feature name, and data to be processed. The target operator type in the encapsulated data can be mapped to the operator type in the structured data. It can be understood that as long as the client encapsulates the data to be processed according to the specified format, it can perform data transformation through the feature center to obtain the output result.

[0112] The client processes the encapsulated data to be processed through a feature center. The feature center is a form of feature engineering, integrating and scheduling features from feature engineering to achieve a data engineering scheduling approach. The feature center maps multiple structured data structures and their corresponding operator implementations. The feature center can process the data to be processed using the corresponding target operator type (raw_ops). When the encapsulated data to be processed calls the feature center, the feature center can map the key fields in the encapsulated data, i.e., the target operator type raw_ops, to the corresponding structured data. The structured data specifies the implementation rules of the corresponding target operator raw_ops. The feature center's implementation module completes the data processing and outputs the transformed target features, thus realizing the entire data transformation process.

[0113] The feature center stores multiple pre-configured structured data sets. To ensure consistent processing capabilities for feature engineering, customizing structured data entity objects is necessary. Each structured data set corresponds to a different operator capability, and each field (i.e., each configuration item) in the structured data set has a clear definition. This embodiment encapsulates the operator type, processing rules, data type to be processed, and target feature type, with the specific format shown in the example above. The feature center also customizes unified parsing rules, which can streamline and unify the structured data processing logic, and also provide better scalability. When expanding new capabilities, it is only necessary to configure based on the existing structured data format (i.e., determine the specific content of each configuration item).

[0114] Each structured data structure in the feature center corresponds to a specific implementation (i.e., operator implementation rule). The implementation can be a custom calculation rule or a mathematical function; examples include numerical feature normalization, categorical features, and data binning. Numerical feature normalization involves performing a linear transformation on the original data so that the resulting mapping falls within the range [0,1], achieving proportional scaling of the original data. Categorical features include, for example, ordinal encoding, which is typically used to handle data where there is a size relationship between categories. Data binning is a data preprocessing technique that groups multiple consecutive values ​​into a smaller number of "buckets."

[0115] In addition, it should be noted that Figure 3 The second-layer feature center representation receives encapsulated data to be processed through an external interface of the feature center, and then processes the data. In fact, the third-layer structured data and the fourth-layer operator implementation capabilities are both contained within the feature center. Figure 3 Dividing it into three layers is simply to better illustrate the data processing flow.

[0116] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0117] Based on the same inventive concept, one embodiment of the present invention provides a data processing device 400, which can be applied to a client. (See reference) Figure 4 , Figure 4 This is a structural block diagram of a data processing apparatus provided in an embodiment of the present invention. Figure 4 As shown, the data processing device 400 includes:

[0118] Module 401 is used to acquire the data to be processed and the target operator type;

[0119] The structured data determination module 402 is used to determine the target structured data configured in the client according to the target operator type; wherein, the target structured data is encapsulated by multiple configuration items related to the target operator type;

[0120] The operator implementation rule determination module 403 is used to determine the target operator implementation rule corresponding to the target structured data based on each configuration item in the target structured data;

[0121] The processing module 404 is used to process the data to be processed according to the target operator implementation rules to obtain target features.

[0122] Optionally, the client deploys a feature center, which includes multiple pre-configured structured data. Each structured data corresponds to an operator type and an operator implementation rule. Each structured data is encapsulated by multiple configuration items related to their respective operator types. The operator implementation rules include: custom calculation rules or mathematical functions.

[0123] The structured data determination module 402 includes:

[0124] The structured data determination submodule is used to determine the target structured data related to the target operator type from the pre-configured multiple structured data based on the target operator type.

[0125] Optionally, the feature center is configured with unified parsing rules for structured data, which can parse all structured data.

[0126] The operator implementation rule determination module 403 includes:

[0127] The first parsing module is used to parse each configuration item in the target structured data according to the parsing rules, and determine the target operator implementation rules corresponding to the target structured data.

[0128] Optionally, the pre-configured structured data included in the feature center is configured by the server and then sent to the client. The data processing device 400 further includes:

[0129] The receiving module is used to receive the newly configured structured data sent by the server;

[0130] When the new structured data includes target structured data corresponding to the target operator type, the first parsing module includes:

[0131] The first parsing submodule is used to parse each configuration item of the target structured data in the new structured data according to the parsing rules, and determine the target operator implementation rules corresponding to the target structured data.

[0132] Optionally, the data to be processed is data generated by the terminal where the client is located;

[0133] The data processing device 400 further includes:

[0134] The input module is used to input the target features into the model invoked by the client;

[0135] And / or, a training module for training a model on the client based on the target features.

[0136] Based on the same inventive concept, another embodiment of the present invention provides an electronic device 500, such as... Figure 5 As shown. Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the computer program is executed by the processor, it implements the steps of the data processing method described in any of the above embodiments of the present invention.

[0137] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the data processing method as described in any of the above embodiments of the present invention.

[0138] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0139] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0141] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

[0142] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0143] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0144] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0145] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0146] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0147] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0148] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, Applied to the client, the client deploys a feature center, which includes multiple pre-configured structured data sets and unified parsing rules for the structured data. These parsing rules can parse all structured data. The structured data is configured by the server and then sent to the client. Each structured data set corresponds to an operator type and an operator implementation rule. Each structured data set is encapsulated by multiple configuration items related to its corresponding operator type. The operator implementation rules include: custom calculation rules or mathematical functions. The custom calculation rules are data processing rules that cannot be implemented using existing calculation functions. The configuration items at least include: the data type to be processed, the operator type, the processing rule, and the target feature type. The data processing method includes: Obtain the data to be processed and the target operator type; The target structured data configured in the client is determined based on the target operator type; wherein the target structured data is encapsulated by multiple configuration items related to the target operator type; Based on each configuration item in the target structured data, determine the target operator implementation rule corresponding to the target structured data, including: parsing each configuration item in the target structured data according to the parsing rule to determine the target operator implementation rule corresponding to the target structured data; The data to be processed is processed according to the target operator implementation rules to obtain the target features.

2. The data processing method according to claim 1, characterized in that, The step of determining the target structured data configured in the client according to the target operator type includes: Based on the target operator type, target structured data related to the target operator type is determined from the pre-configured multiple structured data.

3. The data processing method according to claim 1, characterized in that, The data processing method further includes: Receive the newly configured structured data sent by the server; When the new structured data includes target structured data corresponding to the target operator type, the step of parsing each configuration item in the target structured data according to the parsing rules to determine the target operator implementation rules corresponding to the target structured data includes: The configuration items of the target structured data in the new structured data are parsed according to the parsing rules to determine the target operator implementation rules corresponding to the target structured data.

4. The data processing method according to any one of claims 1-3, characterized in that, The data to be processed is data generated by the terminal where the client is located; The data processing method further includes: The target features are input into the model invoked by the client; And / or, model training is performed on the client based on the target features.

5. A data processing apparatus, characterized in that, Applied to a client, the client deploys a feature center, which includes multiple pre-configured structured data sets and unified parsing rules for the structured data. These parsing rules can parse all structured data. The structured data is configured by the server and then sent to the client. Each structured data set corresponds to an operator type and an operator implementation rule. Each structured data set is encapsulated by multiple configuration items related to its corresponding operator type. The operator implementation rules include: custom calculation rules or mathematical functions. The custom calculation rules are data processing rules that cannot be implemented using existing calculation functions. The configuration items at least include: the data type to be processed, the operator type, the processing rule, and the target feature type. The data processing device includes: The acquisition module is used to acquire the data to be processed and the target operator type; The structured data determination module is used to determine the target structured data configured in the client according to the target operator type; wherein the target structured data is encapsulated by multiple configuration items related to the target operator type; The operator implementation rule determination module is used to determine the target operator implementation rule corresponding to the target structured data based on each configuration item in the target structured data, including: a first parsing module, used to parse each configuration item in the target structured data according to the parsing rule to determine the target operator implementation rule corresponding to the target structured data; The processing module is used to process the data to be processed according to the implementation rules of the target operator to obtain the target features.

6. The data processing apparatus according to claim 5, characterized in that, The structured data determination module includes: The structured data determination submodule is used to determine the target structured data related to the target operator type from the pre-configured multiple structured data based on the target operator type.

7. The data processing apparatus according to claim 5, characterized in that, The data processing device further includes: The receiving module is used to receive the newly configured structured data sent by the server; When the new structured data includes target structured data corresponding to the target operator type, the first parsing module includes: The first parsing submodule is used to parse each configuration item of the target structured data in the new structured data according to the parsing rules, and determine the target operator implementation rules corresponding to the target structured data.

8. The data processing apparatus according to any one of claims 5-7, characterized in that, The data to be processed is data generated by the terminal where the client is located; The data processing device further includes: The input module is used to input the target features into the model invoked by the client; And / or, a training module for training a model on the client based on the target features.

9. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the data processing method as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 4.

Images