Application feature data processing method and apparatus, device, and medium

Abstract

The application relates to an application feature data processing method and device, equipment and medium. The method comprises the following steps: acquiring stream feature data and / or batch feature data corresponding to feature data, performing data preprocessing on the stream feature data and / or batch feature data to determine a plurality of different feature data sets; extracting a feature subset corresponding to each application program from the plurality of feature data sets according to a preset rule, splicing the feature subset corresponding to each application program to determine a feature data set list corresponding to each application program, and sending the feature data set list to a program feature pool; acquiring feature view configuration information corresponding to each application program, determining view data corresponding to each application program from the feature data set list in the program feature pool according to the feature view configuration information; and pushing the view data to a target application program to display a feature view corresponding to each application program. The application can improve the flexibility, reusability and maintainability of feature data.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a method for processing application feature data, a corresponding apparatus, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In business scenarios involving deep learning model inference, it is often necessary to use machine learning algorithms for classification or regression calculations on ultra-large-scale feature data. In the current Internet environment, billions or even hundreds of billions of data points need to be trained. As the business expands, the number of training features will also reach an astonishing level, and the number of features involved in the calculation may reach tens of billions.

[0003] Conventional solutions for processing feature data include manual export and import, batch file transfer, database copying and synchronization, message queues, or API integration for synchronization and publishing. However, when dealing with feature data scales of billions or even hundreds of billions, these methods are inefficient, lack flexibility, and have extremely low maintainability and reusability, failing to meet the personalized and functional requirements of various applications.

[0004] In summary, given the problems of low efficiency, poor flexibility, and extremely low maintainability and reusability in the synchronization and publication of feature data, the applicant has made corresponding explorations to address these issues. Summary of the Invention

[0005] The purpose of this application is to solve the above-mentioned problems by providing an application feature data processing method, a corresponding device, an electronic device, and a computer-readable storage medium.

[0006] To achieve the various objectives of this application, the following technical solution is adopted:

[0007] A method for processing application feature data, proposed to meet one of the purposes of this application, includes the following steps:

[0008] In response to the application's feature data request event, obtain the streaming feature data and / or batch feature data corresponding to the feature data, and perform data preprocessing on the streaming feature data and / or batch feature data to determine multiple different feature datasets;

[0009] According to preset rules, feature subsets corresponding to each application are extracted from multiple feature datasets to correspond to each application. The feature subsets corresponding to each application are concatenated to determine the feature dataset list corresponding to each application, and then sent to the program feature pool.

[0010] obtaining feature view configuration information corresponding to each application program, and determining view data corresponding to each application program from the feature data set list in the program feature pool according to the feature view configuration information;

[0011] pushing the view data specified by the feature view calling instruction submitted by the target application program to the target application program in response to the instruction, to display the feature view corresponding to each application program.

[0012] Optionally, the step of obtaining the streaming feature data and / or batch feature data corresponding to the feature data in response to the feature data request event of the application program comprises the following steps:

[0013] obtaining a streaming feature data identifier in the real-time incremental streaming feature data issued by an upstream system;

[0014] sending the streaming feature data identifier and its corresponding real-time incremental streaming feature data to a streaming system, so that each application program can obtain the changes of the streaming feature data in real time.

[0015] Optionally, the step of pre-processing the streaming feature data and / or batch feature data to determine a plurality of different feature data sets comprises the following steps:

[0016] performing data cleaning on the obtained streaming feature data or batch feature data of each source and type;

[0017] performing normalization processing on the streaming feature data or batch feature data of each type after data cleaning to determine a plurality of different feature data sets.

[0018] Optionally, the step of extracting a feature subset corresponding to each application program from a plurality of the feature data sets according to a preset rule to correspond to each application program, and splicing the feature subset corresponding to each application program to determine a feature data set list corresponding to each application program comprises the following steps:

[0019] obtaining an application program identifier corresponding to a deep learning model in the application program and its corresponding feature data name;

[0020] extracting a feature subset corresponding to the deep learning model from a plurality of different types of the feature data sets according to the application program identifier and its corresponding feature data name, and standardizing the feature subset into a feature data vector or a feature data matrix corresponding to the deep learning model;

[0021] determining a feature data set list corresponding to the deep learning model in each application program according to the feature data vector or the feature data matrix.

[0022] Optionally, the step of determining the view data corresponding to each application program from the list of feature data sets in the program feature pool according to the feature view configuration information comprises the following steps:

[0023] acquiring the feature data name and the feature data version in the feature view configuration information in response to a configuration information parsing instruction;

[0024] determining the feature data corresponding to each application program from the list of feature data sets according to the feature data name and the feature data version, and determining the view data corresponding to each application program according to the feature data.

[0025] Optionally, the step of determining the feature view configuration information comprises the following steps:

[0026] defining the parameter information corresponding to the feature view of the application program in response to a feature view configuration instruction, wherein the parameter information comprises a layout style, a feature data name, and a feature data version;

[0027] encapsulating the layout style, the feature data name, and the feature data version in the parameter information as the feature view configuration information.

[0028] Optionally, the step of pushing the view data specified by the feature view calling instruction submitted by a target application program to the target application program to display the feature view corresponding to each application program in response to the feature view calling instruction comprises the following steps:

[0029] receiving the feature view calling instruction, and extracting the parameter information in the feature view configuration information according to the feature view calling instruction;

[0030] generating a data query request according to the layout style, the feature data name, and the feature data version in the parameter information;

[0031] acquiring the view data corresponding to each application program from the list of feature data sets in the program feature pool according to the data query request;

[0032] displaying the feature view corresponding to each application program in the graphical interface of each application program according to the view data corresponding to each application program.

[0033] To achieve another object of the present application, an application program feature data processing device is provided, comprising:

[0034] a data preprocessing module configured to acquire streaming feature data and / or batch feature data corresponding to the feature data in response to a feature data request event of an application program, and to perform data preprocessing on the streaming feature data and / or batch feature data to determine a plurality of different feature data sets.

[0035] a feature set determination module configured to extract a feature subset corresponding to each application program from the plurality of feature data sets according to a preset rule, splice the feature subset corresponding to each application program to determine a feature data set list corresponding to each application program, and send the feature data set list to a program feature pool;

[0036] a view data determination module configured to obtain feature view configuration information corresponding to each application program, and determine view data corresponding to each application program from the feature data set list in the program feature pool according to the feature view configuration information;

[0037] a feature view display module configured to push view data specified by a feature view calling instruction submitted by a target application program to the target application program in response to the feature view calling instruction, so as to display a feature view corresponding to each application program.

[0038] Another object of the present application is to provide an electronic device comprising a central processing unit and a memory, wherein the central processing unit is configured to invoke a computer program stored in the memory to execute the steps of the application program feature data processing method.

[0039] Another object of the present application is to provide a computer readable storage medium storing a computer program implemented according to the application program feature data processing method in the form of computer readable instructions, wherein the computer program is invoked and run by a computer to execute the steps included in the corresponding method.

[0040] Compared with the prior art, the present application can separate the data processing and data use of the feature data corresponding to each application program, store the feature data in the program feature pool, then parse the parameter information in the feature view configuration information corresponding to each application program by the function view processor, and push the feature data corresponding thereto to the target application program, thereby separating the processing and use of the feature data, and through the separation of the processing and use of the feature data, the following beneficial effects are included but not limited to:

[0041] Firstly, the flexibility of the feature data is significantly improved, the processing logic and the use logic of the feature data corresponding to each application program are separated, each link can be independently modified and optimized, which means that the processing mode of the feature data can be modified according to the actual business requirements without affecting the use of the feature data in the model training or inference of each application program.

[0042] Secondly, the maintainability of the feature data is significantly enhanced. By separating the processing logic of the feature data corresponding to each application program from the usage logic, the feature data is easier to maintain and manage, and is easier to debug, test and modify. The usage logic of the feature data can also clearly express the requirements of each application program model or algorithm.

[0043] Thirdly, the reusability of the feature data is significantly improved. By modularizing the processing logic of the feature data, the feature data can be reused as an independent component. The feature data processing logic that has been developed and verified can be shared and reused in different projects or tasks, greatly improving the development efficiency.

[0044] Fourthly, the uniformity of the feature data is significantly improved. By separating the processing logic of the feature data corresponding to each application program from the usage logic, the same feature processing logic can be used in different models or analysis tasks, thereby maintaining consistency and helping to avoid introducing bias or errors in different data processing sections.

[0045] Further, by separating the processing logic of the feature data corresponding to each application program from the usage logic, the work efficiency of feature data synchronization and release can be significantly improved. The design of the feature view corresponding to each application program incorporates a multi-version management design idea, which is conducive to loading multiple versions of data according to actual needs by downstream application services, and also helps to perform data version rollback and other functions. Different feature views of each application program can combine feature data from different sources and types according to individualized needs and user experience needs, achieving the effect of feature data sharing. BRIEF DESCRIPTION OF DRAWINGS

[0046] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0047] Figure 1 An exemplary network architecture employed by the application program feature data processing method;

[0048] Figure 2 A flowchart of the application program feature data processing method in the embodiment of the present application;

[0049] Figure 3 A flowchart of each application program in the embodiment of the present application acquiring the changes of the streaming feature data in real time;

[0050] Figure 4 A flowchart of data cleaning of the streaming feature data or batch feature data in the embodiment of the present application;

[0051] Figure 5A flowchart for determining a feature data set list corresponding to each application program in the embodiment of the present application;

[0052] Figure 6 A flowchart for determining a view data corresponding to each application program in the embodiment of the present application;

[0053] Figure 7 A flowchart for determining a feature view configuration information of each application program in the embodiment of the present application;

[0054] Figure 8 A flowchart for displaying a feature view corresponding to each application program in the embodiment of the present application;

[0055] Figure 9 A principle block diagram of an application program feature data processing device in the embodiment of the present application;

[0056] Figure 10 A structure schematic diagram of a computer device in the embodiment of the present application. DETAILED DESCRIPTION

[0057] The embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary only, and are used to explain the present application, but cannot be interpreted as a limitation of the present application.

[0058] It can be understood by those skilled in the art that, unless specifically stated, the singular forms “a”, “an” and “the” used herein include plural forms. It should be further understood that the use of the term “include” in the specification of the present application means that the features, integers, steps, operations, elements and / or components exist, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there can be an intermediate element. In addition, “connected” or “coupled” used herein can include wireless connection or wireless coupling. The phrase “and / or” used herein includes all or any single unit and all combinations of the associated listed items.

[0059] As those skilled in the art will appreciate, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further appreciated that terms, such as those defined in commonly used dictionaries, should be given their ordinary and customary meaning, unless explicitly defined otherwise herein and unless the context clearly dictates otherwise.

[0060] As those skilled in the art will appreciate, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further appreciated that terms, such as those defined in commonly used dictionaries, should be given their ordinary and customary meaning, unless explicitly defined otherwise herein and unless the context clearly dictates otherwise. As those skilled in the art will appreciate, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further appreciated that terms, such as those defined in commonly used dictionaries, should be given their ordinary and customary meaning, unless explicitly defined otherwise herein and unless the context clearly dictates otherwise. As those skilled in the art will appreciate, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further appreciated that terms, such as those defined in commonly used dictionaries, should be given their ordinary and customary meaning, unless explicitly defined otherwise herein and unless the context clearly dictates otherwise.

[0061] The hardware referred to by the names "server", "client", "service node" and the like in the present application is essentially an electronic device with the equivalent capabilities of a personal computer, and is a hardware device with the necessary components disclosed by the von Neumann principle, including a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in the memory, the central processing unit calls the program stored in the external memory into the memory for running, executes the instructions in the program, and interacts with the input and output devices to complete a specific function.

[0062] It should be noted that the concept of "server" in the present application can also be extended to the case of a server cluster. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided, and in physical space, these servers can be independent of each other but can be called through an interface, or can be integrated into a physical computer or a computer cluster. Those skilled in the art should understand this variation and should not be restricted by the implementation of the network deployment of the present application.

[0063] One or more technical features of the present application, unless explicitly specified, can be deployed on a server for implementation and accessed by a client remotely calling an online service interface provided by the server, or can be directly deployed and run on a client for implementation and access.

[0064] The neural network model referred to or possibly referred to in the present application, unless explicitly specified, can be deployed on a remote server and remotely called by a client, or can be deployed on a client with sufficient device capability for direct calling. In some embodiments, when it is run on a client, its corresponding intelligence can be obtained through transfer learning to reduce the requirement for client hardware running resources and avoid excessive occupation of client hardware running resources.

[0065] The various data involved in the present application, unless explicitly specified, can be remotely stored on a server or stored on a local terminal device, as long as it is suitable for being called by the technical solutions of the present application.

[0066] Those skilled in the art should know that the various methods of the present application, although based on the same concept and described to present commonality among them, are independently executable unless otherwise specified. Similarly, for each embodiment disclosed in the present application, it is based on the same inventive concept, and therefore, for the same concept of expression, and although the concept of expression is different, it is only for the convenience of appropriately transforming the concept, and should be understood as equivalent.

[0067] Unless otherwise indicated herein, the various disclosed embodiments can be combined in any and all permutations. It is intended that the following claims be construed to include all such combinations.

[0068] Referring to Figure 1 The hardware infrastructure required for implementing the technical solutions of the present application can be deployed according to the architecture shown in the figure. The AlgoFeatureHandler pre-processes the stream data and / or batch data in the original feature data to convert them into feature data required by the algorithms or deep learning models corresponding to each application, and then sends the pre-processed stream data and / or batch data to the MergeFeatureHandler. The MergeFeatureHandler splices the feature data corresponding to the algorithms of each application, and writes the spliced feature data corresponding to the algorithms of each application into the HbaseFeatureStore. The AppFeatureViewPusher reads and parses the configuration information in the AppFeatureViewConfig, reads the spliced feature data corresponding to the algorithms of each application in the HbaseFeatureStore according to the configuration information, and pushes the feature data to the AppFeatureStore. The AppFeatureStore responds to the feature view call instruction submitted by the target application, pushes the view data specified by the instruction to the target application, and displays the feature view corresponding to each application.

[0069] Referring to Figure 2 In one embodiment of the application program feature data processing method of the present application, the following steps are included:

[0070] Step S10, in response to a feature data request event of an application program, acquiring stream feature data and / or batch feature data corresponding to the feature data, and pre-processing the stream feature data and / or batch feature data to determine a plurality of different feature data sets;

[0071] The AlgoFeatureHandler is used to process feature data in an algorithm, and is responsible for pre-processing, converting and extracting input streaming feature data and / or batch feature data to better adapt to the needs of an algorithm model; the streaming feature data represents a continuous data stream; the batch feature data is a data set organized and processed in a batch processing manner, and is used to process a data batch of multiple data samples at the same time. The AlgoFeatureHandler in the server responds to a feature data request event of an application algorithm or a deep learning model, acquires streaming feature data and / or batch feature data corresponding to the feature data, performs data preprocessing on the streaming feature data and / or batch feature data to determine multiple different feature data sets, applies the streaming feature data and / or batch feature data to the multiple different feature data sets required by the application algorithm or the deep learning model for consumption conversion, and then sends the multiple different feature data sets to a MergeFeatureHandler for processing.

[0072] In step S20, a feature subset corresponding to each application is extracted from the multiple feature data sets according to a preset rule to correspond to each application, the feature subset corresponding to each application is spliced to determine a feature data set list corresponding to each application, and the feature data set list is sent to a program feature pool.

[0073] The feature merge handler can be used in the feature engineering stage to fuse different sources or types of feature data to provide richer information for subsequent machine learning or data analysis tasks, to merge and integrate multiple different feature data sets, receive input of multiple different feature data sets, and combine the feature data sets into a larger feature vector or feature matrix according to a preset rule or a preset logic. After receiving the multiple different types or sources of feature data sets in the algorithm feature handler, the feature merge handler in the server can extract the feature subsets corresponding to each application program from the multiple different sources or types of feature data sets according to a preset rule to correspond to each application program. It is not difficult to understand that the algorithms or deep learning models in each application program are different, and the sources and types of the multiple different feature data sets are extensive. For the algorithms or deep learning models in the application program, only one or more feature subsets in the feature data set may be needed, which cannot adapt the entire feature data set to the algorithms or deep learning models of the application program. In order to enrich the training and learning of the algorithms or deep learning models of the application program, fuse feature data of different sources or types, adapt to the personalized needs and functional needs of the application program, the feature merge handler splices the feature subsets corresponding to the algorithms or deep learning models of each application program to determine the feature data set list corresponding to the algorithms or deep learning models of each application program, and sends it to the program feature pool. The program feature pool can be a bottom feature storage system (HbaseFeatureStore) and the like. Those skilled in the art can determine the program feature pool according to actual conditions as needed, which is not limited here.

[0074] In some embodiments, the bottom feature storage system (HbaseFeatureStore) refers to a feature storage system using HBase as the bottom storage. The feature data is stored in the HBase table in the form of key-value pairs by using the characteristics and functions of HBase, which provides fast read-write and query capabilities for feature data and supports horizontal expansion and distributed computing.

[0075] Step S30, acquiring feature view configuration information corresponding to each application program, and determining view data corresponding to each application program from the feature data set list in the program feature pool according to the feature view configuration information;

[0076] The function view pusher (AppFeatureViewPusher) is used to send the feature data or view data of each application to the backend server or other related systems for further processing, storage or display. The function view pusher (AppFeatureViewPusher) in the server can obtain the feature view configuration information corresponding to each application, which includes the setting information and parameter information of the feature view in each application, used to define the layout, style, behavior and other related attributes of the feature view, to ensure that the feature view in the application can be correctly displayed and operated. By configuring the feature view configuration information (AppFeatureViewConfig), the background developer or administrator can define the appearance and behavior of each different feature view to meet the individual needs and user experience needs of the application. The function view pusher (AppFeatureViewPusher) can determine the view data corresponding to each application from the list of feature data sets corresponding to each application in the underlying feature storage system (HbaseFeatureStore) according to the feature view configuration information (AppFeatureViewConfig). After determining the view data corresponding to each application, the function view pusher (AppFeatureViewPusher) pushes it to the application feature storage, which is used to store and manage the feature data in the application, such as Elasticsearch, Redis or Mongodb, etc. It is different from the underlying feature storage system (HbaseFeatureStore), and is often related to the specific attributes of the related business needs of an application, such as query speed, etc.

[0077] Step S40, in response to the feature view calling instruction submitted by the target application, the view data specified by the instruction is pushed to the target application to display the feature view corresponding to each application.

[0078] The function view processor (AppFeatureViewPusher) in the server determines the view data corresponding to each application from the list of feature data sets corresponding to each application in the underlying feature storage system (HbaseFeatureStore) according to the feature view configuration information (AppFeatureViewConfig), and then pushes the view data to the application feature storage (AppFeatureStore) or the feature register center (FeatureRegisterCenter). In response to a feature view calling instruction submitted by a target application, the application feature storage (AppFeatureStore) or the feature register center (FeatureRegisterCenter) can retrieve the view data corresponding to the feature view calling instruction according to the application identifier and the feature data name corresponding to each application, and push the view data specified by the instruction to the target application to display the feature view (FeatureView) corresponding to each application.

[0079] In some embodiments, the function view processor (AppFeatureViewPusher) determines the view data corresponding to each application and sends the view data to the feature register center (FeatureRegisterCenter). The feature register center (FeatureRegisterCenter) is a centralized system or component for managing and registering the view data of the feature view corresponding to each application or system, which provides a centralized way to manage the definition, registration, and access of features, allowing developers or system administrators to register and manage various view data in a unified location and providing an access interface for other modules or components to access the view data of the feature view corresponding to each application or system.

[0080] In some embodiments, the feature view (FeatureView) is used to display feature information related to a specific function, data, or user operation of each application. The feature view (FeatureView) can be a chart, graph, table, card, or other form of visual element, which is used to display the attributes, status, or results of features in an intuitive and easy-to-understand manner.

[0081] Compared with the prior art, the application can separate the data processing and data use of the feature data corresponding to each application program, uniformly store the feature data in the program feature pool, then parse the parameter information in the feature view configuration information corresponding to each application program through the function view processor, and push the corresponding feature data to the target application program, thereby separating the processing and use of the feature data, and achieving the following beneficial effects:

[0082] First, the flexibility of the feature data is significantly improved. By separating the processing logic and use logic of the feature data corresponding to each application program, each link can be independently modified and optimized, which means that the processing method of the feature data can be modified according to actual business needs without affecting the use of the feature data in model training or inference of each application program.

[0083] Second, the maintainability of the feature data is significantly improved. By separating the processing logic and use logic of the feature data corresponding to each application program, the feature data is more easily maintained and managed, and is more easily debugged, tested and modified. The use logic of the feature data can also more clearly express the requirements of each application program model or algorithm.

[0084] Third, the reusability of the feature data is significantly improved. By modularizing the processing logic of the feature data, it can be reused as an independent component. The feature data processing logic that has been developed and verified can be shared and reused in different projects or tasks, greatly improving the development efficiency.

[0085] Fourth, the uniformity of the feature data is significantly improved. By separating the processing logic and use logic of the feature data corresponding to each application program, the same feature processing logic can be used in different models or analysis tasks, thereby maintaining consistency and helping to avoid introducing bias or errors in different data processing links.

[0086] Further, by separating the processing logic and use logic of the feature data corresponding to each application program, the work efficiency of feature data synchronization and release can be significantly improved. The design of the feature view corresponding to each application program incorporates a multi-version management design idea, which is beneficial to loading multiple versions of data according to actual needs of downstream application services, and also helps to perform data version rollback and other functions. Different feature views of each application program can combine feature data from different sources and types according to individual needs and user experience needs, achieving the effect of feature data sharing.

[0087] On the basis of any embodiment of the application, please refer to Figure 3, the step of acquiring the stream feature data and / or the batch feature data corresponding to the feature data in response to a feature data request event of an application program comprises the following steps:

[0088] Step S101, acquiring a stream feature data identifier in real-time incremental stream feature data issued by an upstream system;

[0089] The log file of the business server stores business data of the upstream system, which has the characteristics of "stream". The event-driven log collection can obtain stream incremental real-time data, that is, real-time incremental stream feature data. The real-time incremental stream feature data identifier (StreamFeatureId) can be obtained by performing an analysis operation on the real-time incremental stream feature data. The stream feature data identifier (StreamFeatureId) refers to a unique identifier of the stream feature data, which is used to identify and track a specific stream feature in the data stream. The stream feature data identifier (StreamFeatureId) is usually a unique identifier, which can be associated with a specific feature data. By using the stream feature data identifier (StreamFeatureId), the feature can be marked, indexed, and queried in the stream feature data processing process, so as to analyze, process, or store the specific stream feature.

[0090] Step S103, sending the stream feature data identifier and the real-time incremental stream feature data corresponding thereto to a stream system, so that each application program can acquire the change of the stream feature data in real time.

[0091] In order to facilitate the downstream application program to acquire the change of the feature data in real time and avoid introducing deviation or error in different data processing links, the stream feature data identifier and the real-time incremental stream feature data corresponding thereto are sent to the stream system, so that each application program can acquire the change of the stream feature data in real time. The stream system includes a Kafka or Pulsar system, etc.

[0092] As can be seen from the above embodiments, by writing the real-time incremental stream feature data into the stream system, the downstream application program can acquire the change of the feature data in real time, avoid introducing deviation or error in different data processing links, facilitate marking, indexing, and querying of the feature in the stream feature data processing process, and facilitate analysis, processing, or storage of the specific stream feature.

[0093] On the basis of any embodiment of the present application, please refer to Figure 4 The step of performing data preprocessing on the stream feature data and / or the batch feature data to determine a plurality of different feature data sets comprises the following steps:

[0094] Step S105, data cleaning is performed on the acquired flow feature data or batch feature data of each source and type;

[0095] For the acquired flow feature data or batch feature data of each source and type, since the flow feature data or batch feature data can contain a large number of missing values, a large number of noises, and can also have abnormal data due to manual marking errors, if the flow feature data or batch feature data is not cleaned or the missing values are not filled, the accuracy, reliability and other indicators of the flow feature data or batch feature data applied to the algorithms or deep models corresponding to each application program will be affected. The flow feature data or batch feature data is cleaned or the missing values are filled to ensure the integrity and accuracy of the flow feature data or batch feature data, and the accuracy and reliability of the algorithms or deep models corresponding to each application program.

[0096] Step S107, normalization processing is performed on the flow feature data or batch feature data of each type after data cleaning to determine a plurality of different feature data sets.

[0097] Since the dimensions and orders of magnitude of the flow feature data or batch feature data of each source and type can be inconsistent, in order to eliminate the influence of the dimensions and orders of magnitude, normalization processing needs to be performed on the flow feature data or batch feature data of each source and type after data cleaning or missing value filling.

[0098] As can be seen from the above embodiments, data cleaning is performed on the flow feature data or batch feature data, which can ensure the integrity and accuracy of the flow feature data or batch feature data, and at the same time, ensure the accuracy and reliability of the algorithms or deep models corresponding to each application program. Standardization processing is performed on the flow feature data or batch feature data after data cleaning, which can unify the dimensions and orders of magnitude of the flow feature data or batch feature data, eliminate the influence of the dimensions and orders of magnitude, and ensure the uniformity of the feature data.

[0099] On the basis of any embodiment of the present application, please refer to Figure 5 According to a predetermined rule, each application program corresponding feature subset is extracted from a plurality of feature data sets to correspond to each application program, and each application program corresponding feature subset is spliced to determine a feature data set list corresponding to each application program. The step includes the following steps:

[0100] Step S201, acquiring an application program identifier corresponding to a deep learning model in the application program and a feature data name corresponding thereto;

[0101] The feature merging processor (MergeFeatureHandler) extracts a feature subset corresponding to each application from a plurality of different types or sources of feature data sets according to a preset rule after receiving the plurality of different types or sources of feature data sets in the algorithm feature processor (AlgoFeatureHandler), and can obtain the application identifier corresponding to the algorithm or deep learning model in each application and the corresponding feature data name in the feature view configuration information, so as to accurately and efficiently extract the feature subset corresponding to the algorithm or deep learning model of each application from a plurality of different types of feature data sets, so as to determine the feature data set list corresponding to each application.

[0102] In step S203, the feature subset corresponding to the deep learning model is extracted from a plurality of different types of feature data sets according to the application identifier and the corresponding feature data name, and the feature subset is standardized to a feature data vector or a feature data matrix corresponding to the deep learning model.

[0103] After obtaining the application identifier corresponding to the deep learning model in each application and the corresponding feature data name, since the algorithms or deep learning models in each application are different, and the sources and types of a plurality of different feature data sets are extensive, only one or more feature subsets in the feature data set may be needed for the algorithm or deep learning model in the application, which cannot be generally adapted to the algorithm or deep learning model of the application. In order to enrich the training and learning of the algorithm or deep learning model of the application, to fuse feature data of different sources or types, to adapt to the individualization needs and functional needs of the application, and to make the use logic of the feature data more clearly and explicitly express the needs of the deep learning model, algorithm or data analysis in each application, the feature merging processor (MergeFeatureHandler) extracts the feature subset corresponding to the deep learning model from a plurality of different types of feature data sets according to the application identifier and the corresponding feature data name, and standardizes the feature subset to a feature data vector or a feature data matrix corresponding to the deep learning model.

[0104] In step S205, the feature data set list corresponding to the deep learning model in each application is determined according to the feature data vector or the feature data matrix.

[0105] The feature merge processor (MergeFeatureHandler) normalizes the feature subsets corresponding to the algorithms or deep learning models of the respective applications, determines the feature data vectors or feature data matrices corresponding to the algorithms or deep learning models, and then determines the feature dataset lists corresponding to the algorithms or deep learning models in the respective applications according to the feature data vectors or feature data matrices, and pushes the feature dataset lists corresponding to the algorithms or deep learning models in the respective applications to the underlying feature storage system (HbaseFeatureStore), so that the function view processor (AppFeatureViewPusher) calls according to the feature view configuration information corresponding to the respective applications.

[0106] According to the above embodiments, the application identifiers and the corresponding feature data names of the algorithms or deep learning models in the respective applications are obtained from the feature view configuration information, which facilitates accurate and efficient extraction of the feature subsets corresponding to the algorithms or deep learning models of the respective applications from multiple different types of feature datasets, determination of the feature dataset lists corresponding to the respective applications, pushing of the feature dataset lists to the underlying feature storage system (HbaseFeatureStore), and calling by the function view processor (AppFeatureViewPusher) according to the feature view configuration information corresponding to the respective applications, thereby significantly enhancing the maintainability and flexibility of feature data.

[0107] Based on any embodiment of the present application, please refer to Figure 6 According to the feature view configuration information, the step of determining the view data corresponding to the respective applications from the feature dataset list in the program feature pool comprises the following steps:

[0108] Step S301, in response to a configuration information parsing instruction, obtaining the feature data name and feature data version in the feature view configuration information;

[0109] The function view processor (AppFeatureViewPusher) obtains the feature data name and feature data version in the feature view configuration information corresponding to the respective applications in response to a configuration information parsing instruction, so as to obtain and determine the version information of the feature data corresponding to the respective applications, and adapt to the individualization needs and functional needs of the feature views corresponding to the respective applications.

[0110] Step S303, determining the feature data corresponding to each application program according to the feature data name and feature data version from the feature data set list, and determining the view data corresponding to each application program according to the feature data.

[0111] After obtaining the feature data name and feature data version in the feature view configuration information corresponding to each application program, the function view processor (AppFeatureViewPusher) calls the feature data set list corresponding to the algorithm or deep learning model of each application program in the underlying feature storage system (HbaseFeatureStore) according to the feature data name and feature data version, to determine the feature data corresponding to each application program algorithm or deep learning model, determine the view data corresponding to each application program according to the feature data corresponding to the application program algorithm or deep learning model, and push to the application feature storage (AppFeatureStore). The application feature storage (AppFeatureStore) responds to the feature view call instruction submitted by the target application program to display the feature view (FeatureView) corresponding to each application program.

[0112] From the above embodiments, by analyzing the feature data name and feature data version in the feature view configuration information, the function view processor (AppFeatureViewPusher) can call the view data corresponding to each application program according to the feature view configuration information corresponding to each application program to display the feature view corresponding to each application program. This is beneficial to loading multiple versions of data according to actual needs by downstream application services, and also helps to perform data version rollback and other functions.

[0113] On the basis of any embodiment of the present application, please refer to Figure 7 , the step of determining the feature view configuration information includes the following steps:

[0114] Step S305, defining the parameter information corresponding to the feature view of the application program in response to the feature view configuration instruction, the parameter information including layout style, feature data name and feature data version;

[0115] The feature view configuration information (AppFeatureViewConfig) contains setting information and parameter information of the feature view in each application, which is used to define the layout, style, behavior and other related attributes of the feature view, so as to ensure that the feature view in the application can be correctly displayed and operated. By configuring the feature view configuration information (AppFeatureViewConfig), the background developer or administrator can define the appearance and behavior of each different feature view to meet the individualization needs and user experience needs of the application. The business server defines the parameter information corresponding to the feature view of each application in response to the feature view configuration instruction submitted by the application, and the parameter information includes the layout style, feature data name and feature data version.

[0116] In step S307, the layout style, feature data name and feature data version in the parameter information are encapsulated into the feature view configuration information.

[0117] After defining the layout style, feature data name and feature data version in the parameter information corresponding to the feature view of the application, the layout style, feature data name and feature data version in the parameter information are encapsulated into the feature view configuration information according to a preset format, and are pushed to the external middleware for storage.

[0118] As can be seen from the above embodiments, by encapsulating the feature view configuration information corresponding to each application, it can be ensured that the feature view in the application can be correctly displayed and operated. By configuring the feature view configuration information (AppFeatureViewConfig), the background developer or administrator can define the appearance and behavior of each different feature view to meet the individualization needs and user experience needs of the application, and can adapt to the functional needs of the feature view corresponding to each application,

[0119] On the basis of any embodiment of the present application, please refer to Figure 8 In response to the feature view calling instruction submitted by the target application, the view data specified by the instruction is pushed to the target application to display the feature view corresponding to each application.

[0120] In step S401, the feature view calling instruction is received, and the parameter information in the feature view configuration information is extracted according to the feature view calling instruction.

[0121] The function view processor (AppFeatureViewPusher) receives the feature view calling instruction submitted by the target application, extracts the layout style, feature data name, feature data version and the like in the parameter information in the feature view configuration information corresponding to the target application according to the feature view calling instruction.

[0122] In step S403, a data query request is generated according to the layout style, feature data name and feature data version in the parameter information.

[0123] The function view processor (AppFeatureViewPusher) can generate a data query request of the view data corresponding to the target application according to the layout style, feature data name and feature data version in the parameter information in the feature view configuration information corresponding to the target application.

[0124] In step S405, the view data corresponding to each application is obtained from the feature data set list in the program feature pool according to the data query request.

[0125] The underlying feature storage system (HbaseFeatureStore) calls the view data corresponding to each application from the feature data set list according to the data query request and pushes it to the application feature storage (AppFeatureStore) or feature register center (FeatureRegisterCenter).

[0126] In step S407, the feature view corresponding to each application is displayed in the graphical interface of each application according to the view data corresponding to the application.

[0127] The application feature storage (AppFeatureStore) or feature register center (FeatureRegisterCenter) can call the view data corresponding to the feature view calling instruction according to the layout style, feature data name and feature data version and the like of each application according to the feature view calling instruction submitted by the target application, and push the view data specified by the instruction to the target application. Each application client displays the feature view corresponding to each application in the graphical interface of each application according to the view data corresponding to the application.

[0128] From the above embodiments, the design of the feature view corresponding to each application program integrates the design idea of multi-version management, which is beneficial to the downstream application service to load multiple version data according to actual needs, and also helps to perform data version rollback and other functions. Different feature views of each application program can combine feature data from different sources and types according to individualized needs and user experience needs, to achieve the effect of feature data sharing.

[0129] Please refer to Figure 9 , one of the purposes of the present application is to provide an application feature data processing device, which comprises a data preprocessing module 1100, a feature set determination module 1200, a view data determination module 1300 and a feature view display module 1400. Among them, the data preprocessing module 1100 is set to obtain the streaming feature data and / or batch feature data corresponding to the feature data of the application program in response to the feature data request event of the application program, and to determine a plurality of different feature data sets by preprocessing the streaming feature data and / or batch feature data; the feature set determination module 1200 is set to extract the feature subset corresponding to each application program from the plurality of feature data sets according to a predetermined rule to correspond to each application program, splice the feature subset corresponding to each application program to determine the feature data set list corresponding to each application program, and send it to the program feature pool; the view data determination module 1300 is set to obtain the feature view configuration information corresponding to each application program, and to determine the view data corresponding to each application program from the feature data set list in the program feature pool according to the feature view configuration information; the feature view display module 1400 is set to push the view data specified by the feature view calling instruction submitted by the target application program to the target application program in response to the instruction, to display the feature view corresponding to each application program.

[0130] On the basis of any embodiment of the present application, the data preprocessing module 1100 comprises:

[0131] The acquisition unit is set to acquire the streaming feature data identifier in the real-time incremental streaming feature data issued by the upstream system;

[0132] The data increment unit is set to send the streaming feature data identifier and its corresponding real-time incremental streaming feature data to the streaming system, so that each application program can obtain the changes of the streaming feature data in real time.

[0133] On the basis of any embodiment of the present application, the data preprocessing module 1100 comprises:

[0134] The data cleaning unit is set to clean the acquired streaming feature data or batch feature data of each source and type;

[0135] a normalization unit, configured to perform normalization processing on each type of the stream feature data or the batch feature data after data cleaning to determine a plurality of different feature data sets.

[0136] On the basis of any embodiment of the present application, the feature set determination module 1200 comprises:

[0137] an acquisition unit, configured to acquire an application program identifier corresponding to the deep learning model in the application program and a feature data name corresponding thereto;

[0138] a standardization processing unit, configured to extract a feature subset corresponding to the deep learning model from the plurality of different types of feature data sets according to the application program identifier and the feature data name corresponding thereto, and standardize the feature subset into a feature data vector or a feature data matrix corresponding to the deep learning model;

[0139] a feature set determination unit, configured to determine a feature data set list corresponding to the deep learning model in each application program according to the feature data vector or the feature data matrix.

[0140] On the basis of any embodiment of the present application, the view data determination module 1300 comprises:

[0141] an acquisition unit, configured to acquire a feature data name and a feature data version in the feature view configuration information in response to a configuration information analysis instruction;

[0142] a view data determination unit, configured to determine feature data corresponding to each application program from the feature data set list according to the feature data name and the feature data version, and determine view data corresponding to each application program according to the feature data.

[0143] On the basis of any embodiment of the present application, the application program feature data processing apparatus comprises:

[0144] a parameter information definition module, configured to define parameter information corresponding to a feature view of the application program in response to a feature view configuration instruction, the parameter information comprising a layout style, a feature data name and a feature data version;

[0145] a configuration information determination module, configured to encapsulate the layout style, the feature data name and the feature data version in the parameter information into feature view configuration information.

[0146] On the basis of any embodiment of the present application, the feature view display module 1400 comprises:

[0147] The parameter information extraction unit is configured to receive the feature view calling instruction, and extract parameter information in feature view configuration information according to the feature view calling instruction.

[0148] The query request generation unit is configured to generate a data query request according to the layout style, feature data name and feature data version in the parameter information.

[0149] The view data determination unit is configured to obtain view data corresponding to each application program from the feature data set list in the program feature pool according to the data query request.

[0150] The feature view display unit is configured to display the feature view corresponding to each application program in the graphical interface of the application program according to the view data corresponding to the application program.

[0151] On the basis of any embodiment of the present application, please refer to Figure 10 Another embodiment of the present application also provides an electronic device, which can be implemented by a computer device, as shown in Figure 10 The internal structure of the computer device. The computer device includes a processor, a computer readable storage medium, a memory and a network interface connected by a system bus. Among them, the computer readable storage medium of the computer device stores an operating system, a database and a computer readable instruction, the database can store a control information sequence, and the computer readable instruction is executed by the processor to enable the processor to implement an application feature data processing method. The processor of the computer device is used to provide calculation and control ability to support the operation of the whole computer device. The memory of the computer device can store computer readable instructions, and the computer readable instructions are executed by the processor to enable the processor to execute the application feature data processing method of the present application. The network interface of the computer device is used to connect and communicate with the terminal. Those skilled in the art can understand Figure 10 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or less components than those shown in the figure, or combine certain components, or have different component arrangement.

[0152] The processor in the embodiment is used to execute the specific functions of each module and its sub-modules in Figure 9 The memory stores the program code and various data required to execute the above-mentioned modules or sub-modules. The network interface is used for data transmission between user terminals or servers. The memory in the embodiment stores the program code and data required to execute all modules / sub-modules in the application feature data processing device of the present application. The server can call the program code and data of the server to execute the functions of all sub-modules.

[0153] The application further provides a storage medium storing computer readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the application feature data processing method described in any of the embodiments.

[0154] The application further provides a computer program product, comprising computer programs / instructions, which, when executed by one or more processors, implement the steps of the application feature data processing method described in any of the embodiments.

[0155] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments of the application can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the method. The storage medium can be a computer readable storage medium such as a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM).

[0156] The above only describes some embodiments of the application. It should be noted that those of ordinary skill in the art can make some improvements and refinements without departing from the principles of the application, and these improvements and refinements should also be considered within the protection scope of the application.

[0157] In summary, the design of the feature view corresponding to each application program incorporates the design idea of multi-version management, which is conducive to loading multiple versions of data according to actual needs by downstream application services, and also helps to perform data version rollback and other functions. Different feature views of various application programs can combine feature data from different sources and types according to individualized needs and user experience needs, achieving the effect of feature data sharing.

Claims

1. A method for processing application feature data, characterized in that, Includes the following steps: In response to the application's feature data request event, obtain the streaming feature data and / or batch feature data corresponding to the feature data, and perform data preprocessing on the streaming feature data and / or batch feature data to determine multiple different feature datasets; According to preset rules, feature subsets corresponding to each application are extracted from multiple feature datasets to correspond to each application. The feature subsets corresponding to each application are concatenated to determine the feature dataset list corresponding to each application, and then sent to the program feature pool. Obtain the feature view configuration information corresponding to each application, and determine the view data corresponding to each application from the feature dataset list in the program feature pool based on the feature view configuration information; In response to a feature view call command submitted by a target application, the view data specified by the command is pushed to the target application to display the feature view corresponding to each application.

2. The application feature data processing method according to claim 1, characterized in that, The steps of obtaining streaming feature data and / or batch feature data corresponding to the feature data in response to a feature data request event from an application include the following steps: Obtain the streaming feature data identifier from the real-time incremental streaming feature data issued by the upstream system; The streaming feature data identifier and its corresponding real-time incremental streaming feature data are sent to the streaming system so that various applications can obtain the changes in the streaming feature data in real time.

3. The application feature data processing method according to claim 1, characterized in that, The steps of preprocessing the streaming feature data and / or batch feature data to determine multiple different feature datasets include the following steps: Data cleaning is performed on the acquired streaming feature data or batch feature data from various sources and of various types. The various types of streaming feature data or batch feature data after data cleaning are normalized to determine multiple different feature datasets.

4. The application feature data processing method according to claim 1, characterized in that, The steps of extracting feature subsets corresponding to each application from multiple feature datasets according to preset rules to correspond to each application, and concatenating the feature subsets corresponding to each application to determine the feature dataset list corresponding to each application, include the following steps: Obtain the application identifier and its corresponding feature data name corresponding to the deep learning model in each application; Based on the application identifier and its corresponding feature data name, extract the feature subset corresponding to the deep learning model from multiple different types of feature datasets, and standardize the feature subset into the feature data vector or feature data matrix corresponding to the deep learning model. The feature dataset list corresponding to the deep learning model in each application is determined based on the feature data vector or feature data matrix.

5. The application feature data processing method according to claim 1, characterized in that, The step of determining the view data corresponding to each application from the feature dataset list in the program feature pool based on the feature view configuration information includes the following steps: In response to a configuration information parsing command, the feature data name and feature data version in the feature view configuration information are obtained; Based on the feature data name and feature data version, determine the feature data corresponding to each application from the feature dataset list, and determine the view data corresponding to each application based on the feature data.

6. The application feature data processing method according to claim 1, characterized in that, The steps for determining the feature view configuration information include the following steps: In response to feature view configuration instructions, the parameter information corresponding to the feature view of the application is defined, including layout style, feature data name, and feature data version; The layout style, feature data name, and feature data version in the parameter information are encapsulated into feature view configuration information.

7. The application feature data processing method according to claims 1 to 6, characterized in that, The steps of responding to a feature view call command submitted by a target application and pushing the view data specified by the command to the target application to display the feature view corresponding to each application include the following steps: Receive the feature view call instruction, and extract the parameter information from the feature view configuration information according to the feature view call instruction; A data query request is generated based on the layout style, feature data name, and feature data version in the parameter information. Based on the data query request, retrieve the view data corresponding to each application from the feature dataset list in the program feature pool; Based on the view data corresponding to the application, the corresponding feature view is displayed in the graphical interface of each application.

8. An application feature data processing apparatus, characterized in that, include: The data preprocessing module is configured to obtain streaming feature data and / or batch feature data corresponding to the feature data in response to the feature data request event of the application, and to perform data preprocessing on the streaming feature data and / or batch feature data to determine multiple different feature datasets; The feature set determination module is configured to extract feature subsets corresponding to each application from multiple feature datasets according to preset rules to correspond to each application, concatenate the feature subsets corresponding to each application to determine the feature dataset list corresponding to each application, and send it to the program feature pool. The view data determination module is configured to obtain the feature view configuration information corresponding to each application, and determine the view data corresponding to each application from the feature dataset list in the program feature pool based on the feature view configuration information. The feature view display module is configured to respond to a feature view call command submitted by a target application and push the view data specified by the command to the target application to display the feature view corresponding to each application.

9. An electronic device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 7, which, when invoked by a computer, executes the steps included in the corresponding method.

Images