Intelligent interaction method and device for data processing based on automatic construction

Abstract

The invention discloses an intelligent interaction method and device for data processing based on automatic construction. The method comprises the steps of obtaining a processing instruction input by a user; determining an initial construction path corresponding to the target object according to model information of the target object in the processing instruction and a pre-stored parallel connection rule; encoding the initial construction path by adopting a preset encoding method to obtain an encoding result corresponding to the target object; the coding result is subjected to off-location processing, target path information corresponding to the target object is obtained, the target path information is used for obtaining scene data under different scenes, and the scene data is processed, so that the scene data to be processed can be quickly found according to the obtained target path information, and the situation that the scene data to be processed can be quickly found in the prior art is not needed; different model parameters and data standardization modes need to be adopted for data in different scenes, and the accuracy and efficiency of data processing are improved.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to an intelligent interactive method and device for data processing based on automated construction. Background Art

[0002] One of the foundations of modern science is empirical evidence. Empirical evidence requires that theories be consistent with objective reality. The digital description of objective reality is called observational data. Observational data may come from nature, human society, or controlled experiments. Generally speaking, the amount of observational data is large, the information density is low, and the expression is obscure, making it difficult for people to understand directly. Therefore, the task of data analysis is to extract the information in the observational data to make a general description or display specific information. In this way, people can understand it and participate in the theoretical reasoning process.

[0003] With the development of the field of data analysis, a large number of data analysis algorithms have been developed. However, data analysis is still a tedious task: first, it is necessary to select an algorithm suitable for the target data set, which requires a large number of tests on different parameters of different algorithms and comparison of quality control indicators for selection; secondly, the quality of the data itself also needs to be examined. The quality of the data and the strategy of data normalization are sometimes coupled with the choice of algorithm; thirdly, how to display the data intuitively and comprehensively is also part of the data analysis workflow. Finally, the data set itself is changing. On the one hand, which content in the data set is selected for data analysis is a problem that needs to be considered in the analysis. On the other hand, new data records may continue to be added to the data set. In short, the data analysis workflow is lengthy and is usually divided into multiple steps: problem definition, data collection, data normalization, data feature exploration, data analysis model algorithm selection and model parameter debugging, model training, model evaluation, and result reporting.

[0004] Many data analysis software are dedicated to assisting data analysis. For example, SPSS, SAS, SPSSAU, SPSSPRO, etc. provide users with graphical interactive interfaces. For example, scipy, sklearn, statsmodels in Python, mlpack in C language, etc. are data analysis algorithm libraries based on a certain computer language. For example, MATLAB and other software are specifically built for a computer language. After entering the era of big data, on the one hand, the accumulation speed of observation data has exploded, and on the other hand, different data analysis methods have emerged in an endless stream. This makes data analysis integrated software more and more important.

[0005] The automated build system is a successful attempt to manage complex processes in the field of code compilation. Typical automated build software includes make, cmake, apache-ant, etc. The automated build system organizes all related objects in a complex process into a directed acyclic graph. All related objects here are usually files in the computer file system, including: pending objects that will be created or modified in the process; external objects that will not be created or modified in the process but need to be referenced and may be modified externally.

[0006] The build graph is described by a series of triples called rules. The rule triples consist of a target, a dependency, and a command. The target is an object, and the dependency is a set of several objects. The meaning of the rule triples in the build graph is: if the target object and the dependent object are both nodes in the build graph, there is an edge from the dependent object to the target object; the actual meaning is that the target object can be generated from the dependent object through the command in the rule triple. The automated build system is a special automated process system. Compared with ordinary processes, it is designed to handle situations where external objects are frequently modified. During the code compilation process, modifying the source code and testing is such a situation.

[0007] In automated building, the core operation is recursive update, that is, comparing the generation time of the target object and the dependent objects in the rule. When the generation time of the target object is later than the update time of any dependent object, the command described in the rule is executed to regenerate the target file.

[0008] If the rule where the target is located is marked with a mandatory execution flag, or the target file does not exist, the generation time of the target file is recorded as infinitely early, which means that it must be regenerated.

[0009] If the dependency is an external object, an error will be reported if the corresponding file does not exist. The generation time of the dependency is the last modification time indicated by the file system.

[0010] If the dependency is an internal object, it is updated recursively; if no regeneration is required, the generation time of the dependency is the most recent modification time indicated by the file system; if regeneration is performed, the generation time of the dependency is recorded as infinitely late, that is, the target of this rule must be regenerated. If the dependency is marked with a temporary file marker, the file corresponding to the dependency has been deleted at this time, so the generation time of the target file is passed to the rule where the dependency marked with the temporary file marker is located as the generation time of the target of the rule where the dependency is located.

[0011] In short, the automated construction process can be summarized as: starting from the final goal input by the user, recursively update the operation in a depth-first traversal manner. There are many different data analysis algorithms. Taking the decision tree algorithm as an example, the decision tree is one of the most basic data analysis algorithms. It is divided into two categories: regression tree and classification tree. Among them, the classification tree can further specify the two local sub-algorithms of "mixed index" and "pruning strategy". Each local sub-algorithm requires multiple parameters; therefore, the decision tree algorithm in the sklearn library has as many as 12 parameters. What's more, to date, there are still literatures proposing "improved decision tree" algorithms.

[0012] The continuous development of data analysis algorithms has brought about two problems. First, it is difficult for a single data analysis software or algorithm library to fully cover various machine algorithms. In fact, SPSS makes up for the scalability problem of the software to a certain extent by allowing the installation of python extension plug-ins. On the other hand, the scope of model selection and parameter adjustment continues to increase, which brings higher requirements for the degree of automation of the process. At present, the sklearn library provides the GridSearchCV class to implement the traversal of parameter combinations, and the RandomizedSearchCV class to sample parameter combinations; Matlab functions provide similar fminsearch functions, but they can only try the parameters accepted by the model algorithm, and cannot take the model type selection and data normalization method into consideration.

[0013] At present, the field of data analysis needs to try to use different model algorithms, different parameters of model algorithms, and different data normalization methods to analyze data sets, and even needs to develop model algorithms by itself. The current data analysis integration software is difficult to effectively manage such complex workflows. Therefore, how to effectively manage complex processes with multiple branching paths in data analysis tasks and then realize automation is a problem that needs to be solved urgently. Summary of the invention

[0014] The present application aims to provide an intelligent interactive method and device for data processing based on automated construction to solve the deficiencies in the prior art. The technical problem to be solved by the present application is achieved through the following technical solutions.

[0015] In a first aspect, an embodiment of the present application provides an intelligent interactive method for data processing based on automated construction, the method comprising: Acquire a processing instruction input by a user, wherein the processing instruction at least includes model information of a target object; Determining an initial construction path corresponding to the target object according to the model information of the target object and a pre-stored parallel rule; Encoding the initial construction path using a preset encoding method to obtain an encoding result corresponding to the target object; The encoding result corresponding to the target object is processed in-place to obtain target path information corresponding to the target object, the target path information is used to obtain scene data in different scenes, and the scene data is processed.

[0016] Optionally, determining an initial construction path corresponding to the target object according to the model information of the target object and a pre-stored parallel rule includes: Recursively process each path of the target object according to the parallel rule, and record the modification time of each path corresponding to the parallel rule; The path corresponding to the latest modification time among the modification times of each path is determined as the initial construction path.

[0017] Optionally, encoding the initial construction path using a preset encoding method to obtain an encoding result corresponding to the target object includes: For each parallel rule, encoding each path corresponding to the parallel rule in chronological order to obtain an encoding result; Determining, according to the encoding result, an overall parallel rule corresponding to the target object; According to the overall parallel rule and the dependency relationship, an encoding result corresponding to the target object is determined.

[0018] Optionally, performing off-site processing on the encoding result corresponding to the target object to obtain target path information corresponding to the target object, wherein the target path information is used to obtain scene data in different scenes, and processing the scene data includes: For each target file corresponding to the target object, a version folder corresponding to the target file is created; Different encoding results are stored in corresponding version folders, and the names of the version folders are different.

[0019] Optionally, determining the encoding result corresponding to the target object according to the overall parallel rule and the dependency relationship includes: The paths corresponding to the overall parallel rule are merged according to the dependency relationship to obtain a coding result corresponding to the target object.

[0020] Optionally, the method further comprises: The target objects corresponding to different target path information are summarized and generalized according to the preset induction rules, and the summary results are stored.

[0021] In a second aspect, an embodiment of the present application provides an intelligent interactive device for data processing based on automated construction, the device comprising: An acquisition module, used for acquiring a processing instruction input by a user, wherein the processing instruction at least includes model information of a target object; A determination module, configured to determine an initial construction path corresponding to the target object according to model information of the target object and a pre-stored parallel rule; An encoding module, used to encode the initial construction path using a preset encoding method to obtain an encoding result corresponding to the target object; The off-site module is used to perform off-site processing on the encoding result corresponding to the target object to obtain target path information corresponding to the target object, wherein the target path information is used to obtain scene data under different scenes and process the scene data.

[0022] Optionally, the determining module is used to: Recursively process each path of the target object according to the parallel rule, and record the modification time of each path corresponding to the parallel rule; The path corresponding to the latest modification time among the modification times of each path is determined as the initial construction path.

[0023] Optionally, the encoding module is used to: For each parallel rule, encoding each path corresponding to the parallel rule in chronological order to obtain an encoding result; Determining, according to the encoding result, an overall parallel rule corresponding to the target object; According to the overall parallel rule and the dependency relationship, an encoding result corresponding to the target object is determined.

[0024] Optionally, the off-site module is used to: For each target file corresponding to the target object, a version folder corresponding to the target file is created; Different encoding results are stored in corresponding version folders, and the names of the version folders are different.

[0025] Optionally, the encoding module is used to: The paths corresponding to the overall parallel rule are merged according to the dependency relationship to obtain a coding result corresponding to the target object.

[0026] Optionally, the off-site module is used to: The target objects corresponding to different target path information are summarized and generalized according to the preset induction rules, and the summary results are stored.

[0027] In a third aspect, an embodiment of the present application provides a terminal device, including: at least one processor and a memory; The memory stores a computer program; the at least one processor executes the computer program stored in the memory to implement the intelligent interactive method for data processing based on automated construction provided in the first aspect.

[0028] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed, the intelligent interactive method for data processing based on automated construction provided in the first aspect is implemented.

[0029] The embodiments of the present application include the following advantages: The intelligent interactive method, apparatus, terminal device and storage medium based on data processing of automated construction provided in the embodiments of the present application obtain processing instructions input by a user, wherein the processing instructions at least include model information of a target object; determine an initial construction path corresponding to the target object according to the model information of the target object and a pre-stored parallel rule; encode the initial construction path using a preset encoding method to obtain an encoding result corresponding to the target object; perform off-site processing on the encoding result corresponding to the target object to obtain target path information corresponding to the target object, wherein the target path information is used to obtain scene data in different scenarios and process the scene data, wherein the target path information is used to obtain scene data in different scenarios and process the scene data, in the embodiments of the present application, by introducing parallel rules , describing different ways of generating an object, further introducing a construction path to represent all generation method choices of an object in itself, its direct dependencies and its indirect dependencies. Therefore, after the user inputs a processing instruction, the model information of the target object in the processing instruction is obtained, the model information of the target object is processed using a pre-stored parallel rule, an initial construction path is constructed, and then the initial construction path is encoded and processed off-site, so as to obtain target path information corresponding to different target objects. The target path information is used to obtain scene data in different scenarios and process the scene data. In this way, the scene data to be processed can be quickly found according to the obtained target path information, without the need to use different model parameters and data normalization methods for data in different scenarios as in the prior art, thereby improving the accuracy and efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the existing technical solutions, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0031] Figure 1 This is a flow chart of an intelligent interactive method for data processing based on automated construction in one embodiment of the present application; Figure 2 This is a flowchart of another intelligent interactive method for data processing based on automated construction in one embodiment of the present application; Figure 3 This is a flowchart of another intelligent interactive method for data processing based on automated construction in one embodiment of the present application; Figure 4 The storage method of rules in the memory in one embodiment of the present application; Figure 5 It is a structural block diagram of an embodiment of an intelligent interactive device for data processing based on automated construction of the present application; Figure 6 It is a structural diagram of a terminal device of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0033] An embodiment of the present application provides an intelligent interactive method for data processing based on automated construction, for processing data. The execution subject of this embodiment is an intelligent interactive device for data processing based on automated construction, which is set on a terminal device, for example, the terminal device at least includes a computer terminal, etc.

[0034] Reference Figure 1 , shows a flowchart of an embodiment of an intelligent interactive method for data processing based on automated construction of the present application, the method may specifically include the following steps: S101, obtaining a processing instruction input by a user, wherein the processing instruction at least includes model information of a target object; Specifically, the terminal device receives a processing instruction input by a user, and the processing instruction includes at least model information of the target object.

[0035] S102, determining an initial construction path corresponding to the target object according to model information of the target object and a pre-stored parallel rule; Parallel rules are introduced to indicate that a target object has multiple construction methods to choose from. First of all, parallel rules need to be compatible with existing conventional construction methods. From the content point of view: parallel rules are two-tuples consisting of "target" and "path". You can choose any "path" to achieve the "target"; like the "dependency" of ordinary rules, the "path" is also an object in the construction graph. Parallel rules are also regarded as special rule triples. At this time, the command of the rule takes an empty value, the dependency of the rule is the path of the parallel rule, and the target of the rule is still the target of the parallel rule.

[0036] The target name matching of parallel rules is no different from that of regular target names. Parallel rules execute special generation commands: the files generated by any path are moved to the files indicated by the parallel rule target. Therefore, the update of parallel rules is also different from that of ordinary rules: when recursively updating, the most recent modification time of each path is recorded; when the generation command execution of all paths reports an error, the parallel rule will report an error; then the path with the latest modification time is taken as the path selected by the parallel rule. In particular, if the most recent modification time of a path is returned as new, that is, the generation command for the path has been re-executed, the path can be directly selected, and the attempt to other paths is terminated.

[0037] S103, encoding the initial construction path using a preset encoding method to obtain an encoding result corresponding to the target object; After the parallel rule is introduced, each object in the constructed graph may have different path choices. In order to achieve more complex management functions, it is necessary to encode the different path choices of each object.

[0038] S104: Perform off-bit processing on the encoding result corresponding to the target object to obtain target path information corresponding to the target object, wherein the target path information is used to obtain scene data in different scenes, and process the scene data.

[0039] Specifically, the target object may have multiple construction paths, and the results of different paths have the same name, so they cannot exist at the same time in the computer file system. In order to save and reference the results of different generation paths at the same time, it is necessary to introduce a mechanism to rename the target object according to the path encoding string, that is, out-of-place processing.

[0040] In an embodiment of the present application, by determining the target path information based on the possible existence of multiple construction paths for the target object, the corresponding storage directory can be determined based on the target path information, and then the scene data in different scenarios can be obtained from the storage directory, and then the different scene data can be processed according to user needs. Therefore, there is no need to use different model parameters and data normalization methods for data in different scenarios as in the prior art, so as to improve the accuracy and efficiency of intelligent interaction based on automatically constructed data processing.

[0041] The embodiment of the present application is based on the script specified by the make program, and a new syntax is added to represent parallel rules and inductive rules: The basic triple syntax specified by the make program is:

[0042] Parallel rule syntax:

[0043] Induction rules:

[0044] The intelligent interactive method for data processing based on automated construction provided in an embodiment of the present application obtains processing instructions input by a user, wherein the processing instructions at least include model information of a target object; determines an initial construction path corresponding to the target object according to the model information of the target object and a pre-stored parallel rule; encodes the initial construction path using a preset encoding method to obtain an encoding result corresponding to the target object; performs off-site processing on the encoding result corresponding to the target object to obtain target path information corresponding to the target object, wherein the target path information is used to obtain scene data in different scenes, and processes the scene data, describes different generation methods of an object by introducing parallel rules, and further introduces a construction path to represent all generation method choices of an object among itself, its direct dependencies, and its indirect dependencies. Further, a program function for generating all possible construction paths is introduced to meet the traversal test requirements of model algorithm selection, namely parameter selection. In the embodiment of the present application, different generation methods of an object are described by introducing parallel rules, and a construction path is further introduced to represent all generation method selections of an object in itself, its direct dependencies, and its indirect dependencies. Therefore, after the user inputs a processing instruction, the model information of the target object in the processing instruction is obtained, and the model information of the target object is processed using a pre-stored parallel rule to construct an initial construction path, and then the initial construction path is encoded and processed off-site to obtain target path information corresponding to different target objects. The target path information is used to obtain scene data in different scenarios and process the scene data. In this way, the scene data to be processed can be quickly found based on the obtained target path information, without the need to use different model parameters and data normalization methods for data in different scenarios as in the prior art, thereby improving the accuracy and efficiency of data processing.

[0045] Another embodiment of the present application further supplements the intelligent interaction method for data processing based on automated construction provided in the above embodiment.

[0046] like Figure 2 and Figure 3 As shown, the embodiment of the present application implements the operation control module in the form of an automated construction system, which is the core module of the program. The operation control module interacts with the user through a model configuration file, an automated construction script, and a command line interface. The control module parses the model configuration file to obtain model information, such as the model algorithm executable program path, the observation data accepted by the model, the external parameters accepted by the model, etc. The automated construction script indicates which model algorithms, preprocessing methods, model parameters, and other different options to be tested and compared during operation. The actual operation is performed through a command line interface. On the basis of the above, a graphical interface is encapsulated to prompt the user to operate.

[0047] 1) Model algorithm configuration file 1.1 Model Algorithm Configuration File Retrieval The model algorithm configuration file follows the json format. The program uses the relative file path "* / mod.json" and "*.mod.json" with wildcards to match under one or more fixed model retrieval paths. The matched file is regarded as the model algorithm configuration file. In order to facilitate the retrieval of the model, the main body of the model file name, that is, the content matched by the wildcard, must be the prefix of the model algorithm name.

[0048] 1.2 Model algorithm configuration file content There are many types of model algorithms. This project believes that model algorithms can be summarized by three functions: (1) Based on the training data and parameters, a model instance is established, which is recorded as the initial training train.

[0049] (2) Adjust the model instance based on the training data, parameters, and existing model instances, which is recorded as step training step.

[0050] (3) Apply the model instance to test data or production data to obtain the result, which is recorded as apply.

[0051] The essence of a model instance is a series of parameters, called model internal parameters. The initial model training process is the process of determining the model internal parameters; the model step training process is the process of adjusting the model internal parameters. To distinguish, the input parameters in the initial model training and step training are called external parameters. Internal parameters and external parameters are allowed to have intersections. External parameters are further divided into initialization parameters and step parameters: the parameters of step training are step parameters; the parameters required in the initial training, but not required or not allowed to be changed in step training, are called initialization parameters. In the absence of confusion, initial training is also referred to as training.

[0052] In addition, two auxiliary functions are required, which are not independent: (4) Returns the internal parameters of the model. The internal parameters are also called the state of the model. This function is denoted as state.

[0053] (5) Some models need to be applied directly to the training data. Continuous training and application is called launch.

[0054] This generalization applies to most model algorithms, from the simplest calculation of the mean or variance of data to the most complex deep learning for image recognition.

[0055] The model algorithm needs to implement one or more of these five functions. The model configuration file records the implementation or calling methods of these functions and related parameters. The model configuration file stores a json object, and the description of the model function is the field in this json object, as shown in Table 1.

[0056] Table 1

[0057] Table 1 describes the json fields of the model algorithm configuration file. The model algorithm is referred to as model in the table. "Not filled" means that the value of this field is not accepted as a parameter due to the selection of other fields. "Not filled" is a kind of "required" situation.

[0058] The five Java objects train, step, apply, state, launch, etc. that describe the implementation of the model algorithm function all have the same fixed fields. First, it is necessary to specify the implementation method of the model algorithm function through the enumerated variable funcType. Currently, executable programs "EXE" and script execution "SCRIPT" are supported. For script execution, it is necessary to further specify the parser path through the execPath field, such as a simple "python", or specify the parser version by giving an absolute path, such as "~ / Python310 / python.exe". In particular, the funcType field of the state function can indicate a text file "TEXT" or a binary file "BIN", indicating that the file storage format of the model's internal parameters is transparent, and the embodiment of the present application can be read directly. At this time, the execPath field represents the storage path.

[0059] Next, specify the path for implementing the model algorithm function. Regarding the matching method of the two model algorithm configuration files, there is a difference here: the "* / mod.json" format indicates that the model algorithm-related content is stored in the "* / " path, and the path can use a relative path based on "* / ". The "*.mod.json" format requires an absolute path. Finally, paramTrans is used to specify the form of providing parameters to the program or script. Currently, three forms are supported: "JSON" means that the parameters are input as a whole in the form of a json string; "SEQ" means that the parameters are passed in sequence; "--" means passing in accordance with the specification of the getopt function in the POSIX standard, as shown in Table 2, the json field of the model algorithm function description.

[0060] Table 2

[0061] trainDt, applyDt and outDt describe the input and output data of the model. They are all objects with indefinite fields. The fields in the objects are the parameter names of the input and output data. The data accepted by the model has a table structure, that is, each row represents a record, and each column represents an attribute in the record. The requirements of the parameter on the table structure are expressed by a fixed-length array.

[0062] The first json array element specifies the basic computer data type of the table element. Here, the data types of different columns in the table are required to be the same. If data of different data types are required, the table can be split and passed with different parameters. This is in line with the specific implementation of most model algorithms.

[0063] The second json array element specifies the number of columns in the table. Positive integers are regular values, indicating that the number of columns of input and output data is fixed. 0 is a special value. For input data, it means that any number of columns are accepted; for output data, it means that the number of output data columns varies with the input data. -1 is a special value, indicating that the number of output data columns is specified by the model external parameters.

[0064] The third json element indicates the row alignment. Some models are used for prediction, so the output results will be ahead of the test data by a certain number of days, so an integer is introduced to indicate the row alignment. A non-negative integer is used as a regular value to indicate the number of days ahead; if the number of days ahead of all data is 0, the data can be aligned by row. -1 is a special value, indicating that the number of days ahead depends on external parameters. -2 is a special value, indicating that the data does not need to be aligned with other data.

[0065] The last JSON element is a comment for display only. As shown in Table 3, it is the JSON field of the model input and output data array element.

[0066] Table 3

[0067] initParam and stepParam describe the model parameter type for input, innerParam and outParam are used for output. They are both objects with undefined fields, and the fields in the objects are the parameter names of the input and output model parameters. The description of the parameters is represented by a fixed-length array.

[0068] The first json array element specifies the parameter type. Since the basic computer data types are not sufficient to describe all possible parameter types, the parameter types are extended based on the basic computer types. Currently, two types have been added: enumeration and 32-bit integer array. The string here is divided into sections by colons. The first section is the enumeration of the parameter type, and the second section is a further description of the parameter type: that is, the enumeration type needs to explain the options, and the 32-bit integer array type needs to explain the array length.

[0069] For output parameters, you only need to specify the parameter type. For input parameters, you also need to specify the default value of the parameter for user use, as shown in Table 4, which is the JSON field of the model input parameter.

[0070] Table 4

[0071] 2) Operation control module based on automation construction like Figure 4 As shown, first of all, it is necessary to specifically represent the rule triple in the computer, that is, to clarify the storage form of the rule triple in the computer. On the one hand, the storage space required for the rule triple cannot be determined in advance, and on the other hand, it is usually not necessary to randomly access the rules. Therefore, the method of sequential arrangement of indefinite-length elements is used to store the rule structure and its target, dependency, command and other string lists in the memory. In terms of implementation, there is an extension compared to the theory. The rules can be generalized: the generalized rules describe the target in the form of matching, that is, a generalized rule describes many specific rules; the dependencies and commands of the generalized rules can contain pending content that depends on the target matching results; the target must be actually given to concretize the generalized rules. There are two ways to return the target: matching with regular expressions and taking the regular expression capture group as the pending content; matching with wildcards and taking the wildcard matching content as the pending content. The latter is simpler when the latter is adopted. Tables 5 and 6 are the C++ class fields of the rules, among which the fields in Table 6 are only used in the traversal process.

[0072] Table 5

[0073] Table 6

[0074] The operations based on rule triples are as follows: (a) Target name matching match ,Right now .remember is a rule, which may be specific or general, given a target name T , you can check whether the given target name matches the rule target name. If not, return a null value; if it matches, G is a specific rule, returns itself. G is a generalized rule, and returns the specific rule generated based on the given target name. The returned rule must be a specific rule. R .

[0075] (b) The above matching can be tried on all the rules in the constructed graph, denoted as .Right now If multiple rules are matched at the same time, the one with the least pending content will be matched. If the length of the pending content is the same, an error will be reported.

[0076] (c) Execute the generation command, i.e. .

[0077] (d) Get the last modification time of the file system t , if the file does not exist, a special value is returned Indicates infinitely early. The modification time defines two special values, one of which is infinitely late. , another special value Indicates an error, infinitely early is less than any normal value, infinitely late is greater than any normal value, and an error is greater than infinitely late.

[0078] (e) The above steps together constitute the update step, starting from the user-specified goal. In the update step, the depth-first recursive operation is performed on the graph. That is, .in t is the last modification time of the target, and is passed recursively so that rules marked with the temporary file marker refer to it as their own target creation time.

[0079]

[0080] Optionally, determining an initial construction path corresponding to the target object according to model information of the target object and a pre-stored parallel rule includes: Recursively process each path of the target object according to the parallel rule, and record the modification time of each path corresponding to the parallel rule; The path corresponding to the latest modification time among the modification times of each path is determined as the initial construction path.

[0081] Specifically, parallel rules are introduced to indicate that a target object has multiple construction methods to choose from. First of all, parallel rules need to be compatible with existing conventional construction methods. From the content point of view: parallel rules are two-tuples consisting of "target" and "path". You can choose any "path" to achieve the "target"; like the "dependency" of ordinary rules, the "path" is also an object in the construction graph. Parallel rules are also regarded as special rule triples. At this time, the command of the rule takes an empty value, the dependency of the rule is the path of the parallel rule, and the target of the rule is still the target of the parallel rule.

[0082] The target name matching of parallel rules is no different from the regular target name matching. Parallel rules execute special generation commands: move the files generated by any path to the files indicated by the parallel rule target. Therefore, the update of parallel rules is also different from that of ordinary rules: when recursively updating, the last modification time of each path is recorded; when the generation command execution of all paths fails, the parallel rule will report an error; then the path with the latest modification time is taken as the path selected by the parallel rule. In particular, if the last modification time of a path is returned as , that is, the generation command for the path is re-executed, the path can be directly selected, and the attempt to other paths is terminated.

[0083] Optionally, a preset encoding method is used to encode the initial construction path to obtain an encoding result corresponding to the target object, including: For each parallel rule, encode each path corresponding to the parallel rule in time order to obtain an encoding result; According to the encoding results, determine the overall parallel rule corresponding to the target object; According to the overall parallel rules and dependencies, the encoding result corresponding to the target object is determined.

[0084] Specifically, after the parallel rule is introduced, each object in the constructed graph may have different path choices. In order to achieve more complex management functions, it is necessary to encode the different path choices of each object. The specific method is: (1) In each parallel rule, encode its path in sequence, starting from 1.

[0085] (2) For an object, determine all the parallel rules that may be used.

[0086] (3) Specifying the path selection of these parallel rules is the path selection of the object. It is the first representation method that constitutes the path, the complete array representation: that is, using an array to represent the path, and the length of the array is the number of parallel nodes in the construction graph. The path of a parallel rule is represented by an ordinal starting from 1. The special value -1 indicates that the parallel rule is not an indirect dependency of the object. The special value 0 indicates that the parallel rule is an indirect dependency of the object, but due to the selection of other parallel rule paths, the parallel rule does not need to be executed. In order to do this, two constraints are required: first, different specific nodes of a generalized node cannot directly or indirectly constitute a dependency relationship; second, in a diamond dependency relationship, different branch choices are not allowed for common parallel dependencies.

[0087] (4) Take out the non-negative integer items in the complete array representation of the parallel rule and arrange them in order. This is also enough to completely specify the path selection of the object, which is recorded as the local array representation.

[0088] (5) When outputting to the file system, this path selection is further encoded as a string, concatenating the integer string with an underscore "_" and adding another underscore "_" as a prefix; this is the string representation of the object path. The three representation methods can be converted into each other.

[0089] Table 7 shows the global variables during construction.

[0090] Table 7

[0091] There are three operations for specifying object paths, all of which are based on full array representation: Decomposition: that is, get the path of its child nodes from the path of the node. Use the recursive dependency mask maskm of the child nodes and set the array elements corresponding to the bits with a value of 0 in the mask to -1.

[0092] Comparison: Compare the generated object path to see if it is consistent with the required object path. That is, skip the elements with values ​​of 0 or -1 in the path of the generated object and compare the remaining elements one by one. If they are all equal, the two paths are considered to be consistent.

[0093] Merge: that is, obtain the path of the parent node based on the path of the child node. There are two cases. When the parent node is a parallel node, the ordinal number of the parent node in the parallel node is recorded as p ; First, set the mask maskm to 1 and the elements in the path array that are -1 to 0, and then set the array elements corresponding to the parent node to p , that is, the path array of the parent node is obtained. When the parent node is not a parallel node, the path array of its child nodes is merged element by element; the merging of path elements is specified to satisfy the commutative law and the associative law, which is recorded as V For any non-negative integer j , ; For any positive integer j , ; If two positive integers are compared, if they are equal, the common value of the two is returned. If they are not equal, it is reported that they cannot be merged. Unable to merge means that the two child nodes are associated with the same parallel node in a diamond dependency manner, and different choices are made at the parallel node, so they cannot be merged.

[0094] This allows you to specify a specific path for the final target to be generated. The program will save the path actually executed during the last build in the build cache path, which is the directory where the build script is located by default. In this way, when generating at a specified path, you can compare the object's build path to see if it is consistent with the specified path. If the paths are inconsistent, regenerate the object.

[0095] Optionally, performing off-position processing on the encoding result corresponding to the target object to obtain target path information corresponding to the target object, the target path information is used to obtain scene data in different scenes, and processing the scene data includes: For each target file corresponding to a target object, create a version folder corresponding to the target file; Different encoding results are stored in corresponding version folders, and the names of the version folders are different.

[0096] Specifically, the target object may have multiple build paths, and the results of different paths have the same name, so they cannot exist at the same time in the computer file system. In order to save and reference the results of different build paths at the same time, a mechanism for renaming the target object based on the path encoding string needs to be introduced.

[0097] When instructing to execute out-of-place, a version folder of the target file is created for each target file built in the build cache directory. The version folder is named by adding the ".dir" suffix to the target file name. Target files in different file system paths may have the same file name, which may cause version folder naming conflicts. In this case, a random suffix is ​​added to the version folder. A mapping is established to store the target file name and its version folder name in the cache file. Based on the version folder, the path string encoding is used as the main file name, and the suffix of the target file name is added to form a new file name: <build cache directory> / <target file name>.dir<possible random suffix> / <path encoding string>.<target file name suffix>. Replace the target file name with this new file name as the file name used when executing out-of-place.

[0098] When a specific build path is specified for execution, regardless of whether off-site execution is indicated, the version folder will be checked for target files that can be used directly.

[0099] Optionally, determining a coding result corresponding to the target object according to the overall parallel rule and the dependency relationship includes: The paths corresponding to the overall parallel rule are merged according to the dependency relationship to obtain the encoding result corresponding to the target object.

[0100] For the model selection and parameter adjustment process, the final goal is a comparable value, such as training error or prediction error. Each combination of selection and parameters is a possible construction path. Therefore, all possible paths need to be executed. Full path execution determines all possible paths in a recursive manner: when generating the target object, different paths with different dependencies are attempted to be merged, and all successful merges are generated out of place.

[0101] Optionally, the method further comprises: The target objects corresponding to different target path information are summarized and generalized according to the preset induction rules, and the summary results are stored.

[0102] Specifically, an induction rule is introduced to summarize objects constructed by different paths. The induction rule is relatively simple, with only one dependency and one line of command output. The induction rule generates a line of text for each path of the dependency, and writes these text suffixes into the target file of the induction rule, which makes it easy to compare objects constructed by different paths.

[0103] The intelligent interactive method for data processing based on automated construction provided in an embodiment of the present application obtains a processing instruction input by a user, wherein the processing instruction includes at least model information of a target object; determines an initial construction path corresponding to the target object according to the model information of the target object and a pre-stored parallel rule; encodes the initial construction path using a preset encoding method to obtain an encoding result corresponding to the target object; performs off-site processing on the encoding result corresponding to the target object to obtain target path information corresponding to the target object, wherein the target path information is used to obtain scene data in different scenes and process the scene data. In an embodiment of the present application, by introducing parallel rules to describe different generation methods of an object, a construction path is further introduced to represent an object in All generation mode selections among itself, its direct dependencies and its indirect dependencies. Therefore, after the user inputs the processing instruction, the model information of the target object in the processing instruction is obtained, the model information of the target object is processed by using the pre-stored parallel rule, and an initial construction path is constructed. The initial construction path is then encoded and processed off-site to obtain target path information corresponding to different target objects. The target path information is used to obtain scene data in different scenarios and process the scene data. In this way, the scene data to be processed can be quickly found according to the obtained target path information, without the need to use different model parameters and data normalization methods for data in different scenarios as in the prior art, thereby improving the accuracy and efficiency of data processing.

[0104] Another embodiment of the present application provides an intelligent interaction device based on automated data processing, which is used to execute the intelligent interaction method based on automated data processing provided in the above embodiment.

[0105] Reference Figure 5 , shows a structural block diagram of an embodiment of an intelligent interactive device for data processing based on automated construction of the present application, the device may specifically include the following modules: an acquisition module 501, a determination module 502, an encoding module 503 and a separation module 504, wherein: The acquisition module 501 is used to acquire the processing instruction input by the user, wherein the processing instruction at least includes the model information of the target object; The determination module 502 is used to determine an initial construction path corresponding to the target object according to the model information of the target object and the pre-stored parallel rule; The encoding module 503 is used to encode the initial construction path using a preset encoding method to obtain an encoding result corresponding to the target object; The off-position module 504 is used to perform off-position processing on the encoding result corresponding to the target object to obtain target path information corresponding to the target object. The target path information is used to obtain scene data in different scenes and process the scene data.

[0106] The intelligent interactive device based on data processing of automatic construction provided in the embodiment of the present application obtains the processing instruction input by the user, wherein the processing instruction at least includes the model information of the target object; determines the initial construction path corresponding to the target object according to the model information of the target object and the pre-stored parallel rule; encodes the initial construction path by a preset encoding method to obtain the encoding result corresponding to the target object; performs off-site processing on the encoding result corresponding to the target object to obtain the target path information corresponding to the target object, and the target path information is used to obtain scene data in different scenes and process the scene data. In the embodiment of the present application, by introducing parallel rules to describe different generation methods of an object, a construction path is further introduced to represent an object in All generation mode selections among itself, its direct dependencies and its indirect dependencies. Therefore, after the user inputs the processing instruction, the model information of the target object in the processing instruction is obtained, the model information of the target object is processed by using the pre-stored parallel rule, and an initial construction path is constructed. The initial construction path is then encoded and processed off-site to obtain target path information corresponding to different target objects. The target path information is used to obtain scene data in different scenarios and process the scene data. In this way, the scene data to be processed can be quickly found according to the obtained target path information, without the need to use different model parameters and data normalization methods for data in different scenarios as in the prior art, thereby improving the accuracy and efficiency of data processing.

[0107] Another embodiment of the present application further supplements the intelligent interactive device for data processing based on automated construction provided in the above embodiment.

[0108] Optionally, a module is determined to: Recursively process each path of the target object according to the parallel rule, and record the modification time of each path corresponding to the parallel rule; The path corresponding to the latest modification time among the modification times of each path is determined as the initial construction path.

[0109] Optionally, the encoding module is used to: For each parallel rule, encode each path corresponding to the parallel rule in time order to obtain an encoding result; According to the encoding results, determine the overall parallel rule corresponding to the target object; According to the overall parallel rules and dependencies, the encoding result corresponding to the target object is determined.

[0110] Optionally, an off-site module is used to: For each target file corresponding to a target object, create a version folder corresponding to the target file; Different encoding results are stored in corresponding version folders, and the names of the version folders are different.

[0111] Optionally, the encoding module is used to: The paths corresponding to the overall parallel rule are merged according to the dependency relationship to obtain the encoding result corresponding to the target object.

[0112] Optionally, an off-site module is used to: The target objects corresponding to different target path information are summarized and generalized according to the preset induction rules, and the summary results are stored.

[0113] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0114] The intelligent interactive device based on data processing of automatic construction provided in the embodiment of the present application obtains the processing instruction input by the user, wherein the processing instruction at least includes the model information of the target object; determines the initial construction path corresponding to the target object according to the model information of the target object and the pre-stored parallel rule; encodes the initial construction path by a preset encoding method to obtain the encoding result corresponding to the target object; performs off-site processing on the encoding result corresponding to the target object to obtain the target path information corresponding to the target object, and the target path information is used to obtain scene data in different scenes and process the scene data. In the embodiment of the present application, by introducing parallel rules to describe different generation methods of an object, a construction path is further introduced to represent an object in All generation mode selections among itself, its direct dependencies and its indirect dependencies. Therefore, after the user inputs the processing instruction, the model information of the target object in the processing instruction is obtained, the model information of the target object is processed by using the pre-stored parallel rule, and an initial construction path is constructed. The initial construction path is then encoded and processed off-site to obtain target path information corresponding to different target objects. The target path information is used to obtain scene data in different scenarios and process the scene data. In this way, the scene data to be processed can be quickly found according to the obtained target path information, without the need to use different model parameters and data normalization methods for data in different scenarios as in the prior art, thereby improving the accuracy and efficiency of data processing.

[0115] Yet another embodiment of the present application provides a terminal device for executing the intelligent interaction method for data processing based on automated construction provided in the above embodiment.

[0116] Figure 6 It is a schematic diagram of the structure of a terminal device of the present application, such as Figure 6 As shown, the terminal device includes: at least one processor 601 and a memory 602; The memory stores a computer program; and at least one processor executes the computer program stored in the memory to implement the intelligent interactive method for data processing based on automated construction provided in the above embodiment.

[0117] The terminal device provided in the present embodiment obtains a processing instruction input by a user, wherein the processing instruction includes at least model information of a target object; determines an initial construction path corresponding to the target object according to the model information of the target object and a pre-stored parallel rule; encodes the initial construction path using a preset encoding method to obtain an encoding result corresponding to the target object; performs off-site processing on the encoding result corresponding to the target object to obtain target path information corresponding to the target object, wherein the target path information is used to obtain scene data in different scenes and process the scene data. In the embodiment of the present application, by introducing parallel rules to describe different generation methods of an object, a construction path is further introduced to represent an object in itself, its direct dependencies, All generation method selections in the dependent item and its indirect dependent items. Therefore, after the user inputs the processing instruction, the model information of the target object in the processing instruction is obtained, the model information of the target object is processed by using the pre-stored parallel rule, and an initial construction path is constructed, and then the initial construction path is encoded and processed in situ, so as to obtain target path information corresponding to different target objects. The target path information is used to obtain scene data in different scenarios and process the scene data. In this way, the scene data to be processed can be quickly found according to the obtained target path information, without the need to use different model parameters and data normalization methods for data in different scenarios as in the prior art, so as to improve the accuracy and efficiency of data processing.

[0118] Yet another embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed, the intelligent interactive method for data processing based on automated construction provided in any of the above embodiments is implemented.

[0119] According to the computer-readable storage medium of the present embodiment, by obtaining a processing instruction input by a user, wherein the processing instruction at least includes model information of a target object; determining an initial construction path corresponding to the target object according to the model information of the target object and a pre-stored parallel rule; encoding the initial construction path using a preset encoding method to obtain an encoding result corresponding to the target object; performing off-site processing on the encoding result corresponding to the target object to obtain target path information corresponding to the target object, wherein the target path information is used to obtain scene data in different scenes, and process the scene data. In the embodiment of the present application, by introducing parallel rules to describe different generation methods of an object, a construction path is further introduced to represent an object in itself, its direct Therefore, after the user inputs a processing instruction, the model information of the target object in the processing instruction is obtained, the model information of the target object is processed by using a pre-stored parallel rule, an initial construction path is constructed, and then the initial construction path is encoded and processed in situ, so as to obtain target path information corresponding to different target objects. The target path information is used to obtain scene data in different scenarios and process the scene data. In this way, the scene data to be processed can be quickly found according to the obtained target path information, without the need to use different model parameters and data normalization methods for data in different scenarios as in the prior art, thereby improving the accuracy and efficiency of data processing.

[0120] It should be noted that the above detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present application belongs.

[0121] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should also be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0122] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0123] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0124] For ease of description, spatially relative terms, such as "above", "above", "on the upper surface of", "above", etc., may be used herein to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" may include both "above" and "below". The device may also be positioned in other different ways, such as rotated 90 degrees or in other orientations, and the spatially relative descriptions used herein are interpreted accordingly.

[0125] In the above detailed description, reference is made to the accompanying drawings, which form a part of this document. In the accompanying drawings, similar symbols typically identify similar components unless the context indicates otherwise. The illustrated embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be used, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.

[0126] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An intelligent interactive method for data processing based on automated construction, It is characterized in that The method comprises: Acquire a processing instruction input by a user, wherein the processing instruction at least includes model information of a target object; Determining an initial construction path corresponding to the target object according to the model information of the target object and a pre-stored parallel rule; Encoding the initial construction path using a preset encoding method to obtain an encoding result corresponding to the target object; The encoding result corresponding to the target object is processed in-place to obtain target path information corresponding to the target object, the target path information is used to obtain scene data in different scenes, and the scene data is processed.

2. The intelligent interactive method for data processing based on automated construction according to claim 1, It is characterized in that The determining, according to the model information of the target object and the pre-stored parallel rule, an initial construction path corresponding to the target object comprises: Recursively process each path of the target object according to the parallel rule, and record the modification time of each path corresponding to the parallel rule; The path corresponding to the latest modification time among the modification times of each path is determined as the initial construction path.

3. The intelligent interactive method for data processing based on automated construction according to claim 1, It is characterized in that The using a preset encoding method to encode the initial construction path to obtain an encoding result corresponding to the target object includes: For each parallel rule, encoding each path corresponding to the parallel rule in chronological order to obtain an encoding result; Determining, according to the encoding result, an overall parallel rule corresponding to the target object; According to the overall parallel rule and the dependency relationship, an encoding result corresponding to the target object is determined.

4. The intelligent interactive method for data processing based on automated construction according to claim 1, It is characterized in that The step of performing off-position processing on the encoding result corresponding to the target object to obtain target path information corresponding to the target object, wherein the target path information is used to obtain scene data in different scenes, and processing the scene data includes: For each target file corresponding to the target object, a version folder corresponding to the target file is created; Different encoding results are stored in corresponding version folders, and the names of the version folders are different.

5. The intelligent interactive method for data processing based on automated construction according to claim 3, It is characterized in that The step of determining the encoding result corresponding to the target object according to the overall parallel rule and the dependency relationship includes: The paths corresponding to the overall parallel rule are merged according to the dependency relationship to obtain a coding result corresponding to the target object.

6. The intelligent interactive method for data processing based on automated construction according to claim 1, It is characterized in that The method further comprises: The target objects corresponding to different target path information are summarized and generalized according to the preset induction rules, and the summary results are stored.

7. An intelligent interactive device based on automated data processing, It is characterized in that The device comprises: An acquisition module, used for acquiring a processing instruction input by a user, wherein the processing instruction at least includes model information of a target object; A determination module, configured to determine an initial construction path corresponding to the target object according to model information of the target object and a pre-stored parallel rule; An encoding module, used to encode the initial construction path using a preset encoding method to obtain an encoding result corresponding to the target object; The off-site module is used to perform off-site processing on the encoding result corresponding to the target object to obtain target path information corresponding to the target object, wherein the target path information is used to obtain scene data under different scenes and process the scene data.

8. The intelligent interactive device based on automated data processing according to claim 7, It is characterized in that The determining module is used to: recursively process each path of the target object according to the parallel rule, and record the modification time of each path corresponding to the parallel rule; The path corresponding to the latest modification time among the modification times of each path is determined as the initial construction path.

9. A terminal device, It is characterized in that include: at least one processor and memory; The memory stores a computer program; The at least one processor executes the computer program stored in the memory to implement the intelligent interactive method for data processing based on automated construction according to any one of claims 1 to 6.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, which, when executed, implements the intelligent interactive method for data processing based on automated construction according to any one of claims 1 to 6.

Images