A method, device and storage medium for multi-domain data fusion

Abstract

The present invention provides a method, device and storage medium for multi-domain data fusion, the method comprising: S1, establishing a task rule library in a pre-set multi-domain scheduling fusion area; S2, extracting all selected tasks in the task rule library, determining the corresponding data domain group and the corresponding execution engine group for each selected task, and obtaining a task set related to the same data domain; S3, scheduling tasks in the task set related to the same data domain according to the pre-set task priority, and preferentially calling the execution engine corresponding to the task with a higher priority; S4, for each task, storing the task results returned from the corresponding data domain by each execution engine in the corresponding execution engine group to the multi-domain scheduling fusion area, and fusing them in the multi-domain scheduling fusion area to obtain a fusion result. The above technical solution can achieve efficient cross-domain data fusion.

Description

Technical Field

[0001] The present invention relates to the field of big data processing, and in particular to a method, device and storage medium for multi-domain data fusion. Background Art

[0002] In today's era, big data has become a valuable resource. Existing big data projects often adopt the traditional "standard + centralized" fusion method. This fusion method requires that the data of each domain be converted into a unified standard format and stored in a specific domain. For big data projects with large data volume and complex structure, this processing method has the following shortcomings: centralized storage will lead to repeated conversion and storage of a large amount of data, which will lead to higher project construction costs; due to frequent business changes in each domain, the generated business data will also change frequently, and the centralized storage method according to certain standards is often difficult to match the changing data format in time and be compatible with the old data format, which will lead to a decline in external empowerment capabilities. Summary of the invention

[0003] The embodiments of the present invention provide a method, device and storage medium for multi-domain data fusion, which realizes efficient cross-domain data fusion by setting a multi-domain scheduling fusion area, task normalization and task scheduling.

[0004] In one aspect, a method for multi-domain data fusion is provided, for performing a processing task using data from multiple domains, including:

[0005] S1, establishing a task rule base in a pre-set multi-domain scheduling fusion area, the task rule base including: an identifier of a task to be processed, a data source related to the task to be processed, a plurality of data domains related to the data source, and an execution engine corresponding to each data domain;

[0006] S2, extracting all selected tasks in the task rule library, determining the corresponding data domain group and the corresponding execution engine group for each selected task, and obtaining a task set related to the same data domain;

[0007] S3, scheduling tasks in a task set related to the same data domain according to a preset task priority, and preferentially calling the execution engine corresponding to the task with a higher priority;

[0008] S4, for each selected task, storing the task results returned by each execution engine in the corresponding execution engine group from the corresponding data domain into the multi-domain scheduling fusion area, and fusion is performed in the multi-domain scheduling fusion area to obtain a fusion result.

[0009] Further, in the method, step S2 comprises:

[0010] Extract all the selected tasks in the task rule base to form a task data set Sn to be processed, and create a new normalized task data set Tn;

[0011] Traverse Sn, obtain the data domain group corresponding to the data source related to each task and the corresponding execution engine group list (clyc), and split Sn according to the execution engine group list (clyc) and store it in Tn, where Tn = {Sn, list (clyc)};

[0012] Traverse Tn, aggregate the records in Tn according to the execution engine, merge the tasks in the same data domain, and obtain a set of tasks related to the same data domain.

[0013] Further, in the method, S3 includes:

[0014] Determine the maximum number of processing threads m according to the hardware resources allocated by the multi-domain scheduling fusion zone, where m is a natural number greater than 0;

[0015] According to the preset priorities, the tasks in the task set related to the same data domain are sorted, and the m tasks with higher priorities are selected in turn;

[0016] Call the execution engine corresponding to the m tasks to process the data.

[0017] Furthermore, in the method, after S2 and before S3, the following steps are further included:

[0018] Verify the tasks in the task set related to the same data domain; if the verification passes, set the task status of the corresponding task to available; otherwise, set the task status of the corresponding task to unavailable;

[0019] In S3, the tasks participating in task scheduling are tasks with available task status in the task set related to the same data domain.

[0020] Furthermore, in the method, the step of verifying the task includes:

[0021] Call the execution engine corresponding to the task. If the result returned by the execution engine is 0, retry and increase the number of retries by 1. If the return result is still 0 when the number of retries reaches the predetermined threshold, set the task status to unavailable. If the result returned by the execution engine is 1, the verification is passed and the task status is set to available.

[0022] Furthermore, in this method, the format of the returned task result is dynamically defined by the corresponding execution engine.

[0023] Furthermore, in the method, a storage time limit for the fusion result is set according to a preset data classification.

[0024] Furthermore, the method also includes filtering the fusion results according to the task source and task classification, and distributing the filtered results to the task source party.

[0025] Furthermore, the method also includes: after the distribution is completed, the step of destroying the corresponding task.

[0026] On the other hand, a device for multi-domain data fusion is provided, including a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement the multi-domain data fusion method as described above.

[0027] In another aspect, a computer-readable storage medium is provided, wherein at least one program is stored in the storage medium, and the at least one program is executed by a processor to implement the multi-domain data fusion method as described above. .

[0028] The above technical solution has the following technical effects:

[0029] For the scenario of multi-domain massive data fusion application, the multi-domain data fusion technical solution of the embodiment of the present invention utilizes the pre-set multi-domain multi-domain scheduling fusion area to standardize the data scheduling tasks in various business scenarios through task normalization and task scheduling, forming a unified task pool, executing corresponding execution engines for different data domains, and effectively integrating and accurately distributing the task execution results, thereby forming a physically decentralized and logically unified cross-domain data fusion model, which effectively supports the big data application needs in various business scenarios in real time and improves the coverage of big data dividend sharing. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of a method flow of multi-domain data fusion according to an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram of the overall flow of a method for multi-domain data fusion according to another embodiment of the present invention;

[0032] Figure 3 It is a schematic diagram of the structure of a device for multi-domain data fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] To further illustrate the various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, a person of ordinary skill in the art should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0034] The embodiments of the present invention will now be further described in conjunction with the accompanying drawings and specific implementation methods.

[0035] Embodiment 1:

[0036] Figure 1 FIG. 1 is a flow chart of a method for multi-domain data fusion according to an embodiment of the present invention. Figure 1 The multi-domain data fusion method of this embodiment is used to perform a processing task using data from multiple domains, and includes the following steps:

[0037] S1, establishing a task rule base in a pre-set multi-domain scheduling fusion area, the task rule base includes: an identifier of a task to be processed, a data source related to the task to be processed, multiple data domains related to the data source, and an execution engine corresponding to each data domain; the task rule base can be used to standardize data processing or data scheduling tasks in various business scenarios;

[0038] S2, extract all the tasks selected in the task rule library, determine the corresponding data domain group and the corresponding execution engine group for each selected task, and obtain a task set related to the same data domain; in many business scenarios of big data, a task may correspond to multiple different data domains, and the above-mentioned data domain group may include the data domains involved in the corresponding data source, and these data domains may belong to different owners; using the multi-domain data fusion method of the embodiment of the present invention, a multi-domain scheduling fusion area in an intermediate position can be constructed between different data domains for different data domains, and the corresponding execution engine or processing engine is matched according to the situation of each data domain. The above-mentioned execution engine group includes execution engines matched or corresponding to different data domains related to a task; the execution engine can also be pre-set according to the type of task;

[0039] S3, schedule the tasks in the task set related to the same data domain according to the preset task priority, and give priority to calling the execution engine corresponding to the task with a higher priority; the same data domain may involve multiple tasks, and calling the execution engine according to the task priority when executing the task can realize task scheduling considering different business scenarios and improve the efficiency of data processing;

[0040] S4, for each selected task, the task results returned by each execution engine in the corresponding execution engine group from the corresponding data domain are stored in the multi-domain scheduling fusion area, and fused in the multi-domain scheduling fusion area to obtain a fusion result. The multi-domain scheduling fusion area can be used to conveniently complete the fusion of execution results from different domains for the same task.

[0041] Embodiment 2:

[0042] Figure 2FIG2 is a schematic diagram of the overall process of a method for multi-domain data fusion according to another embodiment of the present invention. As shown in FIG2 , in this embodiment of the present invention, the method for multi-domain data fusion can be performed in a preset multi-domain scheduling fusion area.

[0043] In the multi-domain data fusion method of this embodiment of the present invention, a task rule base is defined, and the rule base stores various attributes related to the tasks to be processed. Table 1 is an example of a feature table in the rule base. Preferably, the task rule base includes: an identifier of the task to be processed, a data source related to the task to be processed, multiple data domains related to the data source, and an execution engine corresponding to each data domain. As needed, the task library may contain other feature information related to the task, such as required time limit, associated features, associated ratio, etc., as shown in Table 1. The task rule base can be used to obtain information such as the characteristics of each task to be processed and the matching execution engine. The attribute LWLC represents a task type label. For example, 1 represents personnel information supplement, 2 represents mobile phone file creation, and 3 represents modeling. This label is only an exemplary description. The task content adapted to the task type label can be set according to the actual business scenario.

[0044] Table 2 is a task scheduling table, which can be used to obtain the scheduling information of each task to be processed. Tables 1 and 2 are both stored in the multi-domain scheduling fusion area. The attribute names in Tables 1 and 2 are only exemplary, and other names can be used as needed.

[0045]

[0046] Table 1

[0047]

[0048]

[0049] Table 2

[0050] Task normalization:

[0051] In this embodiment, the correspondence between the data domain and the execution engine or the correspondence between the task type and the execution engine is pre-established. Tasks are normalized, for example, the subject identifiers of the tasks are normalized, and then a task set for the same data domain is further formed, which will greatly improve the efficiency of scheduling task execution and reduce the utilization rate of computing resources.

[0052] Extract all tasks from the task rule library to form a task data set Sn to be processed, and create a new normalized task data set Tn. Traverse Sn and analyze the execution engine group list (clyc) corresponding to the data source Sn.CJLY involved in the task. Split Sn according to the execution engine and store it in Tn, such as Tn = {Sn, list (clyc)}. For example: There is a data processing task 1. Task 1 requires calling the "hotel accommodation record" resource, and this resource is stored in domain 1 and domain 3, then Tn = {task 1, domain 1 engine}, {task 1, domain 3 engine}.

[0053] Next, traverse Tn, aggregate the records according to the data domain, merge the tasks in the same domain, and form the final task set related to the same data domain, namely, the execution task list Tn{data domain, List(task type label, task identifier, execution engine)}.

[0054] Traditional task scheduling often only considers the execution of the task itself, and cannot meet the data fusion requirements in different business scenarios. The task scheduling in the embodiment of the present invention comprehensively considers task scheduling, authority control, and data fusion, which can meet the management of the entire life cycle from task verification to scheduling and finally destruction, and meet the requirements of on-demand data application and data security in specific business scenarios, making the construction of a multi-domain scheduling fusion zone a better model for the integration of different business scenarios.

[0055] Task Verification:

[0056] In further implementation, after establishing the task rule base and before performing actual task scheduling, the task can be verified first. Specifically, traverse Tn, take the first record of the task identifier array in List as a parameter, call the execution engine in List of [Tn]. data domain, if the return result of the execution engine is 0, set the task status to unavailable, retry, and the number of retries is increased by 1, that is, Tn.zt = 0 and Tn.ZXCC = Tn.ZXCC + 1. If the return result of the execution engine is 1, the status is available, that is, Tn.zt = 1. When the number of retries exceeds 5 and the status is still 0, no retries are made, and the status of the task remains unavailable. Finally, the record with Tn status of 1 is saved as Pn, that is, the task set with available status. Through task verification, the abnormal proportion of the production environment can be greatly reduced. For example, the big data platform pushes the built dynamic model to the fusion area for model testing and verification. After the verification operation is successful, the model is deployed to the specified domain for operation analysis to ensure the stable operation of the production environment.

[0057] Task Scheduling:

[0058] For the task set of the same data domain, task scheduling is performed based on the pre-set priority. The maximum number of processing threads m is obtained according to the hardware resources allocated by the multi-domain scheduling fusion area, Pn is traversed, and the tasks are sorted in descending order according to their priorities. The first m records of Pn are taken in turn, and the task identifier array in the List is used as a parameter. The execution engine in the List of the [Tn] data domain is called to perform formal scheduling processing and return the analysis result Rn. Process m items each time until the execution engines corresponding to all tasks for the same data domain are traversed. Exemplarily, the priority setting can be set in combination with the type of task, the routing of the data, and the authority of the data user.

[0059] Scheduling result storage:

[0060] The method of the embodiment of the present invention supports the execution engine corresponding to the domain to dynamically define the attributes of the returned data, and can also support returning according to the security level of the pre-set domain data item, such as returning data according to pre-set permissions. The results of the same object or the same task returned by different domains can be stored in the multi-domain scheduling fusion area, and fused in the multi-domain scheduling fusion area, such as attribute merging, to obtain the final fusion result. In this way, it can adapt to flexible and changeable business scenario data fusion. For example, for the retrieval task of a mobile phone, a mobile phone file can be established in the multi-domain scheduling fusion area. The execution engines of different domains can characterize the dimensions of the mobile phone according to their respective data characteristics, and provide data separately, which can eventually be merged into a complete file in the multi-domain scheduling fusion area to complete the retrieval task of the mobile phone.

[0061] For this example, the data of each field is as follows:

[0062] Domain 1: mobile phone identification, mobile phone application information, APP package name, APP application software name, APP version number, APP installation time, operating system type, application information;

[0063] Domain 2: mobile phone identification, address book information: name (nickname) of the address book friend, friend's mobile phone number, mobile phone location, friend's notes, group name, data source, person label, number of calls, call duration, and last call time;

[0064] Domain 3: Mobile phone identification, mobile phone associated address information: account type, account number, mobile phone number, name, ID number, authentication account, contact address, data source, data source.

[0065] In further implementation, the retention period of the returned results can also be determined according to the pre-set business classification principle. For example, for personnel files, data on low-risk groups such as the elderly and children can be kept for a shorter period of time, while high-risk data such as key control personnel can be kept for a longer period of time.

[0066] Scheduling result distribution:

[0067] In further implementation, after the task is completed, the fusion result Rn is filtered according to the task source and task classification, and the data items that are not allowed to be viewed in advance are filtered out, and the filtered Rn is distributed to the task source, and log records and audits are done to ensure that the data is used reasonably, compliantly, safely and reliably. For example, the data items that are not allowed to be viewed can be set according to permissions and security.

[0068] Task destruction:

[0069] In a further implementation, after the result distribution is completed, the task status is set to unavailable, that is, Tn. Zt=0, and the task scheduling for the task record is no longer executed.

[0070] The technical solution of the embodiment of the present invention standardizes the data tasks in various business scenarios by constructing a multi-domain fusion scheduling area, establishing a task rule library, performing task normalization and task scheduling, and forming a unified task pool; executing corresponding execution engines for different data domains, and effectively integrating and accurately distributing the execution results of each execution engine, can form a physically decentralized and logically unified cross-domain data fusion model, meet the big data fusion needs in various business scenarios, solve the problems of massive data fusion that have plagued the big data era for a long time, and can effectively support the big data application needs in various business scenarios in real time and improve the coverage of big data dividend sharing.

[0071] Embodiment three:

[0072] The present invention also provides a device for multi-domain data fusion, such as Figure 3 As shown, the device includes a processor 301, a memory 302, a bus 303, and a computer program stored in the memory 302 and executable on the processor 301. The processor 301 includes one or more processing cores. The memory 302 is connected to the processor 301 via the bus 303. The memory 302 is used to store program instructions. When the processor executes the computer program, the steps in the above method embodiment of the first embodiment of the present invention are implemented.

[0073] Furthermore, as an executable solution, the device for identifying microplastics can be a computer unit, which can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The computer unit may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that the composition structure of the above-mentioned computer unit is only an example of a computer unit and does not constitute a limitation on the computer unit. It may include more or less components than the above, or a combination of certain components, or different components. For example, the computer unit may also include input and output devices, network access devices, buses, etc., which are not limited in the embodiments of the present invention.

[0074] Further, as an executable solution, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit, and uses various interfaces and lines to connect various parts of the entire computer unit.

[0075] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the computer unit by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0076] Embodiment 4:

[0077] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the above method in the embodiment of the present invention are implemented.

[0078] If the module / unit integrated in the computer unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Onny Memory), random access memory (RAM, Random Access Memory) and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0079] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. A method for multi-domain data fusion, for performing a processing task using data from multiple domains, characterized in that: include: S1, establishing a task rule base in a pre-set multi-domain scheduling fusion area, wherein the task rule base includes: an identifier of a task to be processed, a data source related to the task to be processed, a plurality of data domains related to each data source, and an execution engine corresponding to each data domain; S2, extracting all selected tasks from the task rule library, determining a corresponding data domain group and a corresponding execution engine group for each selected task, and obtaining a task set related to the same data domain; S3, scheduling tasks in the task set related to the same data domain according to the preset task priority, and preferentially calling the execution engine corresponding to the task with a higher priority; S4, for each selected task, storing the task results returned by each execution engine in the corresponding execution engine group from the corresponding data domain into the multi-domain scheduling fusion area, and fusing them in the multi-domain scheduling fusion area to obtain a fusion result; Wherein, the step S2 comprises: Extract all the selected tasks in the task rule library to form a task data set Sn to be processed, and create a new normalized task data set Tn; Traverse the Sn, obtain the data domain group corresponding to the data source related to each task and the corresponding execution engine group list (clyc), and split the Sn according to the execution engine group list (clyc) and store it in Tn, where Tn = {Sn, list (clyc)}; Traversing the Tn, aggregating the records in the Tn according to the execution engine, merging the tasks in the same data domain, and obtaining a set of tasks related to the same data domain; Wherein, after S2 and before S3, it also includes: Verifying the tasks in the task set related to the same data domain; if the verification passes, setting the task status of the corresponding task to available; otherwise, setting the task status of the corresponding task to unavailable; In S3, the tasks participating in task scheduling are tasks with available task status in the task set related to the same data domain.

2. The method according to claim 1, characterized in that The S3 includes: Determine a maximum number of processing threads m according to the hardware resources allocated by the multi-domain scheduling fusion area, where m is a natural number greater than 0; According to the preset priorities, the tasks in the task set related to the same data domain are sorted, and m tasks with higher priorities are selected in turn; The execution engine corresponding to the m tasks is called to perform data processing.

3. The method according to claim 1, characterized in that The steps to verify the task include: Call the execution engine corresponding to the task to be verified. If the execution engine returns a result of 0, retry and increase the number of retries by 1. If the return result is still 0 when the number of retries reaches the predetermined threshold, set the task status to unavailable. If the execution engine returns a result of 1, the verification is successful and the task status is set to available.

4. The method according to claim 1, characterized in that The format of the returned task results is dynamically defined by the corresponding execution engine.

5. The method according to claim 1, characterized in that The preservation time limit of the fusion result is set according to the preset data classification.

6. The method according to claim 1, characterized in that It also includes filtering the fusion results according to the task source and task classification, and distributing the filtered results to the task source party.

7. The method according to claim 1, characterized in that Also includes: After the distribution is completed, the corresponding task will be destroyed.

8. A device for multi-domain data fusion, characterized in that: The method comprises a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement the method for multi-domain data fusion as claimed in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: At least one program is stored in the storage medium, and the at least one program is executed by the processor to implement the method for multi-domain data fusion as described in any one of claims 1 to 7.

Images