Personnel theme view construction method, device and equipment

Abstract

The invention relates to a personnel theme view construction method, device and equipment, and the method comprises the steps: collecting original data from different data sources, and generating a data set corresponding to each person; the original data comprises original data items and data values of the original data items; for each data set, executing the following operations: matching each original data item in the data set with each reference data item, and determining a corresponding relationship between a target data item and the reference data item, the target data item being the original data item matched with the pre-configured reference data item; for each target data item, based on a target merging mode corresponding to the target data item and a fusion strategy corresponding to each merging mode, performing data fusion on each data value corresponding to the target data item to obtain a target data value; and on the basis of the target data values corresponding to the target data items, the personnel theme view corresponding to the data set is generated, and the method improves the accuracy of data construction in the theme view.

Description

Technical Field

[0001] The present application relates to the field of data fusion technology, and in particular to a method, device and equipment for constructing a personnel theme view. Background Art

[0002] With the rapid development of information technology, various business application systems, Internet platforms and social media in the era of big data generate massive amounts of data every day. Since these data come from various sources, formats and dimensions, in order to achieve more effective data utilization, it is necessary to integrate and analyze these data through data fusion technology. Data fusion technology can integrate data from different data sources to form a unified personnel theme view, which is convenient for subsequent data analysis and mining.

[0003] In the related art, when constructing a personnel subject view, although the data from different data sources can be aggregated into one subject view, the data fusion method used is single, and when performing data fusion, only different types of data corresponding to the same person are associated. In scenarios where data in different data sources usually exist in different forms, this method easily leads to data redundancy and inaccurate data in the aggregated subject view. Summary of the invention

[0004] The present invention provides a method, device and equipment for constructing a personnel subject view, which are used to solve the problems of information redundancy and inaccuracy in the construction of personnel subject views in related technologies.

[0005] In a first aspect, an embodiment of the present application provides a method for constructing a personnel theme view, the method comprising:

[0006] Collect various raw data from different data sources, and select the raw data corresponding to each person, thereby generating data sets corresponding to each person; the raw data includes raw data items and data values ​​of the raw data items;

[0007] For each data set, perform the following operations:

[0008] Matching each original data item in the data set with each reference data item respectively, and determining the corresponding relationship between the target data item and the reference data item, wherein the target data item is: an original data item that matches the pre-configured reference data item;

[0009] For each target data item, based on the target merging mode of the reference data item corresponding to the target data item and the fusion strategies corresponding to each merging mode, data fusion is performed on each data value corresponding to the target data item to obtain a target data value corresponding to the target data item;

[0010] Based on the target data values ​​corresponding to each target data item, a personnel subject view corresponding to the data set is generated.

[0011] The above method determines the target data item by matching the original data item with the reference data item, which can avoid the influence of invalid data in the original data on the subsequent construction of the subject view. Different merging methods are set for different data items, and fusion strategies corresponding to each merging method are set. It can meet the requirements of various types of data merging and improve the accuracy of data merging. By performing data fusion on the data values ​​corresponding to the target data item, the target data value corresponding to the target data item is determined. By comprehensively considering the information of different data sources, the accuracy of the target data item can be guaranteed, and the limitations of a single data source can be compensated, thereby obtaining a more comprehensive and accurate personnel subject view.

[0012] In a possible implementation, each original data item in the data set is matched with each reference data item to determine the corresponding relationship between the target data item and the reference data item, including:

[0013] Each reference data item and each original data item are segmented respectively to obtain a first word segmentation set corresponding to each reference data item and a second word segmentation set corresponding to each original data item; for any reference data item, the semantic similarity between the first word segmentation set corresponding to the reference data item and each second word segmentation set is calculated respectively, and the original data item corresponding to the second word segmentation set whose semantic similarity meets a preset condition is determined as the target data item corresponding to the reference data item.

[0014] The above method determines the correspondence between the reference data item and the target data item based on the semantic similarity between the first segmentation set corresponding to the reference data item and the second segmentation set corresponding to the original data item by segmenting each data item. This method can determine the objectivity and accuracy of the result.

[0015] In a possible implementation manner, before performing data fusion on each data value corresponding to the target data item, the method further includes:

[0016] Based on the preset data extraction rules corresponding to each reference data item and the corresponding relationship between the target data item and the reference data item, determine the target extraction rule corresponding to the target data item; and select the data value that meets the target extraction rule from the data values ​​corresponding to the target data item;

[0017] Performing data fusion on each data value corresponding to the target data item includes: performing data fusion on the data values ​​corresponding to the filtered target data item.

[0018] The above method sets the corresponding data extraction rules for each reference data item, and screens the data values ​​that meet the target extraction rules when determining the data value corresponding to the target data item, which can avoid the influence of invalid data values ​​on subsequent data fusion, thereby improving the accuracy of data fusion.

[0019] In a possible implementation, when the target merging mode is the first merging mode, based on the target merging mode corresponding to the reference data item matched by the target data item and the fusion strategies corresponding to the merging modes, data fusion is performed on each data value corresponding to the target data item to obtain a target data value corresponding to the target data item, including:

[0020] For any data value corresponding to the target data item, obtain the indicator values ​​corresponding to the data value and the research and judgment indicators of different dimensions; the different dimensions include: at least one of the data source dimension, the time dimension and the statistical dimension; for any research and judgment indicator, based on the indicator values ​​corresponding to the data values ​​of the target data item and the research and judgment indicators, calculate the weights of the research and judgment indicators corresponding to each data value; for any data value, based on the indicator values ​​and weights of each research and judgment indicator corresponding to the data value, determine the confidence corresponding to the data value; determine the data value with the highest confidence among the data values ​​corresponding to the target data value as the target data value corresponding to the target data item.

[0021] In the above method, different dimensions of analysis and judgment indicators are set in the embodiment of the present application. For each data value, the corresponding confidence level is determined by the indicator value corresponding to the data value and each analysis and judgment indicator and the weight of each analysis and judgment indicator. The target data value is then selected based on the confidence level. This method can comprehensively consider multi-dimensional information to select the target data value, thereby improving the accuracy of data fusion.

[0022] In a possible implementation, when the analysis and judgment indicator is a data source dimension, the weights of the analysis and judgment indicators corresponding to each data value are calculated based on the indicator values ​​corresponding to each data value of the target data item and the analysis and judgment indicator, including: based on the indicator values ​​corresponding to each data value and the analysis and judgment indicator, the weights of the analysis and judgment indicators corresponding to each data value are calculated using a fuzzy multi-attribute decision method.

[0023] The above method calculates the weights corresponding to each analysis and judgment indicator through the fuzzy attribute decision method for the data source dimension, comprehensively considers the importance of the decision maker's objective comments and subjective preferences in solving practical problems, and ensures the accuracy of the weights of the determined analysis and judgment indicators.

[0024] In one possible implementation, when the analysis and judgment indicator is a time dimension, the weights of the analysis and judgment indicators corresponding to each data value are calculated based on the index values ​​corresponding to each data value of the target data item and the index values, including: for any data value, according to the index value corresponding to any data value, determining the weight corresponding to the analysis and judgment indicator based on an exponential decay function.

[0025] The above method uses an exponential decay function to calculate the corresponding weights for the analysis and judgment indicators in the time dimension, which can ensure that the longer the collection time corresponding to the data value, the smaller the determined weight, thereby ensuring the accuracy of the weight determination.

[0026] In a possible implementation, when the analysis and judgment indicator is a statistical dimension, based on each data value of the target data item and the indicator value corresponding to the analysis and judgment indicator, the weight of the analysis and judgment indicator corresponding to each data value is calculated, including:

[0027] For each data value corresponding to the target data item, the data value is adjusted based on the preset adjustment amount to obtain the adjusted data value; for each adjusted data value, based on the base number corresponding to the analysis and judgment index value, the logarithm of the adjusted data value is determined, and the logarithm is used as the weight of the analysis and judgment index corresponding to the data value.

[0028] The above method uses logarithms to calculate the corresponding weights of the analysis and judgment indicators of the statistical dimension, thereby ensuring the accuracy of the weight determination.

[0029] In a possible implementation, when the target merging mode is the second merging mode, based on the target merging mode of the reference data item corresponding to the target data item and the fusion strategies corresponding to the merging modes, data fusion is performed on the data values ​​corresponding to the target data item to obtain the target data value corresponding to the target data item, including:

[0030] Performing a deduplication operation on each data value corresponding to the target data value;

[0031] Each data value obtained after deduplication is determined as a target data value corresponding to the target data item.

[0032] The above method, for the second merging method, can meet the business requirements corresponding to the reference data items and avoid data redundancy by removing duplicate data values ​​as target data values.

[0033] In a second aspect, an embodiment of the present application provides a device for constructing a personnel theme view, the device comprising:

[0034] The data set generation module is used to collect various raw data from different data sources, and filter the raw data corresponding to each person, and generate data sets corresponding to each person; the raw data includes raw data items and data values ​​of the raw data items;

[0035] The view generation module is used to perform the following operations for each data set:

[0036] Matching each original data item in the data set with each reference data item respectively, and determining the corresponding relationship between the target data item and the reference data item, wherein the target data item is: an original data item that matches the pre-configured reference data item;

[0037] For each target data item, based on the target merging mode of the reference data item corresponding to the target data item and the fusion strategies corresponding to each merging mode, data fusion is performed on each data value corresponding to the target data item to obtain a target data value corresponding to the target data item;

[0038] Based on the target data values ​​corresponding to each target data item, a personnel subject view corresponding to the data set is generated.

[0039] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the method for constructing a personnel subject view when executing the computer program.

[0040] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps in the above-mentioned method for constructing a personnel subject view of the present application are implemented.

[0041] In the fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which is stored in a computer-readable storage medium; when the processor of a memory access device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the memory access device executes the steps in the above-mentioned personnel subject view construction method of the present application.

[0042] For each of the above-mentioned second to fifth aspects and the technical effects that may be achieved by each of the above-mentioned aspects, please refer to the above-mentioned description of the technical effects that can be achieved by the first aspect or various possible schemes in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 A schematic diagram of a process of constructing a theme view provided in an embodiment of the present application;

[0045] Figure 2 A flowchart of a method for constructing a personnel theme view provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of a data item corresponding relationship determination process provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of an example of a personnel theme view construction process provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of a device for constructing a personnel theme view provided in an embodiment of the present application;

[0049] Figure 6 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0051] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0052] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0053] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0054] In order to facilitate those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the following is a brief description of the professional terms involved:

[0055] (1) FMADM (Fuzzy Multi-Attribute Decision Making): is a set of theories, methods and procedures used to select and determine alternative solutions when dealing with fuzzy decision problems. This method is suitable for decision-making scenarios where attribute values ​​are difficult to quantify accurately or are uncertain.

[0056] (2) Multi-source data fusion technology: refers to the technology of integrating all the information obtained through investigation and analysis by relevant means, and making a unified evaluation of this information to finally obtain unified information. The main purpose of this technology is to integrate various different data information, absorb the characteristics of different data sources, and thus extract better and richer information than single data.

[0057] (3) Topic view: refers to the extraction, transformation, and loading (ETL) of relevant data from multiple data sources around a specific topic or business area to form a comprehensive and in-depth understanding of the topic.

[0058] (4) Data access: refers to the process of extracting data from different data sources (such as databases, files, application programming interfaces, etc.) and importing it into the target system or tool for further processing and analysis. Its purpose is to achieve data integration, sharing and utilization, and provide a basis for subsequent data analysis, decision support, etc.

[0059] (5) Cloud storage service: This is a new concept that is extended and developed from the concept of cloud computing. It refers to a system that uses cluster applications, network technology, or distributed file systems to bring together a large number of different types of storage devices in the network through application software to work together and provide external data storage and business access functions.

[0060] (6) Hudi Data Lake: An open source framework for handling large-scale data storage and management. It is built on Hadoop / HDFS (Hadoop Distributed File System) and provides efficient data processing capabilities for scenarios such as data lakes, data warehouses, and streaming ETL.

[0061] With the rapid development of information technology, various business application systems, Internet platforms and social media in the era of big data generate massive amounts of data every day. These data come from various sources, formats and dimensions. In order to make better use of these data, they need to be integrated and analyzed through data fusion technology. Data fusion technology can integrate these data from different data sources to form a unified personnel theme view for subsequent data analysis and mining.

[0062] In the related art, when constructing a personnel subject view, there are the following problems:

[0063] (1) The data fusion algorithm is single and there is a problem of data inconsistency.

[0064] Since data may come from multiple systems, the same information may exist in different forms under different data sources. Data fusion technology can integrate data from multiple different data sources into a unified data set for better analysis and application. However, the current multi-source data fusion algorithm is single, which only associates and maps different types of data without considering the form, credibility and timeliness of data from different sources, resulting in data redundancy, inconsistency, inaccuracy and poor timeliness.

[0065] (2) The data fusion parameter dimensions are relatively small, resulting in poor data credibility.

[0066] In data fusion operations, only the time attribute parameter dimension is used to judge the validity of the data, and the statistical attributes and data source attributes of the data are not considered, resulting in poor data fusion accuracy, low data value density, and confusing and erroneous data in actual data applications.

[0067] (3) The data source is single and the data attributes are few, resulting in incomplete subject view information.

[0068] Although the technology for constructing thematic views is relatively mature, the data source for constructing thematic views is relatively single and has fewer data attributes. It is unable to obtain the static and dynamic attributes of entities and cannot fully reflect the holographic archival information of a subject. Therefore, the constructed thematic views have information omissions and one-sidedness and cannot support more advanced and complex data applications and algorithm models.

[0069] To overcome the above problems, the embodiment of the present application provides a method for constructing a personnel theme view. The method realizes the accurate construction of the theme view based on the technical method of multi-source data fusion of fuzzy multi-attribute decision-making method, and reflects the holographic archive of a theme more comprehensively and accurately through the construction of static attributes and dynamic attributes. This information integration method can improve data integrity, eliminate differences, and enhance data consistency. The fused holographic data can support more advanced data applications such as data mining and intelligent decision-making.

[0070] Figure 1 A schematic diagram of a process of constructing a theme view provided in an embodiment of the present application is shown below in combination with Figure 1 , the construction process of the personnel theme view in the embodiment of the present application is explained in detail.

[0071] like Figure 1 As shown, the above-mentioned subject view construction process mainly includes several steps: data source, data access, data processing, view construction, data extraction and data fusion. The specific execution process corresponding to each step can be found in the relevant description below, which will not be repeated here.

[0072] It should be noted that the execution order of the view building steps can be set based on demand and only needs to be executed before the data extraction step.

[0073] The present application is further described in detail below in conjunction with the accompanying drawings. Figure 2 As shown, it is a flow chart of a method for constructing a personnel theme view provided in an embodiment of the present application. The specific implementation process of the method is as follows:

[0074] Step S201, collecting various original data from different data sources, and filtering the original data corresponding to each person, thereby generating a data set corresponding to each person;

[0075] The original data includes the original data item and the data value of the original data item;

[0076] In some embodiments, the raw data is personnel-related data; the different data sources include at least one of first source data, second source data, industry data, social data, and Internet data, such as Figure 1As shown; industry data mainly include: user registration information and payment information related to water, electricity, gas and heating; social data mainly include: basic population information, education information, personal employment information, medical and health data, cultural data, etc.; Internet data mainly include: basic personal information, social network activities, online behavior records, consumption habits, interest preferences and other information.

[0077] It should be noted that there are overlapping parts and complementary parts between the raw data collected from different data sources. When data fusion is performed based on the collected data, weights can be set for various data sources to obtain accurate attribute information, and comprehensive information can be obtained through data association.

[0078] In some embodiments, the above-mentioned original data is stored in the format of original data items and their corresponding data values. The original data items are used to identify the attribute fields corresponding to the corresponding data values. For example, when the original data is a personnel identity identifier: XXXX (where X refers to a number), the personnel identity identifier is the original data item corresponding to the original field, and XXXX is the data value corresponding to the original data item.

[0079] In some embodiments, when accessing data, since various types of original data are distributed in different departments and come from different business application systems, different data access methods can be used. Specifically, Figure 1 As shown, there are two main data access methods, namely, data synchronous access through real-time data integration and data synchronous access through offline data integration.

[0080] In some embodiments, after completing the multi-source structured data access, the accessed raw data can be stored in a data storage area, such as a hudi data lake, to facilitate subsequent data processing and extraction.

[0081] It is possible that after collecting the raw data from different data sources, before generating the data sets corresponding to each person, the raw data need to be preprocessed (i.e. Figure 1 data processing shown); in some embodiments, the preprocessing operation may include: at least one of data deduplication, data filtering, data standardization, data format conversion, and data cleaning operations, and when the preprocessing operations include multiple operations, the execution order corresponding to each operation is not limited in this application and can be set based on demand.

[0082] Among them, data deduplication refers to: identifying and removing duplicate data in each original data; in some embodiments, key business fields can be extracted from each original data (extraction rules can be set based on needs), a primary key set can be generated based on the extracted key business fields, and the hash value of the primary key set of each original data is calculated separately (such as MD5 hash function value), and duplicate data records are removed by identifying the hash value corresponding to each original data, thereby reducing redundant storage of data and improving data quality and availability.

[0083] Data filtering means: for meaningless identifiers or attribute information in the original data, such as dirty data such as work unit none, unknown, 000, 111, use data filtering technology with specific rules (the rules can be set based on needs) to clean the dirty data, extract high-value information from the original data, and remove irrelevant or unnecessary data.

[0084] Data standardization means: to address the problems of inconsistent data standards and data specifications for raw data from different sources, data definition is standardized, that is, the naming rules, data types, lengths, etc. of the data are uniformly defined to ensure data consistency.

[0085] Data format conversion refers to converting raw data from different sources into a unified format according to data standard content, including date and time standardization, data type conversion, unit unification, non-standard code conversion to standard code value, etc. For example, according to the national standard "Human Gender Code" (GB 2261-1980), male and female in the data are uniformly converted into code "1", and female and female are uniformly converted into code "2".

[0086] Data cleaning means using data processing strategies (which can be set based on needs) to standardize dirty and non-compliant data in the original data. For example, through the standardized mobile phone number strategy, data such as "+86 123-1234-8888" can be processed into a unified "12312348888".

[0087] After preprocessing the raw data, it is also necessary to perform data association on the preprocessed raw data, that is, to select the raw data corresponding to each person from the preprocessed raw data to generate a data set corresponding to each person.

[0088] In some embodiments, when performing data association, the original data of the same person can be associated through the primary key to obtain the data sets corresponding to each person. Specifically, the associated primary key corresponding to each original data is determined, and when the associated primary keys are consistent, it is determined that the original data belongs to the same person, and it is divided into the same data set; wherein, in the embodiment of the present application, there is no restriction on the selection of the associated primary key, for example, the unique identifier of the person can be used as the associated primary key. Through data association, relevant information scattered in different data sets can be integrated together to form a more complete and valuable data set. For example: use the unique identifier of the person as the associated primary key, supplement the person's educational information (part of the original data) to the basic information of the person (part of the original data), fill in missing values, and improve the integrity of the data.

[0089] Step S202: for each data set, perform the following operations:

[0090] Step S2021, matching each original data item in the data set with each reference data item respectively, and determining the corresponding relationship between the target data item and the reference data item;

[0091] Wherein, the target data item is: an original data item that matches a pre-configured reference data item;

[0092] In some embodiments, Figure 3 As shown, the process of matching each original data item in the data set with each reference data item to determine the corresponding relationship between the target data item and the reference data item specifically includes:

[0093] Step S301, segmenting each reference data item and each original data item respectively to obtain a first segmentation set corresponding to each reference data item and a second segmentation set corresponding to each original data item;

[0094] Optionally, the above-mentioned original data items are composed of qualifiers and data elements, wherein the qualifier refers to the person subject corresponding to the original data item, for example: oneself, father, mother and spouse, etc.; the data element refers to the corresponding attribute parameters under the subject, such as citizen identification number, name, date of birth, etc.

[0095] In some embodiments, the above-mentioned reference data items refer to the data elements in the reference data items, that is, the part excluding the qualifiers. The above-mentioned splitting of the reference data items refers to: splitting the data elements in the reference data items. The specific splitting method is not limited in this application. Optionally, the data elements can be split into multiple phrases. It should be noted that the splitting method corresponding to the reference data items is the same as that of the original data items.

[0096] In some embodiments, before each reference data item and each original data item are segmented separately, each original data item needs to be identified, that is, multiple data groups are generated based on the data set through the qualifiers and data elements in each data item in the data set, where each group corresponds to a person subject, and each group includes at least one data element corresponding to the corresponding person subject, and then, for each data group, each data element therein is split to obtain a second word segmentation set corresponding to each data group.

[0097] For example, suppose a data set contains data items such as [citizen ID number, name, date of birth, father_citizen ID number, father_name, mother_citizen ID number, mother_name, spouse_citizen ID number, spouse_name, ...]; after semantic analysis of the data items, each data item can be split into a qualifier set of [self, father, mother, spouse, ...] and a data element set of [citizen ID number, name, date of birth]. The person subjects included in the qualifier set are: self, father, mother, spouse, etc., and the data grouping under each person subject includes: [citizen ID number, name, date of birth].

[0098] Step S302, for any reference data item, respectively calculate the semantic similarity between the first word segmentation set corresponding to the reference data item and each second word segmentation set, and determine the original data item corresponding to the second word segmentation set whose semantic similarity meets the preset condition as the target data item corresponding to the reference data item.

[0099] In the embodiments of the present application, the method for calculating the semantic similarity between the word segmentation sets is not limited. In some embodiments, the semantic similarity can be calculated using Jaccard Similarity, and the calculation formula is as follows:

[0100] Among them, A∩B is the intersection of sets A and B, and A∪B is the union of sets A and B.

[0101] Among them, A is the second word segmentation set corresponding to any original data item (specifically, the second word segmentation set corresponding to the data elements under any of the above original data items), and B is the first word segmentation set corresponding to any of the above reference data items.

[0102] In this application, Jaccard similarity is used to calculate the semantic similarity between data items. This method is not affected by the size of the vocabulary set and performs data mapping based on the same logic and standards, which can ensure the objectivity and accuracy of the results.

[0103] In some embodiments, the above-mentioned preset conditions include: a second set of word segmentations with the highest semantic similarity. To further ensure the accuracy of the match, the embodiments of the present application can be further set to include: a second set of word segmentations with the highest semantic similarity, and a semantic similarity value greater than a preset threshold, wherein the value of the preset threshold can be set based on demand.

[0104] For example: suppose the set of original data items in the data set is: [citizen ID number a1, name a2, gender a3, date of birth a4, nationality a5, father_citizen ID number a6, father_name a7, father_nationality a8, father_date of birth a9, …], and the data groups obtained after processing include: the data group corresponding to the person is [citizen ID number a1, name a2, date of birth a3, date of birth a4, nationality a5], and the data group corresponding to the father is [citizen ID number a6, name a7, date of birth a8, date of birth a9, …]; the set of reference data items is: [citizen ID number b1, name b2, gender b3, date of birth b4, nationality b5, …].

[0105] After the semantic similarity calculation, the semantic similarity between each data group and each reference data item can be obtained as shown in the following table:

[0106] Fields b1 b2 b3 b4 b5 …… a1 <![CDATA[J 本人 =1]]> 0 0 0 a2 0 <![CDATA[J 本人 =1]]> 0 0 0 a3 0 0 <![CDATA[J 本人 =1]]> 0 0 a4 0 0 0 <![CDATA[J 本人 =1]]> 0 a5 0 0 0 <![CDATA[J 本人 =1]]> Fields b1 b2 b3 b4 b5 …… a6 <![CDATA[J 父亲 =1]]> 0 0 0 0 a7 0 <![CDATA[J 父亲 =1]]> 0 0 0 a8 0 0 0 0 <![CDATA[J 父亲 =1]]> a9 0 0 0 <![CDATA[J 父亲 =1]]> 0 ……

[0107] Based on this, the target data items that can be finally obtained are: [citizen ID number, citizen ID number, name, gender, date of birth, ethnicity, …], [father_citizen ID number, father_citizen ID number, father_name, father_date of birth, father_ethnicity, …].

[0108] In some embodiments, after determining the correspondence between the target data item and the reference data item, the embodiment of the present application also needs to extract the data value corresponding to the target data item in the data set, specifically:

[0109] Based on the preset data extraction rules corresponding to each reference data item and the corresponding relationship between the target data item and the reference data item, determine the target extraction rule corresponding to the target data item;

[0110] From each data value corresponding to the target data item, filter the data value that meets the target extraction rule;

[0111] Optionally, in an embodiment of the present application, data extraction rules corresponding to each reference data item are pre-set, wherein an extraction rule identifier (ID) corresponding to each reference data item, a source data set (identifier) ​​corresponding to the data item, a target data set (identifier) ​​corresponding to the reference data item, extraction rules, personnel subjects, and data sources may be set, wherein the data extraction rules may be set based on demand, for example, the data extraction rules corresponding to the reference data item may be set as follows: the data value corresponding to the data item is in a non-empty state (such as NULL in the following table).

[0112] The following table is a possible data extraction rule table provided in an embodiment of the present application:

[0113]

[0114]

[0115] After determining the data extraction rules corresponding to each reference data item and the correspondence between the target data item and the reference data item, a mapping table corresponding to the data extraction rules can be generated. The mapping table can store the identifier of the source data source corresponding to the current data set (i.e. the source field name and the source field Chinese (name) in the table below), the identifier of the target data item (i.e. the extraction field name and the extraction field Chinese (name) in the table below), the identifier of the reference data item corresponding to the target data item (i.e. the target field name and the target field Chinese (name) in the table below), and the data extraction rule identifier corresponding to the reference data item (i.e. the extraction rule ID in the table below).

[0116] The following table is a schematic diagram of a mapping table of data extraction rules provided in this application:

[0117]

[0118] After obtaining the mapping table of the above data extraction rules, the data values ​​that meet the target extraction rules can be screened from the data values ​​corresponding to each target data item in the data set based on the information stored in the mapping table. In some embodiments, after obtaining the data values ​​corresponding to each target data item (i.e., the screened data values), they can be stored as HDFS files through the cloud storage service.

[0119] Step S2022, for each target data item, based on the target merging mode of the reference data item corresponding to the target data item and the fusion strategies corresponding to the merging modes, data fusion is performed on each data value corresponding to the target data item to obtain a target data value corresponding to the target data item;

[0120] In some embodiments, when the target merging mode is the first merging mode (including update merging and joint merging), based on the target merging mode corresponding to the reference data item matched by the target data item and the fusion strategies corresponding to each merging mode, data fusion is performed on each data value corresponding to the target data item to obtain a target data value corresponding to the target data item, including:

[0121] For any data value corresponding to the target data item, obtain the indicator values ​​corresponding to the data value and the research and judgment indicators of different dimensions; the different dimensions include: at least one of the data source dimension, the time dimension and the statistical dimension;

[0122] For any research and judgment indicator, based on each data value of the target data item and the indicator value corresponding to the research and judgment indicator, calculate the weight of the research and judgment indicator corresponding to each data value;

[0123] For any data value, the confidence level corresponding to the data value is determined based on the indicator values ​​and weights of each analysis indicator corresponding to the data value;

[0124] The data value with the highest confidence among the data values ​​corresponding to the target data value is determined as the target data value corresponding to the target data item.

[0125] In the related art, during the process of data fusion, when a data item has multiple data values ​​from multiple data sources, the most recently updated data value is generally directly selected. However, this approach does not take into account the reliability and effectiveness of each data source and cannot reflect the value of multi-source data fusion. In response to this, the embodiment of the present application proposes a method based on a multi-attribute decision-making method to determine the weight coefficients corresponding to each data value and the judgment index, and combines the expert experience method to set the weight value of each judgment index for data judgment, and uses the data value with the highest credibility as the target data value corresponding to the target data item to ensure the reliability and accuracy of data fusion.

[0126] In the embodiment of the present application, during the data fusion process corresponding to the target data item, a plurality of analysis and judgment indicators are pre-set as the standard for the subsequent calculation of the confidence corresponding to each data value. The plurality of analysis and judgment indicators correspond to different dimensions (i.e., the data source dimension, time dimension, and statistical dimension shown in the following table), and each dimension has a corresponding attribute type (i.e., qualitative and quantitative in the following table). The following table shows a possible analysis and judgment indicator:

[0127]

[0128] In the embodiments of the present application, the time dimension, statistical dimension, and data source dimension are introduced into the fusion attributes to ensure data integrity and timeliness; by establishing a comprehensive model for obtaining the optimization weights of scheme attributes, the model changes the orientation of the model by adjusting the model parameters, and can highlight the importance of objective optimization ranking or the subjective preferences of decision makers in solving practical problems according to actual conditions, so that the model has a strong generalization ability.

[0129] In some embodiments, when the target merging mode is the second merging mode (i.e., multi-value merging), based on the target merging mode of the reference data item corresponding to the target data item and the fusion strategies corresponding to each merging mode, data fusion is performed on each data value corresponding to the target data item to obtain a target data value corresponding to the target data item, including:

[0130] Performing a deduplication operation on each data value corresponding to the target data value;

[0131] Each data value obtained after deduplication is determined as a target data value corresponding to the target data item.

[0132] In some embodiments, multi-value merging means that when the extracted data values ​​come from multiple data sources, all data values ​​are retained during merging. Therefore, when the target merging mode is the second merging mode, in the embodiment of the present application, each data value is determined as a target data value corresponding to the target data item. For example, the target data item of a multi-value attribute may include: a former name, an alias / nickname, etc.

[0133] In the specific implementation, by modifying the hudi data merging mechanism, all source data values ​​are retained during data fusion to achieve data merging in the second merging method.

[0134] In the embodiment of the present application, the technology of Hudi data lake in cloud computing service is used to merge and fuse multi-source detailed data. During the fusion process, the merging mechanism of Hudi is modified, and methods such as multi-value merging, joint merging, and update merging are introduced to ensure the consistency, integrity, availability, and consistency of the data.

[0135] Step S2023: Generate a personnel subject view corresponding to the data set based on the target data value corresponding to each target data item.

[0136] In some embodiments, after the target data values ​​corresponding to the respective target data items are determined, a person subject view corresponding to the data set (corresponding person) may be generated based on a predefined person subject view template.

[0137] After generating the personnel theme view, the theme view is stored, that is, the integrated theme view data is stored in the hudi data lake for unified management and maintenance. Since a large amount of computing resources and storage resources are required in the process of integrating multiple types of data sources, this application implements the choice of using the hudi data lake in the cloud storage service for big data storage. On the one hand, cloud storage space can be expanded according to demand, and there is no need to worry about insufficient storage space; on the other hand, cloud storage service providers usually provide automatic backup and recovery functions, which can ensure the security of data and avoid losses caused by data loss or damage; it not only realizes the effective storage of large amounts of data but also ensures the security of data. Based on the integrated theme view, it can be used for data analysis and application, generate valuable information and support decision-making and business applications.

[0138] In the embodiment of the present application, different weight calculation methods are set for the evaluation indicators of different dimensions in the above step S2022, specifically:

[0139] In some embodiments, when the analysis indicator is a data source dimension, the weight of the analysis indicator corresponding to each data value based on each data value of the target data item and the indicator value corresponding to the analysis indicator is calculated, including:

[0140] Based on the index values ​​corresponding to each data value and the research and judgment index, the fuzzy multi-attribute decision-making method is used to calculate the weights of the research and judgment indexes corresponding to each data value.

[0141] In the specific implementation, we invited several data engineers, data analysts and business analysts to give qualitative descriptions of the indicator values ​​corresponding to each data and each data source dimension, and obtained the interval number matrix A = [a ij ] n×m , i=1,2,…,n,j=1,2,…,m;where a ij is the number of intervals corresponding to the indicator value of the jth analysis indicator of the ith data value, n is the number of data values ​​corresponding to the target data item, and m is the number of analysis indicators of the data source dimension.

[0142] For example, when the data value corresponding to the target data item is 5 (i.e., the evaluation schemes X1 to X5 in the table below), and the evaluation indicators of the data source dimension include data source type, data resources, data collection method, industry type, and collection area, the determination process of an example interval number matrix is ​​as follows:

[0143] We invited several data engineers, data analysts, and business analysts to use the 9-level quantitative method to give a qualitative description of the indicator values ​​of each research and judgment indicator corresponding to each data value. After summarizing, we obtained 5 sets of data fusion research and judgment evaluation schemes (i.e., the interval number matrix A = [a ij ] 5×5), the values ​​of each element are shown in the following table:

[0144] Evaluation plan Data source type Data resources Data collection method Industry Category Collection area <![CDATA[X1]]> Generally reliable Poor quality High systematization High confidence A lot of data <![CDATA[X2]]> Relatively reliable Good quality Low systematization Medium confidence Average data volume <![CDATA[X3]]> Generally reliable Average quality Highly systematic High confidence Large amount of data <![CDATA[X4]]> Relatively reliable Better quality Very low systematization Medium confidence Small amount of data <![CDATA[X5]]> Unreliable Poor quality Systematization in general High confidence Large amount of data

[0145] Then, the above interval number matrix is ​​normalized to obtain the normalized interval number decision matrix R = [r ij ] n×m , i=1,2,…,n,j=1,2,…,m;where, For r ij The corresponding lower limit of the interval is For r ij The corresponding upper limit of the interval.

[0146] Specifically, the above as well as The calculation method is as follows:

[0147] in, for a ij The corresponding upper limit of the interval is for a ij The corresponding lower limit of the interval, the values ​​of the upper limit and lower limit of the interval are both preset values, N is the data value, and M is the judgment indicator of the data source dimension.

[0148] Then, calculate the preference degree g of multiple decision makers for the i-th evaluation scheme i , preference degree g i It is equal to the product of the average value of the decision-maker's authority coefficient, the standardized value of the plan, and the weighted sum of the relative importance of the research and judgment indicators. The calculation formula is as follows:

[0149] b p ∈(0,1),p=1,2,…,d; where ω j is the importance of the jth evaluation indicator, which is regarded as the initial weight of the evaluation indicator. Its value can be set based on demand. p is the authority coefficient of the pth decision maker, and its value can be set based on demand. d is the number of decision-making group members.

[0150] For example, for the example shown in the table above, the interval numbers are normalized according to the above formula, and the normalized interval number decision matrix R = [r ij ] 5×5 , the values ​​of each element of the matrix are shown in the following table:

[0151]

[0152]

[0153] Assume that four decision makers are invited to participate in the evaluation decision, and the corresponding authorities of the four decision makers are 0.83, 0.74, 0.80, and 0.94 respectively. The initial weights of each evaluation index obtained by the AHP method are 0.25, 0.28, 0.15, 0.21, and 0.11 respectively. According to the above calculation method, the preference degree of each evaluation scheme corresponding to the above table is: [[0.293, 0.363], [0.359, 0.407], [0.375, 0.418], [0.334, 0.375], [0.246, 0.295]], and it can be seen from the preference degree that the expert group has the greatest preference for evaluation scheme X3, followed by scheme X2, and finally scheme X5.

[0154] In the embodiment of the present application, the quantitative value of the qualitative indicator is expressed in the form of an interval number, and the definition of the expected value of the interval number is as follows: The expected value of the jth judgment indicator is Where j = 1, 2,…, m.

[0155] For the jth evaluation indicator, the sum of squares of the deviations between the evaluation values ​​of each scheme and the expected value is:

[0156] Where i∈N,j∈M;

[0157] Then, the objective weight vector is calculated based on the above evaluation values ​​and expected values. The objective weight vector should be selected so that the total deviation between the evaluation values ​​of all solutions and the expected solution values ​​is minimized, that is, the following optimization model constraints are satisfied:

[0158]

[0159] At the same time, since it is also necessary to refer to the subjective preference information of each decision maker, and since there is usually a certain gap between the subjective preference and objective preference of the decision maker, in order to make the decision reasonable, the selection of the decision attribute weight ω should make the total deviation between the objective preference and the subjective preference of the decision as small as possible, that is, to establish an optimization model for obtaining the subjective attribute weight considering the decision maker's preference:

[0160]

[0161] Based on the above model, the linear weighted sum method in multi-objective analysis is used to construct a bi-objective decision-making problem to obtain the decision weight ω:

[0162] Among them, α is the weight coefficient corresponding to model F1, and β is the weight coefficient corresponding to model F2, and their values ​​can be set based on requirements.

[0163] After determining the optimal weight of the target attribute, the comprehensive evaluation value of each evaluation scheme can be calculated and sorted to obtain the best evaluation scheme. The comprehensive evaluation value of each evaluation scheme is:

[0164] For example, based on the above calculation method, the optimal attribute weight distribution and solution attribute comprehensive values ​​under different model parameters are shown in the following table:

[0165] Model Parameters α, β <![CDATA[Attribute optimization weight ω j > <![CDATA[Comprehensive value z of solution attributes i > F1 α＝1,β＝0 (0.41,0.13,0.14,0.21,0.11) (0.421,0.443,0.500,0.431,0.340) F1&F2 α＝0.5,β＝0.5 (0.37,0.15,0.12,0.22,0.14) (0.434,0.440,0.491,0.427,0.348) F1&F2 α＝0.2,β＝0.8 (0.36,0.16,0.13,0.20,0.15) (0.425,0.438,0.492,0.424,0.344) F2 α＝0,β＝1 (0.34,0.19,0.13,0.18,0.16) (0.422,0.444,0.506,0.425,0.335)

[0166] In the process of data fusion, the subjective experience of data analysts, data engineers and business analysts plays a relatively important role in the attribute weight evaluation model. The F1 and F2 models can be combined to obtain the comprehensive weight score from an objective perspective and adjust the attribute weight from the subjective perspective of expert experience. For example, the value of the F1 model parameter α is set to 0.2, and the value of the F2 model parameter β is set to 0.8. At this time, the attribute optimization weight distribution is (0.36, 0.16, 0.13, 0.20, 0.15).

[0167] In the process of data fusion, the embodiment of the present application introduces a fuzzy multi-attribute decision-making method to obtain a reliable comprehensive weight coefficient. This model takes into account the advantages of objective optimization sorting and decision makers' preferences for solutions to a great extent, so that the attribute weight information can more accurately reflect the actual situation and reduce the influence of the analyst's subjective and single weighting on data reliability.

[0168] In some embodiments, when the analysis indicator is a statistical dimension, the weight of the analysis indicator corresponding to each data value based on each data value of the target data item and the indicator value corresponding to the analysis indicator is calculated, including:

[0169] For each data value corresponding to the target data item, adjusting the data value based on a preset adjustment amount to obtain an adjusted data value;

[0170] For each adjusted data value, the logarithm of the adjusted data value is determined based on the base number corresponding to the research and judgment index value, and the logarithm is used as the weight of the research and judgment index corresponding to the data value.

[0171] In a specific implementation, when there are multiple analysis and judgment indicators in a statistical dimension, a corresponding base number and a preset adjustment amount are set for different analysis and judgment indicators in the embodiment of the present application, wherein the base numbers (or preset adjustment amounts) corresponding to different analysis and judgment indicators may be the same or different.

[0172] In some embodiments, after a large amount of data analysis practice, in the embodiment of the present application, the base number corresponding to the number of data sources is set to 4, the base number corresponding to the number of data resources is set to 6, and the base number corresponding to the number of data collection times is set to 11. The preset adjustment amount corresponding to each judgment indicator is 1. At this time, the weights corresponding to each judgment indicator are respectively:

[0173] The weight corresponding to the number of data sources is: log4(count+1), where count is the number of data sources corresponding to the data value (i.e., the indicator value). That is, more than 3 data sources reach the full value, and 2 data sources reach the median value. The base number can be adjusted according to the data situation.

[0174] The weight corresponding to the number of data resources is log6(count+1), where count is the number of data resources corresponding to the data value (i.e., the indicator value). That is, the full value is reached when there are more than 5 data set resources, and the median value is reached when there are 3 data resources. The base number can be adjusted according to the data situation.

[0175] The weight log corresponding to the number of data collection times 11 (count+1), where count is the number of data collection times (i.e., indicator value) corresponding to the data value. That is, the full value is reached when it appears 10 times, and the median value is reached when it appears 2 times. The base number can be adjusted according to the data situation.

[0176] In some embodiments, after obtaining the weights corresponding to the various evaluation indicators, each data value may be comprehensively measured according to the qualitative and quantitative dimensions, that is, the confidence level S corresponding to the data value = ∑ i∈N a i ω i +∑ j∈M b j ω j , where N is the qualitative evaluation index, M is the quantitative evaluation index, and a i is the index value of the i-th qualitative evaluation index, b j is the indicator value corresponding to each j quantitative analysis indicators, and ω is the weight corresponding to the analysis indicator.

[0177] For example, the following table is a schematic table of possible confidence levels corresponding to various data values:

[0178]

[0179] In some embodiments, when the judgment indicator is a time dimension, the weight of the judgment indicator corresponding to each data value based on each data value of the target data item and the indicator value corresponding to the judgment indicator is calculated, including:

[0180] For any data value, according to the indicator value corresponding to any data value, the weight corresponding to the analysis and judgment indicator is determined based on the exponential decay function.

[0181] In a specific implementation, for the judgment index of the time dimension, the following exponential decay function can be used in the embodiment of the present application to determine the weight Decay Factor (t) corresponding to the threshold:

[0182] Decay Factor(t)=e -λt ,

[0183] Among them, t is the time interval between the corresponding time of the data value (that is, the generation or update time corresponding to the data value) and the current time, λ is the decay rate, which is used to determine the decay speed and can be set based on demand. In some embodiments, the value of λ can be set to 2 through expert experience. The calculation formula for the value of t is: (current timestamp seconds - data timestamp seconds) / 10 years of seconds. If the value of the above formula is greater than 1, the value of t is 1. At this time, the weight of about 3.4 years can be reduced by half.

[0184] When constructing a subject view, this application considers the influencing factors of static and dynamic attributes, introduces an exponential decay function algorithm, and dynamically updates the subject view, which can ensure that the information in the archive is always kept up to date and guarantee the integrity and timeliness of the data.

[0185] After obtaining the confidence corresponding to each data value, the data value with the highest confidence among the data values ​​corresponding to the target data value is determined as the target data value corresponding to the target data item. In specific implementation, this step can be achieved by modifying the mechanism of hudi data merging.

[0186] It should be noted that the above description is a specific implementation method for the target merging method being update merging. If the target merging method is joint merging, its confidence determination process is of the same type. However, before obtaining the index values ​​corresponding to the data value and the analysis and judgment indicators of different dimensions for any data value corresponding to the target data item, it is necessary to treat the target data item and other target data items associated with it as a data item group. For example, longitude and latitude are treated as a data item group, administrative divisions and detailed addresses are treated as a data item group, file names and file storage paths are treated as a data item group, etc.; and the data values ​​corresponding to the target data items of the same data source in the group are concatenated in a preset order to obtain the data values ​​corresponding to the data item group, and the target data value corresponding to each data item group is determined in a manner similar to the above-mentioned update merging method.

[0187] In some embodiments, before executing step S202, it is necessary to pre-establish a personnel theme view template so as to perform subsequent data extraction and data fusion based on the personnel theme view template. Specifically, the establishment process of the personnel theme view template may be:

[0188] Extract high-quality (here, the high-quality judgment standard can be set based on demand, usually can be set to a more comprehensive type of corresponding data items, etc.) original data of personnel objects from various source data, including object unique identification attributes and other attribute information, mine and reorganize the collected original data, and store independent personnel objects, integrate the attribute information, association information, relationship information and behavior information between various personnel objects, and generate a theme view template for each personnel object. Specifically, when constructing a personnel theme view template, it is necessary to first determine the unique identification of the person. In some embodiments, a combination key of certificate type + certificate number is set in the embodiments of the present application as the unique identification of the person. Among them, certificate types generally include: citizen identity card number, passport, Mainland Travel Permit for Hong Kong and Macao Residents, Mainland Travel Permit for Taiwan Residents, military officer certificate, etc.

[0189] In some embodiments, the generated theme view template may include: attribute classification, attributes included in each attribute classification, and attribute types corresponding to each attribute, source data type, merging method, etc. It should be noted that the specific format and content in the theme view template are not restricted in the embodiments of the present application. The above is only an example, and it can be set based on needs in the specific implementation.

[0190] Specifically, attribute classification may include: basic attributes, household registration information, residence information, biometric information, health status information, education information, employment information, associated identity information, contact information, family status information, social relationship information, network behavior information, offline (i.e., offline) behavior information, case-related event information, personnel label information, etc.; attribute types usually include static attributes and dynamic attributes, which are helpful to understand and analyze personnel characteristics and behaviors; source data types mainly include: first source (data), second source (data), industry (data), society (data), Internet (data), etc., to obtain various original data of personnel from various data sources, and obtain personnel detailed data (i.e., target data value) content after data extraction and mapping; merging methods include: multi-value merging (first merging method), joint merging (second merging method) and update merging (second merging method), etc.

[0191] The following table shows a possible structure of a personnel theme view template provided in an embodiment of the present application:

[0192]

[0193]

[0194]

[0195]

[0196] Among them, the attributes in the above table are the reference data items described in step S201 of this application.

[0197] The following combination Figure 4 The above-mentioned personnel subject view construction process is described in detail with a specific example, and the process specifically includes:

[0198] Step S401, collecting original data from different data sources;

[0199] Step S402, for each raw data, perform data deduplication, data filtering, data standardization, data format conversion and data cleaning in sequence to obtain pre-processed raw data;

[0200] Step S403, for the pre-processed original data, based on the associated primary key corresponding to each original data, the original data are associated to obtain the data sets corresponding to each person;

[0201] It should be noted that step S404 and subsequent steps are all performed on one data set.

[0202] Step S404, for each data set, splitting each original data item in the data set to obtain a qualifier set and a data element set corresponding to each qualifier in the set;

[0203] Step S405, segmenting the reference data elements in each reference data item and each data element in the data element set to obtain a first segmentation set corresponding to the reference data element and a second segmentation set corresponding to each data element;

[0204] Step S406, respectively calculating the semantic similarity between the first word segmentation set corresponding to any reference data item and each second word segmentation set, and determining the original data item corresponding to the second word segmentation set with the highest semantic similarity as the target data item corresponding to the reference data item;

[0205] Step S407, based on the preset data extraction rules corresponding to the reference data items and the corresponding relationship between the target data items and the reference data items, obtaining the data value that meets the target extraction rules;

[0206] The target extraction rule is as follows: the target data item corresponds to the target extraction rule;

[0207] Step S408, when the target data item corresponds to the first merging mode, for each data value, based on the index value of each judgment index corresponding to the data value and the weight of each judgment index, the confidence corresponding to each data value is determined, and the target data value corresponding to the target data item is determined based on the confidence;

[0208] In some embodiments, the above step S408 specifically includes:

[0209] Step S4081 (not shown in the figure), for each data value corresponding to the target data item, obtain the indicator values ​​corresponding to the data value and the research and judgment indicators of different dimensions;

[0210] The different dimensions include: at least one of a data source dimension, a time dimension, and a statistical dimension;

[0211] In some embodiments, when the first merging method is an update merging method, each data value corresponding to the target data item is directly obtained; when the first merging method is a joint merging method, each data value corresponding to the target data item is obtained by taking the target data item and other target data items associated with it as a data item group, and concatenating the data values ​​corresponding to the target data items of the same data source in the group in a preset order to obtain the data value corresponding to the data item group.

[0212] Step S4082 (not shown in the figure), for any judgment indicator, based on each data value of the target data item and the indicator value corresponding to the judgment indicator, calculate the weight of the judgment indicator corresponding to each data value;

[0213] In some embodiments, for the analysis and judgment indicators of the data source dimension, the fuzzy multi-attribute decision method is used to calculate the weights of the analysis and judgment indicators corresponding to each data value; for the analysis and judgment indicators of the time dimension, the exponential decay function is used to calculate the weights corresponding to the analysis and judgment indicators; for the analysis and judgment indicators of the statistical dimension, the logarithmic formula is used to calculate the weights corresponding to the analysis and judgment indicators.

[0214] Step S4083 (not shown in the figure), for any data value, based on the index values ​​and weights of each analysis and judgment index corresponding to the data value, determine the confidence level corresponding to the data value;

[0215] Step S4084 (not shown in the figure), determining the data value with the highest confidence among the data values ​​corresponding to the target data value as the target data value corresponding to the target data item;

[0216] Step S409, when the target data item corresponds to the second merging mode, performing a deduplication operation on each data value corresponding to the target data value, and determining each data value obtained after deduplication as the target data value corresponding to the target data item;

[0217] Step S410: generating a personnel subject view corresponding to the data set based on the target data value corresponding to each target data item.

[0218] Based on the same inventive concept, the present application also provides a device for constructing a personnel theme view, see Figure 5 , the device comprises:

[0219] The data set generation module 501 is used to collect various raw data from different data sources, and filter the raw data corresponding to each person, and generate a data set corresponding to each person; the raw data includes raw data items and data values ​​of the raw data items;

[0220] The view generation module 502 includes a matching submodule 5021, a fusion submodule 5022, and a generation submodule 5023, and is used to perform the following operations for each data set:

[0221] Through the matching submodule 5021, each original data item in the data set is matched with each reference data item respectively to determine the corresponding relationship between the target data item and the reference data item, where the target data item is: the original data item that matches the pre-configured reference data item;

[0222] Through the fusion submodule 5022, for each target data item, based on the target merging mode of the reference data item corresponding to the target data item and the fusion strategy corresponding to each merging mode, data fusion is performed on each data value corresponding to the target data item to obtain a target data value corresponding to the target data item;

[0223] Through the generation submodule 5023, based on the target data values ​​corresponding to each target data item, a personnel subject view corresponding to the data set is generated.

[0224] In a possible implementation, the matching submodule 5021 is specifically used to: segment each reference data item and each original data item respectively to obtain a first word segmentation set corresponding to each reference data item and a second word segmentation set corresponding to each original data item; for any reference data item, calculate the semantic similarity between the first word segmentation set corresponding to the reference data item and each second word segmentation set respectively, and determine the original data item corresponding to the second word segmentation set whose semantic similarity meets a preset condition as the target data item corresponding to the reference data item.

[0225] In a possible implementation, the fusion submodule 5022 is specifically used to: determine the target extraction rule corresponding to the target data item based on the preset data extraction rules corresponding to each reference data item and the correspondence between the target data item and the reference data item; filter the data values ​​that meet the target extraction rule from the data values ​​corresponding to the target data item; and perform data fusion on the data values ​​corresponding to the filtered target data items.

[0226] In a possible implementation, when the target merging method is the first merging method, the fusion submodule 5022 is specifically used for: for any data value corresponding to the target data item, obtaining the indicator values ​​corresponding to the data value and the analysis and judgment indicators of different dimensions; the different dimensions include: at least one of the data source dimension, the time dimension and the statistical dimension; for any analysis and judgment indicator, based on the indicator values ​​corresponding to the data values ​​of the target data item and the analysis and judgment indicators, calculating the weights of the analysis and judgment indicators corresponding to the data values; for any data value, based on the indicator values ​​and weights of the analysis and judgment indicators corresponding to the data value, determining the confidence corresponding to the data value; determining the data value with the highest confidence among the data values ​​corresponding to the target data value as the target data value corresponding to the target data item.

[0227] In a possible implementation, when the above-mentioned analysis and judgment indicators are data source dimensions, the above-mentioned fusion submodule 5022 is specifically used to: based on the indicator values ​​corresponding to each data value and the analysis and judgment indicator, use the fuzzy multi-attribute decision method to calculate the weights of the analysis and judgment indicators corresponding to each data value.

[0228] In a possible implementation, when the above-mentioned analysis and judgment indicator is a time dimension, the above-mentioned fusion submodule 5022 is specifically used to: for any data value, according to the indicator value corresponding to any data value, determine the weight corresponding to the analysis and judgment indicator based on the exponential decay function.

[0229] In a possible implementation, when the above-mentioned analysis and judgment index is a statistical dimension, the above-mentioned fusion submodule 5022 is specifically used to: for each data value corresponding to the target data item, adjust the data value based on a preset adjustment amount to obtain an adjusted data value; for each adjusted data value, based on the base number corresponding to the analysis and judgment index value, determine the logarithm of the adjusted data value, and use the logarithm as the weight of the analysis and judgment index corresponding to the data value.

[0230] In a possible implementation, when the target merging method is the second merging method, the fusion submodule 5022 is specifically used to: perform deduplication operations on each data value corresponding to the target data value; and determine each data value obtained after deduplication as the target data value corresponding to the target data item.

[0231] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application, and the electronic device can realize the functions of the aforementioned personnel theme view construction device, referring to Figure 6 , the electronic device comprises:

[0232] At least one processor 601, and a memory 602 connected to the at least one processor 601. The specific connection medium between the processor 601 and the memory 602 is not limited in the embodiment of the present application. Figure 6In the example, the processor 601 and the memory 602 are connected via a bus 600. The bus 600 is Figure 6 The connection between other components is shown by bold lines, and is not intended to be limiting. The bus 600 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 601 can also be called a controller, and there is no limitation on the name.

[0233] In the embodiment of the present application, the memory 602 stores instructions that can be executed by at least one processor 601. The at least one processor 601 can execute the method for constructing a person-themed view discussed above by executing the instructions stored in the memory 602. The processor 601 can implement Figure 5 The functions of each module in the device shown.

[0234] Among them, the processor 601 is the control center of the device, and can use various interfaces and lines to connect the various parts of the entire control device. By running or executing instructions stored in the memory 602 and calling the data stored in the memory 602, the various functions of the device and processing data, the device can be monitored as a whole.

[0235] In one possible design, the processor 601 may include one or more processing units, and the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 601. In some embodiments, the processor 601 and the memory 602 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0236] Processor 601 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the personnel theme view construction method disclosed in the embodiments of the present application can be directly embodied as a hardware processor execution, or a combination of hardware and software modules in the processor.

[0237] The memory 602 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 602 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 602 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 602 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0238] By designing and programming the processor 601, the code corresponding to the application anomaly detection method described in the above embodiment can be fixed into the chip, so that the chip can execute the code when running. Figure 2 The steps of the method for constructing the personnel theme view of the embodiment shown are as follows: How to design and program the processor 601 is a technology well known to those skilled in the art and will not be described in detail here.

[0239] An embodiment of the present application also provides a computer-readable storage medium that stores computer-executable instructions required to execute the above-mentioned processor, which includes a program required to execute the above-mentioned processor.

[0240] In some possible implementations, various aspects of the personnel subject view construction method provided in the present application may also be implemented in the form of a program product, which includes a program code. When the above-mentioned program product is run on an electronic device, the above-mentioned program code is used to enable the above-mentioned electronic device to execute the steps of the personnel subject view construction method according to various exemplary implementations of the present application described above in this specification.

[0241] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0242] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (apparatus), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0243] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0244] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0245] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0246] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for constructing a personnel theme view, characterized in that: The method comprises: Collecting original data from different data sources, and filtering the original data corresponding to each person, to generate data sets corresponding to each person; the original data includes original data items and data values ​​of the original data items; For each data set, perform the following operations: Matching each original data item in the data set with each reference data item respectively, and determining a corresponding relationship between a target data item and a reference data item, wherein the target data item is: an original data item that matches a pre-configured reference data item; For each target data item, based on the target merging mode of the reference data item corresponding to the target data item and the fusion strategies corresponding to each merging mode, data fusion is performed on each data value corresponding to the target data item to obtain a target data value corresponding to the target data item; Based on the target data values ​​corresponding to the target data items, a personnel subject view corresponding to the data set is generated.

2. The method according to claim 1, characterized in that The matching of each original data item in the data set with each reference data item to determine the corresponding relationship between the target data item and the reference data item includes: Segmenting the reference data items and the original data items respectively to obtain first segmentation sets corresponding to the reference data items and second segmentation sets corresponding to the original data items; For any reference data item, the semantic similarity between the first word segmentation set corresponding to the reference data item and each second word segmentation set is calculated respectively, and the original data item corresponding to the second word segmentation set whose semantic similarity meets the preset conditions is determined as the target data item corresponding to the reference data item.

3. The method according to claim 1, characterized in that Before performing data fusion on each data value corresponding to the target data item, the method further includes: Based on the preset data extraction rules corresponding to each reference data item and the corresponding relationship between the target data item and the reference data item, determining a target extraction rule corresponding to the target data item; Filtering, from the data values ​​corresponding to the target data item, data values ​​that meet the target extraction rule; The performing data fusion on each data value corresponding to the target data item includes: Data fusion is performed on the data values ​​corresponding to the filtered target data items.

4. The method according to any one of claims 1 to 3, characterized in that: When the target merging mode is the first merging mode, the target merging mode corresponding to the reference data item matched by the target data item and the fusion strategies corresponding to the merging modes are used to fuse the data values ​​corresponding to the target data item to obtain the target data value corresponding to the target data item, including: For any data value corresponding to the target data item, obtain the indicator values ​​corresponding to the data value and the research and judgment indicators of different dimensions respectively; the different dimensions include: at least one of the data source dimension, the time dimension and the statistical dimension; For any research and judgment indicator, based on each data value of the target data item and the indicator value corresponding to the research and judgment indicator, calculate the weight of the research and judgment indicator corresponding to each data value; For any of the data values, based on the indicator values ​​and weights of the analysis indicators corresponding to the data value, determine the confidence level corresponding to the data value; The data value with the highest confidence among the data values ​​corresponding to the target data value is determined as the target data value corresponding to the target data item.

5. The method according to claim 4, characterized in that When the research and judgment indicator is a data source dimension, the weight of the research and judgment indicator corresponding to each data value is calculated based on each data value of the target data item and the indicator value corresponding to the research and judgment indicator, including: Based on the index values ​​corresponding to the data values ​​and the analysis and judgment indicators, the fuzzy multi-attribute decision method is used to calculate the weights of the analysis and judgment indicators corresponding to the data values.

6. The method according to claim 4, characterized in that When the research and judgment indicator is a time dimension, the weight of the research and judgment indicator corresponding to each data value is calculated based on each data value of the target data item and the indicator value corresponding to the research and judgment indicator, including: For any data value, according to the indicator value corresponding to the any data value, the weight corresponding to the analysis and judgment indicator is determined based on the exponential decay function.

7. The method according to claim 4, characterized in that When the research and judgment indicator is a statistical dimension, the weight of the research and judgment indicator corresponding to each data value is calculated based on each data value of the target data item and the indicator value corresponding to the research and judgment indicator, including: For each data value corresponding to the target data item, adjusting the data value based on a preset adjustment amount to obtain an adjusted data value; For each adjusted data value, the logarithmic value of the adjusted data value is determined based on the base number corresponding to the analysis and judgment index value, and the logarithmic value is used as the weight of the analysis and judgment index corresponding to the data value.

8. The method according to any one of claims 1 to 3, characterized in that: When the target merging mode is the second merging mode, the target merging mode based on the reference data item corresponding to the target data item and the fusion strategies corresponding to the merging modes respectively perform data fusion on the data values ​​corresponding to the target data item to obtain the target data value corresponding to the target data item, including: Performing a deduplication operation on each data value corresponding to the target data value; Each data value obtained after deduplication is determined as a target data value corresponding to the target data item.

9. A device for constructing a personnel theme view, characterized in that: The device comprises: A data set generation module is used to collect various raw data from different data sources, and filter the raw data corresponding to each person, and generate a data set corresponding to each person; the raw data includes raw data items and data values ​​of the raw data items; The view generation module is used to perform the following operations for each data set: Matching each original data item in the data set with each reference data item respectively, and determining a corresponding relationship between a target data item and a reference data item, wherein the target data item is: an original data item that matches a pre-configured reference data item; For each target data item, based on the target merging mode of the reference data item corresponding to the target data item and the fusion strategies corresponding to each merging mode, data fusion is performed on each data value corresponding to the target data item to obtain a target data value corresponding to the target data item; Based on the target data values ​​corresponding to the target data items, a personnel subject view corresponding to the data set is generated.

10. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in any one of the methods of claims 1-8 according to the obtained program instructions.

Images