A method and system for monitoring heterogeneous data

Abstract

The present application relates to the technical field of data processing, in particular to a heterogeneous data monitoring method and system, the method comprising: obtaining target user-provided monitoring data; extracting key feature data in the monitoring data, so as to construct a knowledge graph of the key features based on the key feature data, to realize monitoring of the heterogeneous data; being able to establish a correlation between time and features for different heterogeneous data, and then construct a knowledge graph based on the correlation to monitor, which can more effectively observe the update state of the heterogeneous data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of heterogeneous data monitoring, in particular to a heterogeneous data monitoring method and system. BACKGROUND

[0002] With the continuous development of industrialization and informatization, many fields have different data, especially the data accumulation in the industrial field to a certain order of magnitude, which exceeds the processing capacity of traditional technology, so it is necessary to use big data technology to improve the processing capacity and efficiency. Therefore, industrial big data provides technical and management support for the massive data in the industrial field. The main sources of industrial big data include business data related to production and operation, equipment and internet of things data, and external internet data, so there are many isolated heterogeneous databases (time series database, graph database, relational database and unstructured database, etc.). Because the data structure and database management system of the heterogeneous database are different, it is difficult to monitor the heterogeneous database under industrial big data by a monitoring system.

[0003] In the prior art, the original data of the monitoring target is obtained, the original data is converted to obtain first format data, and the first format data is verified. The verification process includes the following steps: determining the second index attribute of the data layer, the second index attribute being the same as the first index attribute; determining the timestamp information of the index data; associating the first format data after verification with a plurality of business dimensions to obtain second format data; performing alarm prediction according to the second format data, and visually displaying the results of the alarm prediction and the second format data. In the above prior art, the updating and changes of the data cannot be accurately and real-timely monitored. SUMMARY

[0004] The present application aims to provide a heterogeneous data monitoring method, which comprises the following steps:

[0005] receiving a to-be-processed data set sent by a target client, wherein the to-be-processed data set comprises a first initial data subset, a second initial data subset and a third initial data subset; the first initial data subset comprises a plurality of first initial data, and the type of each first initial data is text type; the second initial data subset comprises a plurality of second initial data, and the type of each second initial data is image type; the third initial data subset comprises a plurality of third initial data, and the type of each third initial data is table type;

[0006] aligning the to-be-processed data set to obtain an intermediate data set, wherein the intermediate data set comprises a first intermediate data subset, a second intermediate data subset and a third intermediate data subset, and the second intermediate data subset and the third intermediate data subset have a time correlation relationship with the first intermediate data subset;

[0007] extracting features from the intermediate data set to obtain a target feature set of the intermediate data set, and constructing a target knowledge graph according to the target feature set;

[0008] monitoring the to-be-processed data set according to update information of the target knowledge graph.

[0009] The application further provides a heterogeneous data monitoring system, which comprises:

[0010] a first execution module configured to receive a to-be-processed data set sent by a target client, wherein the to-be-processed data set comprises a first initial data subset, a second initial data subset and a third initial data subset; the first initial data subset comprises a plurality of first initial data, and each first initial data is of a text type; the second initial data subset comprises a plurality of second initial data, and each second initial data is of an image type; and the third initial data subset comprises a plurality of third initial data, and each third initial data is of a table type;

[0011] a second execution module configured to align the to-be-processed data set to obtain an intermediate data set, wherein the intermediate data set comprises a first intermediate data subset, a second intermediate data subset and a third intermediate data subset, and the second intermediate data subset and the third intermediate data subset have a time correlation relationship with the first intermediate data subset;

[0012] a third execution module configured to extract features from the intermediate data set to obtain a target feature set of the intermediate data set, and construct a target knowledge graph according to the target feature set;

[0013] a fourth execution module configured to monitor the to-be-processed data set according to update information of the target knowledge graph.

[0014] The application provides a heterogeneous data monitoring method, which comprises the following steps: receiving a to-be-processed data set sent by a target client, wherein the to-be-processed data set comprises a first initial data subset, a second initial data subset and a third initial data subset; the first initial data subset comprises a plurality of first initial data, and each first initial data is of a text type; the second initial data subset comprises a plurality of second initial data, and each second initial data is of an image type; the third initial data subset comprises a plurality of third initial data, and each third initial data is of a table type; performing data alignment processing on the to-be-processed data set to obtain an intermediate data set, wherein the intermediate data set comprises a first intermediate data subset, a second intermediate data subset and a third intermediate data subset, and the second intermediate data subset and the third intermediate data subset have a time correlation relationship with the first intermediate data subset; performing feature extraction on the intermediate data set to obtain a target feature set of the intermediate data set, so as to construct a target knowledge graph according to the target feature set; and monitoring the to-be-processed data set according to update information of the target knowledge graph. According to the time correlation relationship between different types of data, the same or similar data of different types can be queried, and the knowledge graph is constructed based on the time correlation relationship to monitor the update state of the heterogeneous data, so that the update state of the heterogeneous data can be observed more effectively and quickly. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 A flow chart of a heterogeneous data monitoring method provided by the first embodiment of the present application is shown in the figure.

[0017] Figure 2 A structural schematic diagram of a heterogeneous data monitoring system provided by the second embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0019] Embodiment one

[0020] like Figure 1 As shown, this embodiment provides a method for monitoring heterogeneous data, which includes the following steps:

[0021] A data set to be processed sent by a target client is received, wherein the data set to be processed includes a first type of initial data subset, a second type of initial data subset, and a third type of initial data subset.

[0022] Specifically, the first type of initial data subset includes a plurality of first initial data, wherein each of the first initial data is of text type.

[0023] Specifically, the second type of initial data subset includes a plurality of second initial data, wherein each second initial data is of an image type.

[0024] Specifically, the third type of initial data subset includes a plurality of third initial data, wherein each of the third initial data is of a table type.

[0025] Perform data alignment processing on the data set to be processed to obtain an intermediate data set, wherein the intermediate data set includes a first intermediate data subset, a second intermediate data subset and a third intermediate data subset, and the second intermediate data subset and the third intermediate data subset are respectively temporally associated with the first intermediate data subset.

[0026] Furthermore, the first intermediate data subset includes a plurality of first intermediate data.

[0027] Furthermore, the second intermediate data subset includes a plurality of second intermediate data.

[0028] Furthermore, the third intermediate data subset includes a plurality of third intermediate data.

[0029] In a specific embodiment, the step of performing data alignment on the data set to be processed to obtain an intermediate data set further includes the following steps:

[0030] S100, obtain the time information set A={A1, ..., A x ,……,A p}, A x is the collection time of the xth first initial data, and the time information set of the second type of initial data set B={B1,...,B y ,……,B q}, B y is the collection time of the yth second initial data, and the time information set of the third type of initial data set C={C1, ..., C Z ,……,Cg},C Z is the collection time of the zth third initial data, the value range of x is 1 to p and p≥p 0 , p is the number of the first initial data, p 0 is a preset first threshold value, the value range of y is 1 to q and q≥q 0 , q is the number of the second initial data, q 0 is a preset second threshold value, the value range of z is 1 to g and g≥g 0 , g is the number of the third initial data, g 0 is a preset third threshold value; the first threshold value, the second threshold value and the third threshold value set by the person skilled in the art according to the actual demand, which will not be repeated here.

[0031] S101, when , it is determined that the corresponding second initial data of B y and the corresponding first initial data of A x have a time correlation relationship and are marked with a label;

[0032] S102, when , it is determined that the corresponding third initial data of C z and the corresponding first initial data of A x have a time correlation relationship and are marked with a label;

[0033] S103, the first initial data, the second initial data and the third initial data with the same label are obtained from the first type initial data set, the second type initial data set and the third type initial data set, and the first initial data, the second initial data and the third initial data with the same label are taken as the first intermediate data, the second intermediate data and the third intermediate data respectively.

[0034] Preferably, △T0 meets the following conditions:

[0035] , the gcd() function is a common divisor function.

[0036] The above, the time correlation relationship between the image type data and the table type data and the text type data can be determined through the collection time interval of different types of data, the time correlation relationship reflects that the smaller the time interval is, the more consistent the content between the image type data and the table type data and the text type data is, and then the associated heterogeneous data can be obtained. In order to further construct a knowledge graph for the convenience of monitoring.

[0037] Feature extraction is performed on the intermediate data set to obtain a target feature set of the intermediate data set, so as to construct a target knowledge graph according to the target feature set.

[0038] The target feature set includes an intermediate title feature set of the first intermediate data subset, an intermediate image feature set of the second intermediate data subset, and an intermediate name feature set of the third intermediate data subset.

[0039] In a specific embodiment, feature extraction is performed on the intermediate data set to obtain a target feature set of the intermediate data set, and a target knowledge graph is constructed according to the target feature set, and the method further includes the following steps:

[0040] Title feature extraction is performed on the first intermediate data subset to obtain an intermediate title feature set AT = {AT1,..., ATn} of the first intermediate data subset, where ATi is an i-th title feature of an r-th first intermediate data, i is in a range of 1 to n, n is a number of title features, and r is in a range of 1 to s, s is a number of first intermediate data. r ,..., AT s}, AT r = {AT 1 r ,..., AT i r ,..., AT n r}, AT i r is an i-th title feature of an r-th first intermediate data, i is in a range of 1 to n, n is a number of title features, and r is in a range of 1 to s, s is a number of first intermediate data. Any title feature extraction method in the prior art is known to those skilled in the art, and will not be described here.

[0041] Image feature extraction is performed on the second intermediate data subset to obtain an intermediate image feature set BT = {BT1,..., BTm} of the second intermediate data subset, where BTj is a j-th image feature of an r-th second intermediate data, j is in a range of 1 to m, and m is a number of image features. r ,..., BT s}, BT r = {BT 1 r ,..., BT j r ,..., BT m r}, BT j r is a j-th image feature of an r-th second intermediate data, j is in a range of 1 to m, and m is a number of image features. Any image feature extraction method in the prior art is known to those skilled in the art, and will not be described here.

[0042] Name feature extraction is performed on the third intermediate data subset to obtain an intermediate name feature set CT = {CT1,..., CTn} of the third intermediate data subset, where CTi is an i-th name feature of an r-th third intermediate data, i is in a range of 1 to n, and n is a number of name features. r ,..., CT s}, CT r = {CT 1 r ,..., CT er ,..., CT f r}, CT e r is the e-th name feature of the r-th third intermediate data, e ranges from 1 to f, and f is the number of name features; wherein the name features include table names and field names of the table type data; any table name and field name extraction method in the prior art is known to those skilled in the art, and will not be described here.

[0043] Based on the AT and the BT, a first similarity set F = {F1,..., F r ,..., F s} is obtained, F r = {F 1 r ,..., F i r ,..., F n r} is obtained, F i r = {F i1 r ,..., F ir r ,..., F is r} is obtained, F ir r = {F ir r1 ,..., F ir rj ,..., F ir rm} is obtained, F ir rj is the similarity between the AT i r and the BT j r ; wherein the title and the image feature are both in the form of a string; any method for determining the similarity between two strings in the prior art is known to those skilled in the art, and will not be described here, for example, Mahalanobis distance, Euclidean distance, cosine distance, etc.

[0044] Based on the AT and the CT, a second similarity set D = {D1,..., D r ,..., D s} is obtained, D r = {D 1 r ,..., D i r ,..., D n r} is obtained, D i r = {Di1 r , …, D ir r , …, D is r}, D ir r { D ir r1 , …, D ir re , …, D ir rf}, D ir re is AT i r and CT e r ; wherein, the title and the name are in the form of string; any method for determining the similarity between two strings in the prior art is known to those skilled in the art, which will not be described here, for example, Mahalanobis distance, Euclidean distance, cosine distance, etc.

[0045] When the sum of the mean of F and the mean of D is less than ΔH 1 , it is recommended to explain the intermediate data set as a monitorable data set, wherein ΔH 1 is the first similarity threshold, which is set by those skilled in the art according to actual needs.

[0046] When the sum of the mean of F and the mean of D is not less than ΔH 1 , the intermediate data set is not taken as a monitorable data set.

[0047] The above, in addition to establishing a time correlation in the collection time for the target client's to-be-processed data set, a content correlation is also established between the names or features of the text, data table and picture, so that the heterogeneous data in the to-be-processed data provided by the user has a higher relationship, which is conducive to monitoring the heterogeneous data according to the correlation.

[0048] When the sum of the mean of F and the mean of D is less than ΔH 1 , a target knowledge graph is constructed according to the target feature set; further understanding is that: when the sum of the mean of F and the mean of D is less than ΔH 1 , each target feature in the target feature set is taken as an entity, and the similarity between two target features is taken as an edge to construct a target knowledge graph.

[0049] In another embodiment, when F+D≥ΔH 1 , after the intermediate data set is not taken as a monitorable data set, the following steps are further included:

[0050] Extract key text feature data U={U1,...,U r ,……,U s},U r ={U 1 r ,……,U t r ,……,U v r},U t r is the t-th text feature in the r-th first intermediate data, where t ranges from 1 to v, and v is the number of text features, where a text feature is a keyword;

[0051] Based on U and BT, obtain the third similarity set Q={Q1, ..., Q r ,……,Q s}, Q r ={Q 1 r ,……,Q t r ,……,Q v r}, Q t r ={Q t1 r ,……,Q tr r ,……,Q ts r}, Q tr r ={Q tr r1 ,……,Q tr rj ,……,Q tr rm}, Q tr rj It's U t r With BT j r The similarity between them; wherein the text features and the image features are both in the form of character strings; those skilled in the art are aware of any method for determining the similarity between two character strings in the prior art, which will not be described here, for example, Mahalanobis distance, Euclidean distance, cosine distance, etc.

[0052] Based on U and CT, the fourth similarity set W={W1, ..., W r ,……,W s}, W r ={W 1 r ,……,W tr , …, W v r}, W t r = {W t1 r , …, W tr r , …, W ts r}, W ir r = {W tr r1 , …, W tr re , …, W tr rf}, W tr re is U t r and CT e r ; wherein, the text features and the names are in the form of strings; any method for determining the similarity between two strings in the prior art is known to those skilled in the art, and will not be described here, for example, Mahalanobis distance, Euclidean distance, cosine distance, etc.

[0053] When the sum of the mean of W and the mean of Q is less than ΔH 2 , the intermediate data set is taken as a monitorable data set, and a target knowledge graph is constructed according to the target feature set; ΔH 2 is a second similarity threshold, which is set by those skilled in the art according to actual needs, and will not be described here; it is further understood that: when Q+W< ΔH 2 , each target feature in the target feature set is taken as an entity, and the similarity between two target features is taken as an edge, and a target knowledge graph is constructed.

[0054] Preferably, the target features can be a combination of title features, image features, name features, and text features.

[0055] According to the update information of the target knowledge graph, the to-be-processed data set is monitored; it can be understood that the update information of the target knowledge graph refers to the update of the target features and the similarity between two target features, and thus the change of each to-be-processed data can be monitored through the change of the knowledge graph.

[0056] The embodiment provides a heterogeneous data monitoring method, which comprises the following steps: receiving a to-be-processed data set sent by a target client, wherein the to-be-processed data set comprises a first initial data subset, a second initial data subset and a third initial data subset; the first initial data subset comprises a plurality of first initial data, and the type of each first initial data is a text type; the second initial data subset comprises a plurality of second initial data, and the type of each second initial data is an image type; the third initial data subset comprises a plurality of third initial data, and the type of each third initial data is a table type; performing data alignment processing on the to-be-processed data set to obtain an intermediate data set, wherein the intermediate data set comprises a first intermediate data subset, a second intermediate data subset and a third intermediate data subset, and the second intermediate data subset and the third intermediate data subset have a time correlation relationship with the first intermediate data subset; performing feature extraction on the intermediate data set to obtain a target feature set of the intermediate data set, so as to construct a target knowledge graph according to the target feature set; monitoring the to-be-processed data set according to update information of the target knowledge graph; it can be known that different types of data with the same or similar content are queried out through the time correlation relationship between different types of data, and the knowledge graph is constructed based on the time correlation relationship to monitor, so that the update state of the heterogeneous data can be observed more effectively and quickly.

[0057] Embodiment two

[0058] As Figure 2 shown, the embodiment two provides a heterogeneous data monitoring system, which comprises:

[0059] A first execution module is configured to receive a to-be-processed data set sent by a target client, wherein the to-be-processed data set comprises a first initial data subset, a second initial data subset and a third initial data subset.

[0060] Specifically, the first initial data subset comprises a plurality of first initial data, and the type of each first initial data is a text type.

[0061] Specifically, the second initial data subset comprises a plurality of second initial data, and the type of each second initial data is an image type.

[0062] Specifically, the third initial data subset comprises a plurality of third initial data, and the type of each third initial data is a table type.

[0063] The second execution module is used to perform data alignment processing on the data set to be processed to obtain an intermediate data set, wherein the intermediate data set includes a first intermediate data subset, a second intermediate data subset and a third intermediate data subset, wherein the second intermediate data subset and the third intermediate data subset are respectively time-related to the first intermediate data subset.

[0064] Furthermore, the first intermediate data subset includes a plurality of first intermediate data.

[0065] Furthermore, the second intermediate data subset includes a plurality of second intermediate data.

[0066] Furthermore, the third intermediate data subset includes a plurality of third intermediate data.

[0067] In a specific embodiment, the second execution module includes:

[0068] The data acquisition module is used to obtain the time information set A={A1, ..., A x ,……,A p}, A x is the collection time of the xth first initial data, and the time information set of the second type of initial data set B={B1,...,B y ,……,B q}, B y is the collection time of the yth second initial data, and the time information set of the third type of initial data set C={C1, ..., C Z ,……,C g}, C Z is the acquisition time of the zth third initial data, the value range of x is 1 to p and p ≥ p 0 , p is the number of the first initial data, p 0 is the preset first threshold, the value range of y is 1 to q and q≥q 0 , q is the number of the second initial data, q 0 is the preset second threshold, the value range of z is 1 to g and g≥g 0 , g is the number of the third initial data, g 0 It is a preset third threshold; the first threshold, second threshold and third threshold set by those skilled in the art according to actual needs are not described here in detail.

[0069] The first determining module is used when When B y The corresponding second initial data and A x The corresponding first initial data have a time association relationship and are marked with labels;

[0070] The second determining module is configured to determine C when z The corresponding third initial data has a time correlation relationship with the A x corresponding first initial data and is labeled with a label.

[0071] The data determining module is configured to obtain the first initial data, the second initial data and the third initial data with the same label from the first initial data set, the second initial data set and the third initial data set, and take the first initial data, the second initial data and the third initial data with the same label as the first intermediate data, the second intermediate data and the third intermediate data respectively.

[0072] Preferably, ΔT0 satisfies the following condition:

[0073] The gcd() function is a greatest common divisor function.

[0074] The above, the time correlation relationship between the image type data and the table type data and the text type data can be determined by the collection time interval of different types of data, and the time correlation relationship reflects that the smaller the time interval is, the more consistent the content between the image type data and the table type data and the text type data is, and then the associated heterogeneous data can be obtained. In order to further construct a knowledge graph for the convenience of monitoring.

[0075] The third execution module is configured to perform feature extraction on the intermediate data set to obtain a target feature set of the intermediate data set, and construct a target knowledge graph according to the target feature set.

[0076] The target feature set includes an intermediate title feature set of the first intermediate data subset, an intermediate image feature set of the second intermediate data subset and an intermediate name feature set of the third intermediate data subset.

[0077] In one specific embodiment, the third execution module includes:

[0078] The first obtaining module is configured to perform title feature extraction on the first intermediate data subset to obtain an intermediate title feature set AT = {AT1, …, AT r , …, AT s} of the first intermediate data subset. x {AT 1 r , …, AT i r , …, AT n r} of the first intermediate data subset. i ris the i-th title feature of the r-th first intermediate data, i ranges from 1 to n, n is the number of title features, r ranges from 1 to s, s is the number of first intermediate data; any title feature extraction method in the prior art is known to those skilled in the art, and will not be described here.

[0079] The second acquisition module is configured to perform image feature extraction on the second intermediate data subset to obtain an intermediate image feature set BT of the second intermediate data subset, where BT = {BT1, …, BT r , …, BT s}, BT r = {BT 1 r , …, BT j r , …, BT m r}, BT j r is the j-th image feature of the r-th second intermediate data, j ranges from 1 to m, m is the number of image features; any image feature extraction method in the prior art is known to those skilled in the art, and will not be described here.

[0080] The third acquisition module is configured to perform name feature extraction on the third intermediate data subset to obtain an intermediate name feature set CT of the third intermediate data subset, where CT = {CT1, …, CT r , …, CT s}, CT r = {CT 1 r , …, CT e r , …, CT f r}, CT e r is the e-th name feature of the r-th third intermediate initial data, e ranges from 1 to f, f is the number of name features; the name features include table names and field names of table type data; any table name and field name extraction method in the prior art is known to those skilled in the art, and will not be described here.

[0081] The fourth acquisition module is configured to obtain a first similarity set F based on AT and BT, where F = {F1, …, F r , …, F s}, F r = {F 1 r , …, F i r , …, F n r}, F ir ={F i1 r ,……,F ir r ,……,F is r}, F ir r ={F ir r1 ,……,F ir rj ,……,F ir rm}, F ir rj It is AT i r With BT j r The similarity between them; wherein, the title and the image features are both in the form of character strings; those skilled in the art are aware of any method for determining the similarity between two character strings in the prior art, which will not be described here, for example, Mahalanobis distance, Euclidean distance, cosine distance, etc.

[0082] The fifth acquisition module is used to acquire a second similarity set D={D1, ..., D r ,……,D s}, D r ={D 1 r ,……,D i r ,……,D n r}, D i r ={D i1 r ,……,D ir r ,……,D is r}, D ir r ={D ir r1 ,……,D ir re ,……,D ir rf}, D ir re It is AT i r With CT e r The similarity between them; wherein the title and the name are both in the form of character strings; those skilled in the art are aware of any method for determining the similarity between two character strings in the prior art, which will not be described here, for example, Mahalanobis distance, Euclidean distance, cosine distance, etc.

[0083] The first module is used when the sum of the mean of F and the mean of D is less than ΔH 1 When , the intermediate data set is used as the monitorable data set, where ΔH 1 It is the first similarity threshold, and those skilled in the art can set the similarity threshold according to actual needs.

[0084] The second module is used when the sum of the mean of F and the mean of D is not less than ΔH 1 When the intermediate data set is used, it is not used as a monitorable data set.

[0085] As described above, in addition to establishing a time correlation relationship based on the collection time for the target client's unprocessed data set, a content correlation relationship can also be established through the names or features between texts, data tables, and pictures, so that the heterogeneous data in the unprocessed data provided by the user have a higher relationship, which is conducive to monitoring heterogeneous data according to the correlation relationship.

[0086] When the sum of the mean of F and the mean of D is less than ΔH 1 When , the target knowledge graph is constructed according to the target feature set; further understood as: when the sum of the mean of F and the mean of D is less than ΔH 1 When , each target feature in the target feature set is taken as an entity, and the similarity between two target features is taken as an edge to construct the target knowledge graph.

[0087] In another embodiment, the second module includes:

[0088] Feature extraction module, used to extract key text feature data U={U1, ..., U r ,……,U s},U r ={U 1 r ,……,U t r ,……,U v r},U t r is the t-th text feature in the r-th first intermediate data, where t ranges from 1 to v, and v is the number of text features, where a text feature is a keyword;

[0089] The first similarity acquisition module is used to obtain a third similarity set Q={Q1, ..., Q r ,……,Q s}, Q r ={Q 1 r ,……,Q t r..., Q v r ..., Q t r ..., Q t1 r ..., Q tr r ..., Q ts r ..., Q tr r ..., Q tr r1 ..., Q tr rj ..., Q tr rm ..., Q tr rj is U t r and BT j r ; wherein, the text feature and the image feature are both in the form of string; any method for determining the similarity between two strings in the prior art is known to those skilled in the art, and will not be described here again, for example, Mahalanobis distance, Euclidean distance, cosine distance, etc.

[0090] a second similarity obtaining module, configured to obtain a fourth similarity set W={W1,..., W r ..., W s}, W r ={W 1 r ..., W t r ..., W v r}, W t r ={W t1 r ..., W tr r ..., W ts r}, W ir r ={W tr r1 ..., W tr re ..., W tr rf}, W tr re is U t r and CT e rsimilarity between W and Q; wherein, the text feature and the name are both in string form; any method for determining the similarity between two strings in the prior art is known to those skilled in the art, and will not be described here, for example, Mahalanobis distance, Euclidean distance, cosine distance, etc.

[0091] a monitorable data set determination module, configured to determine the monitorable data set when the sum of the mean of W and the mean of Q is less than ΔH 2 , and construct a target knowledge graph according to the target feature set; ΔH 2 is a second similarity threshold, which is set by those skilled in the art according to actual needs, and will not be described here; it is further understood that when the sum of the mean of W and the mean of Q is less than ΔH 2 , each target feature in the target feature set is taken as an entity, and the similarity between two target features is taken as an edge, to construct a target knowledge graph.

[0092] Preferably, the target feature can be a combination of title feature, image feature, name feature and text feature.

[0093] monitoring the to-be-processed data set according to the update information of the target knowledge graph; it can be understood that the update information of the target knowledge graph refers to the update of the target feature and the similarity between two target features, and thus the change of each to-be-processed data can be monitored through the change of the knowledge graph.

[0094] The second embodiment provides a heterogeneous data monitoring system, the system comprising: a first execution module configured to receive a to-be-processed data set sent by a target client, wherein the to-be-processed data set comprises a first initial data subset, a second initial data subset and a third initial data subset; the first initial data subset comprises a plurality of first initial data, each of the first initial data being of a text type; the second initial data subset comprises a plurality of second initial data, each of the second initial data being of an image type; and the third initial data subset comprises a plurality of third initial data, each of the third initial data being of a table type; a second execution module configured to perform data alignment processing on the to-be-processed data set to obtain an intermediate data set, wherein the intermediate data set comprises a first intermediate data subset, a second intermediate data subset and a third intermediate data subset, and the second intermediate data subset and the third intermediate data subset have a time correlation relationship with the first intermediate data subset; a third execution module configured to perform feature extraction on the intermediate data set to obtain a target feature set of the intermediate data set, and construct a target knowledge graph based on the target feature set; and a fourth execution module configured to monitor the to-be-processed data set based on update information of the target knowledge graph. It can be known that, through the time correlation relationship between different types of data, different types of data with the same or similar content are queried out, and then the knowledge graph is constructed based on the time correlation relationship to monitor, so that the update state of the heterogeneous data can be observed more effectively and quickly.

[0095] Although some specific embodiments of the present application have been described in detail by examples, those skilled in the art should understand that the above examples are only for illustration, but not for limiting the scope of the present application. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A method for monitoring heterogeneous data, characterized in that: The method comprises the following steps: Receive a data set to be processed sent by a target client, wherein the data set to be processed includes a first type of initial data subset, a second type of initial data subset, and a third type of initial data subset; the first type of initial data subset includes a plurality of first initial data, each of which is of a text type; the second type of initial data subset includes a plurality of second initial data, each of which is of an image type; the third type of initial data subset includes a plurality of third initial data, each of which is of a table type; Performing data alignment processing on the data set to be processed to obtain an intermediate data set, wherein the intermediate data set includes a first intermediate data subset, a second intermediate data subset, and a third intermediate data subset, and the second intermediate data subset and the third intermediate data subset are respectively time-correlated with the first intermediate data subset; the first intermediate data subset includes a plurality of first intermediate data; the second intermediate data subset includes a plurality of second intermediate data; and the third intermediate data subset includes a plurality of third intermediate data; performing data alignment processing on the data set to be processed to obtain the intermediate data set further includes the following steps: S100, obtain the time information set A={A1, ..., A x ,……,A p }, A x is the collection time of the xth first initial data, and the time information set of the second type of initial data set B={B1,...,B y ,……,B q }, B y is the collection time of the yth second initial data, and the time information set of the third type of initial data set C={C1, ..., C Z ,……,C g }, C Z is the acquisition time of the zth third initial data, the value range of x is 1 to p and p ≥ p 0 , p is the number of the first initial data, p 0 is the preset first threshold, the value range of y is 1 to q and q≥q 0 , q is the number of the second initial data, q 0 is the preset second threshold, the value range of z is 1 to g and g≥g 0 , g is the number of the third initial data, g 0 is the preset third threshold; S101, when When B y The corresponding second initial data and A x The corresponding first initial data have a time correlation relationship and are marked with labels, and ΔT0 is a preset correction time; S102, when When C z The corresponding third initial data and A x The corresponding first initial data have a time correlation relationship and are labeled; S103, obtaining first initial data, second initial data, and third initial data with the same label from the first type of initial data set, the second type of initial data set, and the third type of initial data set, and using the first initial data, second initial data, and third initial data with the same label as first intermediate data, second intermediate data, and third intermediate data, respectively; Performing feature extraction on the intermediate data set to obtain a target feature set of the intermediate data set, and constructing a target knowledge graph based on the target feature set; The data set to be processed is monitored according to the update information of the target knowledge graph.

2. The method for monitoring heterogeneous data according to claim 1, characterized in that: △T0 meets the following conditions: , the gcd() function is the common divisor function.

3. The method for monitoring heterogeneous data according to claim 1, characterized in that: Performing feature extraction on the intermediate data set to obtain a target feature set of the intermediate data set, and constructing a target knowledge graph based on the target feature set, further comprising the following steps: Extract title features from the first intermediate data subset to obtain an intermediate title feature set AT={AT1, ..., AT r ,……,AT s }, AT r ={AT 1 r ,……,AT i r ,……,AT n r }, AT i r is the i-th title feature of the r-th first intermediate data, i ranges from 1 to n, n is the number of title features, r ranges from 1 to s, s is the number of first intermediate data; Perform image feature extraction on the second intermediate data subset to obtain an intermediate image feature set BT={BT1, ..., BT r ,……,BT s }, BT r ={BT 1 r ,……,BT j r ,……,BT m r }, BT j r is the jth image feature of the rth second intermediate data, where j ranges from 1 to m, and m is the number of image features; Perform name feature extraction on the third intermediate data subset to obtain an intermediate name feature set CT of the third intermediate data subset = {CT1, ..., CT r ,……,CT s }, CT r ={CT 1 r ,……,CT e r ,……,CT f r }, CT e r is the e-th name feature of the r-th third intermediate data, where the value of e ranges from 1 to f, and f is the number of name features; wherein the name feature includes the table name and field name of the table type data; Based on AT and BT, obtain the first similarity set F={F1, ..., F r ,……,F s }, F r ={F 1 r ,……,F i r ,……,F n r }, F i r ={F i1 r ,……,F ir r ,……,F is r }, F ir r ={F ir r1 ,……,F ir rj ,……,F ir rm }, F ir rj It is AT i r With BT j r similarity between Based on AT and CT, obtain the second similarity set D={D1, ..., D r ,……,D s }, D r ={D 1 r ,……,D i r ,……,D n r }, D i r ={D i1 r ,……,D ir r ,……,D is r }, D ir r ={D ir r1 ,……,D ir re ,……,D ir rf }, D ir re It is AT i r With CT e r similarity between When the sum of the mean of F and the mean of D is less than ΔH 1 When , the intermediate data set is used as the monitorable data set, where ΔH 1 is the first similarity threshold; When the sum of the mean of F and the mean of D is not less than ΔH 1 When the intermediate data set is used, it is not used as a monitorable data set.

4. A monitoring system for heterogeneous data, characterized in that: The system comprises: A first execution module is configured to receive a data set to be processed sent by a target client, wherein the data set to be processed includes a first type of initial data subset, a second type of initial data subset, and a third type of initial data subset; the first type of initial data subset includes a plurality of first initial data, each of which is of a text type; the second type of initial data subset includes a plurality of second initial data, each of which is of an image type; the third type of initial data subset includes a plurality of third initial data, each of which is of a table type; The second execution module is configured to perform data alignment processing on the data set to be processed to obtain an intermediate data set, wherein the intermediate data set includes a first intermediate data subset, a second intermediate data subset, and a third intermediate data subset, and the second intermediate data subset and the third intermediate data subset are respectively time-correlated with the first intermediate data subset; the first intermediate data subset includes a plurality of first intermediate data; the second intermediate data subset includes a plurality of second intermediate data; and the third intermediate data subset includes a plurality of third intermediate data; the second execution module includes: The data acquisition module is used to obtain the time information set A={A1, ..., A x ,……,A p }, A x is the collection time of the xth first initial data, and the time information set of the second type of initial data set B={B1,...,B y ,……,B q }, B y is the collection time of the yth second initial data, and the time information set of the third type of initial data set C={C1, ..., C Z ,……,C g }, C Z is the acquisition time of the zth third initial data, the value range of x is 1 to p and p ≥ p 0 , p is the number of the first initial data, p 0 is the preset first threshold, the value range of y is 1 to q and q≥q 0 , q is the number of the second initial data, q 0 is the preset second threshold, the value range of z is 1 to g and g≥g 0 , g is the number of the third initial data, g 0 is the preset third threshold; The first determining module is used when When B y The corresponding second initial data and A x The corresponding first initial data have a time correlation relationship and are marked with labels, and ΔT0 is a preset correction time; The second determining module is used when When C z The corresponding third initial data and A x The corresponding first initial data have a time correlation relationship and are labeled; a data determination module, configured to obtain first initial data, second initial data, and third initial data with the same label from the first type of initial data set, the second type of initial data set, and the third type of initial data set, and use the first initial data, second initial data, and third initial data with the same label as first intermediate data, second intermediate data, and third intermediate data, respectively; A third execution module is configured to perform feature extraction on the intermediate data set to obtain a target feature set of the intermediate data set, so as to construct a target knowledge graph based on the target feature set; The fourth execution module is used to monitor the data set to be processed according to the update information of the target knowledge graph.

5. The heterogeneous data monitoring system according to claim 4, characterized in that: △T0 meets the following conditions: , the gcd() function is the common divisor function.

6. The heterogeneous data monitoring system according to claim 4, characterized in that: The third execution module includes: The first acquisition module is used to extract title features from the first intermediate data subset to obtain an intermediate title feature set AT={AT1, ..., AT r ,……,AT s }, AT r ={AT 1 r ,……,AT i r ,……,AT n r }, AT i r is the i-th title feature based on the r-th first intermediate data, i ranges from 1 to n, n is the number of title features, r ranges from 1 to s, s is the number of first intermediate data; The second acquisition module is used to extract image features from the second intermediate data subset to obtain an intermediate image feature set BT={BT1, ..., BT r ,……,BT s }, BT r ={BT 1 r ,……,BT j r ,……,BT m r }, BT j r is the jth image feature of the rth second intermediate data, where j ranges from 1 to m, and m is the number of image features; The third acquisition module is used to extract name features from the third intermediate data subset to obtain an intermediate name feature set CT of the third intermediate data subset = {CT1, ..., CT r ,……,CT s }, CT r ={CT 1 r ,……,CT e r ,……,CT f r }, CT e r is the e-th name feature of the r-th third intermediate data, where the value of e ranges from 1 to f, and f is the number of name features; wherein the name feature includes the table name and field name of the table type data; The fourth acquisition module is used to acquire a first similarity set F={F1, ..., F r ,……,F s }, F r ={F 1 r ,……,F i r ,……,F n r }, F i r ={F i1 r ,……,F ir r ,……,F is r }, F ir r ={F ir r1 ,……,F ir rj ,……,F ir rm }, F ir rj It is AT i r With BT j r similarity between The fifth acquisition module is used to acquire a second similarity set D={D1, ..., D r ,……,D s }, D r ={D 1 r ,……,D i r ,……,D n r }, D i r ={D i1 r ,……,D ir r ,……,D is r }, D ir r ={D ir r1 ,……,D ir re ,……,D ir rf }, D ir re It is AT i r With CT e r similarity between The first module is used when the sum of the mean of F and the mean of D is less than ΔH 1 When , the intermediate data set is used as the monitorable data set, where ΔH 1 is the first similarity threshold; The second module is used when the sum of the mean of F and the mean of D is not less than ΔH 1 When the intermediate data set is used, it is not used as a monitorable data set.

Images