A storage method and system for multi-source heterogeneous target data

Abstract

The present invention discloses a storage method for multi-source heterogeneous target data. The multi-source heterogeneous target data is collected through a web crawler program module, and the image data is converted into text data, unifying the format of the multi-source heterogeneous data. Combining a clustering algorithm and a TextCNN model for target feature extraction, the effective fusion of multi-source heterogeneous data features is realized, and the target feature information is stored, providing real and effective data for the subsequent judgment and analysis model of market competition monopoly behavior.

Description

Technical Field

[0001] This application belongs to the technical field of intelligent processing of multi-source heterogeneous data, and particularly relates to a storage method and system for multi-source heterogeneous target data. Background Art

[0002] The concept of multi-source heterogeneous data mainly includes two characteristics: one is multi-source. For example, descriptions and evaluations of the same theme or object are given from different perspectives by different people; the other is heterogeneity, that is, the types and forms of these data are not the same and are complex, such as text, images, videos, etc.

[0003] Currently, the data generated by different e-commerce platforms not only have different storage modes and information contents, but also have different generation times, user perspectives, generation locations, code rules followed, etc. It can be seen that the big data field of e-commerce platforms has wide data sources, complex data structures, and multi-source heterogeneous characteristics. In the prior art, it is difficult to effectively fuse and store the above-mentioned various types of big data, which in turn leads to the inability to fully consider the correlation between different data when using relevant models for data mining and analysis later, reducing the efficiency of big data analysis. Summary of the Invention

[0004] Aiming at the defects in the prior art, the present invention provides a storage method and system for multi-source heterogeneous target data. By unifying the multi-source heterogeneous data format and combining the clustering algorithm and TextCNN model for target feature extraction, the effective fusion of multi-source heterogeneous data features is realized, and the target feature information is stored, providing real and effective data for the subsequent judgment and analysis model of market competition monopoly behavior. To achieve the above object, the present invention provides the following technical solutions:

[0005] A storage method and system for multi-source heterogeneous target data, which is applied to a storage system for multi-source heterogeneous market-sensitive data, is characterized in that the storage method for multi-source heterogeneous target data includes the following steps:

[0006] (1) Collecting multiple target data through a web crawler program module, where the target data is multi-source heterogeneous and at least includes text type data and image type data;

[0007] (2) Obtaining the text information corresponding to the target data through a text information extraction module;

[0008] (3) Clustering and fusing the text information through a clustering module to obtain multiple text information groups;

[0009] (4) Extracting features from multiple text information data under each group through a target feature extraction module to obtain multiple groups of target feature information;

[0010] (5) Store the multiple groups of target feature information in a feature database for subsequent judgment and analysis of market competition monopoly behaviors.

[0011] Further, obtaining the text information corresponding to the target data in step (2) includes: for text-based multi-source heterogeneous data, directly obtaining the text information; for image-based multi-source heterogeneous data, using an image-to-text conversion tool to convert it into corresponding text information.

[0012] Further, the image-to-text conversion tool calls the General Scene OCR API in the "Baidu Smart Cloud" platform by writing a Python script to convert image-based data into text information.

[0013] Further, clustering and fusing the text information in step (3) to obtain multiple text information groups is achieved through the K-means clustering algorithm, and its specific implementation steps are as follows:

[0014] (3-1) Assume that the text information corresponding to the target data constitutes a data set , given the number of clusters K, set the maximum number of iterations N, and randomly select K data as the initial cluster centers ;

[0015] (3-2) Calculate the distance between the remaining different data in the data set D and the cluster center m , where ; is the Euclidean distance or Manhattan distance;

[0016] If the distance between the data and a certain cluster center satisfies , then the data is divided into ;

[0017] (3-3) If the data in the data set D is divided into a certain category, then adjust the classification and recalculate the cluster center , where r represents the number of data contained in the cluster category ;

[0018] (3-4) Calculate the error criterion function Z. If there is a satisfaction with the error requirement, it means that the clustering criterion function converges, and the K-means algorithm clustering stops here; otherwise, return to step (3-2) to start clustering again until the clustering iteration times end. The calculation formula of the error criterion function Z is as follows:

[0019] ;

[0020] Further, after obtaining multiple text data groups through the K-means clustering algorithm, in step (4), the target feature extraction module extracts features from multiple text information data under each group to obtain multiple groups of target feature information, and its specific implementation is as follows:

[0021] (4-1) Input the text information data under each group into the Word2vec vector generation model to generate multiple word vectors;

[0022] (4-2) Concatenate the word vectors into a two-dimensional matrix according to the sentence order;

[0023] (4-3) Input the two-dimensional matrix into the TextCNN model to extract target feature information.

[0024] Further, in step (4-3), the convolutional layer of the TextCNN model uses multiple convolutional kernels of different sizes to extract local features of sentences in different ranges, and the sliding of the convolutional kernel only acts in the sentence order direction; the pooling layer of the TextCNN model converts the dimension to a fixed length through one-dimensional pooling operation, and finally combines the local features through the fully connected layer to obtain the target feature information of the text data.

[0025] Another object of the present invention is to provide a storage system for multi-source heterogeneous market-sensitive data, including a network crawler program module, a text information extraction module, a clustering module, a target feature extraction module, and a feature database; the storage system for multi-source heterogeneous market-sensitive data is used to execute the above-mentioned storage method for multi-source heterogeneous market-sensitive data.

[0026] Another object of the present invention is to provide a computer-readable storage medium storing one or more programs, and the one or more programs cause a computer to execute the above-mentioned storage method for multi-source heterogeneous market-sensitive data.

[0027] Combining all the above technical solutions, the present invention has the following advantages compared with the prior art:

[0028] By unifying the multi-source heterogeneous data format, combining the clustering algorithm and the TextCNN model for target feature extraction, the effective fusion of multi-source heterogeneous data features is realized, and the target feature information is stored, providing real and effective data for the subsequent judgment and analysis model of market competition monopoly behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic flowchart of the implementation of the storage method for multi-source heterogeneous market-sensitive data of the present invention.

[0030] Figure 2 is a schematic structural diagram of the storage system for multi-source heterogeneous market-sensitive data of the present invention. Detailed implementation manners

[0031] The following is further described by way of embodiments in combination with the attached Figure 1 and 2 Obviously, the described embodiments are partial embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the protection scope of the present invention.

[0032] The present invention provides a storage method and system for multi-source heterogeneous market-sensitive data, which is applied to the Figure 2 shown storage system for multi-source heterogeneous market-sensitive data. According to Figure 1 , the storage method for the multi-source heterogeneous market-sensitive data includes the following steps:

[0033] (1) Collect market-sensitive data of multiple e-commerce platforms through a web crawler program module, where the market-sensitive data is data suspected of market competition monopoly behavior; the market-sensitive data is multi-source heterogeneous and at least includes text data and image data;

[0034] (2) Obtain the text information corresponding to the market-sensitive data through a text information extraction module;

[0035] (3) Cluster and fuse the text information through a clustering module to obtain multiple text information groups;

[0036] (4) Extract features from multiple text information data under each group through a target feature extraction module to obtain multiple groups of target feature information;

[0037] (5) Store the multiple groups of target feature information through a feature database for subsequent judgment and analysis of market competition monopoly behavior.

[0038] Among them, the suspected market monopoly competition behavior covers suspected monopoly agreements, failure to declare business concentration according to law, unfair prices, sales below cost, and differential treatment scenarios.

[0039] Preferably, obtaining the text information corresponding to the market-sensitive data in step (2) includes: for text multi-source heterogeneous data, directly obtaining the text information, and for image multi-source heterogeneous data, using an image-to-text conversion tool to convert it into corresponding text information.

[0040] Preferably, the image-to-text conversion tool calls the general scene text recognition API in the "Baidu Smart Cloud" platform by writing a Python script to convert image data into text information.

[0041] The present invention realizes the unification of two heterogeneous data formats, namely text and image, by converting image data into text information, so as to facilitate subsequent data fusion processing, and avoids the problem of increased system complexity caused by separately extracting and fusing features of text data and image data.

[0042] Preferably, in step (3), the clustering and fusion of the text information to obtain multiple text information groups is realized by the K-means clustering algorithm, and its specific implementation steps are as follows:

[0043] (3-1) Assume that the text information corresponding to the market-sensitive data constitutes a data set , given the number of clusters K, set the maximum number of iterations N, and randomly select K data as the initial cluster centers ; where K and N are natural numbers, and the specific values are adjusted according to the actual situation;

[0044] (3-2) Calculate the distance between the remaining different data in the data set D and the cluster center m , where ; is the Euclidean distance or the Manhattan distance;

[0045] If the distance between the data and a certain cluster center satisfies , then the data is divided into ;

[0046] (3-3) If the data in the data set D is divided into a certain category, then adjust the classification and recalculate the cluster center , where r represents the number of data contained in the cluster category ;

[0047] (3-4) Calculate the error criterion function Z. If there is a satisfaction of the error requirement, it means that the clustering criterion function converges, and the K-means algorithm clustering stops here; otherwise, return to step (3-2) to continue clustering again until the clustering iteration is completed. The calculation formula of the error criterion function Z is as follows:

[0048] .

[0049] The principle of the K-means clustering analysis algorithm is to use distance to measure the similarity between different data, that is, the greater the distance between different data, the weaker their similarity. It believes that data with close distances should be merged into the same category, and the ultimate goal of the K-means algorithm is to find appropriate several independent and non-interfering categories. The present invention realizes the classification and fusion of multi-source heterogeneous data through the K-means clustering analysis algorithm.

[0050] Preferably, after obtaining multiple text data groups through the K-means clustering algorithm, in step (4), the target feature extraction module extracts features from multiple text information data under each group to obtain multiple groups of target feature information. The specific implementation is as follows:

[0051] (4-1) Input the text information data under each group into the Word2vec vector generation model to generate multiple word vectors;

[0052] (4-2) Concatenate the word vectors into a two-dimensional matrix according to the sentence order;

[0053] (4-3) Input the two-dimensional matrix into the TextCNN model to extract target feature information.

[0054] Preferably, in step (4-3), the convolutional layer of the TextCNN model uses multiple convolutional kernels of different sizes to extract local features of statements in different ranges, and the sliding of the convolutional kernel only acts in the statement order direction; the pooling layer of the TextCNN model converts the dimension to a fixed length through one-dimensional pooling operation, and finally combines the local features through the fully connected layer to obtain the target feature information of the text data.

[0055] In the TextCNN model, the most important is the convolutional layer. As an example, the model parameters are set so that the convolutional kernel size of its filter is set to 2, 3, and 4 to extract features at different levels. The calculation method is shown in the following formula:

[0056] ;

[0057] Where: represents the activation function. Preferably, the activation function is the ReLU function; represents the node of the filter, represents the filter value; represents the bias term corresponding to the t-th node, represents the input node of the filter corresponding to the t-th node in the output matrix The weight it has. By setting three filters with different convolutional kernels, the purpose of extracting local features is achieved, and finally is the output result of the convolutional layer.

[0058] The core purpose of the pooling layer of TextCNN is to make the model pay more attention to certain features, rather than the specific position of the features, and at the same time reduce the feature vector and the size of the network parameters, so as to achieve the purpose of reducing the dimension. Preferably, the max-pooling method is used to implement the output of the convolutional layer operation Pooling processing. Then, the max-pooling strategy is adopted in the pooling layer to obtain more comprehensive local features.

[0059] In addition, the structural schematic diagram of the storage system for multi-source heterogeneous market-sensitive data of the present invention is shown in Figure 2 .

[0060] According to Figure 2 , the present invention provides a storage system for multi-source heterogeneous market-sensitive data, including a web crawler program module, a text information extraction module, a clustering module, a target feature extraction module, and a feature database; the storage system for multi-source heterogeneous market-sensitive data is used to execute the above-mentioned storage method for multi-source heterogeneous market-sensitive data.

[0061] It should be noted that those skilled in the art should understand that the implementation functions of each module shown in the implementation manner of the storage system for multi-source heterogeneous market-sensitive data can be understood with reference to the relevant descriptions of the storage method for multi-source heterogeneous market-sensitive data. The functions of each module shown in the implementation manner of the storage system for multi-source heterogeneous market-sensitive data can be realized by a program (executable instructions) running on a processor, or can also be realized by specific logic circuits.

[0062] Correspondingly, the present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, the various method embodiments of the present application are realized. The computer-readable storage medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassette tapes, magnetic disk storage, or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0063] In addition, it should be understood that the above are only the preferred embodiments of this specification, and are not used to limit the protection scope of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included within the protection scope of one or more embodiments of this specification.

Claims

1. A storage method for multi-source heterogeneous target data, characterized in that, It includes the following steps: (1) Collect multiple target data through a web crawler program module. The target data is multi-source heterogeneous and at least includes text data and image data; (2) Obtain the text information corresponding to the target data through a text information extraction module; (3) Cluster and fuse the text information through a clustering module to obtain multiple text information groups; (4) Use the TextCNN model in the target feature extraction module to extract features from multiple text information data under each group to obtain multiple groups of target feature information; (5) Store the multiple groups of target feature information in a feature database for subsequent judgment and analysis of market competition monopoly behaviors; among them, the market competition monopoly behaviors cover scenarios of suspected monopoly agreements, failure to declare business concentration in accordance with the law, unfair prices, sales below cost, and differential treatment; Among them, obtaining the text information corresponding to the target data in step (2) includes: for multi-source heterogeneous text data, directly obtain the text information; for multi-source heterogeneous image data, use an image-to-text conversion tool to convert it into corresponding text information; the image-to-text conversion tool calls the general scene text recognition API in the "Baidu Smart Cloud" platform by writing a Python script to convert the image data into text information; Among them, clustering and fusing the text information in step (3) to obtain multiple text information groups is achieved through the K-means clustering algorithm; Among them, in the TextCNN model in step (4), the convolution kernel sizes of the convolution layer filters are set to 2, 3, and 4 to extract features at different levels, and the calculation method is shown in the following formula: ； Wherein: is the ReLU function; represents the node of the filter, represents the filter value; represents the bias term corresponding to the t-th node, represents, in the output matrix, the input node of the filter corresponding to the t-th node weight it has; The purpose of extracting local features is achieved by setting three filters with different convolutional kernels, and finally is the output result of the convolutional layer, and the max pooling method is used to implement the pooling process of the output of the convolutional layer operation.

2. The storage method of multi-source heterogeneous target data according to claim 1, characterized in that, Clustering and fusing the text information in step (3) to obtain multiple text information groups is achieved through the K-means clustering algorithm, and its specific implementation steps are as follows: (3-1) Assume that the text information corresponding to the target data constitutes a data set , given the number of clusters K, set the maximum number of iterations N, and randomly select K data as the initial cluster centers ; (3 - 2) Calculate the distance between the remaining different data in the dataset D and the cluster center m , where ; is the Euclidean distance or the Manhattan distance; If the distance between the data and a certain cluster center satisfies , then the data is classified as ; If the data in the dataset D is classified into a certain category in (3-3), then adjust the classification and recalculate the cluster centers , where r represents the clustering category and the number of data contained in (3-4) Calculate the error criterion function Z. If there is a satisfaction with the error requirement, it means that the clustering criterion function converges, and the K-means algorithm clustering stops here; otherwise, return to step (3-2) to start clustering again until the clustering iteration times end. The calculation formula of the error criterion function Z is as follows: 。 3. The storage method of multi-source heterogeneous target data according to claim 2, characterized in that, After obtaining multiple text data groups through the K-means clustering algorithm, in step (4), the target feature extraction module extracts features from multiple text information data under each group to obtain multiple groups of target feature information, and its specific implementation is as follows: (4-1) Input the text information data under each group into the Word2vec vector generation model to generate multiple word vectors; (4-2) Concatenate the word vectors into a two-dimensional matrix according to the sentence order; (4-3) Input the two-dimensional matrix into the TextCNN model to extract target feature information.

4. The storage method of multi-source heterogeneous target data according to claim 3, characterized in that Among them, in step (4-3), the convolution layer of the TextCNN model uses multiple convolution kernels of different sizes to extract local features of statements in different ranges, and the sliding of the convolution kernel only acts in the statement order direction; the pooling layer of the TextCNN model converts the dimension to a fixed length through a one-dimensional pooling operation, and finally combines the local features through a fully connected layer to obtain the target feature information of the text data.

5. A storage method for multi-source heterogeneous target data according to any one of claims 1-4, characterized in that, The target data is market-sensitive data of an e-commerce platform, and the market-sensitive data is data suspected of market competition monopoly behavior.

6. A storage system for multi-source heterogeneous target data, characterized in that, It includes a web crawler program module, a text information extraction module, a clustering module, a target feature extraction module, and a feature database, and the storage system is used to execute the method according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, Store one or more programs, and the one or more programs cause a computer to execute the method according to any one of claims 1-5.

Images