Unstructured content similarity determination method, apparatus, and electronic device

Abstract

The application provides a non-structured content similarity determination method and device and electronic equipment, which normalizes extracted content obtained from non-structured content to obtain normalized content; sets a label for the normalized content based on a specified feature extracted from the normalized content, performs semantic similarity matching on the normalized content with the set label to obtain a first matching result; performs relationship calculation of metric fusion on the normalized content with the set label based on context correlation of the normalized content to obtain relationship metric information; performs context similarity matching according to the relationship metric information to obtain a second matching result; and performs content similarity matching on the non-structured content according to the normalized content, the label set for the normalized content and the relationship metric information to obtain a third matching result. This method comprehensively utilizes multi-dimensional methods of content analysis to perform similarity matching on non-structured content, so that each dimension method can complement each other, and the accuracy of analysis is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, and in particular to a method and device for determining similarity of unstructured content and an electronic device. BACKGROUND

[0002] In the field of content analysis, rule-based, statistical methods and artificial intelligence methods have gradually developed. According to the expert knowledge of semantic rules and industry knowledge, the content is analyzed by rules, the theme of the content is judged by mathematical statistical models, and the content is analyzed and classified by artificial intelligence according to local features and global features. The method of content analysis is gradually mature and available.

[0003] In the production and operation activities of enterprises, with the collection, transmission, storage, processing, exchange and destruction of business data, files with sensitive information produce multiple versions in the entire cycle of data survival along with the business process. Data collection, identification and content analysis of inventory files have always been a difficulty and focus, especially for unstructured content. There are usually insufficient labels, random storage and security problems such as leakage. In business scenarios such as data classification, document tracing detection and content leakage detection, it is urgent to calculate the similarity of sensitive information in unstructured content to avoid security problems. SUMMARY

[0004] The purpose of the present application is to provide a method and device for determining similarity of unstructured content and an electronic device to determine similarity of unstructured content from multiple dimensions, thereby improving data security.

[0005] In a first aspect, the present application provides a method for determining similarity of unstructured content, comprising: extracting content from the unstructured content to obtain extracted content; normalizing the extracted content to obtain normalized content; extracting specified features from the normalized content, setting labels for the normalized content based on the specified features; performing semantic similarity matching on the normalized content with the set labels to obtain a first matching result; performing relationship calculation of measurement fusion on the normalized content with the set labels based on context relevance of the normalized content to obtain relationship measurement information; performing context similarity matching according to the relationship measurement information to obtain a second matching result; wherein the context relevance includes word frequency, sequence features and long-distance structure patterns; and performing content similarity matching on the unstructured content according to the normalized content, the set labels of the normalized content and the relationship measurement information to obtain a third matching result.

[0006] In an optional implementation, the unstructured content includes one or more of text, pictures, audio and video, information flow, and encryption status; the step of extracting content from the unstructured content to obtain extracted content includes: extracting content from the unstructured content by one or more of preset extraction methods to obtain extracted content; the preset extraction methods include embedded extraction, encryption detection, text extraction, metadata extraction, audio extraction, video frame extraction, thumbnail generation, PDF generation, and OCR processing.

[0007] In an optional implementation, the step of normalizing the extracted content to obtain normalized content includes: defining a model for the extracted content by using a content definition analysis tool or a preset framework, so that the extracted content is normalized in a standard format in the model, and the normalized extracted content is determined as the normalized content.

[0008] In an optional implementation, the step of extracting a specified feature from the normalized content and setting a label for the normalized content based on the specified feature includes: extracting the specified feature from the normalized content based on a preset feature extraction algorithm; the specified feature includes content entity, keyword, theme, relationship information, and attribute information; and at least one label is set for the normalized content according to the extracted specified feature.

[0009] In an optional implementation, after the step of performing relationship calculation on the normalized content with the set label based on the context relevance of the normalized content to obtain relationship measurement information, the method further includes: generating multi-dimensional relationship information of the normalized content with the set label according to a preset quantification method and a specified strategy.

[0010] In an optional implementation, the step of performing relationship calculation on the normalized content with the set label based on the context relevance of the normalized content to obtain relationship measurement information includes: retaining the context relevance of the normalized content, and performing relationship calculation on the multi-dimensional relationship information and the set label of the normalized content to obtain relationship measurement information.

[0011] In an optional implementation, the step of performing content similarity matching on the unstructured content based on the normalized content, the set label of the normalized content, and the relationship measurement information to obtain a third matching result includes: extracting a target feature based on the normalized content, the set label of the normalized content, and the relationship measurement information; performing feature vectorization on the normalized content for the target feature to perform content similarity matching on the vectorized content to obtain the third matching result.

[0012] In a second aspect, the present application provides a non-structured content similarity determination apparatus, comprising: a content extraction module configured to perform content extraction on the non-structured content to obtain extracted content; a normalization module configured to perform normalization processing on the extracted content to obtain normalized content; a tagging module configured to extract a specified feature from the normalized content, and set a tag for the normalized content based on the specified feature; perform semantic similarity matching on the normalized content with the set tag to obtain a first matching result; a feature relationship module configured to perform metric fusion relationship calculation on the normalized content with the set tag based on context correlation of the normalized content to obtain relationship metric information; perform context similarity matching based on the relationship metric information to obtain a second matching result; wherein the context correlation comprises word frequency, sequence feature and long-distance structure pattern; and a feature vectorization module configured to perform content similarity matching on the non-structured content based on the normalized content, the set tag of the normalized content and the relationship metric information to obtain a third matching result.

[0013] In a third aspect, the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions capable of being executed by the processor, and the processor executes the machine executable instructions to implement the non-structured content similarity determination method described above.

[0014] In a fourth aspect, the present application provides a machine readable storage medium, wherein the machine readable storage medium stores machine executable instructions, and when the machine executable instructions are invoked and executed by a processor, the machine executable instructions cause the processor to implement the non-structured content similarity determination method described above.

[0015] The embodiments of the present application bring the following beneficial effects:

[0016] The present application provides a non-structured content similarity determination method, apparatus and electronic device, which first performs content extraction on the non-structured content to obtain extracted content, then performs normalization processing on the extracted content to obtain normalized content, extracts a specified feature from the normalized content, sets a tag for the normalized content based on the specified feature, performs semantic similarity matching on the normalized content with the set tag to obtain a first matching result, then performs metric fusion relationship calculation on the normalized content with the set tag based on context correlation of the normalized content to obtain relationship metric information, performs context similarity matching based on the relationship metric information to obtain a second matching result, and finally performs content similarity matching on the non-structured content based on the normalized content, the set tag of the normalized content and the relationship metric information to obtain a third matching result. This method comprehensively utilizes multi-dimensional methods of content analysis to perform similarity matching on the non-structured content, which can make the various dimensional methods complementary to each other and improve the accuracy of analysis.

[0017] In addition, the application adopts a content similarity method to process sensitive files (equivalent to files containing non-structured content), directly compares a large number of files generated by a business process with historical known file analysis results, calculates the similarity, and directly generates a file sensitivity level. In the scenes of data classification, document tracing detection, content leakage detection, etc., the judgment complexity is greatly reduced.

[0018] The application provides a non-structured content similarity determination method and device and an electronic device,

[0019] Other features and advantages of the present application will be described in the following description, or can be known from the description, or can be determined without doubt, or can be known by implementing the above-mentioned technologies of the present application.

[0020] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0022] Figure 1 A flowchart of a non-structured content similarity determination method provided by an embodiment of the present application is shown in the figure.

[0023] Figure 2 A flowchart of another non-structured content similarity determination method provided by an embodiment of the present application is shown in the figure.

[0024] Figure 3 A structural schematic diagram of a non-structured content similarity determination device provided by an embodiment of the present application is shown in the figure.

[0025] Figure 4 A structural schematic diagram of a non-structured content similarity determination device provided by an embodiment of the present application is shown in the figure.

[0026] Figure 5 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0027] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the drawings in the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0028] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0029] Based on the problems involved in the background art, the embodiments of the present application provide a non-structured content similarity determination method, device and electronic equipment, which can be applied to the scene of similarity matching of non-structured data, especially the filtering scene of sensitive words or illegal words. In order to facilitate the understanding of the present embodiment, first, a non-structured content similarity determination method disclosed by the present embodiment is described in detail, as shown in the following figure, which includes the following specific steps: Figure 1

[0030] Step S102, content extraction is performed on the non-structured content to obtain extracted content.

[0031] The above-mentioned non-structured content includes but is not limited to text, picture, audio and video, information flow, health status and table, etc. The extracted content extracted from the non-structured content includes the content corresponding to the object required for normalization, wherein the object required for normalization can be set according to the research and development requirements, which is not specifically limited here.

[0032] Step S104, the extracted content is normalized to obtain normalized content.

[0033] In specific implementation, the extracted content can be modeled according to the needs of user attention business and information that may affect the business, so as to uniformly and completely represent various types of extracted content, and obtain normalized content. The normalized content can also be understood as content converted to a standard format, so as to provide content definition objects for subsequent content detection and matching, and facilitate subsequent data processing.

[0034] Step S106, extracting a specified feature in the normalized content, setting a label for the normalized content based on the specified feature; performing semantic similarity matching on the normalized content with the set label to obtain a first matching result.

[0035] ​The specified features can include one or more of the following: content entities, keywords, in-text, subjects, and relationships. In a specific implementation, a label corresponding to each specified feature is pre-set, so that at least one label can be set for the normalized content according to the specified features. For example, if the normalized content contains illegal information (e.g., illegal information in the keywords), the label of the normalized content is set as illegal.

[0036] The first matching result is a similarity degree (or a similarity level) obtained by performing semantic similarity matching on the normalized content with a set label and specified content. The specified content can be set according to user requirements, for example, it can be an analysis result of content contained in a historical file, that is, unstructured content. The analysis result corresponding to the historical file can be an analysis result obtained by the present application, which can include a label corresponding to the unstructured content, relationship metric information, and a feature vector.

[0037] In step S108, relationship calculation of metric fusion is performed on the normalized content with a set label based on the context relevance of the normalized content, to obtain relationship metric information. Context similarity matching is performed according to the relationship metric information, to obtain a second matching result.

[0038] The context relevance includes word frequency, sequence features, and long-distance structure patterns. In a specific implementation, the label of the normalized data with a set label is input, metric fusion relationship calculation is performed on the normalized content, to obtain relationship metric information, and the context relevance is retained for context similarity matching on the normalized content and specified content, to obtain a second matching result.

[0039] In step S110, content similarity matching is performed on the unstructured content according to the normalized content, the label set for the normalized content, and the relationship metric information, to obtain a third matching result.

[0040] In a specific implementation, a typical feature is selected from the label set for the normalized content, the relationship metric information, and the specified features, and then the typical feature is vectorized to obtain a vectorized content. Content similarity matching is then performed on the vectorized content and the specified content, to obtain a third matching result.

[0041] The embodiment of the present application provides a non-structured content similarity determination method, which comprises the following steps: firstly, extracting content of non-structured content to obtain extracted content; then, performing normalization processing on the extracted content to obtain normalized content; extracting a specified feature in the normalized content, and setting a label for the normalized content based on the specified feature; performing semantic similarity matching on the normalized content with the label to obtain a first matching result; then, performing relationship calculation of measurement fusion on the normalized content with the label based on context correlation of the normalized content to obtain relationship measurement information; performing context similarity matching according to the relationship measurement information to obtain a second matching result; and finally, performing content similarity matching on the non-structured content according to the normalized content, the label set for the normalized content and the relationship measurement information to obtain a third matching result. The method comprehensively utilizes multi-dimensional methods of content analysis to perform similarity matching on the non-structured content, so that the multi-dimensional methods can complement each other, and the accuracy of analysis is improved.

[0042] The embodiment of the present application also provides another non-structured content similarity determination method, which is implemented based on the method of the above embodiment; the method mainly describes a specific process of extracting content of non-structured content to obtain extracted content (implemented through the following step S202), a specific process of performing normalization processing on the extracted content to obtain normalized content (implemented through the following step S204), and a specific process of setting a label for the normalized content based on a specified feature (implemented through the following steps S206-S208); as shown in the method comprises the following specific steps: Figure 2

[0043] In step S202, one or more of preset extraction methods are used to extract content of the non-structured content to obtain extracted content.

[0044] Specifically, the non-structured content comprises one or more of text, pictures, information flow and encryption status. The preset extraction methods comprise embedded extraction, encryption detection, text extraction, metadata extraction, audio extraction, video frame extraction, thumbnail generation, PDF (Portable Document Format) generation and OCR (Optical Character Recognition) processing. In the specific implementation, different non-structured content can be extracted by using different preset extraction methods, for example, audio extraction and video frame extraction can be used to extract content of audio and video to obtain audio data or video data.

[0045] ​In step S204, the extracted content is defined by a content definition analysis tool or a preset framework, so as to normalize the extracted content in a standard format in the model, and the normalized extracted content is determined as normalized content.

[0046] In the model definition of the content, the information that is needed by the business and may affect the business in the extracted content is mainly focused, so as to normalize the extracted content in a standard format, for example, the extracted content can include text content and description information (which is used to indicate the string time and source of the text, etc.), and can be expressed in the form of key, value attribute pair.

[0047] In step S206, a specified feature is extracted from the normalized content based on a preset feature extraction algorithm.

[0048] In the specific implementation, the specified feature includes content entity, keyword, theme, relationship information and attribute information, etc., and the content entity can include name, number, department and industry, etc. The feature extraction algorithm includes keyword detection, text classification, mark detection, fingerprint detection, and, or, not detection, theme extraction, document element extraction, word segmentation, mark and document range extraction, picture classification, picture seal identification, picture document detection, picture mark detection, picture filtering, entity extraction, keyword extraction, attribute determination and script extraction, etc.

[0049] In step S208, at least one label is set for the normalized content according to the extracted specified feature.

[0050] In step S210, the normalized content with the set label is subjected to semantic similarity matching, and a first matching result is obtained.

[0051] In step S212, the normalized content with the set label is subjected to relationship calculation of measurement fusion based on the context correlation of the normalized content, and relationship measurement information is obtained; and context similarity matching is performed according to the relationship measurement information, and a second matching result is obtained.

[0052] In step S214, the unstructured content is subjected to content similarity matching according to the normalized content, the label set for the normalized content and the relationship measurement information, and a third matching result is obtained.

[0053] The unstructured content similarity determination method uses a content similarity method to process sensitive files (which are equivalent to files containing unstructured content), directly compares a large number of files generated by a business process with historical known file analysis results, calculates the similarity, and directly generates a file sensitivity level. In the scenes of data classification, document tracing detection, content leakage detection, etc., the judgment complexity is greatly reduced.

[0054] The embodiment of the present application also provides another unstructured content similarity determination method, which is implemented based on the method in the above embodiment; the method focuses on the specific process of relationship calculation of the measurement fusion of the normalized content with the label (implemented through step S310) based on the context correlation of the normalized content, and the specific process of content similarity matching of the unstructured content based on the normalized content, the label of the normalized content and the relationship measurement information to obtain the third matching result (implemented through steps S312-S314); as shown in the following formula: Figure 3 The method comprises the following specific steps:

[0055] Step S302: content extraction is performed on the unstructured content to obtain extracted content.

[0056] Step S304: the extracted content is normalized to obtain normalized content.

[0057] Step S306: a specified feature in the normalized content is extracted, a label is set for the normalized content based on the specified feature, and semantic similarity matching is performed on the normalized content with the label to obtain a first matching result.

[0058] Step S308: multi-dimensional relationship information of the normalized content with the label is generated according to a preset quantization method and a specified strategy.

[0059] The multi-dimensional relationship information contains relationships between the specified features and relationships between the labels, and the specified features include content entities, keywords, themes, relationship information, text types and attribute information.

[0060] Step S310: the context correlation of the normalized content is retained, and relationship calculation of the measurement fusion of the multi-dimensional relationship information and the label set for the normalized content is performed to obtain relationship measurement information.

[0061] Step S312: a target feature is extracted based on the normalized content, the label set for the normalized content and the relationship measurement information.

[0062] Step S314: the normalized content is vectorized for the target feature, and content similarity matching is performed on the vectorized content to obtain a third matching result.

[0063] In the specific implementation, a typical feature is selected from the label set for the normalized content, the relationship measurement information and the specified feature, and then the typical feature is vectorized to obtain vectorized content; then the vectorized content is matched with specified content in terms of content similarity to obtain a third matching result. Specifically, the vectorized content can be represented by the following formula:

[0064]

[0065] wherein a1, b1, c1, d1 and e1 represent different typical features respectively.

[0066] The above non-structured content similarity determination method extracts content from the non-structured content, normalizes the extracted content, extracts main feature information from the normalized content and labels the main feature information, performs semantic similarity matching, performs context similarity matching based on the labels of the content, preserves context correlation, performs relationship calculation of measurement fusion, and performs content similarity matching by vectorizing the feature and relationship information of the content. The above normalization, labeling, relationship and vectorization operations are combined to complete multi-dimensional content similarity matching, thereby improving the accuracy of non-structured content analysis.

[0067] Corresponding to the above method embodiment, the embodiment of the present application also provides a non-structured content similarity determination device, as shown in Figure 4 The device comprises:

[0068] A content extraction module 40 is configured to extract content from the non-structured content to obtain extracted content.

[0069] A normalization module 41 is configured to normalize the extracted content to obtain normalized content.

[0070] A labeling module 42 is configured to extract a specified feature from the normalized content, set a label for the normalized content based on the specified feature, and perform semantic similarity matching on the normalized content with the set label to obtain a first matching result.

[0071] A feature relationship module 43 is configured to perform relationship calculation of measurement fusion on the normalized content with the set label based on context correlation of the normalized content to obtain relationship measurement information, and perform context similarity matching based on the relationship measurement information to obtain a second matching result; wherein the context correlation includes word frequency, sequence feature and long-distance structure pattern.

[0072] A feature vectorization module 44 is configured to perform content similarity matching on the non-structured content based on the normalized content, the label set for the normalized content and the relationship measurement information to obtain a third matching result.

[0073] The above non-structured content similarity determination apparatus first extracts content from the non-structured content to obtain extracted content; then normalizes the extracted content to obtain normalized content; extracts specified features from the normalized content, and sets a label for the normalized content based on the specified features; performs semantic similarity matching on the normalized content with the set label to obtain a first matching result; then performs relationship calculation of metric fusion on the normalized content with the set label based on the context correlation of the normalized content to obtain relationship metric information; performs context similarity matching according to the relationship metric information to obtain a second matching result; and finally performs content similarity matching on the non-structured content according to the normalized content, the set label of the normalized content, and the relationship metric information to obtain a third matching result. This way, the multi-dimensional method of content analysis is comprehensively utilized to perform similarity matching on the non-structured content, so that the advantages of each dimension method are complementary, and the accuracy of the analysis is improved.

[0074] In a specific implementation, the non-structured content includes one or more of text, pictures, audio and video, information flow, and encryption status; the content extraction module 40 is configured to extract content from the non-structured content by one or more of preset extraction methods to obtain extracted content; and the preset extraction methods include embedded extraction, encryption detection, text extraction, metadata extraction, audio extraction, video frame extraction, thumbnail generation, PDF generation, and OCR processing.

[0075] Further, the normalization module 41 is further configured to define a model for the extracted content by a content definition analysis tool or a preset framework, so that the extracted content is normalized in a standardized format in the model, and the normalized extracted content is determined as the normalized content.

[0076] Further, the labeling module 42 is further configured to extract specified features from the normalized content based on a preset feature extraction algorithm, wherein the specified features include content entities, keywords, topics, relationship information, and attribute information; and set at least one label for the normalized content according to the extracted specified features.

[0077] In a specific implementation, the apparatus further includes a relationship determination module configured to, after obtaining the relationship metric information by performing relationship calculation of metric fusion on the normalized content with the set label based on the context correlation of the normalized content, generate multi-dimensional relationship information of the normalized content with the set label according to a preset quantification method and a specified strategy.

[0078] Further, the feature relationship module 43 is configured to retain the context correlation of the normalized content, and perform relationship calculation of metric fusion on the multi-dimensional relationship information and the set label of the normalized content to obtain relationship metric information.

[0079] In some embodiments, the feature vectorization module 44 is further configured to: extract target features based on the normalized content, the labels of the normalized content settings, and the relationship measurement information; and perform feature vectorization on the normalized content with respect to the target features to perform content similarity matching on the vectorized content to obtain a third matching result.

[0080] The unstructured content similarity determination apparatus provided by the embodiments of the present application has the same implementation principle and technical effects as the foregoing method embodiments. For brevity, the part of the apparatus embodiments not mentioned in the foregoing method embodiments can be referred to the corresponding content in the foregoing method embodiments.

[0081] The embodiments of the present application further provide an electronic device, as shown in the accompanying drawings, which comprises a processor 101 and a memory 100, the memory 100 storing machine executable instructions capable of being executed by the processor 101, and the processor 101 executes the machine executable instructions to implement the foregoing unstructured content similarity determination method. Figure 5

[0082] Further, Figure 5 The electronic device shown in the accompanying drawings further comprises a bus 102 and a communication interface 103, and the processor 101, the communication interface 103 and the memory 100 are connected through the bus 102.

[0083] The memory 100 can include a high-speed random access memory (RAM) and can also include a non-volatile memory such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used. The bus 102 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of brevity, Figure 5 In the accompanying drawings, only one bidirectional arrow is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0084] ​The processor 101 can be an integrated circuit chip with processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 101 or the instruction in the form of software. The processor 101 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiment of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiment of the present application can be directly embodied as a hardware code processor to execute, or be executed by a combination of hardware and software modules in the code processor. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100, and combines the hardware to complete the steps of the method of the above embodiment.

[0085] The embodiment of the present application also provides a machine readable storage medium, the machine readable storage medium stores machine executable instructions, when the machine executable instructions are called and executed by the processor, the machine executable instructions cause the processor to realize the above-mentioned unstructured content similarity determination method. For specific implementation, please refer to the method embodiment, which will not be repeated here.

[0086] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0087] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit them. The protection scope of the present application is not limited to this. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for unstructured content similarity determination, the method comprising: The method comprises: content extraction is performed on the unstructured content to obtain extracted content; wherein the unstructured content comprises one or more of text, pictures and audio and video; the step of performing content extraction on the unstructured content to obtain extracted content comprises: performing content extraction on the unstructured content by one or more of preset extraction methods to obtain the extracted content; wherein the preset extraction methods comprise embedded extraction, encryption detection, text extraction, metadata extraction, audio extraction, video frame extraction, thumbnail generation, PDF generation and OCR processing; the extracted content is normalized to obtain normalized content; specified features in the normalized content are extracted, and tags are set for the normalized content based on the specified features; semantic similarity matching is performed on the normalized content with the set tags to obtain a first matching result; the first matching result is the degree of similarity obtained after the normalized content with the set tags is subjected to semantic similarity matching with specified content; the specified content refers to an analysis result of content contained in a historical file, i.e., tags, relationship measurement information and feature vectors corresponding to the unstructured content; based on the context relevance of the normalized content, relationship calculation of measurement fusion is performed on the normalized content with the set tags to obtain relationship measurement information; multi-dimensional relationship information of the normalized content with the set tags is generated according to a preset quantization method and a specified strategy; wherein the context relevance is retained, and relationship calculation of measurement fusion is performed on the multi-dimensional relationship information and the tags set for the normalized content to obtain relationship measurement information; context similarity matching is performed according to the relationship measurement information to obtain a second matching result; wherein the context relevance comprises word frequency, sequence features and long-distance structure patterns; content similarity matching is performed on the unstructured content according to the normalized content, the tags set for the normalized content and the relationship measurement information to obtain a third matching result; the content similarity matching on the unstructured content according to the normalized content, the tags set for the normalized content and the relationship measurement information to obtain a third matching result comprises: selecting typical features from the normalized content, the tags set for the normalized content and the relationship measurement information; vectorizing the typical features to obtain vectorized content; performing content similarity matching on the vectorized content and the specified content to obtain a third matching result.

2. The method of claim 1, wherein, the step of normalizing the extracted content to obtain normalized content comprises: a model definition is performed on the extracted content by a content definition analysis tool or a preset framework, so that the extracted content is normalized in a standard format in the model; the extracted content in the normalized representation is determined as the normalized content.

3. The method of claim 1, wherein, the step of extracting specified features in the normalized content and setting tags for the normalized content based on the specified features comprises: extract the specified feature from the normalized content based on a preset feature extraction algorithm; wherein the specified feature comprises a content entity, a keyword, a theme, relationship information and attribute information; set at least one label for the normalized content according to the extracted specified feature.

4. The method of claim 1, wherein, The step of performing content similarity matching on the unstructured content according to the normalized content, the label set for the normalized content and the relationship measurement information to obtain a third matching result comprises: extracting a target feature based on the normalized content, the label set for the normalized content and the relationship measurement information; performing feature vectorization on the normalized content for the target feature to perform content similarity matching on the vectorized content to obtain a third matching result.

5. An unstructured content similarity determination apparatus characterized by comprising: The device comprises: a content extraction module configured to perform content extraction on unstructured content to obtain extracted content; wherein the unstructured content comprises one or more of text, pictures and audio and video; the content extraction module is configured to perform content extraction on the unstructured content by one or more of preset extraction methods to obtain the extracted content; wherein the preset extraction methods comprise embedded extraction, encryption detection, text extraction, metadata extraction, audio extraction, video frame extraction, thumbnail generation, PDF generation and OCR processing; a normalization module configured to perform normalization processing on the extracted content to obtain normalized content; a labeling module configured to extract a specified feature from the normalized content, set a label for the normalized content based on the specified feature, perform semantic similarity matching on the normalized content with the set label to obtain a first matching result; the first matching result is the degree of similarity obtained after performing semantic similarity matching on the normalized content with the set label and specified content; the specified content refers to an analysis result of content contained in a historical file, i.e. a label, relationship measurement information and feature vector corresponding to unstructured content; a feature relationship module configured to perform relationship calculation of measurement fusion on the normalized content with the set label based on context correlation of the normalized content to obtain relationship measurement information, generate multi-dimensional relationship information of the normalized content with the set label according to a preset quantization method and a specified strategy; wherein the context correlation comprises word frequency, sequence feature and long-distance structure pattern; perform relationship calculation of measurement fusion on the multi-dimensional relationship information and the label set for the normalized content to obtain relationship measurement information; perform context similarity matching according to the relationship measurement information to obtain a second matching result; a feature vectorization module configured to perform content similarity matching on the unstructured content according to the normalized content, the label set for the normalized content and the relationship measurement information to obtain a third matching result; The feature vectorization module is specifically configured to: select a typical feature from the normalized content, the label set for the normalized content and the relationship measurement information; vectorizing the typical features to obtain vectorized content; performing content similarity matching on the vectorized content and the specified content to obtain a third matching result.

6. An electronic device, comprising: The electronic device includes a processor and a memory, the memory stores machine executable instructions capable of being executed by the processor, and the processor executes the machine executable instructions to implement the unstructured content similarity determination method of any one of claims 1 to 4.

7. A machine-readable storage medium, characterized in that, The machine readable storage medium stores machine executable instructions, and when the machine executable instructions are called and executed by the processor, the machine executable instructions cause the processor to implement the unstructured content similarity determination method of any one of claims 1 to 4.

Images