Content management method and device

Abstract

The invention discloses a content management method and device, and relates to the technical field of artificial intelligence. A specific embodiment of the content management method comprises the steps of determining a data type of each target content in response to receiving a plurality of target contents, and determining an embedding method suitable for each target content according to the data type; executing a corresponding embedding method on each target content by using a preset Chinese embedding model to obtain an embedding vector of each target content; and determining the similarity among the plurality of target contents according to the embedded vector, and de-duplicating the plurality of target contents according to the similarity. According to the embodiment, the Chinese embedding model is used for executing the corresponding embedding method on the target contents, then the similarity among the multiple target contents is determined according to the embedding vector output by the Chinese embedding model, duplicate removal is carried out on the target contents with high similarity, the content management efficiency can be improved, and the labor cost can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for content management. Background Art

[0002] The content platform has content in multiple fields, such as health science, travel notes, etc. The content of the content platform comes from many sources, including: original generation by users of the content platform, purchase and introduction from outside the platform, etc. Therefore, the content platform usually has duplicate content or content with high similarity. Duplicate or highly similar content increases the difficulty of content management, and it is necessary to deduplicate the content on the content platform. When deduplicating content, it is usually manually labeled, classified, reindexed, and similarity judged, or external interfaces are called to find duplicate or highly similar content.

[0003] In the process of implementing the present invention, the inventors found that the prior art has at least the following problems:

[0004] Manual deduplication consumes a lot of manpower costs, leads to low content management efficiency, and unstable external interfaces, which also reduces the efficiency of content management. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and apparatus for content management, which can improve content management efficiency and reduce labor costs.

[0006] To achieve the above object, according to a first aspect of an embodiment of the present invention, a content management method is provided, comprising:

[0007] In response to receiving a plurality of target contents, determining a data type of each target content, and determining an embedding method applicable to each target content according to the data type;

[0008] Use the preset Chinese embedding model to perform the corresponding embedding method on each target content to obtain the embedding vector of each target content;

[0009] The similarities between the multiple target contents are determined according to the embedding vectors, and the multiple target contents are deduplicated according to the similarities.

[0010] Optionally, before using the preset Chinese embedding model to execute the corresponding embedding method on each target content, the method further includes: determining a content segmentation strategy applicable to each target content respectively; segmenting the corresponding target content according to the content segmentation strategy;

[0011] Using a preset Chinese embedding model to execute a corresponding embedding method on each target content includes: using a preset Chinese embedding model to execute a corresponding embedding method on a segmentation result of each target content.

[0012] Optionally, segmenting the corresponding target content according to the content segmentation strategy includes:

[0013] Dividing the target content into a plurality of non-overlapping target sub-contents;

[0014] Determine the shared content of each target sub-content;

[0015] According to the arrangement order of the multiple target sub-contents, the shared content of the previous target sub-content is added to the head of each target sub-content, and / or the shared content of the next target sub-content is added to the tail of each target sub-content.

[0016] Optionally, the embedding vector of each target content includes: embedding vectors of multiple segmentation results of each target content; before determining the similarity between the multiple target contents according to the embedding vectors, the method further includes:

[0017] The embedding vectors of multiple segmentation results for each target content are merged into a single vector.

[0018] Optionally, the embedding vectors of multiple segmentation results of each target content are merged into a single vector, including:

[0019] Determine a segment weight for each segment result of each target content;

[0020] weighting the corresponding embedding vector according to the segment weight;

[0021] The weighted embedding vectors are merged to obtain the weighted average vector.

[0022] Optionally, the method further comprises:

[0023] storing the embedding vector of each target content and the similarities between multiple target contents in a distributed system;

[0024] In response to receiving the target content set, the distributed system is queried for similarities between a plurality of target contents included in the target content set.

[0025] Optionally, querying the distributed system for similarities between a plurality of target contents included in the target content set includes:

[0026] parsing query conditions associated with the target content set;

[0027] Screening multiple target contents included in the target content set according to the query condition;

[0028] The similarity between the filtered target contents is queried from the distributed system.

[0029] According to a second aspect of an embodiment of the present invention, there is provided a content management device, including:

[0030] a determination module, configured to determine a data type of each target content in response to receiving a plurality of target contents, and determine an embedding method applicable to each target content according to the data type;

[0031] An embedding module, used to use a preset Chinese embedding model to execute a corresponding embedding method on each target content to obtain an embedding vector for each target content;

[0032] A deduplication module is used to determine the similarity between the multiple target contents according to the embedding vector, and to dedupe the multiple target contents according to the similarity.

[0033] Optionally, the device further comprises: a policy module, configured to respectively determine a content segmentation strategy applicable to each target content; a segmentation module, configured to segment the corresponding target content according to the content segmentation strategy;

[0034] Using a preset Chinese embedding model to execute a corresponding embedding method on each target content includes: using a preset Chinese embedding model to execute a corresponding embedding method on a segmentation result of each target content.

[0035] Optionally, segmenting the corresponding target content according to the content segmentation strategy includes:

[0036] Dividing the target content into a plurality of non-overlapping target sub-contents;

[0037] Determine the shared content of each target sub-content;

[0038] According to the arrangement order of the multiple target sub-contents, the shared content of the previous target sub-content is added to the head of each target sub-content, and / or the shared content of the next target sub-content is added to the tail of each target sub-content.

[0039] Optionally, the embedding vector of each target content includes: the embedding vectors of multiple segmentation results of each target content; the device further includes:

[0040] A merging module is used to merge the embedding vectors of multiple segmentation results of each target content into a single vector.

[0041] Optionally, the embedding vectors of multiple segmentation results of each target content are merged into a single vector, including:

[0042] Determine a segment weight for each segment result of each target content;

[0043] weighting the corresponding embedding vector according to the segment weight;

[0044] The weighted embedding vectors are merged to obtain the weighted average vector.

[0045] Optionally, the device further comprises:

[0046] A storage module, used for storing the embedding vector of each target content and the similarity between multiple target contents in a distributed system;

[0047] The query module is configured to query the distributed system for similarities between a plurality of target contents included in the target content set in response to receiving the target content set.

[0048] Optionally, querying the distributed system for similarities between a plurality of target contents included in the target content set includes:

[0049] parsing query conditions associated with the target content set;

[0050] Screening multiple target contents included in the target content set according to the query condition;

[0051] The similarity between the filtered target contents is queried from the distributed system.

[0052] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0053] one or more processors;

[0054] a storage device for storing one or more programs,

[0055] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any one of the above embodiments.

[0056] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described in any one of the above embodiments is implemented.

[0057] An embodiment of the above invention has the following advantages or beneficial effects: using a Chinese embedding model to execute a corresponding embedding method on a target content, and then determining the similarity between multiple target contents according to the embedding vector output by the Chinese embedding model, and removing duplicates of target contents with high similarity, which can improve content management efficiency and reduce labor costs; before executing the embedding method, segmenting the target content according to the content segmentation strategy can ensure that the segmented content meets the input conditions of different Chinese embedding models and improve the flexibility of content input; when segmenting, the segmentation results retain the contextual semantics before and after the segmentation, which can ensure that the segmented content has complete and coherent semantics, and avoid the Chinese embedding model not understanding the segmented content or generating ambiguity; merging multiple embedding vectors can reduce the storage space of the embedding vector and improve the processing efficiency of the embedding vector; storing the embedded target content and similarity in a distributed system, providing similarity query services to the outside, which can improve the query efficiency of the target content similarity, and store content of multiple data types together to reduce storage costs; meeting different query requirements according to query conditions can improve the accuracy of similarity queries, more flexibly query the similarity between different contents, and save query time.

[0058] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0060] Figure 1 is a schematic diagram of the main process of the content management method according to an embodiment of the present invention;

[0061] Figure 2 is a schematic diagram of the overall process of content management according to a reference embodiment of the present invention;

[0062] Figure 3 is a schematic diagram of the main process of generating an embedding vector according to a reference embodiment of the present invention;

[0063] Figure 4 is a schematic diagram of the relationship between content categories and repeated content according to a reference embodiment of the present invention;

[0064] Figure 5 is a schematic diagram of the main process of content segmentation according to a reference embodiment of the present invention;

[0065] Figure 6 is a schematic diagram of the main process of content management according to a reference embodiment of the present invention;

[0066] Figure 7is a schematic diagram of main modules of a content management apparatus according to an embodiment of the present invention;

[0067] Figure 8 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0068] Fig. 9 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0070] It should be noted that in the technical solution of the present invention, the collection, use, storage, sharing and transfer of user personal information involved are in compliance with the provisions of relevant laws and regulations, and it is necessary to inform the user and obtain the user's consent or authorization. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted.

[0071] The content platform has content in multiple fields, such as health science, travel notes, etc. The content of the content platform comes from many sources, including: original generation by users of the content platform, purchase and introduction from outside the platform, etc. Therefore, the content platform usually has duplicate content or content with high similarity. Duplicate or highly similar content increases the difficulty of content management, and it is necessary to deduplicate the content on the content platform. When deduplicating content, it is usually manually labeled, classified, reindexed, and similarity judged, or external interfaces are called to find duplicate or highly similar content.

[0072] Manual deduplication consumes a lot of manpower costs and has low content management efficiency. Manual deduplication may result in misjudgment and wrong judgment, which reduces the accuracy of content deduplication. The external interface is unstable, and connections are often disconnected or requests are blocked, which also reduces the efficiency of content management. In addition, the cost of calling the external interface is too high.

[0073] In view of this, according to a first aspect of an embodiment of the present invention, a content management method is provided.

[0074] Figure 1 FIG. 1 is a schematic diagram of the main process of the content management method according to an embodiment of the present invention. Figure 1As shown, the content management method according to the embodiment of the present invention mainly includes the following steps S101 to S103.

[0075] Step S101 : in response to receiving a plurality of target contents, determining a data type of each target content, and determining an embedding method suitable for each target content according to the data type.

[0076] The target content is the content of the content platform. The user selects a part of the content from the content platform as the target content and sends it to the execution subject of the embodiment of the present invention, or the execution subject of the embodiment of the present invention directly selects the target content from the content platform. The target content can belong to one or more content fields, for example, the financial field, shopping field, logistics field, medical field, etc. Multiple target contents can come from the same content field or from multiple content fields.

[0077] After receiving the target content, determine the data type of the target content. For example, the execution subject of the embodiment of the present invention receives multiple target contents in the form of files, determines the suffix of each file in turn according to the order of reception, and determines the data type of the target content included in the file according to the suffix of the file; for example, when the suffix of the file is "doc", "docx", or "txt", the data type of the target content is determined to be text, and when the suffix of the file is "jpg", "png", or "jpeg", the data type of the target content is determined to be image. For another example, the target content has a content tag, and the content tag is used to indicate the data type of the target content. The execution subject of the embodiment of the present invention searches for the content tag of the target content and determines the data type of the target content according to the content tag. For example, when the content tag is "wenben", the data type of the target content is determined to be text, and when the content tag is "tuxiang", the data type of the target content is determined to be image.

[0078] Pre-set corresponding embedding methods for different data types, establish an association relationship between data types and embedding methods, and the embedding method is used to convert the target content into a vector form, which is easy to process. For example, the pre-set embedding methods include: openai.Embed, openai.Davinci, etc., among which the embedding method "openai.Embed" is associated with the target content of text type and is applicable to the single vector mode, and the embedding method "openai.Davinci" is associated with the target content of image type and is applicable to the multi-task mode. The execution subject of the embodiment of the present invention determines the embedding method corresponding to the data type of each target content and applicable to each target content according to the association relationship between the data type and the embedding method.

[0079] Exemplarily, the execution subject of the embodiment of the present invention receives three files, takes each file as a target content, and the suffixes of the three files are all "doc", then it is determined that the data types of the three target contents are all text; according to the association between the data type and the embedding method, it is determined that the embedding method applicable to the three target contents is "openai.Embed".

[0080] By determining the embedding method suitable for the target content according to the data type of the target content, a paradigm for converting the target content into a vector can be established, the efficiency of converting the target content into a vector can be improved, the labor cost can be reduced, and manual labeling and indexing of the target content can be avoided.

[0081] Step S102: Use a preset Chinese embedding model to execute a corresponding embedding method on each target content to obtain an embedding vector for each target content.

[0082] A Chinese embedding model is pre-set, and the Chinese embedding model can simultaneously process data of multiple modalities (i.e., target content of multiple data types), such as text, image, voice, etc. The Chinese embedding model applicable to the embodiment of the present invention includes: M3E (Moka Massive Mixed Embedding) model. Specifically, the M3E model is open source, and the M3E model is obtained from the Internet to the local computer. The M3E model is fine-tuned according to the business requirements of different content fields to obtain multiple fine-tuned M3E models applicable to different content fields.

[0083] Send a request to the Chinese embedding model so that it executes the corresponding embedding method for each target content. Specifically, the embedding methods "openai.Embed", "openai.Davinci", etc. are all OpenAI's application interfaces. When converting the target content into an embedding vector, first call OpenAI's application interface to send a request to the Chinese embedding model, and then the Chinese embedding model converts the target content into an embedding vector. For example, the request generated by OpenAI's application interface carries the target content. The Chinese embedding model parses the received request to obtain the target content, performs feature extraction and vector generation on the target content, and converts the target content into an embedding vector. It should be noted that the content input into the Chinese embedding model is a string. For example, when the target content is an image, the image is first converted into a string, and then the string representing the image is input into the Chinese embedding model.

[0084] When only OpenAI is used to convert the target content into vectors, network disconnection or network delay often occurs, which greatly reduces the efficiency of the target content's vector conversion, and the budget cost of using OpenAI is high. Using the Chinese embedding model to convert the target content into an embedded vector can reduce the cost and overhead of vector conversion and improve the efficiency of vector conversion; fine-tuning the Chinese embedding model based on different content fields can further improve the accuracy of vector conversion.

[0085] According to a reference embodiment of the present invention, before using a preset Chinese embedding model to execute a corresponding embedding method for each target content, the method further includes: determining a content segmentation strategy applicable to each target content, respectively, and the content segmentation strategy is used to represent the segmentation method of the target content. In the case where the Chinese embedding model has restrictions on the length of the input content, it is necessary to segment the target content whose length is greater than the input length threshold of the Chinese embedding model. The content segmentation strategy includes: dividing the target content into a plurality of segmented contents of equal length, so that the length of each segmented content of the target content is less than the input length threshold of the Chinese embedding model, or, in the case where the target content is text, taking each sentence and each paragraph of the target content as a segmented content, and ensuring that the length of each sentence or each paragraph is less than the input length threshold of the Chinese embedding model, or in the case where the target content is an image, dividing the target content into a plurality of segmented contents of equal area, and then converting each segmented content into a character string, so that the length of the character string is less than the input length threshold of the Chinese embedding model.

[0086] The corresponding target content is segmented according to the determined content segmentation strategy to obtain the segmentation result of each target content. When the corresponding embedding method is executed on each target content using the preset Chinese embedding model, the corresponding embedding method is executed on the segmentation result of each target content using the preset Chinese embedding model, that is, the segmentation result of each target content is converted into an embedding vector using the Chinese embedding model, and each segmentation result of the target content corresponds to an embedding vector.

[0087] Exemplarily, the input rule of the Chinese embedding model is "the string input into the model each time does not exceed 512 characters", that is, the length of the target content is less than or equal to 512. When the length of the target content is greater than 512, the target content is segmented; specifically, when the target content is text, the target content is divided according to the periods included in the target content, and the content between every two periods is a segmentation result; when the target content is an image, the image is divided into multiple sub-images of equal area, and each sub-image is converted into a string. When the length of the string is greater than 512, the string is segmented again until the length of the segmented string is less than 512, and the string with a length less than 512 is used as the segmentation result.

[0088] Segmenting the target content according to the content segmentation strategy can ensure that the segmented content meets the input conditions of the Chinese embedding model, ensure that the embedding vector is obtained quickly and accurately, set a variety of content segmentation strategies, and improve the flexibility of content input.

[0089] According to another reference embodiment of the present invention, when the corresponding target content is segmented according to the content segmentation strategy, the target content is first divided into a plurality of non-repeating target sub-contents, and all the target sub-contents are spliced ​​to obtain the complete target content. Then the shared content of each target sub-content is determined, and the shared content is a part of the target sub-content. The shared content includes most of the features of the target sub-content. For example, when the target sub-content is text, the shared content is the content including the explanation of terms in the target sub-content. When the target sub-content is an image, the shared content is the content with more color types in the target sub-content. The target sub-content may include one or more shared contents.

[0090] According to the arrangement order of multiple target sub-contents, the shared content of the previous target sub-content is added to the head of each target sub-content, and the shared content of the next target sub-content is added to the tail of each target sub-content. Exemplarily, the text-type target contents are divided into target sub-content A1, target sub-content A2 and target sub-content A3; the shared content B1 of target sub-content A1 is added to the head of target sub-content A2, wherein the shared content B1 is the text of the tail part of target sub-content A1; the shared content B2 of target sub-content A2 is added to the tail of target sub-content A1, and the shared content B3 of target sub-content A2 is added to the head of target sub-content A3, wherein the shared content B2 is the text of the starting part of target sub-content A2, and the shared content B3 is the text of the ending part of target sub-content A2; the shared content B4 of target sub-content A3 is added to the tail of target sub-content A2, wherein the shared content B4 is the text of the starting part of target sub-content A3.

[0091] When segmenting, the segmentation results retain the contextual content before and after the segmentation, which can ensure that the segmented content has complete and coherent semantics, avoid the Chinese embedding model not understanding the segmented content or causing ambiguity, and improve the efficiency and accuracy of vector conversion.

[0092] Step S103: determining similarities between the multiple target contents according to the embedding vectors, and removing duplicates from the multiple target contents according to the similarities.

[0093] Receive the embedding vector output by the Chinese embedding model, each target content corresponds to an embedding vector, and calculate the similarity between multiple target contents based on the embedding vector. Specifically, select two target contents arbitrarily from the target content, obtain the embedding vectors of the two target contents, bring the two obtained embedding vectors into the calculation formulas such as Euclidean distance, cosine similarity, Hamming distance, etc., and calculate the similarity of the above two target contents. When determining the similarity between the target contents, compare the similarity with the pre-set similarity threshold, and use the similarity greater than the similarity threshold as the target similarity, determine the similar target content corresponding to the target similarity, determine the target content that needs to be retained from the similar target content, and remove the content similar to the target content that needs to be retained.

[0094] Exemplarily, the similarity between target content C1 and target content C2 is 0.3, the similarity between target content C1 and target content C3 is 0.4, and the similarity between target content C2 and target content C3 is 0.9; the pre-set similarity threshold is 0.5, and since the similarity between target content C2 and target content C3 is greater than the similarity threshold, it means that target content C2 is relatively similar to target content C3; since target content C2 is original content created by users of the content platform, and target content C3 is content purchased from outside the content platform, target content C3 is removed.

[0095] It should be noted that, in addition to determining the similarities between different target contents, the execution subject of the embodiment of the present invention may also perform tasks such as clustering and classifying the target contents according to the embedding vectors of the target contents.

[0096] Determining the similarity between multiple target contents and deduplicating target contents with high similarity can improve the deduplication efficiency and accuracy of target contents and reduce labor costs.

[0097] According to a reference embodiment of the present invention, when the target content input into the Chinese embedding model is segmented, the Chinese embedding model executes the corresponding embedding method on each segmentation result of the target content, and converts each segmentation result into a corresponding embedding vector. Therefore, the embedding vector of each target content includes: the embedding vector of each segmentation result of the target content, that is, each segmentation result of the target content corresponds to an embedding vector.

[0098] Before determining the similarity between multiple target contents according to the embedding vector, the method also includes: merging the embedding vectors of multiple segmentation results of each target content into a single vector. Exemplarily, the embedding vectors of multiple segmentation results of the same target content are added one by one, and then divided by the number of segmentation results to calculate the average vector of the multiple embedding vectors. The above operation can evenly integrate the semantic information of each segmentation result into the average vector. For example, vecs is a list of embedding vectors containing n segmentation results, and the embedding vectors of n segmentation results are merged into an average vector avg_vector, avg_vector = sum(vecs(i)) / len(vecs), where "vecs(i)" represents the embedding vector of the i-th segmentation result, "sum(vecs(i))" represents the sum of the embedding vectors of multiple segmentation results of the same target content, and "len(vecs)" represents the number of segmentation results.

[0099] As another example, the embedding vectors of multiple segmentation results of the same target content are concatenated together in the segmentation order to obtain a concatenated vector. The above operation retains the independence of each segment. For example, vector is a list containing multiple segmentation vectors, concat_vector = np.concatenate(vector,axis = 0), where "concat_vector" represents the concatenated vector obtained after concatenation, "np.concatenate" represents the function method used to concatenate the vector, and "axis = 0" means "concatenate vectors by axis axis", and the default value of axis is 0.

[0100] It should be noted that when merging embedding vectors, it is necessary to ensure that the dimensions of the embedding vectors of each segmentation result are the same. When the dimensions of the embedding vectors are different, the embedding vectors are dimensionalized, for example, by dimensionality reduction, so that the embedding vectors can be merged. According to the specific application scenario and requirements, select an appropriate merging strategy. The method of using average merging to obtain an average vector is suitable for situations where you want to evenly fuse the segmented semantic information. Concatenating the embedding vectors retains the independence of each segment.

[0101] Merging multiple embedded vectors can reduce the storage space of the embedded vectors and improve the processing efficiency of the embedded vectors.

[0102] According to another reference embodiment of the present invention, when merging the embedding vectors of multiple segmentation results of each target content into a single vector, first determine the segmentation weight of each segmentation result of each target content. The segmentation weight is related to factors such as the content field of the target content, the length of the segmentation result, and the types of characters included in the segmentation result. The segmentation weight of the segmentation result can also be pre-set, and the segmentation weight of the segmentation result is determined according to the generation order of the segmentation results. The corresponding embedding vector is weighted according to the segmentation weight, that is, the segmentation weight is multiplied by the segmentation result. The weighted embedding vectors are merged. Specifically, the weighted embedding vectors are added to obtain a weighted embedding vector, the segmentation weights are summed, and the weighted embedding vector is divided by the segmentation weight to obtain a weighted average vector.

[0103] Exemplarily, the segmentation weight of the segmentation result E1 is F1, the segmentation weight of the segmentation result E2 is F2, and the segmentation weight of the segmentation result E3 is F3. The segmentation results are weighted according to the segmentation weights, and the weighted segmentation results include: E1*F1, E2*F2, E3*F3. The segmentation weights are summed to obtain F1+F2+F3. The sum of the weighted segmentation results is divided by the sum of the segmentation weights to obtain the weighted average vector: (E1*F1+E2*F2+E3*F3) / (F1+F2+F3).

[0104] Weighted average vector merging is suitable for assigning weights according to the importance of segmentation results, so that more important segmentation results have a greater impact on the merged weighted average vector; merging embedded vectors can reduce the storage space of embedded vectors and improve the processing efficiency of embedded vectors.

[0105] According to another reference embodiment of the present invention, the method further includes: storing the embedding vector of each target content and the similarity between multiple target contents in a distributed system, a relational database or a cache. Among them, the distributed system applicable to the embodiment of the present invention includes: Vearch, which is an elastic distributed system for high-performance similarity search of large-scale deep learning vectors. The user can send a similarity query request to the execution subject of the embodiment of the present invention, and the execution subject of the embodiment of the present invention parses the similarity query request to obtain a target content set, which is a set of target contents for which similarity needs to be queried, for example, the target content set includes content encodings of multiple target contents; the execution subject of the embodiment of the present invention queries the similarity between multiple target contents included in the target content set from the distributed system, specifically, determines the target content to be queried according to the content encoding included in the target content set, and queries the similarity between the target contents to be queried.

[0106] Exemplarily, the segmentation results, embedding vectors, content codes, and similarities between multiple target contents of the target content are stored in Vearch; the corresponding content code is determined according to the received query request, the corresponding target content is determined according to the content code, and the similarity associated with the target content is queried in Vearch. In the case of multiple target contents, the similarity between the multiple target contents is queried.

[0107] Storing the embedded target content and similarity in a distributed system and providing similarity query services to the outside world can improve the query efficiency of target content similarity, store content, vectors, and similarities of multiple data types together, and reduce storage costs.

[0108] According to another reference embodiment of the present invention, when querying the similarity between multiple target contents included in the target content set from a distributed system, a query statement sent by a user is received, each distributed system has a different query statement, and the query condition is parsed from the query statement. For example, when the distributed system is Vearch, the received query statement is "curl-H "content-type:application / json"-XPOST–d "{...}"", where the parameters in the brackets "{...}" are query conditions, specifically, the query conditions include: numerical filtering conditions, field sorting conditions, return result quantity conditions, etc.

[0109] The multiple target contents included in the target content set are screened according to the query conditions, specifically, the query conditions include: the length of the target content is greater than or equal to the query length, the content field of the target content is the same as the content field included in the pre-query conditions, etc. The similarity between the screened target contents is queried from the distributed system. Exemplarily, the received query request is parsed to obtain content coding G1, content coding G2, content coding G3 and query conditions, wherein content coding G1 corresponds to target content H1, content coding G2 corresponds to target content H2, and content coding G3 corresponds to target content H3; the query conditions include: querying the similarity of target contents belonging to content field I1, wherein target content H1 and target content H2 belong to content field I1, and target content H3 belongs to content field I2, therefore, querying the similarity between target content H1 and target content H2.

[0110] Meeting different query requirements according to query conditions can improve the accuracy of similarity queries, more flexibly query the similarities between different contents, and save query time.

[0111] Figure 2 FIG. 1 is a schematic diagram of the overall process of content management according to a reference embodiment of the present invention. Figure 2As shown, the target content is selected from the content platform's own content (for example, user-generated content of the content platform) and externally purchased content, the target content is segmented, and the segmentation results of the target content are input into the Chinese embedding model. Specifically, the application interface of OpenAI is called to send a request to the M3E model, and the M3E model is used to execute the corresponding embedding method on the segmentation results of the target content to obtain the embedding vector of each segmentation result of the target content, and the embedding vectors of the segmentation results of the same target content are merged; according to the merged embedding vector as the embedding vector of the target content, the embedding vector of the target content is input into the target content application module, and the similarity is calculated according to the embedding vectors of different target contents, the target contents are clustered, the target contents are subject-classified, and keywords of the target contents are generated, etc.; the target contents, the embedding vectors of the target contents, the subject classifications of the target contents, the keywords, the similarities between different target contents, and the clustering results are stored in the distributed system Vearch; a query interface is generated to provide query services to the outside, specifically, a query request sent by a user is received, and the similarities between different target contents are queried, and the keywords, subjects, and other target contents of the same category as the target contents are queried.

[0112] Figure 3 FIG. 1 is a schematic diagram of the main process of generating an embedding vector according to a reference embodiment of the present invention. For example, Figure 3 As shown, the target content that needs to be deduplicated is obtained, the OpenAI application interface is called using the obtained target content, a processing request is generated, the processing request is sent to the M3E model, the M3E model executes the embedding method on the target content, and the embedding vector output by the M3E model is received.

[0113] Figure 4 It is a schematic diagram of the relationship between content categories and repeated content according to a reference embodiment of the present invention. Exemplarily, the content categories are distinguished by internal and external content platforms, and the original content of the content platform and externally purchased content are obtained. There are repeated contents or content with high similarity between the two, that is, the content in the shaded part between the original content of the content platform and the externally purchased content; the content categories are distinguished by content fields, and the financial field content, e-commerce field content, logistics field content and medical field content are obtained, among which there are repeated contents or content with high similarity between the financial field content and the e-commerce field content, between the e-commerce field content and the logistics field content, and between the e-commerce field content and the medical field content.

[0114] Figure 5 FIG. 1 is a schematic diagram of the main process of content segmentation according to a reference embodiment of the present invention. Figure 5As shown, the target content is split to obtain target sub-content 1, target sub-content 2 and target sub-content 3, wherein target sub-content 1 includes shared content 1-1, target sub-content 2 includes shared content 2-1 and shared content 2-2, and target sub-content 3 includes shared content 3-1. The target sub-content 1 and shared content 2-1 are spliced ​​to obtain a splicing result 501 of the target content, the target sub-content 2, shared content 1-1 and shared content 3-1 are spliced ​​to obtain another splicing result 502 of the target content, and the target sub-content 3 and shared content 2-2 are spliced ​​to obtain another splicing result 503 of the target content.

[0115] Figure 6 Schematic diagram of the main process of content management according to a reference embodiment of the present invention. Figure 6 As shown, the content management method may include:

[0116] Step S601, in response to receiving a plurality of target contents, determining a data type of each target content, and determining an embedding method applicable to each target content according to the data type;

[0117] Step S602, determining a content segmentation strategy applicable to each target content respectively;

[0118] Step S603, segmenting the corresponding target content according to the content segmentation strategy;

[0119] Step S604, using a preset Chinese embedding model to execute a corresponding embedding method on each segmentation result of the target content to obtain an embedding vector for each segmentation result;

[0120] Step S605, merging the embedding vectors of the multiple segmentation results of each target content into the embedding vector of the target object;

[0121] Step S606, storing the embedding vector of each target content and the similarity between the multiple target contents in a distributed system;

[0122] Step S607, in response to receiving the query request, querying the similarities between the corresponding multiple target contents from the distributed system;

[0123] Step S608: determine the similarity between the multiple target contents according to the embedding vectors, and remove duplicates from the multiple target contents according to the similarity.

[0124] The specific implementation content of the content management method of the above-mentioned reference embodiment of the present invention has been described in detail in the above-mentioned content management method, so the repeated content will not be described again here.

[0125] Figure 7is a schematic diagram of main modules of a content management device according to an embodiment of the present invention, such as Figure 7 As shown, the content management device 700 mainly includes:

[0126] A determination module 701 is used to determine the data type of each target content in response to receiving a plurality of target contents, and determine an embedding method suitable for each target content according to the data type;

[0127] An embedding module 702 is used to use a preset Chinese embedding model to execute a corresponding embedding method on each target content to obtain an embedding vector for each target content;

[0128] The deduplication module 703 is configured to determine the similarity between the multiple target contents according to the embedding vectors, and deduplication the multiple target contents according to the similarity.

[0129] According to a reference embodiment of the present invention, the content management device 700 further includes: a policy module, used to determine a content segmentation strategy applicable to each target content; a segmentation module, used to segment the corresponding target content according to the content segmentation strategy;

[0130] Using a preset Chinese embedding model to execute a corresponding embedding method on each target content includes: using a preset Chinese embedding model to execute a corresponding embedding method on a segmentation result of each target content.

[0131] According to another reference embodiment of the present invention, segmenting the corresponding target content according to the content segmentation strategy includes:

[0132] Dividing the target content into a plurality of non-overlapping target sub-contents;

[0133] Determine the shared content of each target sub-content;

[0134] According to the arrangement order of the multiple target sub-contents, the shared content of the previous target sub-content is added to the head of each target sub-content, and / or the shared content of the next target sub-content is added to the tail of each target sub-content.

[0135] According to another reference embodiment of the present invention, the embedding vector of each target content includes: the embedding vectors of multiple segmentation results of each target content; the content management device 700 also includes:

[0136] A merging module is used to merge the embedding vectors of multiple segmentation results of each target content into a single vector.

[0137] According to another reference embodiment of the present invention, merging the embedding vectors of multiple segmentation results of each target content into a single vector includes:

[0138] Determine a segment weight for each segment result of each target content;

[0139] weighting the corresponding embedding vector according to the segment weight;

[0140] The weighted embedding vectors are merged to obtain the weighted average vector.

[0141] According to another reference embodiment of the present invention, the content management device 700 further includes:

[0142] A storage module, used for storing the embedding vector of each target content and the similarity between multiple target contents in a distributed system;

[0143] The query module is configured to query the distributed system for similarities between a plurality of target contents included in the target content set in response to receiving the target content set.

[0144] According to a reference embodiment of the present invention, querying the similarity between the plurality of target contents included in the target content set from the distributed system includes:

[0145] parsing query conditions associated with the target content set;

[0146] Screening multiple target contents included in the target content set according to the query condition;

[0147] The similarity between the filtered target contents is queried from the distributed system.

[0148] It should be noted that the specific implementation content of the content management device in the embodiment of the present invention has been described in detail in the content management method described above, so the repeated content will not be described here.

[0149] According to the technical solution of the embodiment of the present invention, a Chinese embedding model is used to execute a corresponding embedding method on the target content, and then the similarity between multiple target contents is determined according to the embedding vector output by the Chinese embedding model, and the target contents with high similarity are deduplicated, which can improve the content management efficiency and reduce the labor cost; before executing the embedding method, the target content is segmented according to the content segmentation strategy, which can ensure that the segmented content meets the input conditions of different Chinese embedding models and improve the flexibility of content input; when segmenting, the segmentation results retain the contextual semantics before and after the segmentation, which can ensure that the segmented content has complete and coherent semantics, and avoid the Chinese embedding model not understanding the segmented content or generating ambiguity; merging multiple embedding vectors can reduce the storage space of the embedding vector and improve the processing efficiency of the embedding vector; storing the embedded target content and similarity in a distributed system and providing similarity query services to the outside, which can improve the query efficiency of the target content similarity, store content of multiple data types together, and reduce storage costs; meeting different query requirements according to query conditions can improve the accuracy of similarity queries, more flexibly query the similarity between different contents, and save query time.

[0150] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the first aspect of the embodiment of the present invention.

[0151] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method provided by the first aspect of the embodiment of the present invention is implemented.

[0152] Figure 8 An exemplary system architecture 800 is shown to which the content management method or content management apparatus according to an embodiment of the present invention may be applied.

[0153] like Figure 8 As shown, system architecture 800 may include terminal devices 801, 802, 803, a network 804, and a server 805. Network 804 is used to provide a medium for communication links between terminal devices 801, 802, 803 and server 805. Network 804 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0154] Users can use terminal devices 801, 802, and 803 to interact with server 805 through network 804 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 801, 802, and 803, such as content management applications, content query applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0155] The terminal devices 801 , 802 , and 803 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0156] The server 805 may be a server that provides various services, such as a background management server (for example only) that supports content management requests sent from the upstream terminal devices 801, 802, and 803. In response to receiving multiple target contents, the background management server may determine the data type of each target content, determine the embedding method applicable to each target content according to the data type; use a preset Chinese embedding model to execute the corresponding embedding method for each target content to obtain an embedding vector for each target content; determine the similarity between the multiple target contents according to the embedding vector, and deduplicate the multiple target contents according to the similarity; and feed back the content management situation (for example only) to the terminal device.

[0157] It should be noted that the content management method provided in the embodiment of the present invention is generally executed by the server 805, and accordingly, the content management device is generally set in the server 805. The content management method provided in the embodiment of the present invention can also be executed by the terminal devices 801, 802, and 803, and accordingly, the content management device can be set in the terminal devices 801, 802, and 803.

[0158] It should be understood that Figure 8 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0159] Reference below Fig. 9 , which shows a schematic diagram of the structure of a computer system 900 of a terminal device suitable for implementing an embodiment of the present invention. Fig. 9 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0160] like Fig. 9As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the system 900 are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0161] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage section 908 as needed.

[0162] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-mentioned functions defined in the system of the embodiment of the present invention are executed.

[0163] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In an embodiment of the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0164] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0165] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a determination module, an embedding module, and a deduplication module, wherein the names of these modules do not constitute a limitation on the modules themselves under certain circumstances, for example, the determination module may also be described as a "module for determining an embedding method suitable for target content".

[0166] As another aspect, an embodiment of the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device implements the following method:.

[0167] According to the technical solution of the embodiment of the present invention, a Chinese embedding model is used to execute a corresponding embedding method on the target content, and then the similarity between multiple target contents is determined according to the embedding vector output by the Chinese embedding model, and the target contents with high similarity are deduplicated, which can improve the content management efficiency and reduce the labor cost; before executing the embedding method, the target content is segmented according to the content segmentation strategy, which can ensure that the segmented content meets the input conditions of different Chinese embedding models and improve the flexibility of content input; when segmenting, the segmentation results retain the contextual semantics before and after the segmentation, which can ensure that the segmented content has complete and coherent semantics, and avoid the Chinese embedding model not understanding the segmented content or generating ambiguity; merging multiple embedding vectors can reduce the storage space of the embedding vector and improve the processing efficiency of the embedding vector; storing the embedded target content and similarity in a distributed system and providing similarity query services to the outside, which can improve the query efficiency of the target content similarity, store content of multiple data types together, and reduce storage costs; meeting different query requirements according to query conditions can improve the accuracy of similarity queries, more flexibly query the similarity between different contents, and save query time.

[0168] The above specific implementations do not constitute a limitation on the protection scope of the embodiments of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the embodiments of the present invention should be included in the protection scope of the embodiments of the present invention.

Claims

1. A method for content management, characterized in that: include: In response to receiving a plurality of target contents, determining a data type of each target content, and determining an embedding method applicable to each target content according to the data type; Use the preset Chinese embedding model to perform the corresponding embedding method on each target content to obtain the embedding vector of each target content; The similarities between the multiple target contents are determined according to the embedding vectors, and the multiple target contents are deduplicated according to the similarities.

2. The method according to claim 1, characterized in that Before executing the corresponding embedding method on each target content using the preset Chinese embedding model, the method further includes: determining a content segmentation strategy applicable to each target content respectively; segmenting the corresponding target content according to the content segmentation strategy; Using a preset Chinese embedding model to execute a corresponding embedding method on each target content includes: using a preset Chinese embedding model to execute a corresponding embedding method on a segmentation result of each target content.

3. The method according to claim 2, characterized in that Segmenting the corresponding target content according to the content segmentation strategy includes: Dividing the target content into a plurality of non-overlapping target sub-contents; Determine the shared content of each target sub-content; According to the arrangement order of the multiple target sub-contents, the shared content of the previous target sub-content is added to the head of each target sub-content, and / or the shared content of the next target sub-content is added to the tail of each target sub-content.

4. The method according to claim 2, characterized in that: The embedding vector of each target content includes: the embedding vectors of multiple segmentation results of each target content; before determining the similarity between the multiple target contents according to the embedding vectors, the method further includes: The embedding vectors of multiple segmentation results for each target content are merged into a single vector.

5. The method according to claim 4, characterized in that Merge the embedding vectors of multiple segmentation results for each target content into a single vector, including: Determine a segment weight for each segment result of each target content; weighting the corresponding embedding vector according to the segment weight; The weighted embedding vectors are merged to obtain the weighted average vector.

6. The method according to claim 1, characterized in that The method further comprises: storing the embedding vector of each target content and the similarities between multiple target contents in a distributed system; In response to receiving the target content set, the distributed system is queried for similarities between a plurality of target contents included in the target content set.

7. The method according to claim 6, characterized in that Querying the distributed system for similarities between a plurality of target contents included in the target content set includes: parsing query conditions associated with the target content set; Screening multiple target contents included in the target content set according to the query condition; The similarity between the filtered target contents is queried from the distributed system.

8. A content management device, characterized in that: include: a determination module, configured to determine a data type of each target content in response to receiving a plurality of target contents, and determine an embedding method applicable to each target content according to the data type; An embedding module, used to use a preset Chinese embedding model to execute a corresponding embedding method on each target content to obtain an embedding vector for each target content; A deduplication module is used to determine the similarity between the multiple target contents according to the embedding vector, and to dedupe the multiple target contents according to the similarity.

9. An electronic device, characterized in that: include: One or more processors; A storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images