Information deduplication processing method, device and computer readable storage medium

Abstract

The present application provides an information processing method, device and storage medium. After obtaining the newly uploaded file to be processed, the feature vectors of different dimensions of the file to be processed are constructed, so as to realize the similarity calculation between the file to be processed and the sample file in different dimensions, obtain the similarity relationship chains corresponding to these multiple dimensions, and then deduplicate the files in these multiple similarity relationship chains according to the preset deduplication decision. It can be seen that this parallel deduplication method greatly improves the deduplication efficiency and accuracy, and avoids the inability to obtain the similarity of the file to be processed and the eliminated sample file in other dimensions after similar deduplication of serial historical dimensions, resulting in the inability to meet the deduplication processing requirements of specific dimensions, and can better meet business needs.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology and specifically to a method, device and computer-readable storage medium for deduplication of information. Background Art

[0002] In the era of rapid development of the Internet, as the threshold for content production is lowered, the upload and release volume of videos, pictures, texts and other information is growing at an exponential rate. Taking short video content as an example, due to its simple production process, low production threshold and strong participation, it can come from PGC (Professionally-generated Content), UGC (User-generated Content), user uploads, various organizations or applications specializing in short video production, MCN (Multi-Channel Network), etc., making short videos one of the important ways of dissemination for content entrepreneurship and social media platforms.

[0003] However, in actual applications, in order to encourage content creation, various short video platforms have relevant subsidies and incentive mechanisms for content. As a result, content creators will upload a large amount of similar or directly copied and plagiarized content from other account owners in order to increase their own income. They will even upload some high-risk content and its variants, such as bloody and violent content and safety hazards, to increase their attractiveness, which is not conducive to the healthy development of the entire content ecosystem.

[0004] Therefore, in order to ensure the security and timeliness of the distributed content and the interests of the copyright source of the content itself, it is usually necessary to review the uploaded content and eliminate the content that is not conducive to the healthy development of the entire content ecosystem. If the uploaded content is reviewed directly, it will often take a very long time to review due to the large amount of content to review, which will increase the review cost and reduce the review accuracy. In this regard, the existing technology usually uses a similarity detection algorithm to eliminate duplicate content first, and then reviews other content. However, in the process of deduplication, a large number of files are deduplicated in sequence based on the calculation results of the similarity of the title or image. The deduplication efficiency and accuracy are low, and often due to excessive deduplication, it cannot meet business needs. Summary of the invention

[0005] In view of this, in order to improve the accuracy and efficiency of deduplication of large-scale stored files, the present application proposes an information processing method, the method comprising:

[0006] Get the files to be processed;

[0007] Constructing feature vectors of different dimensions of the file to be processed;

[0008] Using the feature vectors of different dimensions, respectively calculate the similarity of the file to be processed and the sample files in the sample library to obtain a similarity relationship chain of the file to be processed under the corresponding dimension, wherein the similarity relationship chain includes the file to be processed and the sample files under the corresponding dimension whose similarity with the file to be processed is greater than a corresponding threshold;

[0009] A deduplication decision for the to-be-processed file is obtained, and deduplication processing is performed on the files in the obtained multiple similar relationship chains according to the deduplication decision.

[0010] In some embodiments, constructing feature vectors of different dimensions of the to-be-processed file includes:

[0011] According to different feature extraction algorithms, feature vectors of corresponding dimensions of the to-be-processed file are obtained; wherein the different feature extraction algorithms include multiple combinations of hash algorithms, twin neural network algorithms, bidirectional encoding representation of transformers BERT algorithms, and edit distance algorithms; the feature vectors include one or more combinations of summary feature vectors, title feature vectors, text content feature vectors, text semantic vectors, picture semantic vectors, picture feature vectors, and semantic fingerprint vectors of videos;

[0012] The using the feature vectors of different dimensions to respectively calculate the similarity between the file to be processed and the sample files in the sample library includes:

[0013] The similarity calculation is performed on the to-be-processed file and the sample files in the sample library respectively by using the feature extraction algorithm based on which the feature vectors of different dimensions are constructed and the feature vectors of corresponding dimensions.

[0014] In some embodiments, the method further comprises:

[0015] Obtaining sample libraries corresponding to the different dimensions;

[0016] The using the feature vectors of different dimensions to respectively calculate the similarity between the file to be processed and the sample files in the sample library includes:

[0017] Reading the feature vector of each sample file from the sample libraries corresponding to the different dimensions in a distributed retrieval manner;

[0018] The similarity between the feature vector of the to-be-processed file and the feature vector of the sample file in the same dimension is calculated.

[0019] In some embodiments, the method further comprises:

[0020] For each of the sample libraries, first index information is constructed using the historical feature vectors obtained within the first time from the current moment, where the first index information is configured as a read-only attribute;

[0021] Using the feature vector obtained within a second time from the current moment, constructing second index information, wherein the second index information is configured as a read-write separation attribute, and the second time is less than the first time;

[0022] A double buffering method is adopted to realize the updating and reconstruction of the first index information and the second index information.

[0023] On the other hand, the present application also proposes an information processing device, the device comprising:

[0024] A module for obtaining files to be processed, used for obtaining files to be processed;

[0025] A feature vector construction module, used to construct feature vectors of different dimensions of the file to be processed;

[0026] A similarity relationship chain obtaining module is used to calculate the similarity between the to-be-processed file and the sample files in the sample library respectively by using the feature vectors of different dimensions, so as to obtain a similarity relationship chain of the to-be-processed file under the corresponding dimension, wherein the similarity relationship chain includes the to-be-processed file and the sample files under the corresponding dimension whose similarity with the to-be-processed file is greater than the corresponding threshold value;

[0027] The deduplication processing module is used to obtain a deduplication decision for the file to be processed, and perform deduplication processing on the files in the obtained multiple similar relationship chains according to the deduplication decision.

[0028] On the other hand, the present application also proposes a computer-readable storage medium on which a computer program is stored, characterized in that the computer program is executed by a processor to implement the steps of the above-mentioned information processing method.

[0029] Based on the above technical solution, after obtaining the newly uploaded to-be-processed file, the present application will construct feature vectors corresponding to multiple dimensions of the to-be-processed file, respectively, so as to use the feature vectors of different dimensions to calculate the similarity of the to-be-processed file and the sample file, respectively. Compared with the single-dimensional similar deduplication processing method, the similarity recall rate and deduplication accuracy rate are improved. Compared with the multi-dimensional serial deduplication processing method, after realizing the similarity calculation of the to-be-processed file and the sample file in one dimension, the present application does not directly perform deduplication processing, but adopts a hierarchical decoupling structure, and obtains the similarity relationship chains corresponding to the multiple dimensions respectively from the similarity calculation results of multiple dimensions, and performs deduplication processing on the files in the multiple similar relationship chains according to the preset deduplication decision. It can be seen that this parallel deduplication method greatly improves the deduplication efficiency and accuracy, and avoids the situation that after the similarity deduplication of the serial historical dimension, the similarity of the to-be-processed file and the eliminated sample file in other dimensions cannot be obtained subsequently, resulting in the inability to meet the deduplication processing requirements of specific dimensions, and can better meet business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0031] Figure 1 A schematic diagram of a system architecture of an optional example of implementing the information processing method is shown;

[0032] Figure 2 A schematic diagram of a system architecture of an optional example of implementing the information processing method proposed in this application is shown;

[0033] Figure 3 A schematic diagram of a system architecture of another optional example applicable to the information processing method proposed in the present application is shown;

[0034] Figure 4 A schematic diagram of a flow chart in an optional system scenario applicable to the information processing method proposed in this application is shown;

[0035] Figure 5 A flowchart showing an optional example of the information processing method proposed in the present application is shown;

[0036] Figure 6 A schematic diagram showing the operation principle of the twin neural algorithm used in the information processing method proposed in this application is shown;

[0037] Figure 7A schematic diagram of an optional scenario of the information processing method proposed in this application is shown;

[0038] Figure 8 A flowchart showing another optional example of the information processing method proposed in the present application is shown;

[0039] Fig. 9 A schematic diagram showing the principle of an optional similarity search method in the information processing method proposed in the present application is shown;

[0040] Fig.10 A schematic diagram showing another optional scenario of the information processing method proposed in this application;

[0041] Fig.11 The schematic diagram of the principle structure of distributed retrieval in the information processing method proposed in this application is shown;

[0042] Fig.12 A schematic diagram showing another optional scenario of the information processing method proposed in this application;

[0043] Fig.13 A schematic structural diagram of an optional example of the information processing device proposed in the present application is shown. DETAILED DESCRIPTION

[0044] In response to the technical problems raised by the background technology, this application hopes to utilize artificial intelligence (AI) technology, such as natural language processing (NLP), machine learning (ML) / deep learning, computer vision technology (CV), etc., to integrate the feature vectors of multiple dimensions (such as title, cover, text content, etc.) of the files to be processed (such as pictures, picture albums, videos, short content, etc.) to achieve similarity retrieval and parallel deduplication processing. Compared with manual deduplication processing methods and sequential deduplication processing based on single dimensional features, the deduplication processing efficiency and accuracy are greatly improved, which helps to reduce the subsequent review costs of other content and improve the review efficiency and accuracy.

[0045] Among them, natural language processing technology is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language; computer vision technology studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval and other technologies. Machine learning is a multi-domain interdisciplinary subject that specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance; the concept of deep learning originates from the study of artificial neural networks, and a multilayer perceptron with multiple hidden layers is a deep learning structure. Deep learning combines low-level features to form more abstract high-level representation attribute categories or features to discover distributed feature representations of data. In this application, appropriate artificial intelligence technology can be used to obtain feature vectors of different dimensions of the files to be processed, and the specific acquisition process is not limited in this application.

[0046] Based on the above-mentioned inventive concept, in the preliminary study of the information processing method, first, for the traditional retrieval database system, such as SQL (Structured Query Language) database, it is usually composed of a structure table containing symbolic information. For example, an image set can be represented by a list with an index photo in each row, and each row contains information such as image identification and description statements. Each row can also be associated with entries in other tables (such as lists of photos and names). This data storage structure is applied to the deduplication retrieval process. Facing billions of vectors, searching for similar entries, that is, searching for similar high-dimensional vectors, for existing standard retrieval languages, the efficiency is very low, and it can even be said to be impossible to achieve. Therefore, in large-scale data retrieval applications, the availability of traditional retrieval database systems is not high, which leads to very low efficiency of deduplication processing based on retrieval results.

[0047] In this regard, this application proposes to use the Faiss database system to implement data retrieval, where Faiss is an open source clustering and similarity search library for the Facebook AI team, which provides efficient similarity search and clustering for dense vectors, and supports an approximate neighbor search library for searching billions of vectors. It usually contains a variety of algorithms for searching vector sets of any size, as well as support code for algorithm evaluation and parameter adjustment. In practical applications, the similarity calculation methods commonly used by Faiss mainly include Euclidean distance and dot product, but are not limited to these two similarity calculation methods.

[0048] It can be seen that the present application can use Faiss as the basic component of the underlying vector retrieval. In this way, the content stored in the sample library can include the feature vectors of each sample file, which can specifically be feature vectors of various dimensions obtained by different feature extraction methods. The specific vector content is not limited in this application. In this way, following the above analysis, the present application can determine the similarity between the file to be processed and the sample file by operating on the feature vectors of each of the file to be processed and the sample file. The specific operation method is not described in detail. Compared with traditional retrieval database systems, the efficiency of similarity retrieval is greatly improved, and it is convenient to carry out parallel expansion of the sample library, such as using a new similarity algorithm to obtain the feature vector of the sample file, and constructing a sample library for the new similarity algorithm. The present application does not limit the method for expanding the sample library.

[0049] Furthermore, in order to solve the technical problem that the existing deduplication processing method performs deduplication processing on a single dimension, resulting in inaccurate deduplication results, a multi-dimensional deduplication processing implementation method is proposed, such as Figure 1 The schematic diagram of an optional system architecture suitable for the information processing method shown is a serial filtering architecture, which, after obtaining the file to be processed (i.e., the content uploaded by the producer, such as various articles, which may contain videos or pictures, or various pure video files, etc.), can usually first perform deduplication based on the title, and then perform text deduplication on the obtained similar sample files (i.e., the files stored in the sample library), and then perform image deduplication. However, in this serial processing process, although deduplication of multiple dimensions (such as title, text, pictures, etc.) is involved, in a certain processing link, the file to be processed is not deduplicated, then the remaining other dimensions of the file to be processed cannot be deduplicated similarly to other sample files.

[0050] visible, Figure 1 In the deduplication application scenario, the business of each dimension maintains its own sample library, which is difficult to align accurately. In addition, due to the deep coupling with historical rules, any file may be filtered out as long as it is similar to other files in any dimension. However, in actual applications, taking a file with short text and multiple images as an example, it is usually necessary to skip text deduplication and only perform image deduplication. Figure 1 The system architecture shown will not be able to meet the deduplication requirement; if it is directly distributed to multiple deduplication services, the subsequent maintenance cost will be very high.

[0051] In addition, as the sample library stores more content (i.e., user-uploaded content), Figure 1 The similarity detection algorithm used in each processing stage may not be accurate enough, resulting in poor deduplication effect. If a new similarity detection algorithm is needed, Figure 1The above problem will become more and more serious if the new similarity detection algorithm is used to achieve deduplication service.

[0052] In order to improve the above problems, the present application further proposes a hierarchical decoupling structure, and transforms the deduplication service into a similarity chain calculation service. In this way, after similarity calculation is performed for each dimension, a corresponding similarity chain is formed. However, deduplication is not performed directly on each similarity chain. Instead, after obtaining the similarity chains of multiple dimensions, the decision layer performs deduplication on the similarity chains of multiple dimensions (such as Figure 2 In the similarity relationship chain 1, similarity relationship chain 2, similarity relationship chain 3, etc., each similarity relationship chain can indicate that the file to be processed is similar to sample file 1, sample file 3, sample file 9, sample file m, etc., and this application does not limit the content of each similarity relationship chain and its output form) to perform deduplication processing on the files, such as Figure 2 The system architecture diagram shown is an optional example of implementing the information processing method proposed in the present application. The present application can calculate similar relationship chains in various dimensions in parallel, thereby improving the efficiency of information processing, and can perform deduplication processing according to at least one dimension as required, thereby improving the reliability and accuracy of deduplication processing.

[0053] And, for Figure 2 The parallel similarity relationship chain calculation method shown facilitates the parallel expansion of the dimensional similarity detection algorithm, better copes with large-scale content deduplication needs, and facilitates the configuration of similarity relationship chains for different purposes such as high-risk content deduplication, specific copyright content protection such as big Vs, etc., to carry out targeted deduplication processing and meet various deduplication needs.

[0054] In addition, since the file status of all files can be maintained uniformly to inform the decision-making layer which files are involved in deduplication, this not only makes it easier for the decision-making layer to implement various customized rules, but also solves the problem of serial filtering being difficult to align accurately.

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] It should be noted that, for the convenience of description, only the parts related to the relevant invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0057] It should be understood that the "system", "device", "unit" and / or "module" used in this application is a method for distinguishing different components, elements, parts, parts or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

[0058] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

[0059] Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" refers to two or more than two. The following terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0060] In addition, flow charts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, the various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0061] Reference Figure 3 , which is a schematic diagram of a system architecture of another optional example applicable to the information processing method provided in the present application. In practical applications, the system may include: a content production end 10, an interface server 20, a content database 30, a content storage service device 40, a scheduling center service device 50, a manual review system 60, a content distribution service device 70, a download file system 80, a frame extraction service device 90, a multimodal content vector generation service device 100, a distributed vector retrieval service device 110, a global deduplication relationship calculation service device 120 and a content consumption end 130, wherein:

[0062] The content production end 10 can be PGC Professionally-generated Content, professionally produced content or UGC (User-generated Content), MCN (Multi-Channel Network) and other content producers, which upload the content to be uploaded, such as pictures, videos, short content, and text files to be processed, to the interface server 20 through the communication interface (such as API interface) of the electronic device. This application does not limit the content uploaded to the interface server 20 and its source.

[0063] The interface server 20 may include upstream and downstream content communication interfaces. The sources of different types of content may be different. For example, the source of graphic content may be a lightweight publishing terminal and an editing content portal, and the source of transmitted video content may be a shooting terminal, such as an electronic device with image shooting capabilities, etc., and the captured video content may be accompanied by music, filter templates, and video beautification functions, etc. The present application may configure communication interfaces for different types of content in the interface server 20 to realize the transmission of corresponding types of content. The present application does not limit the type of the communication interface.

[0064] As above Figure 3 As shown, after the interface server 20 obtains the uploaded content from the content production end 10, such as the source file and meta-information of the uploaded content, such as the title, publisher, summary, cover image, publishing time, file size, bit rate, file format, video file size, video format, whether it is original or not, etc., the meta-information of the uploaded content can be written into the content database 30 for storage, and the source file of the uploaded content, that is, the content entity information other than the meta-information of the content, such as the video source file, the picture source file, etc., can be written into the content storage service device 40 for storage; the published content can also be synchronously submitted to the scheduling center service device 50 for subsequent content processing and circulation.

[0065] In the embodiments of the present application, combined with the above analysis, the content database may be a business database that stores the metadata of all content published by producers. In addition to storing the metadata listed above, it may also store the classification of content during the manual review process, such as first-, second-, and third-level classifications and label information. For example, for an article about mobile phone A, the first-level classification may be technology; the second-level classification may be smart phones; the third-level classification may be domestic mobile phones, and the label information may be A, x series, etc.

[0066] In addition, during the manual review process of the uploaded content by the manual review system 60, the information in the content database 30 is usually read, and the review result and content status of the manual review can be fed back to the content database 30 for storage.

[0067] The dispatch center service device 50 is mainly used to implement the scheduling of the control order and priority of the manual review and machine processing of the uploaded content, as well as the scheduling of the content flow, such as receiving the stored content through the interface server 20, obtaining the meta information of the content from the content database 30, and updating the meta information, etc.; the deduplication service of the global deduplication relationship calculation service device 120 can be called to execute the deduplication service to achieve deduplication processing of the uploaded content and delete duplicate content; after starting the manual review system 60, the display page of the review result is sent directly to the electronic device output of the content consumption end 130 through the content distribution service device 70 (which can be a recommendation engine or search engine or operation engine, etc.), that is, the content index information is sent to the content consumption end 130, etc., and the processing result can be fed back to the content database. The dispatch center service device 50 can specifically be an independent server or a service set composed of multiple servers, and this application does not limit its structure.

[0068] In the actual application of this embodiment, completely repeated content in the content database 30 will not be sent to the manual review system 60, thereby avoiding repeated review; in addition, according to application requirements, the scheduling center service device 50 can be used to realize video content and tag expansion service communication, enrich and expand the tags of video content, and improve the efficiency of content cold start and operation. The specific implementation method is not limited in this application.

[0069] The above-mentioned manual review system 60, as a carrier of manual service capabilities, can be mainly used in this application to review and filter sensitive, violent, legally prohibited and other content that cannot be determined by machines. It also labels and reconfirms video content. The specific review process and the composition structure of the manual review system are not described in detail in this application.

[0070] The download file system 80 can be used to download the required content metadata from the content database 30 and to download the original content, i.e., source files, from the content storage service device 40. A group of parallel servers composed of related task scheduling and distribution clusters can be utilized to control the speed and progress of downloading content.

[0071] In some embodiments, the download request device in the download file system 80 can send a corresponding file download request to the content storage service device 40 based on the file download requirements, so that the content storage service device 40 responds to the file download request, queries the requested file and feeds back to the download request device or other download devices. This application does not elaborate on the composition structure of the download file system 80 and its execution process for implementing file downloads.

[0072] In actual application, if the downloaded content is a video file, the frame extraction service device 90 can be called to obtain necessary file key frames from the source file for subsequent construction of the video fingerprint vector. The specific implementation process can refer to but is not limited to the description of the corresponding part of the embodiment below.

[0073] Among them, the frame extraction processing of the video file by the frame extraction service device 90 can be a primary processing step for constructing the semantic fingerprint vector of the video. According to actual needs, a uniform frame extraction strategy (i.e., the video file is subjected to frame extraction processing at a fixed time interval) or a variable-length frame extraction strategy (i.e., the video file is subjected to frame extraction processing at a variable time interval) can be adopted to implement the frame extraction processing. Since the uniform frame extraction strategy may result in insufficient sampling frequency, increase the burden and calculation amount of frame extraction, lead to a sharp increase in calculation cost, and expand the comparison space, the variable-length frame extraction strategy may be preferably used in this application.

[0074] In one possible implementation, in the implementation of frame extraction processing on video files using a variable-length frame extraction strategy, scene switching frames with obvious brightness changes can be extracted, and the frames can be filled in at equal intervals before and after. The specific implementation process is not described in detail in this application.

[0075] The multimodal content vector generation service device 100 can read the uploaded files to be processed from the content database 30 to construct corresponding feature vectors from multiple dimensions (i.e., multi-angle features) of the files to be processed (such as uploaded content), such as text Simhash vectors, title editing distance vectors, BERT (Bidirectional EncoderRepresentations from Transformers) vectors of titles and texts, semantic vectors of images based on twin networks, semantic vectors of videos, etc.

[0076] The text Simhash vector can be a text vector constructed using a hash algorithm such as Simhash. In practical applications, the Simhash algorithm can be used to calculate a hash value (64-bit integer) to determine the similarity between two files. For example, the Hamming distance calculation method can be used to obtain the distance between the simhash values ​​of the two files. Specifically, the two simhash values ​​can be XORed and the number of bits equal to 1 can be counted. Then the number of bits (such as N bits equal to 1) can represent the distance between the two simhash values. If the distance between the two simhash values ​​is ≤3, the two files can be considered similar, but it is not limited to the value 3.

[0077] In some embodiments, the Simhash algorithm can be used to obtain the text Simhash vector of the short text for deduplication of the title short text. For the body content of the file, the similarity detection result of the body Simhash vector can be used to achieve the preliminary recall of the sample files in the sample library, that is, to preliminarily query the sample files similar to the body of the file to be processed. After that, the preliminary recall result can be fine-grained by obtaining the body BERT vector, that is, using the body BERT vector of each sample file preliminarily recalled, query the sample files similar to the file to be processed from each sample file preliminarily recalled, and sequentially form the corresponding similarity relationship chain.

[0078] Among them, BERT is a bidirectional Transformer Encoder, which belongs to the NLP model. Compared with the bidirectional recurrent neural network, it is more efficient and can capture longer-distance dependencies, that is, it further enhances the generalization ability of the word vector model and fully describes the characteristics of character-level, word-level, sentence-level and even inter-sentence relations. Therefore, BERT is a sentence-level language model that can obtain a unique vector representation of an entire sentence, and it realizes true context through bidirectional encoding.

[0079] Specifically, BERT is essentially a two-stage NLP model. The first stage can be Pre-training, which is similar to the process of obtaining Word Embedding. A language model can be trained using existing unlabeled corpus (i.e., sample files); the second stage can be Fine-tuning, that is, using the pre-trained language model to complete specific NLP downstream tasks. The specific processing procedures of these two stages are not described in detail in this application. In this application, BERT can be used to vectorize the title content and the body text content in the preprocessing stage to provide vector input for subsequent judgment of deduplication and similarity relationship calculation, that is, to obtain the title BERT vector and the body BERT vector. The specific implementation process is not described in detail.

[0080] The title edit distance vector can be a title vector constructed using the Minimum Edit Distance (MED) algorithm, which is usually suitable for vector extraction of short texts to achieve short text similarity calculation. The specific operation principle will not be described in detail; the BERT vectors of the title and text can be title vectors and text vectors constructed using the BERT algorithm. The specific construction process can be determined by referring to the above description of the operation principle of the BERT algorithm. Combined with the above analysis, this application uses BERT to extract feature vectors for similarity calculation, which can increase the recall of deduplication, especially for content with similar semantics, and improves the accuracy and reliability of deduplication.

[0081] The acquisition process of other semantic vectors listed above can be determined according to the operation principle of the corresponding algorithm, and will not be described in detail in this embodiment.

[0082] In summary, for feature vectors of the same dimension, different feature extraction algorithms may be used to obtain different feature vectors, such as the above-mentioned title edit distance vector and title BERT vector. This application does not limit the content of the feature vectors of each dimension and their construction methods, which can be determined according to actual needs and are not limited to the several feature extraction algorithms (or similarity detection algorithms) listed above. Figure 2 As shown in the system architecture diagram, according to business needs, the acquired feature vectors can be expanded in parallel to obtain feature vectors of other dimensions to meet the deduplication requirements of other dimensions. The specific implementation process is not described in detail, and the method for obtaining feature vectors of other dimensions can adopt but is not limited to the method described above.

[0083] The distributed vector search service device 110 may use a distributed search method to search and match each constructed feature vector. Depending on the length and storage space of the vector, the storage and search methods may be different, which can be determined according to actual conditions. This application does not describe in detail the implementation process of distributed management and search of vectors.

[0084] Combined with Figure 2 The system construction shown can use a distributed retrieval method to perform similarity retrieval on sample files in the sample library corresponding to each dimension in the process of obtaining similarity relationship chains of different dimensions, so as to obtain one or more sample files similar to the file to be processed under the corresponding dimension, and form a similarity relationship chain corresponding to the dimension. Compared with the above-mentioned serial filtering method, the efficiency of multi-dimensional similarity calculation is greatly improved, thereby improving the efficiency of deduplication processing. This application does not elaborate on how to implement distributed retrieval.

[0085] The global deduplication relationship calculation service device 120 may be a hierarchical structure that decouples judgment and decision after obtaining the similarity relationship chains of multiple dimensions, and perform global deduplication calculation on multiple similarity relationship chains. In this process, double buffering and read-write separation may be used to improve the similarity relationship chain calculation performance, and the specific implementation process will not be described in detail.

[0086] The content consumption terminal 130 may be a terminal for acquiring and viewing content, such as Figure 3As shown, it can access the index information of the content by communicating with the uplink and downlink content interface server 20, and further communicate with the content storage service device 40 to obtain the corresponding source file, which may include recommended content and thematic subscribed content. In actual applications, the content consumption end 130 can browse content data through the Feeds stream mode, especially for browsing video files.

[0087] Among them, Feeds, or news sources, is a data format through which websites transmit the latest information to users, usually arranged in a timeline format. Timeline is the most primitive, intuitive and basic display form of Feed. In practical applications, an aggregator can be used to converge feeds in one place, which is called aggregation. For end users, an aggregator can be software specifically used to subscribe to websites, which is generally also called an RSS reader, feed reader, news reader, etc.

[0088] Based on the above Figure 3 The system architecture described, refer to Figure 4 As shown in the signaling flow chart of an optional example of an information processing method applicable to the system, as well as the functional description of the system components above, the distribution process for content (i.e., the various files described above) is generally as follows: the content production end uploads the content to the interface server, and after determining that the upload is successful, the metadata of the content can be sent to the content database for storage, and the source file of the content can be sent to the content storage service device for storage, so that the content consumption end can read the required source file therefrom; in order to deduplicate the stored content and avoid wasting storage space due to duplicate content, the scheduling center service device can call the manual review system and / or the global deduplication relationship calculation service device to realize deduplication of the stored content.

[0089] Among them, for the machine deduplication processing scheme implemented by the global deduplication relationship calculation service device, the download file system actually downloads the source files to be deduplicated (which can be called sample files) from the content storage service device. If the source file contains a video file, the frame extraction service device can extract the video file to obtain multiple video frames, and then the multimodal content vector generation service device reads the multiple video frames to construct corresponding feature vectors, such as the semantic fingerprint vector of the video, the picture feature vector, etc. Of course, the multimodal content vector generation service device can also read the metadata corresponding to each source file from the content database to construct feature vectors of multiple dimensions. In this way, when deduplication processing is required, the distributed The quantity detection service device can read the required feature vectors of multiple dimensions from the multimodal content vector generation service device, so that the global de-duplication relationship calculation service device can call the feature retrieval service of the distributed vector detection service device, and use a distributed retrieval method to generate a similarity relationship chain for each dimension. Afterwards, the global de-duplication relationship calculation service device can read the similarity relationship chains of each of the multiple dimensions, and according to the de-duplication decisions determined based on the content status of each content, deduplicate the files in the multiple similarity relationship chains, and delete the duplicate content in the content storage service device, such as sample files or uploaded files to be processed. The specific implementation process can refer to the description of the corresponding part of the method embodiment below, and this embodiment will not be repeated here.

[0090] It should be noted that the various component devices described in the above system may be independent individual devices, or at least some of the component devices may be integrated in the same computer device. The computer device may be a server or a terminal with certain data processing capabilities. This application does not limit the product type of the computer device.

[0091] in addition, Figure 3 The system architecture shown does not constitute a limitation on the system in the embodiment of the present application. In actual applications, the system may include Figure 3 More or fewer components shown, or combinations of certain components, can be determined based on the application scenario requirements of the system, and this application does not list them one by one here.

[0092] Reference Figure 5 , is a flowchart of an optional example of the information processing method proposed in this application. This embodiment can be applied to the system described in the above embodiment. This application does not limit the composition structure of the system, which can include but is not limited to the system structure described in the above embodiment. It should be noted that this embodiment mainly describes the process of how to automatically implement content deduplication processing, and does not elaborate on the manual review method in this embodiment. Figure 5 As shown, the method may include but is not limited to the following steps:

[0093] Step S11, obtaining the file to be processed;

[0094] In combination with the description of the above embodiments, the file to be processed may be content uploaded by the producer and may be downloaded from the content storage service by downloading the file system. Specifically, it may be graphics, video files, plain text files, segment content files, etc. This application does not limit the type of the file to be processed.

[0095] Step S12, constructing feature vectors of different dimensions of the file to be processed;

[0096] Combined with the above description of the inventive concept of the present application, in order to meet different deduplication requirements and improve deduplication efficiency and accuracy, the present application proposes a multi-dimensional deduplication solution. Therefore, after obtaining the file to be processed, the present embodiment can process the file to be processed by the multimodal content vector generation service device to obtain feature vectors of different dimensions. Figure 7 As shown in the structural diagram, the present application can adopt different types of deep learning algorithms to realize the construction of feature vectors of various dimensions, and the specific construction method is not limited.

[0097] In some embodiments, the above-mentioned deep learning algorithm may include one or more combinations of a hash algorithm (such as a Simhash algorithm), a twin neural network algorithm, a bidirectional encoder representation from transformer (BERT) algorithm, and an edit distance algorithm, to form a network structure for generating feature vectors of different dimensions, and input information of the corresponding dimension of the file to be processed into the network structure to obtain the feature vector of the dimension, such as Figure 7 The calculation of the text simhash vector, the calculation of the image hash vector, the calculation of the text BERT vector, the calculation of the image vector, etc. shown in the figure can be determined in combination with the operation principle of the corresponding deep learning algorithm.

[0098] Combined with the description of the corresponding part of the above embodiment, for the deduplication of files in the text dimension, the Simhash algorithm and the BERT algorithm can be used to calculate the text vector. Figure 7, the calculation method of the text simhash vector can be used to construct the title text vector (i.e., the title Simhash vector) and the body text vector (i.e., the body Simhash vector). The specific construction process can refer to the description of the operation principle of the Simhash algorithm above, which will not be repeated in this embodiment; similarly, the calculation method of the text BERT vector can also be used to construct the title text vector (i.e., the title BERT vector) and the body text vector (i.e., the body BERT vector). The specific construction process can be determined in combination with the analysis of the operation principle of BERT above, which will not be described in detail in this application.

[0099] It can be seen that for the same dimension of the file to be processed, different vector construction methods can be used to obtain multiple feature vectors corresponding to the dimension, which is not limited to Figure 7 The construction method of the description; and for the main text part of the file, it usually contains more sentences. In some embodiments, a preliminary similarity calculation can be performed on the file based on the dimension of the main text content, and sample files with a certain similarity to the file to be processed can be recalled, and then other methods can be used to perform further similarity recall to obtain the final feature vector corresponding to the dimension of the main text content.

[0100] In one possible implementation, as described above, the Simhash algorithm can be used to obtain the text Simhash vector of the file's text content, so as to complete the preliminary similarity recall of the files to be processed. Afterwards, the BERT algorithm is used to obtain the text BERT vector of each recalled file to achieve fine-grained recall of the files to be processed. The specific implementation process can be determined in combination with the operating principles of the corresponding algorithms above, and the embodiments of this application will not be elaborated.

[0101] For the twin neural network algorithm (also known as the twin network algorithm), refer to Figure 6 The twin network structure diagram shown in FIG. 1 can be used to calculate the semantic similarity between sentences. Therefore, the twin network can be input in the form of sentence pairs. After that, the input can be mapped to a new space through the two network structures to form a representation of the input in the new space. The similarity of the two inputs is evaluated by calculating the loss function Loss. Specifically, the spatial similarity between the two sentences can be measured by Manhattan distance, Euclidean distance, cosine similarity, etc. It should be noted that the two network structures can be the same and share parameters. When the two sentences are from the same field and have great similarity in structure, the twin network can be selected for calculation, and the loss function of the twin network generally selects the Contrastive loss Function, but is not limited to this loss function.

[0102] Among them, the contrast loss function L can be:

[0103]

[0104] In the above formula (1), N can represent the number of input files, and its specific value is not limited; d can represent the Euclidean distance between the feature vectors of two input files, y=1 can represent that the two input files are similar, y=0 can represent that the two input files are not similar, and margin can represent a preset threshold for judging whether the files are similar. Therefore, the judgment of whether the two input files are repeated or similar can be achieved by adjusting the size of the margin value. The specific value of margin is not limited in this application. Usually, when the obtained Euclidean distance exceeds the margin, the obtained loss can be 0, that is, the farther the dissimilar features are, the lower their loss is; while for similar features, the farther they are, the greater their loss is, thereby continuously updating the matching degree of paired input files. This application does not elaborate on the implementation process of the model for training and extracting feature vectors.

[0105] It can be seen that in the embodiments of the present application, the text in the to-be-processed file and the sample file can be encoded by the twin network algorithm to obtain a text semantic vector representing the semantics of the main text. If the input contains pictures, the picture semantic vector representing the semantics of the main text picture can be obtained according to this processing method. Regarding the specific construction process of the two feature vectors, the text semantic vector and the picture semantic vector, this application does not go into detail. It should be understood that the input of the twin network algorithm is not limited to the main text and pictures listed above, but may also be the title of the file, short text and other dimensions. The implementation process of obtaining the semantic vector of the corresponding dimension is similar, and this application does not go into detail one by one.

[0106] In addition, if the obtained files to be processed include video files, this application needs to extract frames for processing, and then use the obtained multiple video frames for vectorization to obtain the semantic fingerprint vector of the video file. The specific implementation process can refer to but is not limited to the description of the corresponding part of the following embodiment.

[0107] Among them, since each video frame can be used as a picture, this application uses hash algorithms or other picture feature extraction methods to construct picture hash vectors, picture vectors, etc. For the pictures corresponding to each video frame, a key frame can be selected as the cover picture, and the cover picture can be used as a dimension of the video file to construct the corresponding cover picture hash vector and cover picture vector. The specific construction process is not described in detail.

[0108] In summary, for the method of constructing feature vectors of different dimensions of the files to be processed, the deep learning algorithm to be used can be determined according to factors such as the dimension type and the type of file to be processed, including but not limited to the implementation methods listed above, and according to actual needs, the present application can also expand other deep learning algorithms to realize the construction of feature vectors of more dimensions of the files to be processed, so as to improve the accuracy of file similarity recall, and thereby improve the accuracy of file deduplication. The specific expansion process is not described in detail in this application.

[0109] Step S13, using feature vectors of different dimensions, respectively calculate similarities between the file to be processed and the sample files in the sample library, and obtain a similarity relationship chain of the file to be processed under the corresponding dimension;

[0110] In this embodiment, each similarity relationship chain obtained may include sample files whose similarity with the file to be processed is greater than the corresponding threshold under the corresponding dimension. For example, the file to be processed A is similar to the sample files B, D, and E, and the similarity relationship chain is formed by the file to be processed A and the sample files B, D, and E. This application does not limit the content of the similarity relationship chain and its output form.

[0111] Following the above description, combined with Figure 7 As shown in the structural diagram, after obtaining the feature vectors of different dimensions of the file to be processed, the corresponding deep learning algorithm can be used to continue to calculate the similarity of the feature vectors of the corresponding dimensions, that is, the feature vector of each dimension of the file to be processed and the feature vector stored in the sample library of the corresponding dimension are used to realize the similarity calculation between the file to be processed and the sample file in the corresponding dimension. Specifically, the deep learning algorithm based on which the feature vectors of different dimensions are constructed and the feature vectors of the corresponding dimensions can be used to calculate the similarity of the file to be processed and the sample files in the sample library, respectively, so as to obtain the similarity between the file to be processed and each sample file in the sample library from different dimensions, and for each dimension, the obtained similarity can be used to generate a corresponding similarity relationship chain.

[0112] like Figure 7 The simhash algorithm is used to calculate the similarity chain of the title dimension and the text dimension; the BERT algorithm is used to calculate the similarity chain of the title dimension and the text dimension; the hash algorithm is used to calculate the similarity chain of the picture dimension and the cover image; the picture feature extraction algorithm is used to calculate the similarity chain of the picture dimension and the cover image, etc. This application does not limit how to use the feature vector of the file to calculate the similarity between files, and can use but is not limited to the distance calculation method between vectors.

[0113] For the calculation process of the similarity relationship chains of the above-mentioned different dimensions, an example is given. For the file A to be processed, the sample files B, C, D, E, F, etc. in the sample library are calculated according to the similarity of different dimensions. It can be determined that from dimension 1, A is similar to B and E respectively, and AB and AE can be recorded for this dimension 1; from dimension 2, A is similar to C and D respectively, and AC and AD are obtained, and so on. Sample files similar to A in the sample library under different dimensions are obtained, thereby generating a similarity relationship chain for each dimension.

[0114] It should be understood that the similarity relationship chains of the files to be processed in different dimensions may contain the same sample files; and for the same dimension, the sample files contained in the similarity relationship chains of the files to be processed obtained by using different deep learning algorithms may be the same or different, and this application does not limit this. In addition, the calculation method of the feature vectors of each sample file is similar to the calculation method of the feature vectors of the files to be processed mentioned above. The feature vectors obtained in different dimensions can be stored in the sample library corresponding to the dimension, so that each sample library stores the feature vectors of one dimension of each sample file, but it is not limited to this method of constructing the sample library. Among them, for the feature vectors of the files to be processed and the sample files of the same dimension, the same construction method or different construction methods can be adopted, and this application does not limit this.

[0115] Combined with the above analysis, in the actual application of this application, corresponding sample libraries can be constructed for different business scenarios. However, in the process of similarity judgment of feature vectors of sample files in different sample libraries, the judgment thresholds used may be different. Especially in the application of deduplication of copyright content, the similarity judgment of the copyright dimension may be more stringent. Therefore, the required judgment threshold may be higher than the judgment threshold of other dimensions. The value of each judgment threshold can be determined according to actual business needs.

[0116] Moreover, the storage period of sample files stored in different sample libraries may also be different, which will affect the status information of the corresponding sample files, that is, after exceeding their storage period, they are often no longer usable and can be automatically deleted to reduce the workload of deduplication. This application does not limit the storage period value of each sample file. For example, the storage period of a sample library storing hot content can be 7 days, and the storage period of a sample library storing general content can be 30 days, etc. This application does not elaborate on them one by one.

[0117] Therefore, the present application can also obtain sample libraries corresponding to different dimensions. The above step S13 may specifically include: according to the distributed retrieval method, read the feature vector of each sample file from the sample libraries corresponding to different dimensions, and perform similarity calculation between the feature vector of the file to be processed under the same dimension and the feature vector of the sample file, such as calculating the distance between the vectors to realize the similarity calculation between the file to be processed and the sample file. For the implementation process of distributed retrieval, please refer to the description of the corresponding part of the embodiment below.

[0118] It should be noted that the similarity judgments of the above different dimensions are independent of each other. In practical applications, the similarity judgments of each dimension can be regarded as a layer, and the processing process of each layer can be optimized and expanded separately to meet the needs of practical applications, solving the following problems: Figure 1 The serial filtering approach shown here causes scaling issues.

[0119] Step S14, obtaining a deduplication decision for the file to be processed, and performing deduplication processing on the files in the obtained multiple similar relationship chains according to the deduplication decision.

[0120] In an embodiment of the present application, corresponding deduplication decisions can be pre-configured for different deduplication requirements of files to be processed. In this way, after obtaining similarity relationship chains of multiple different dimensions, the multiple similarity relationship chains obtained can be deduplicated according to the deduplication decisions, that is, a comprehensive judgment is made on which one or more of the file to be processed A and sample files B, C, D, E, etc. in the similarity relationship chain of the file to be processed A should be retained. The present application does not elaborate on the specific implementation process of the deduplication processing.

[0121] Refer to Figure 7 In some embodiments, the sample files that need to participate in the deduplication process can be determined based on the status of each sample file, and the corresponding deduplication decisions can be configured to be generated. In this way, after responding to the deduplication request and obtaining the similarity relationship chains of the files to be processed corresponding to multiple dimensions, deduplication will be performed according to the deduplication decision, and the deduplication results will be fed back to the HBase database, that is, a distributed, column-oriented open source database. In this application, the HBase database can be used to store the published content and metadata, and can be used to build the content database and content storage service equipment in the system.

[0122] In summary, after obtaining the newly uploaded file to be processed, the present application will construct feature vectors corresponding to multiple dimensions of the file to be processed, thereby using the feature vectors of different dimensions to calculate the similarity of the file to be processed and the sample file, respectively. Compared with the single-dimensional similarity deduplication processing method, the similarity recall rate and deduplication accuracy are improved, and compared with the multi-dimensional serial deduplication processing method, after realizing the similarity calculation of the file to be processed and the sample file in one dimension, the present application does not directly perform deduplication processing, but adopts a hierarchical decoupling structure, and obtains the similarity relationship chains corresponding to these multiple dimensions from the similarity calculation results of multiple dimensions, and performs deduplication processing on the files in these multiple similarity relationship chains according to the preset deduplication decision. It can be seen that this parallel deduplication method greatly improves the deduplication efficiency and accuracy, and avoids the inability to obtain the similarity of the file to be processed and the eliminated sample file in other dimensions after the similarity deduplication of the serial historical dimension, resulting in the inability to meet the deduplication processing requirements of specific dimensions, and can better meet business needs.

[0123] In practical applications, short videos have gradually become one of the dominant content forms of mobile Internet, replacing the consumption of graphic content to a certain extent, and gradually gaining a dominant position in graphic media such as news and social platforms. It can include video content played on various new media platforms, suitable for viewing in mobile and short leisure states, and pushed frequently, ranging from a few seconds to a few minutes, and can integrate themes such as skill sharing, humor, fashion trends, social hot spots, street interviews, public welfare education, advertising creativity, and commercial customization. As for the deduplication of video files, manual review is usually used at present, which is costly, inefficient, and has low accuracy. It can be seen that efficient and reliable deduplication of video files has become a technical problem that needs to be solved urgently.

[0124] In order to solve the above technical problems, in combination with the information processing method described in the above embodiment, refer to Figure 8 , which is a flowchart of another optional example of the information processing method proposed in this application. This embodiment mainly describes the implementation process of vectorizing the video file contained in the to-be-processed file. For the vectorization process of other types of file contents, reference can be made to the description of the corresponding part of the above embodiment, which will not be described in detail in this embodiment. Figure 8 As shown, the information processing method provided in this embodiment may include:

[0125] Step S21, extracting frames from the video file to obtain multiple video frames;

[0126] A frame is the smallest unit of a single image in an animation, equivalent to each frame on a film. On the timeline of animation software, a frame is represented as a frame or a mark. A key frame is equivalent to the original drawing in a two-dimensional animation, and refers to the frame where the key action of a character or object is moving or changing. The animation between key frames can be created by software, which is called a transition frame or an intermediate frame.

[0127] This embodiment can obtain key frame images by extracting frames from video files, so as to construct a semantic fingerprint vector of the video through video frames to represent the video itself, thereby realizing the vectorization of the video file. Combined with the description of the frame extraction process in the above embodiment, this application can select uniform frame extraction, variable length frame extraction or other frame extraction methods according to actual needs. This embodiment does not describe the specific implementation process of step S21 in detail.

[0128] It should be noted that in the process of extracting multiple video frames from a video file, there may be small segments with overlapping content, such as 5 seconds, 1 second or 2 seconds as the overlapping unit, which is related to the video length. The more overlaps, the greater the amount of calculation, but the better the effect. Whether content overlap is required and how long the content overlap should be can be determined based on actual application requirements.

[0129] Step S22, obtaining the picture semantic vectors corresponding to each of the multiple video frames;

[0130] Since each video frame is equivalent to a separate picture, the picture semantic vector of each video frame can be obtained according to the picture vector processing method. The specific implementation process can refer to the description of the corresponding part of the above embodiment, which will not be repeated in this embodiment.

[0131] Step S23, concatenating the obtained multiple image semantic vectors to obtain a semantic fingerprint vector of the video file.

[0132] In some embodiments, in the process of constructing the semantic fingerprint vector of a video file based on multiple video frames obtained by frame extraction, the picture semantic vectors of the multiple video frames obtained above can be directly spliced, but the amount of data is large. In order to reduce the amount of data, the present application can also merge the picture semantic vectors corresponding to several video frames adjacent in time series, such as merging the picture semantic vectors of 5 to 10 frames, etc. The present application does not limit the implementation method of how to construct the semantic fingerprint vector of a video file, and is not limited to the implementation method described in this embodiment.

[0133] To sum up, for the video files contained in the files to be processed, this embodiment can use multiple video frames obtained by frame extraction to realize the vectorization of the video file, that is, to obtain a semantic fingerprint vector that represents the video file, so as to realize the automatic similarity calculation between the video file and the sample video file, and obtain the similarity relationship chain of the video files, and then realize the deduplication processing of the sample video files. Compared with the traditional manual review method, it greatly improves the efficiency and accuracy of video deduplication processing, and reduces the deduplication cost.

[0134] Furthermore, in some embodiments, after completing the calculation of similarity relationship chains of different dimensions of the to-be-processed files, the obtained similarity relationship chains may be further verified, especially the verification of similarity relationship chains of the same dimension obtained by using different deep learning algorithms, such as Figure 7 As shown, the similarity relationship chains of the text dimension obtained by using the simhash algorithm and the BERT algorithm respectively can be verified, so that based on the verification results, the similarity relationship chains with low similarity accuracy can be deleted, thereby reducing the data processing volume of subsequent deduplication processing and improving the efficiency and accuracy of deduplication processing.

[0135] Among them, for the verification of the above-mentioned similarity relationship chain, the present application can be implemented using a twin neural network algorithm, but is not limited to this deep learning algorithm, and can be determined based on factors such as the dimension type corresponding to the similarity relationship chain to be verified.

[0136] In addition, combined with the description of the above embodiments, since the calculation process of the similarity relationship chain of each dimension is independent of each other, the present application can expand the calculation of feature vectors and similarity relationship chains in parallel according to business needs, that is, adopt a new deep learning algorithm to obtain a new feature vector of a certain dimension of the file to be processed, and obtain a new similarity relationship chain of the file to be processed under this dimension. Figure 7 The title edit distance relationship chain calculation process shown, that is, for the title dimension of the file to be processed, the edit distance algorithm can be used to construct a title edit distance vector, and then the similarity calculation between the file to be processed and the sample file in the title dimension is realized, and the similarity relationship chain of the file to be processed in the title dimension is obtained.

[0137] It should be understood that in the extended application of the similarity detection algorithm, it may be necessary to implement the expansion of the sample library for new business scenarios. Combined with the above description of the sample library corresponding to different dimension configurations, the information processing method proposed in this application may also include:

[0138] A deduplication request for the first business scenario is obtained, and it is detected that there is no sample library for the first business scenario in the existing sample library, and a feature vector of a sample file associated with the first business scenario is obtained, and the first business scenario is determined as the first dimension (which may be a dimension different from the corresponding dimension of the existing sample library, and the specific content is not limited), and the obtained feature vector is used to construct a sample library corresponding to the first dimension. Of course, the sample library may also include sample files associated with the first business scenario, and the specific method for obtaining the sample files is not limited. Afterwards, the deduplication request can be responded to, and the files to be processed for this deduplication processing can be obtained, so as to meet the deduplication requirements of the first business scenario according to the deduplication processing process described in the above embodiment.

[0139] In some embodiments, during the deduplication retrieval process of the sample files in the sample library, a suitable retrieval method can be selected according to the amount of data stored in the sample library to achieve similar retrieval of each sample file. Take simhash vector deduplication as an example to illustrate, that is, after the system responds to the deduplication request in real time, uses the simhash algorithm to extract the features of different dimensions of the files to be processed, and constructs the simhash vectors of the corresponding dimensions, how to use the simhash vector to perform similar retrieval process on the sample files of the corresponding sample library is explained. Assuming that the number of sample files to be deduplicated in the sample library is 30 million, and the Hamming distance between the simhash vectors is less than 4 (i.e., the similarity threshold), it is considered that the corresponding sample file is similar to the file to be processed, thereby achieving a preliminary recall of the sample files to be deduplicated in the sample library.

[0140] In a possible implementation, in order to improve the retrieval speed, the present application may adopt a retrieval method based on the drawer principle, referring to Fig. 9 As shown in the schematic diagram of the retrieval principle, since the length of a simhash vector is usually 64 bits, this embodiment can divide the obtained 64-bit hash value into 4 parts, and store the sample files in buckets, but it is not limited to 4 parts, and can also be 5 parts, etc. Generally, the more parts are divided, the larger the storage space occupied, but the faster the retrieval speed. The number of parts can be determined according to actual needs. This embodiment only takes 4 parts as an example for explanation. According to this division method, for the amount of data in the above sample library, each part can contain 65536 buckets, with an average of 450 elements (i.e., sample files or their feature vectors) per bucket.

[0141] Based on this, after obtaining the simhash value of the file to be processed, it is necessary to find the nearest sample simhash values ​​and corresponding sample files from the massive sample simhash values ​​stored in the sample library, and then divide the simhash value into 4 segments, each with 16 bits, and put the simhash value and file ID with the same segment into the same bucket. The bucket ID is the segment simhash value, thus forming 4 groups of buckets. For any two simhash values ​​with a Hamming distance less than 4, there must be at least one 16-bit segment that is exactly the same, which means that the two must hit at least the same bucket. Therefore, for any file to be processed, you only need to take out all the sample files in the 4 buckets for comparison, and you can find all similar sample files with a Hamming distance less than 4. 30 million articles are scattered in 4 copies. After being stored in buckets, there are only 450 elements in each bucket, which can be quickly traversed and compared, thus solving the time-consuming problem of real-time retrieval. For text vectors with relatively short hash values, similarity relationship chains can be calculated in this way. The specific implementation process will not be described in detail in this application.

[0142] If the number of sample files that need to be deduplicated in the sample library is over 100 million, and the corresponding content vector is relatively long, such as the semantic BERT vectors of images, videos, and text based on the twin network mentioned above, which are usually 1536 dimensions long, and each dimension is represented by a 4-byte integer, the memory space occupied by the vector will expand rapidly, and a single machine will no longer be able to solve the problem. At the same time, the fault tolerance and disaster tolerance of the service will decrease, and it cannot be expanded in parallel.

[0143] In order to improve the above problems, in the scenario of large-scale deduplication, a scalable distributed index can be used, that is, a distributed vector retrieval service device is required to provide support. For example, each FaissNode node shown in Figure 10 can provide an entry for query retrieval index through SearchEngine (search engine), and each node is expanded through a consistent hashing algorithm. The specific expansion process is not described in detail.

[0144] Among them, Faiss is an open source library for clustering and similarity search developed by the Facebook AI team. It provides efficient similarity search and clustering for dense vectors, supports searches of billions of vectors, and is currently the most mature approximate neighbor search library. This application does not elaborate on the implementation of content retrieval and index organization.

[0145] Simply put, the consistent hashing algorithm can change the mapping relationship between existing service requests and request processing servers as little as possible when removing or adding a server, and meet the monotonicity requirement as much as possible. In a common distributed cluster, there is a one-to-one correspondence between service requests and request processing servers, that is, the mapping relationship between service requests and processing servers is fixed, and a certain request is processed by a fixed server.

[0146] Among them, for Fig.10 Other components, such as the extraction of multimodal features, may include but are not limited to Fig.10 The feature extraction of multiple dimensions such as video, picture, audio, text, etc. is shown to realize the vectorization of the features of the corresponding dimensions and obtain the feature vectors of the corresponding dimensions. The specific implementation process can refer to the description of the corresponding part of the above embodiment. Afterwards, combined with the description of the acquisition process of the similarity relationship chain in the above embodiment, this embodiment can adopt the deduplication service of the corresponding dimension, access each sample library, namely FaissNode, through the query entry provided by the search engine, and realize the calculation of the global similarity relationship chain. The specific calculation process is not described in detail in this embodiment. Under normal circumstances, the above information processing process can be implemented by the dispatch center service equipment for unified scheduling, and the specific scheduling method is not limited.

[0147] In some embodiments, for the system described above that is applicable to the information processing method proposed in this application, in addition to supporting real-time online deduplication processing of massive vectors, it can have higher read and write performance. Based on this, the present application has studied and found that if the FaissNode node performs read and write operations at the same time, especially when a large amount of data is written, the performance of reading similar relationship chains must be ensured, and in the process of rebuilding the index, the node performance will be seriously affected, and it will not be able to cope with large-scale deduplication. Therefore, the present application proposes to adopt a partial read-write separation method to achieve distributed retrieval. Taking the text BERT vector as an example (the video content vector can also be processed in the same way), the newly uploaded content flow can be recorded by the hippo (a message middleware for a distributed scheduling system) message queue cache, and the specific caching process will not be described in detail.

[0148] It can be seen that referring to Fig.11As shown in the schematic diagram of the read-write separation principle structure, the present application can use the historical feature vectors (such as the text BERT vector) obtained within the first time from the current moment to construct the first index information (that is, to construct a large index), and the first index information is configured as a read-only attribute, that is, only the read function is supported for the first index information; at the same time, the feature vector (text BERT vector) obtained within the second time from the current moment (such as the same day) can be used to construct the second index information (that is, to construct a small index), and the second index information is configured as a read-write separation attribute, and the second time is less than the first time. It can be seen that the second index information supports read and write functions and can respond to read requests and write requests, while the first index information can only respond to read requests. The present application does not elaborate on the specific construction process of these two types of indexes.

[0149] Among them, in the process of deduplication of sample library storage data, such as Fig.11 As shown, the present application can use a double buffering method to implement the update and reconstruction of the first index information and the second index information. Specifically, the hierarchical query index plug-in EF_index_mgr control switch register can be used to eliminate old data by using double buffer switching to rebuild the size index. Fig.11 As shown, the reconstruction process does not affect the read operation of the first index information and the read and write operations of the second index information.

[0150] Based on the information processing methods described in the above embodiments, Fig.12 As shown, a flow chart of an optional scenario applicable to the information processing method is provided. Faced with a large number of files (such as pictures, texts, atlases, videos, segment contents, etc.), the present application establishes a multi-dimensional and multi-level distributed content deduplication architecture to support the online efficient and accurate deduplication of a large number of files, including the elimination of physical duplication, semantic duplication and thematic duplication in various content dimensions. As analyzed above, a variety of similarity algorithms can be used to obtain similarity relationship chains in different dimensions to ensure that the obtained similarity relationship chains are reliable. In the similarity retrieval process, a distributed and read-write separated retrieval method is adopted to support the deduplication of a large number of files with different validity periods. Compared with the serial retrieval method, the similarity retrieval efficiency is improved and the read-write performance is improved.

[0151] Specifically, combined with the above vectorized description of features of different dimensions, which include text vectors, image vectors, and video vectors, etc., in order to improve the accuracy and recall rate of deduplication, the twin network and BERT are used to generate semantic vectors, and audio vector features can also be integrated, which greatly improves the diversity of the obtained feature vectors. Among them, in the vectorized deduplication processing of multiple dimensions such as the title, cover, and text content of the file, hierarchical vectorized deduplication is adopted, which greatly improves the efficiency of deduplication, and greatly improves the timeliness of file link processing and the efficiency of manual processing; at the same time, combined with business needs, this application can perform multi-level granularity processing on the deduplication sample library, achieving a good balance between effect and efficiency.

[0152] Also, if Fig.12 As shown, the present application divides deduplication judgment into two stages: duplicate judgment and deduplication decision, realizing decision coupling contact. It no longer uses the traditional solution to directly deduplication after completing similarity detection in one dimension, thus avoiding repeated development of the same deduplication requirement and facilitating subsequent targeted optimization of the two stages.

[0153] In addition, the duplicate relations obtained through deduplication can be applied to the recommendation system, so that it can directly read the required duplicate relations and recommend similar files to users, greatly improving the recommendation efficiency and accuracy. Multi-dimensional monitoring of duplicate relations can also be achieved through monitoring equipment and manual work, so as to detect decline in advance, maintain the system in time, and ensure the reliability of deduplication. The specific implementation process will not be described in detail.

[0154] Combined with the above analysis, it should be understood that Fig.12 The system formed by the devices that execute each step does not have a single point, and each layer can be horizontally scaled. It is not limited to the storage layer and recall layer listed above, and does not rely on local files. An anti-avalanche mechanism can be used between layers, and upgrades will not interrupt services. The information generated by the deduplication process can be statistically analyzed and recorded to make the deduplication process traceable.

[0155] Reference Fig.13 , is a schematic diagram of an optional example of the structure of the information processing device proposed in this application, and the device may include:

[0156] The module m10 for obtaining files to be processed is used to obtain files to be processed;

[0157] A feature vector construction module m20 is used to construct feature vectors of different dimensions of the file to be processed;

[0158] A similarity relationship chain obtaining module m30 is used to calculate the similarity between the to-be-processed file and the sample files in the sample library respectively by using the feature vectors of different dimensions, and obtain a similarity relationship chain of the to-be-processed file under the corresponding dimension, wherein the similarity relationship chain includes the to-be-processed file and the sample files under the corresponding dimension whose similarity with the to-be-processed file is greater than the corresponding threshold value;

[0159] The deduplication processing module m40 is used to obtain a deduplication decision for the file to be processed, and perform deduplication processing on the files in the obtained multiple similar relationship chains according to the deduplication decision.

[0160] In a possible implementation, the deduplication processing module m40 may include:

[0161] A status information acquisition unit, used to acquire current status information of sample files in each sample library, wherein the current status information can indicate whether the corresponding sample file participates in deduplication processing;

[0162] The deduplication decision configuration unit is used to configure the deduplication decision for the to-be-processed file according to the current status information of each sample file obtained.

[0163] In some embodiments, the feature vector construction module m20 may include:

[0164] A feature vector acquisition unit, used for acquiring a feature vector of a corresponding dimension of the to-be-processed file according to different feature extraction algorithms;

[0165] Among them, the different feature extraction algorithms may include multiple combinations of hash algorithms, twin neural network algorithms, bidirectional encoding representation of transformers BERT algorithms, and edit distance algorithms, etc.; the feature vectors obtained above may include one or more combinations of summary feature vectors, title feature vectors, text content feature vectors, text semantic vectors, picture semantic vectors, picture feature vectors, video semantic fingerprint vectors, audio feature vectors, etc. For the operation principles of each feature extraction algorithm and the implementation process of obtaining the feature vectors of the corresponding dimensions, reference may be made to the description of the corresponding parts of the above embodiments.

[0166] Accordingly, the similarity relationship chain obtaining module m30 may include:

[0167] A first similarity calculation unit is used to calculate the similarity of the to-be-processed file and the sample files in the sample library respectively by using the feature extraction algorithm based on which the feature vectors of different dimensions are constructed and the feature vectors of corresponding dimensions;

[0168] The similarity relationship chain obtaining unit is used to obtain the similarity relationship chains of the to-be-processed files in different dimensions according to the similarity calculation results.

[0169] In some embodiments, if the to-be-processed file includes a video file, the feature vector construction module m20 may further include:

[0170] A frame extraction unit, used for performing frame extraction processing on the video file to obtain multiple video frames;

[0171] The semantic fingerprint vector construction unit is used to construct the semantic fingerprint vector of the video file according to the multiple video frames.

[0172] In a possible implementation, the semantic fingerprint vector construction unit may include:

[0173] A picture semantic vector acquisition unit, used to acquire the picture semantic vectors corresponding to each of the multiple video frames;

[0174] A vector concatenation unit is used to concatenate multiple acquired image semantic vectors to obtain a semantic fingerprint vector of the video file.

[0175] In some embodiments, the information processing device proposed in this application may also include:

[0176] A sample library acquisition unit is used to acquire sample libraries corresponding to different dimensions;

[0177] The similarity relationship chain obtaining module m30 may include:

[0178] A vector reading unit, used for reading the feature vector of each sample file from the sample library corresponding to the different dimensions in a distributed retrieval manner;

[0179] The second similarity calculation unit is used to calculate the similarity between the feature vector of the to-be-processed file and the feature vector of the sample file in the same dimension.

[0180] In order to improve the reading and writing efficiency in the distributed retrieval process, the above device may further include:

[0181] A first index building module, configured to build first index information for each sample library using a historical feature vector obtained within a first time from a current moment, wherein the first index information is configured as a read-only attribute;

[0182] A second index building module, used to build second index information using a feature vector obtained within a second time from the current moment, wherein the second index information is configured as a read-write separation attribute, and the second time is less than the first time;

[0183] The index reconstruction module is used to adopt a double buffering method to realize the update and reconstruction of the first index information and the second index information.

[0184] In order to achieve parallel expansion of the sample library, the information processing device proposed in this application may also include:

[0185] A deduplication request acquisition module, used to acquire a deduplication request for a first business scenario;

[0186] A detection module, configured to detect that no sample library for the first business scenario exists in the existing sample library, and obtain a feature vector of a sample file associated with the first business scenario;

[0187] A sample library construction module, used to determine the first business scenario as a first dimension, and use the acquired feature vector to construct a sample library corresponding to the first dimension;

[0188] The request response module is used to respond to the deduplication request and obtain the file to be processed for the first business scenario.

[0189] It should be noted that the various modules, units, etc. in the above-mentioned device embodiments can be stored in the memory as program modules, and the processor executes the above-mentioned program modules stored in the memory to implement the corresponding functions. For the functions implemented by each program module and its combination, as well as the technical effects achieved, please refer to the description of the corresponding parts of the above-mentioned method embodiments, which will not be repeated in this embodiment.

[0190] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the steps of the above-mentioned information processing method. The implementation process of the information processing method can refer to the description of the above-mentioned method embodiment, and will not be repeated in this embodiment.

[0191] Finally, it should be noted that the various embodiments in this specification are described in a progressive or parallel manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0192] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0193] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0194] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the core idea or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An information processing method, characterized in that: The method comprises: Get the files to be processed; Constructing feature vectors of different dimensions of the to-be-processed file, comprising: obtaining feature vectors of corresponding dimensions of the to-be-processed file according to different feature extraction algorithms, wherein, for the same dimension of the to-be-processed file, different feature extraction algorithms are used to obtain multiple feature vectors of the to-be-processed file in the same dimension; Obtaining sample libraries corresponding to the different dimensions; Using the feature vectors of different dimensions, respectively performing similarity calculations on the to-be-processed file and the sample files in the sample library, specifically comprising: reading the feature vectors of each sample file from the sample library corresponding to the different dimensions in a distributed retrieval manner, performing similarity calculations on the feature vectors of the to-be-processed file in the same dimension obtained by using different feature extraction algorithms and the feature vectors of the sample files to obtain similarity calculation results, and obtaining similarity relationship chains of the to-be-processed file in the corresponding dimension according to the similarity calculation results, wherein the similarity relationship chains include the to-be-processed file and the sample files in the corresponding dimension whose similarity to the to-be-processed file is greater than a corresponding threshold value; obtaining the similarity relationship chains is used to achieve hierarchical decoupling of similarity judgment and deduplication decision; wherein, after obtaining the similarity relationship chains of different dimensions, verifying the similarity relationship chains of the same dimension obtained by using different feature extraction algorithms, and based on the verification results, deleting the similarity relationship chains with low similarity accuracy to obtain global similarity relationship chains, wherein the global similarity relationship chains include multiple similarity relationship chains corresponding to different dimensions; Acquire current status information of sample files in each sample library, where the current status information can indicate whether the corresponding sample files participate in deduplication processing; Based on the current status information of each sample file obtained, a deduplication decision for the file to be processed is configured, and hierarchical parallel deduplication processing is performed on the files in the obtained multiple similar relationship chains according to the deduplication decision, wherein a similar relationship chain of one dimension is regarded as a level, and the processing processes of each level are independent of each other.

2. The method according to claim 1, characterized in that The different feature extraction algorithms include multiple combinations of hash algorithms, twin neural network algorithms, BERT algorithms, and edit distance algorithms; the feature vectors include multiple combinations of summary feature vectors, title feature vectors, text content feature vectors, text semantic vectors, picture semantic vectors, picture feature vectors, and semantic fingerprint vectors of videos.

3. The method according to claim 2, characterized in that If the file to be processed includes a video file, the step of constructing feature vectors of different dimensions of the file to be processed further includes: Performing frame extraction processing on the video file to obtain multiple video frames; A semantic fingerprint vector of the video file is constructed based on the multiple video frames.

4. The method according to claim 3, characterized in that The step of constructing a semantic fingerprint vector of the video file based on the multiple video frames includes: Obtaining picture semantic vectors corresponding to each of the multiple video frames; The obtained multiple image semantic vectors are spliced ​​to obtain a semantic fingerprint vector of the video file.

5. The method according to claim 1, characterized in that The method further comprises: For each of the sample libraries, first index information is constructed using the historical feature vector obtained within the first time from the current moment, wherein the first index information is configured as a read-only attribute; Using the feature vector obtained within a second time from the current moment, constructing second index information, wherein the second index information is configured as a read-write separation attribute, and the second time is less than the first time; A double buffering method is adopted to realize the updating and reconstruction of the first index information and the second index information.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Obtain a deduplication request for the first business scenario; Detecting that no sample library for the first business scenario exists in an existing sample library, and obtaining a feature vector of a sample file associated with the first business scenario; Determine the first business scenario as a first dimension, and use the acquired feature vector to construct a sample library corresponding to the first dimension; In response to the deduplication request, a file to be processed for the first business scenario is obtained.

7. An information processing device, characterized in that: The device comprises: A module for obtaining files to be processed, used for obtaining files to be processed; A feature vector construction module, used to construct feature vectors of different dimensions of the file to be processed; A similarity relationship chain obtaining module is used to use the feature vectors of different dimensions to respectively calculate the similarity of the to-be-processed file and the sample files in the sample library to obtain similarity calculation results, and obtain the similarity relationship chain of the to-be-processed file in the corresponding dimension based on the similarity calculation results, wherein the similarity relationship chain includes the to-be-processed file and the sample files whose similarity with the to-be-processed file in the corresponding dimension is greater than the corresponding threshold value; the similarity relationship chain is obtained to achieve hierarchical decoupling of similarity judgment and deduplication decision; wherein, after obtaining the similarity relationship chains of different dimensions, the similarity relationship chains of the same dimension obtained by using different feature extraction algorithms are verified, and based on the verification results, the similarity relationship chains with low similarity accuracy are deleted to obtain a global similarity relationship chain, wherein the global similarity relationship chain includes multiple similarity relationship chains corresponding to different dimensions; A status information acquisition unit, used to acquire current status information of sample files in each sample library, wherein the current status information can indicate whether the corresponding sample file participates in deduplication processing; A deduplication decision configuration unit, configured to configure a deduplication decision for the to-be-processed file according to the current status information of each sample file obtained, and to perform hierarchical parallel deduplication processing on the files in the obtained multiple similar relationship chains according to the deduplication decision, wherein a similar relationship chain of one dimension is regarded as a layer, and the processing processes of each layer are independent of each other; The feature vector building module includes: A feature vector acquisition unit, used to acquire feature vectors of corresponding dimensions of the to-be-processed file according to different feature extraction algorithms, wherein for the same dimension of the to-be-processed file, different feature extraction algorithms are used to acquire multiple feature vectors of the to-be-processed file in the same dimension; A sample library acquisition unit, used to acquire sample libraries corresponding to the different dimensions; The similarity relationship chain obtaining module includes: A vector reading unit, used for reading the feature vector of each sample file from the sample library corresponding to the different dimensions in a distributed retrieval manner; The second similarity calculation unit is used to calculate the similarity between the feature vector of the to-be-processed file in the same dimension obtained by using different feature extraction algorithms and the feature vector of the sample file.

8. The device according to claim 7, characterized in that The different feature extraction algorithms include multiple combinations of hash algorithms, twin neural network algorithms, BERT algorithms, and edit distance algorithms; the feature vectors include multiple combinations of summary feature vectors, title feature vectors, text content feature vectors, text semantic vectors, picture semantic vectors, picture feature vectors, and semantic fingerprint vectors of videos.

9. The device according to claim 8, characterized in that If the file to be processed includes a video file, the feature vector construction module further includes: A frame extraction unit, used for performing frame extraction processing on the video file to obtain multiple video frames; The semantic fingerprint vector construction unit is used to construct the semantic fingerprint vector of the video file according to the multiple video frames.

10. The device according to claim 9, characterized in that The semantic fingerprint vector construction unit includes: A picture semantic vector acquisition unit, used to acquire the picture semantic vectors corresponding to each of the multiple video frames; The vector concatenation unit is used to concatenate the acquired multiple image semantic vectors to obtain the semantic fingerprint vector of the video file.

11. The device according to claim 7, characterized in that The device also includes: A first index building module, configured to build first index information for each sample library using a historical feature vector obtained within a first time from a current moment, wherein the first index information is configured as a read-only attribute; A second index building module, used to build second index information using a feature vector obtained within a second time from the current moment, wherein the second index information is configured as a read-write separation attribute, and the second time is less than the first time; The index reconstruction module is used to adopt a double buffering method to realize the update and reconstruction of the first index information and the second index information.

12. The device according to any one of claims 7 to 11, characterized in that: The device also includes: A deduplication request acquisition module, used to acquire a deduplication request for a first business scenario; A detection module, configured to detect that no sample library for the first business scenario exists in the existing sample library, and obtain a feature vector of a sample file associated with the first business scenario; A sample library construction module, used to determine the first business scenario as a first dimension, and use the acquired feature vector to construct a sample library corresponding to the first dimension; The request response module is used to respond to the deduplication request and obtain the file to be processed for the first business scenario.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the steps of the information processing method according to any one of claims 1 to 6.

Images