File reuse detection method, device and equipment

Abstract

The present application discloses a file reuse detection method, device and equipment. The method includes: obtaining a target file to be detected; determining a target detection scheme for the target file from a plurality of reuse detection schemes, wherein the reuse detection scheme includes at least: a fingerprint type to be detected, and different reuse detection schemes correspond to different fingerprint types; determining a target fingerprint type corresponding to the target detection scheme, and determining the first fingerprint information corresponding to the target fingerprint type from a plurality of fingerprint information corresponding to the target file; determining the second fingerprint information of each file in the target database, and determining the similarity between the second fingerprint information and the first fingerprint information, and when the similarity is greater than a preset threshold, determining that the file corresponding to the second fingerprint information in the target database is a reused file of the target file. The present application solves the technical problem of the lack of accuracy and flexibility in the detection of file reuse in file security protection.

Description

Technical Field

[0001] The present application relates to the field of file security technology, and in particular to a file reuse detection method, device and equipment. Background Art

[0002] With the development of enterprises and the increasingly changing information security situation, a large number of sensitive files are involved in the work process, so an intelligent means is needed to accurately identify and properly protect them. However, in the relevant technology, there are several difficulties in identifying sensitive files: (1) As a program running on the terminal device, the source code must be invisible, and the running and storage resources must be low, and cross-operating system support must be available; (2) It must be traceable after the fact. For example, if the file has been leaked due to lack of protection in the early stage, it is necessary to trace back to which file it was leaked through; (3) It must adapt to various changes in the situation, that is, file detection is only related to the content and has nothing to do with other things.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a file reuse detection method, device and equipment to at least solve the technical problem of lack of accuracy and flexibility in file reuse detection in file security protection.

[0005] According to one aspect of an embodiment of the present application, a file reuse detection method is provided, comprising: obtaining a target file to be detected; determining a target detection scheme for the target file from multiple reuse detection schemes, wherein the reuse detection scheme includes at least: a fingerprint type to be detected, and different reuse detection schemes correspond to different fingerprint types; determining a target fingerprint type corresponding to the target detection scheme, and determining first fingerprint information corresponding to the target fingerprint type from multiple fingerprint information corresponding to the target file; determining second fingerprint information of each file in a target database, and determining a similarity between the second fingerprint information and the first fingerprint information, and when the similarity is greater than a preset threshold, determining that the file in the target database corresponding to the second fingerprint information is a reused file of the target file.

[0006] According to another aspect of an embodiment of the present application, a file reuse detection device is also provided, including: an acquisition module, used to acquire a target file to be detected; a first determination module, used to determine a target detection scheme for the target file from multiple reuse detection schemes, wherein the reuse detection scheme includes at least: a fingerprint type that needs to be detected, and different reuse detection schemes correspond to different fingerprint types; a second determination module, used to determine the target fingerprint type corresponding to the target detection scheme, and determine the first fingerprint information corresponding to the target fingerprint type from the multiple fingerprint information corresponding to the target file; a third determination module, used to determine the second fingerprint information of each file in the target database, and determine the similarity between the second fingerprint information and the first fingerprint information, and when the similarity is greater than a preset threshold, determine that the file corresponding to the second fingerprint information in the target database is a reused file of the target file.

[0007] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, wherein the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above-mentioned file reuse detection method.

[0008] According to another aspect of an embodiment of the present application, a file reuse detection device is also provided, including: a processor and a memory, the memory being connected to the processor and being used to provide the processor with instructions for processing the following processing steps: obtaining a target file to be detected; determining a target detection scheme for the target file from a plurality of reuse detection schemes, wherein the reuse detection scheme at least includes: a fingerprint type to be detected, and different reuse detection schemes correspond to different fingerprint types; determining a target fingerprint type corresponding to the target detection scheme, and determining a first fingerprint information corresponding to the target fingerprint type from a plurality of fingerprint information corresponding to the target file; determining second fingerprint information for each file in a target database, and determining a similarity between the second fingerprint information and the first fingerprint information, and when the similarity is greater than a preset threshold, determining that the file in the target database corresponding to the second fingerprint information is a reused file of the target file.

[0009] In the embodiment of the present application, firstly, the target file to be detected is obtained; then, the target detection scheme of the target file is determined from multiple reuse detection schemes, wherein the reuse detection scheme includes at least the fingerprint type to be detected, and different reuse detection schemes correspond to different fingerprint types; then, the target fingerprint type corresponding to the target detection scheme is determined, and the first fingerprint information corresponding to the target fingerprint type is determined from the multiple fingerprint information corresponding to the target file; finally, the second fingerprint information of each file in the target database is determined, and the similarity between the second fingerprint information and the first fingerprint information is determined. When the similarity is greater than a preset threshold, the file corresponding to the second fingerprint information in the target database is determined to be a reused file of the target file. Among them, the user can select the corresponding reuse detection scheme according to his own needs for file similarity detection, and the terminal determines the corresponding type of fingerprint information from the target file according to the reuse detection scheme selected by the user, and then compares the similarity with the fingerprint information in the target database to obtain the detection result. This process not only improves the flexibility of file reuse detection, but also improves the accuracy of the detection result because multiple types of fingerprint information can be detected, thereby solving the technical problem of lack of accuracy and flexibility in file reuse detection in file security protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0011] Figure 1 is a schematic diagram of the structure of an optional computer terminal according to an embodiment of the present application;

[0012] Figure 2 is a flowchart of an optional file reuse detection method according to an embodiment of the present application;

[0013] Figure 3 is a schematic diagram of an optional file reuse classification according to an embodiment of the present application;

[0014] Figure 4 It is a structural schematic diagram of an optional file reuse detection device according to an embodiment of the present application. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0016] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0017] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0018] File fingerprint: used to identify a file and convert standard forms into sensitive information types

[0019] File reuse: A reused file is obtained by directly copying and pasting the original file or simply transforming and modifying the content of the original file. The reused file is essentially the same file as the original file. The detection of file reuse is an important detection link for file security and leakage prevention.

[0020] Example 1

[0021] According to an embodiment of the present application, a file reuse detection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0022] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a file reuse detection method. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0023] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0024] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the file reuse detection method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the vulnerability detection method of the above-mentioned application. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0025] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0026] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0027] In the above operating environment, the present application embodiment provides an optional file reuse detection method, such as Figure 2 As shown, the method at least includes steps S202-S208, wherein:

[0028] Step S202, obtaining the target file to be detected.

[0029] The file reuse detection method in the embodiment of the present application can be executed by the cloud or by the terminal. The cloud mainly refers to the network side device, and the terminal can be a terminal device such as a computer or a tablet computer. Taking the terminal as an example, the terminal first obtains the target file that needs to be detected for file reuse. For example, in an office scene, when the user needs to send a target file in the computer, the computer will automatically perform a file reuse detection on the target file; optionally, the user can also enter the target file to be detected in the interactive interface provided by the computer, and the terminal will perform a file reuse detection on it.

[0030] Figure 3 An optional classification of file reuse types is shown, in which file reuse is mainly divided into two categories: low-fuzzy reuse and high-fuzzy reuse. Specifically, low-fuzzy reuse can usually be divided into: 1) copy and paste, which can be further divided into copy and paste of the entire content of the file and copy and paste of part of the content; 2) approximate copying, including inserting, deleting, and replacing operations in the file content, or splitting and merging sentences in the file; 3) modified copying, mainly refers to reordering sentences or paragraphs in the file, or modifying the syntax. High-fuzzy reuse can usually be divided into: 1) text reuse, mainly including synonym reuse in the file, or summary reuse for keywords and other information; 2) file reuse, mainly including conversion of file types, or file-level reorganization, etc.

[0031] Step S204, determining a target detection scheme for the target file from a plurality of multiplexed detection schemes, wherein the multiplexed detection schemes at least include: a fingerprint type to be detected, and different multiplexed detection schemes correspond to different fingerprint types.

[0032] For the above-mentioned multiple file reuse types, the cloud / terminal provides multiple reuse detection schemes, wherein each reuse detection scheme is used to perform at least the following steps: setting a preset fingerprint type and generating fingerprint information corresponding to the preset fingerprint type; performing similarity matching based on the fingerprint information corresponding to the preset fingerprint type and the fingerprint information of each file in the target database, and determining the reused file from the target database based on the matching results.

[0033] Specifically, the above-mentioned multiple reuse detection schemes at least include: a first detection scheme for low-fuzziness reuse detection and a second detection scheme for high-fuzziness reuse detection, wherein the first detection scheme mainly detects the target file based on the fingerprint information of the first fingerprint type, wherein the fingerprint information of the first fingerprint type includes: fingerprint information in paragraph units, and the fingerprint information of the first fingerprint type is also called hash fingerprint; the second detection scheme mainly detects the target file based on the fingerprint information of the second fingerprint type, wherein the fingerprint information of the second fingerprint type includes: fingerprint information in keyword units, also called term fingerprint, or fingerprint information in text semantic vector units, also called embedding fingerprint. Among them, by combining hash fingerprints, term fingerprints and embedding fingerprints, it is possible to achieve layered protection and realize customized protection measures of different levels, taking into account both efficiency and effectiveness.

[0034] In some optional embodiments of the present application, when determining the target detection scheme for the target file, the target detection scheme for the target file can be pre-set in multiple multiplexed detection schemes. For example, the target detection scheme is set by default to the first detection scheme or the second detection scheme. Of course, the target detection scheme can also be set by default to the first detection scheme and the second detection scheme, that is, the target file is detected according to the first detection scheme and the second detection scheme at the same time.

[0035] Optionally, in response to the first selection instruction of the target object, a target detection scheme corresponding to the first selection instruction can be selected from multiple reuse detection schemes. For example, after the user inputs the target file to be detected in the computer, multiple reuse detection schemes are displayed in the computer interaction interface for the user to choose, such as displaying a high-fuzzy reuse detection scheme and a low-fuzzy reuse detection scheme for the user to choose, and then in response to the user's selection instruction, one or more reuse detection schemes selected by the user are used as the target detection scheme. Optionally, the computer interaction interface can also directly display the specific fingerprint type that needs to be detected for the user to choose.

[0036] Step S206, determining a target fingerprint type corresponding to the target detection solution, and determining first fingerprint information corresponding to the target fingerprint type from the multiple fingerprint information corresponding to the target file.

[0037] In some optional embodiments of the present application, after determining the target detection scheme, it is also possible to select a target multiplexed text type corresponding to the second selection instruction from the target detection scheme in response to a second selection instruction of the target object, wherein the target multiplexed text type includes at least one of the following: a first multiplexed text type corresponding to the first fingerprint type, a second multiplexed text type corresponding to the second fingerprint type, or a third multiplexed text type.

[0038] Among them, the first reused text type mainly includes file reuse types such as copy and paste, approximate copy, and modified copy under low-fuzzy reuse. This type of file reuse is mainly detected through Hash fingerprints; the second reused text type mainly includes summary reuse in text reuse under high-fuzzy reuse. This type of file reuse is mainly detected through term fingerprints; the third reused text type mainly includes synonymous reuse in text reuse under high-fuzzy reuse. This type of file reuse is mainly detected through embedding fingerprints.

[0039] After determining the target detection scheme, the specific target fingerprint type can be determined according to the target reused text type selected by the user, and then the first fingerprint information corresponding to the target fingerprint type in the target file is determined, which can be one of the hash fingerprint, term fingerprint, embedding fingerprint, or any combination thereof.

[0040] Step S208, determining the second fingerprint information of each file in the target database, and determining the similarity between the second fingerprint information and the first fingerprint information, and when the similarity is greater than a preset threshold, determining that the file in the target database corresponding to the second fingerprint information is a reused file of the target file.

[0041] It is understandable that when performing file reuse detection on the target file, it is first necessary to determine a file database, and by detecting the similarity between the target file and each file in the file database, determine whether there is file reuse and find the corresponding reused file. In this application, the detection is mainly based on file fingerprint information, so the above-mentioned target database mainly refers to the file fingerprint database, which is mainly generated by the cloud based on the protected files set by the user. Usually, the file fingerprint database includes at least three major categories: Hash fingerprint library, term fingerprint library and embedding fingerprint library.

[0042] In some optional embodiments of the present application, the cloud will directly send the entire file fingerprint database to the terminal. In this way, the file fingerprint databases of the cloud and the terminal can be guaranteed to be completely consistent, so that the processing results of file reuse detection of the target file by the cloud and the terminal are the same.

[0043] Considering that in some scenarios, the terminal cannot detect all types of fingerprint information due to limited computing resources, or according to user needs, there is no need to detect the fingerprint information corresponding to certain protected files. In order to save computing resources, the cloud can only send a part of the file fingerprint database to the terminal as the target database.

[0044] Specifically, the scene type in which the target terminal is located can be determined, and the scene type can be sent to the network side device, that is, sent to the cloud, and then the target terminal receives the target database corresponding to the scene type from the network side device.

[0045] For example, in a certain office scenario, the user only chooses to perform low-fuzzy reuse detection on the file. At this time, in order to save terminal computing resources, the cloud can only send the hash fingerprint library in the file fingerprint database to the terminal.

[0046] For another example, the cloud generates a file fingerprint database X for all protected files L of the head office A, where the head office A includes subsidiaries B and C, and all protected files L include protected files M of subsidiary B and protected files N of subsidiary C. If M and N do not intersect, then when the terminal of subsidiary B performs file reuse detection, it actually only needs to use part of the file fingerprint database Y corresponding to protected file M, and does not need to use part of the file fingerprint database Z corresponding to protected file N. Therefore, when it is determined that the scene where the target terminal is located is subsidiary B, the cloud can only send part of the file fingerprint database Y to the target terminal as the target database. Similarly, when it is determined that the scene where the target terminal is located is subsidiary C, the cloud can also only send part of the file fingerprint database Z to the target terminal as the target database.

[0047] When determining the similarity between the second fingerprint information and the first fingerprint information, the classification rules corresponding to the target reused text type are mainly used to calculate the similarity between the first fingerprint information and the second fingerprint information of the target file, wherein the specific classification rules can use common decision tree classification, Bayesian algorithm, etc., to calculate the similarity between the first fingerprint information and the second fingerprint information.

[0048] Specifically, when the target multiplexed text type is the first multiplexed text type, a first hash value of the first fingerprint information and a second hash value of the second fingerprint information are calculated; and a classification rule is used to calculate the similarity between the first hash value and the second hash value.

[0049] When the target multiplexed text type is the second multiplexed text type, a first keyword vector of the first fingerprint information and a second keyword vector of the second fingerprint information are calculated; and a classification rule is used to calculate the similarity between the first keyword vector and the second keyword vector.

[0050] When the target reused text type is the third reused text type, the first text semantic vector of the first fingerprint information and the second text semantic vector of the second fingerprint information are calculated; and the similarity of the first text semantic vector and the second text semantic vector is calculated using the classification rule. The text semantic vector corresponding to the embedding fingerprint is obtained by pre-training the model.

[0051] In some optional embodiments of the present application, when it is detected that the similarity between the second fingerprint information corresponding to a certain file and the first fingerprint information of the target file is greater than a preset threshold (the preset threshold is set by the user), it is determined that file reuse exists. At this time, the file can be output as a reused file. At the same time, specific similarity type, similarity degree, similarity content ratio and other information can also be output to achieve classification and stratification of file similarity.

[0052] If the similarities of multiple files are all greater than a preset threshold, all the files can be output as reused files; or the similarities can be sorted and the files with the highest similarity can be output as reused files.

[0053] In the embodiment of the present application, firstly, the target file to be detected is obtained; then, the target detection scheme of the target file is determined from multiple reuse detection schemes, wherein the reuse detection scheme includes at least the fingerprint type to be detected, and different reuse detection schemes correspond to different fingerprint types; then, the target fingerprint type corresponding to the target detection scheme is determined, and the first fingerprint information corresponding to the target fingerprint type is determined from the multiple fingerprint information corresponding to the target file; finally, the second fingerprint information of each file in the target database is determined, and the similarity between the second fingerprint information and the first fingerprint information is determined. When the similarity is greater than a preset threshold, the file corresponding to the second fingerprint information in the target database is determined to be a reused file of the target file. Among them, the user can select the corresponding reuse detection scheme according to his own needs for file similarity detection, and the terminal determines the corresponding type of fingerprint information from the target file according to the reuse detection scheme selected by the user, and then compares the similarity with the fingerprint information in the target database to obtain the detection result. This process not only improves the flexibility of file reuse detection, but also improves the accuracy of the detection result because multiple types of fingerprint information can be detected, thereby solving the technical problem of lack of accuracy and flexibility in file reuse detection in file security protection.

[0054] Example 2

[0055] According to an embodiment of the present application, a file reuse detection device for implementing the above-mentioned file reuse detection method is also provided. Figure 4 As shown, the device at least includes an acquisition module 40, a first determination module 42, a second determination module 44 and a third determination module 46, wherein:

[0056] The acquisition module 40 is used to acquire the target file to be detected.

[0057] The file reuse detection method in the embodiment of the present application can be executed by the cloud or by the terminal. The cloud mainly refers to the network side device, and the terminal can be a terminal device such as a computer or a tablet computer. Taking the terminal as an example, the terminal first obtains the target file that needs to be detected for file reuse. For example, in an office scene, when the user needs to send a target file in the computer, the computer will automatically perform a file reuse detection on the target file; optionally, the user can also enter the target file to be detected in the interactive interface provided by the computer, and the terminal will perform a file reuse detection on it.

[0058] Figure 3 An optional classification of file reuse types is shown, in which file reuse is mainly divided into two categories: low-fuzzy reuse and high-fuzzy reuse. Specifically, low-fuzzy reuse can usually be divided into: 1) copy and paste, which can be further divided into copy and paste of the entire content of the file and copy and paste of part of the content; 2) approximate copying, including inserting, deleting, and replacing operations in the file content, or splitting and merging sentences in the file; 3) modified copying, mainly refers to reordering sentences or paragraphs in the file, or modifying the syntax. High-fuzzy reuse can usually be divided into: 1) text reuse, mainly including synonym reuse in the file, or summary reuse for keywords and other information; 2) file reuse, mainly including conversion of file types, or file-level reorganization, etc.

[0059] The first determination module 42 is used to determine a target detection scheme for a target file from a plurality of multiplexed detection schemes, wherein the multiplexed detection scheme at least includes: a fingerprint type to be detected, and different multiplexed detection schemes correspond to different fingerprint types.

[0060] For the above-mentioned multiple file reuse types, the cloud / terminal provides multiple reuse detection schemes, wherein each reuse detection scheme is used to perform at least the following steps: setting a preset fingerprint type and generating fingerprint information corresponding to the preset fingerprint type; performing similarity matching based on the fingerprint information corresponding to the preset fingerprint type and the fingerprint information of each file in the target database, and determining the reused file from the target database based on the matching results.

[0061] Specifically, the above-mentioned multiple reuse detection schemes at least include: a first detection scheme for low-fuzziness reuse detection and a second detection scheme for high-fuzziness reuse detection, wherein the first detection scheme mainly detects the target file based on the fingerprint information of the first fingerprint type, wherein the fingerprint information of the first fingerprint type includes: fingerprint information in paragraph units, and the fingerprint information of the first fingerprint type is also called hash fingerprint; the second detection scheme mainly detects the target file based on the fingerprint information of the second fingerprint type, wherein the fingerprint information of the second fingerprint type includes: fingerprint information in keyword units, also called term fingerprint, or fingerprint information in text semantic vector units, also called embedding fingerprint. Among them, by combining hash fingerprints, term fingerprints and embedding fingerprints, it is possible to achieve layered protection and realize customized protection measures of different levels, taking into account both efficiency and effectiveness.

[0062] In some optional embodiments of the present application, when determining the target detection scheme for the target file, the target detection scheme for the target file can be pre-set in multiple multiplexed detection schemes. For example, the target detection scheme is set by default to the first detection scheme or the second detection scheme. Of course, the target detection scheme can also be set by default to the first detection scheme and the second detection scheme, that is, the target file is detected according to the first detection scheme and the second detection scheme at the same time.

[0063] Optionally, in response to the first selection instruction of the target object, a target detection scheme corresponding to the first selection instruction can be selected from multiple reuse detection schemes. For example, after the user inputs the target file to be detected in the computer, multiple reuse detection schemes are displayed in the computer interaction interface for the user to choose, such as displaying a high-fuzzy reuse detection scheme and a low-fuzzy reuse detection scheme for the user to choose, and then in response to the user's selection instruction, one or more reuse detection schemes selected by the user are used as the target detection scheme. Optionally, the computer interaction interface can also directly display the specific fingerprint type that needs to be detected for the user to choose.

[0064] The second determination module 44 is used to determine the target fingerprint type corresponding to the target detection scheme, and determine the first fingerprint information corresponding to the target fingerprint type from the multiple fingerprint information corresponding to the target file.

[0065] In some optional embodiments of the present application, after determining the target detection scheme, it is also possible to select a target multiplexed text type corresponding to the second selection instruction from the target detection scheme in response to a second selection instruction of the target object, wherein the target multiplexed text type includes at least one of the following: a first multiplexed text type corresponding to the first fingerprint type, a second multiplexed text type corresponding to the second fingerprint type, or a third multiplexed text type.

[0066] Among them, the first reused text type mainly includes file reuse types such as copy and paste, approximate copy, and modified copy under low-fuzzy reuse. This type of file reuse is mainly detected through Hash fingerprints; the second reused text type mainly includes summary reuse in text reuse under high-fuzzy reuse. This type of file reuse is mainly detected through term fingerprints; the third reused text type mainly includes synonymous reuse in text reuse under high-fuzzy reuse. This type of file reuse is mainly detected through embedding fingerprints.

[0067] After determining the target detection scheme, the specific target fingerprint type can be determined according to the target reused text type selected by the user, and then the first fingerprint information corresponding to the target fingerprint type in the target file is determined, which can be one of the hash fingerprint, term fingerprint, embedding fingerprint, or any combination thereof.

[0068] The third determination module 46 is used to determine the second fingerprint information of each file in the target database, and determine the similarity between the second fingerprint information and the first fingerprint information. When the similarity is greater than a preset threshold, the file in the target database corresponding to the second fingerprint information is determined to be a reused file of the target file.

[0069] It is understandable that when performing file reuse detection on the target file, it is first necessary to determine a file database, and by detecting the similarity between the target file and each file in the file database, determine whether there is file reuse and find the corresponding reused file. In this application, the detection is mainly based on file fingerprint information, so the above-mentioned target database mainly refers to the file fingerprint database, which is mainly generated by the cloud based on the protected files set by the user. Usually, the file fingerprint database includes at least three major categories: Hash fingerprint library, term fingerprint library and embedding fingerprint library.

[0070] In some optional embodiments of the present application, the cloud will directly send the entire file fingerprint database to the terminal. In this way, the file fingerprint databases of the cloud and the terminal can be guaranteed to be completely consistent, so that the processing results of file reuse detection of the target file by the cloud and the terminal are the same.

[0071] Considering that in some scenarios, the terminal cannot detect all types of fingerprint information due to limited computing resources, or according to user needs, there is no need to detect the fingerprint information corresponding to certain protected files. In order to save computing resources, the cloud can only send a part of the file fingerprint database to the terminal as the target database.

[0072] Specifically, the scene type in which the target terminal is located can be determined, and the scene type can be sent to the network side device, that is, sent to the cloud, and then the target terminal receives the target database corresponding to the scene type from the network side device.

[0073] For example, in a certain office scenario, the user only chooses to perform low-fuzzy reuse detection on the file. At this time, in order to save terminal computing resources, the cloud can only send the hash fingerprint library in the file fingerprint database to the terminal.

[0074] For another example, the cloud generates a file fingerprint database X for all protected files L of the head office A, where the head office A includes subsidiaries B and C, and all protected files L include protected files M of subsidiary B and protected files N of subsidiary C. If M and N do not intersect, then when the terminal of subsidiary B performs file reuse detection, it actually only needs to use part of the file fingerprint database Y corresponding to protected file M, and does not need to use part of the file fingerprint database Z corresponding to protected file N. Therefore, when it is determined that the scene where the target terminal is located is subsidiary B, the cloud can only send part of the file fingerprint database Y to the target terminal as the target database. Similarly, when it is determined that the scene where the target terminal is located is subsidiary C, the cloud can also only send part of the file fingerprint database Z to the target terminal as the target database.

[0075] When determining the similarity between the second fingerprint information and the first fingerprint information, the classification rules corresponding to the target reused text type are mainly used to calculate the similarity between the first fingerprint information and the second fingerprint information of the target file, wherein the specific classification rules can use common decision tree classification, Bayesian algorithm, etc., to calculate the similarity between the first fingerprint information and the second fingerprint information.

[0076] Specifically, when the target multiplexed text type is the first multiplexed text type, a first hash value of the first fingerprint information and a second hash value of the second fingerprint information are calculated; and a classification rule is used to calculate the similarity between the first hash value and the second hash value.

[0077] When the target multiplexed text type is the second multiplexed text type, a first keyword vector of the first fingerprint information and a second keyword vector of the second fingerprint information are calculated; and a classification rule is used to calculate the similarity between the first keyword vector and the second keyword vector.

[0078] When the target reused text type is the third reused text type, the first text semantic vector of the first fingerprint information and the second text semantic vector of the second fingerprint information are calculated; and the similarity of the first text semantic vector and the second text semantic vector is calculated using the classification rule. The text semantic vector corresponding to the embedding fingerprint is obtained by pre-training the model.

[0079] In some optional embodiments of the present application, when it is detected that the similarity between the second fingerprint information corresponding to a certain file and the first fingerprint information of the target file is greater than a preset threshold (the preset threshold is set by the user), it is determined that file reuse exists. At this time, the file can be output as a reused file. At the same time, specific similarity type, similarity degree, similarity content ratio and other information can also be output to achieve classification and stratification of file similarity.

[0080] If the similarities of multiple files are all greater than a preset threshold, all the files can be output as reused files; or the similarities can be sorted and the files with the highest similarity can be output as reused files.

[0081] It should be noted that each module in the file reuse detection device in the embodiment of the present application corresponds one by one to the implementation steps of the file reuse detection method in Example 1. Since a detailed description has been given in Example 1, some details not reflected in this embodiment can be referred to Example 1 and will not be repeated here.

[0082] Example 3

[0083] According to an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above-mentioned file reuse detection method.

[0084] Optionally, when the program is running, the device where the non-volatile storage medium is located is controlled to execute the following steps: obtain the target file to be detected; determine the target detection scheme for the target file from multiple reuse detection schemes, wherein the reuse detection scheme at least includes: the fingerprint type to be detected, and different reuse detection schemes correspond to different fingerprint types; determine the target fingerprint type corresponding to the target detection scheme, and determine the first fingerprint information corresponding to the target fingerprint type from the multiple fingerprint information corresponding to the target file; determine the second fingerprint information of each file in the target database, and determine the similarity between the second fingerprint information and the first fingerprint information. When the similarity is greater than a preset threshold, determine that the file corresponding to the second fingerprint information in the target database is a reused file of the target file.

[0085] Example 4

[0086] According to an embodiment of the present application, an electronic device is also provided, which includes a processor and a memory, wherein: the memory is connected to the processor, and is used to provide the processor with instructions for processing the following processing steps: obtaining a target file to be detected; determining a target detection scheme for the target file from multiple reuse detection schemes, wherein the reuse detection scheme at least includes: a fingerprint type that needs to be detected, and different reuse detection schemes correspond to different fingerprint types; determining a target fingerprint type corresponding to the target detection scheme, and determining a first fingerprint information corresponding to the target fingerprint type from multiple fingerprint information corresponding to the target file; determining the second fingerprint information of each file in the target database, and determining the similarity between the second fingerprint information and the first fingerprint information, and when the similarity is greater than a preset threshold, determining that the file corresponding to the second fingerprint information in the target database is a reused file of the target file.

[0087] In some optional embodiments of the present application, the specific structure of the electronic device can refer to Figure 1 The computer terminal shown. It should be noted that Figure 1 The structure shown is for illustration only and does not limit the structure of the electronic device. The structure of the electronic device may be different from that of Figure 1 The computer terminal shown may also include Figure 1 More or fewer components are shown, and there may also be Figure 1 Different configurations are shown.

[0088] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0089] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0091] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0092] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0093] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0094] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A file reuse detection method, characterized in that: include: Get the target file to be detected; Determine a target detection scheme for the target file from a plurality of multiplexed detection schemes, wherein the plurality of multiplexed detection schemes at least include: a first detection scheme and a second detection scheme, the first detection scheme detects the target file based on fingerprint information of a first fingerprint type, the fingerprint information of the first fingerprint type includes: fingerprint information in paragraph units, the second detection scheme detects the target file based on fingerprint information of a second fingerprint type, the fingerprint information of the second fingerprint type includes: fingerprint information in keyword units, or fingerprint information in text semantic vector units; Determine a target fingerprint type corresponding to the target detection scheme, and determine first fingerprint information corresponding to the target fingerprint type from the multiple fingerprint information corresponding to the target file; Determine the second fingerprint information of each file in the target database, and determine the similarity between the second fingerprint information and the first fingerprint information. When the similarity is greater than a preset threshold, determine that the file in the target database corresponding to the second fingerprint information is a reused file of the target file.

2. The method according to claim 1, characterized in that Determining a target detection scheme for the target file from a plurality of multiplexed detection schemes includes: presetting the target detection scheme of the target file in the multiple multiplexed detection schemes; or, In response to a first selection instruction of a target object, the target detection scheme corresponding to the first selection instruction is selected from the plurality of multiplexed detection schemes.

3. The method according to claim 1, characterized in that Determining the similarity between the second fingerprint information and the first fingerprint information includes: In response to a second selection instruction of the target object, a target multiplexed text type corresponding to the second selection instruction is selected from the target detection scheme, wherein the target multiplexed text type includes at least one of the following: a first multiplexed text type corresponding to the first fingerprint type, a second multiplexed text type or a third multiplexed text type corresponding to the second fingerprint type; The similarity between the first fingerprint information and the second fingerprint information of the target file is calculated using the classification rule corresponding to the target multiplexed text type.

4. The method according to claim 3, characterized in that Calculating the similarity between the first fingerprint information and the second fingerprint information of the target file by using the classification rule corresponding to the target multiplexed text type includes: When the target multiplexed text type is the first multiplexed text type, calculating a first hash value of the first fingerprint information and a second hash value of the second fingerprint information; and calculating the similarity between the first hash value and the second hash value using the classification rule; When the target multiplexed text type is the second multiplexed text type, calculating a first keyword vector of the first fingerprint information and a second keyword vector of the second fingerprint information; and calculating a similarity between the first keyword vector and the second keyword vector using the classification rule; When the target multiplexed text type is the third multiplexed text type, a first text semantic vector of the first fingerprint information and a second text semantic vector of the second fingerprint information are calculated; and the similarity between the first text semantic vector and the second text semantic vector is calculated using the classification rule.

5. The method according to claim 1, characterized in that Before determining the second fingerprint information corresponding to the target fingerprint type for each file in the target database, the method further includes: Determine the scene type of the target terminal, and send the scene type to the network side device; The target terminal receives the target database corresponding to the scenario type from the network side device.

6. The method according to claim 1, characterized in that Each of the multiplexed detection schemes is used to perform at least the following steps: Setting a preset fingerprint type and generating fingerprint information corresponding to the preset fingerprint type; A similarity match is performed based on the fingerprint information corresponding to the preset fingerprint type and the fingerprint information of each file in the target database, and a reused file is determined according to the matching result.

7. A file reuse detection device, characterized in that: include: An acquisition module is used to acquire the target file to be detected; A first determination module is used to determine a target detection scheme for the target file from a plurality of multiplexed detection schemes, wherein the plurality of multiplexed detection schemes at least include: a first detection scheme and a second detection scheme, wherein the first detection scheme detects the target file based on fingerprint information of a first fingerprint type, wherein the fingerprint information of the first fingerprint type includes: fingerprint information in paragraph units, and the second detection scheme detects the target file based on fingerprint information of a second fingerprint type, wherein the fingerprint information of the second fingerprint type includes: fingerprint information in keyword units, or fingerprint information in text semantic vector units; A second determination module is used to determine a target fingerprint type corresponding to the target detection scheme, and determine first fingerprint information corresponding to the target fingerprint type from the multiple fingerprint information corresponding to the target file; The third determination module is used to determine the second fingerprint information of each file in the target database, and determine the similarity between the second fingerprint information and the first fingerprint information. When the similarity is greater than a preset threshold, determine that the file in the target database corresponding to the second fingerprint information is a reused file of the target file.

8. A non-volatile storage medium, the non-volatile storage medium comprising a stored program, wherein: When the program is running, the device where the non-volatile storage medium is located is controlled to execute the file reuse detection method described in any one of claims 1 to 6.

9. A file reuse detection device, comprising: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Get the target file to be detected; Determine a target detection scheme for the target file from a plurality of reuse detection schemes, wherein the plurality of reuse detection schemes include at least: a first detection scheme and a second detection scheme, the first detection scheme detects the target file based on fingerprint information of a first fingerprint type, the fingerprint information of the first fingerprint type includes: fingerprint information in paragraph units, the second detection scheme detects the target file based on fingerprint information of a second fingerprint type, the fingerprint information of the second fingerprint type includes: fingerprint information in keyword units, or fingerprint information in text semantic vector units; determine a target fingerprint type corresponding to the target detection scheme, and determine first fingerprint information corresponding to the target fingerprint type from a plurality of fingerprint information corresponding to the target file; determine second fingerprint information of each file in a target database, and determine a similarity between the second fingerprint information and the first fingerprint information, and when the similarity is greater than a preset threshold, determine that the file in the target database corresponding to the second fingerprint information is a reused file of the target file.

Images