An application processing method and apparatus, a storage medium, and an electronic device

Abstract

Embodiments of the present specification disclose an application processing method and device, a storage medium and an electronic device, wherein the method comprises: obtaining a target file hash value of a target application file for first-level deduplication detection processing to obtain a first deduplication detection result; if the target application file is of the same type, performing deduplication filtering processing; if the target application file is of different types, determining file metadata features of the target application file, performing multi-level deduplication detection processing based on the file metadata features to obtain a multi-level deduplication detection result, and executing an application file detection process on the target application file based on the multi-level deduplication detection result.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to an application processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] With the rapid development of the mobile internet, the emergence of online mobile application packaging technology has greatly simplified the application development process. Today, individuals or small teams can easily create and publish their own mobile applications. This convenience has led to a booming application market, but it has also brought a series of security issues, especially the proliferation of risky applications such as fraudulent and malicious apps. Developers often repeatedly submit the same or similar applications by changing the application name, resources, and description. This rapid iteration not only allows risky applications such as fraudulent apps to spread on a large scale on the internet, but also makes traditional application handling methods ineffective. Summary of the Invention

[0003] This specification provides an application processing method, apparatus, storage medium, and electronic device, the technical solutions of which are as follows:

[0004] Firstly, embodiments of this specification provide an application processing method, the method comprising:

[0005] Identify the target application file to be tested;

[0006] Obtain the target file hash value of the target application file, and perform a first-level deduplication detection process based on the target file hash value to obtain the first deduplication detection result;

[0007] If the first deduplication detection result is of the same type, then the target application file is subjected to deduplication filtering processing;

[0008] If the first deduplication detection result is of a different type, the file metadata characteristics of the target application file are determined, and multi-level deduplication detection processing is performed based on the file metadata characteristics to obtain multi-level deduplication detection results. Based on the multi-level deduplication detection results, the application file detection process is executed on the target application file.

[0009] In one feasible implementation, the steps of determining the file metadata features of the target application file, performing multi-level deduplication detection based on the file metadata features to obtain multi-level deduplication detection results, and executing an application file detection process on the target application file based on the multi-level deduplication detection results include:

[0010] The attribute combination hash value and / or application resource hash value of the target application file are parsed, and a second-level deduplication detection process is performed based on the attribute combination hash value and / or application resource hash value to obtain a second deduplication detection result.

[0011] If the second deduplication detection result is of the same type, then the target application file is subjected to deduplication filtering processing;

[0012] If the second deduplication detection result is of a different type, then the file code fingerprint of the target application file is determined, and a third-level deduplication detection process is performed based on the file code fingerprint to obtain a third deduplication detection result;

[0013] If the third deduplication detection result is of the same type, then the target application file is subjected to deduplication filtering. If the third deduplication detection result is of different types, then the target application file is subjected to the application file detection process.

[0014] In one feasible implementation, obtaining the target file hash value of the target application file and performing a first-level deduplication detection process based on the target file hash value to obtain a first deduplication detection result includes:

[0015] Obtain the target original file hash value and / or target list file hash value of the target application file, and perform historical file deduplication comparison processing based on the target original file hash value and / or target list file hash value to obtain the first deduplication detection result.

[0016] In one feasible implementation, the step of performing historical file deduplication comparison based on the hash value of the target original file and / or the hash value of the target list file to obtain a first deduplication detection result includes:

[0017] The hash value of the target original file is compared with the hash value of the target file in the historical detection file hash set to obtain the historical file hash comparison result.

[0018] If the hash comparison results of the historical files are of the same type, then a first deduplication detection result of the same type is generated;

[0019] If the historical file hash comparison result is a different result type, then the target decompressed file list information of the target application file is determined, the target decompressed file list information is combined with information characters to obtain the target list string, the target list file hash value corresponding to the target list string is calculated, and the target list file hash value is compared with the historical file list in the historical list file detection record to obtain the historical list hash comparison result;

[0020] If the hash comparison results of the historical list are of the same type, then a first deduplication detection result of the same type is generated; if the hash comparison results of the historical list are of different types, then a first deduplication detection result of different types is generated.

[0021] In one feasible implementation, the step of parsing the attribute combination hash value and / or application resource hash value of the target application file, and performing a second-level deduplication detection process based on the attribute combination hash value and / or the application resource hash value to obtain a second deduplication detection result includes:

[0022] Extract key metadata from the target application file, extract preset attribute character combinations from the key metadata, calculate the attribute combination hash value of the preset attribute character combinations, perform historical attribute deduplication comparison processing based on the attribute combination hash value, and obtain the attribute deduplication detection result; and / or,

[0023] Extract target application resource data from the target application file, calculate the application resource hash value of the target application resource data, perform historical resource deduplication comparison processing based on the application resource hash value, and obtain resource deduplication detection results.

[0024] A second deduplication result is generated based on the attribute deduplication result and the resource deduplication result.

[0025] In one feasible implementation, the step of performing historical attribute deduplication comparison processing based on the attribute combination hash value to obtain the attribute deduplication detection result includes: performing historical attribute deduplication comparison processing on the attribute combination hash value in the historical attribute file detection record to obtain the historical attribute hash comparison result; and / or,

[0026] The step of performing historical resource deduplication comparison based on the application resource hash value to obtain resource deduplication detection results includes: performing historical resource deduplication comparison on the application resource hash value in the historical resource file detection record to obtain historical resource hash comparison results; and / or,

[0027] The method further includes:

[0028] If the historical attribute hash comparison results are of the same type, then a second deduplication detection result of the same type is generated;

[0029] If the historical attribute hash comparison result is a different result type, then the step of extracting target application resource data from the target application file is executed;

[0030] If the historical resource hash comparison results are of the same type, a second deduplication detection result of the same type is generated; if the historical resource hash comparison results are of different types, a second deduplication detection result of different types is generated.

[0031] In one feasible implementation, determining the file code fingerprint of the target application file and performing a third-level deduplication detection process based on the file code fingerprint to obtain a third deduplication detection result includes:

[0032] Extract a preset core code segment from the target application file, calculate the file code fingerprint corresponding to the preset core code segment, and perform historical code fingerprint deduplication and comparison processing on the file code fingerprint in the historical file code detection record to obtain the historical code fingerprint comparison result.

[0033] If the historical code fingerprint comparison results are of the same type, then a third deduplication detection result of the same type is generated;

[0034] If the historical code fingerprint comparison results are of different types, then a third deduplication detection result of different types is generated.

[0035] In one feasible implementation, the deduplication filtering process for the target application file includes:

[0036] Based on the deduplication detection results, similar application files corresponding to the target application file are determined, and a similar file association relationship is established between the target application file and the similar application files;

[0037] The target application files are sampled and detected based on the aforementioned similar file association relationships;

[0038] The deduplication detection results include at least one of the first deduplication detection results, the second deduplication detection results, and the third deduplication detection results.

[0039] Secondly, embodiments of this specification provide an application processing apparatus, the apparatus comprising:

[0040] The file determination module is used to determine the target application file to be detected.

[0041] The deduplication detection module is used to obtain the target file hash value of the target application file, perform a first-level deduplication detection process based on the target file hash value, and obtain a first deduplication detection result.

[0042] The deduplication detection module is used to perform deduplication filtering on the target application file if the first deduplication detection result is of the same type.

[0043] The deduplication detection module is used to determine the file metadata features of the target application file if the first deduplication detection result is of different types, perform multi-level deduplication detection processing based on the file metadata features to obtain multi-level deduplication detection results, and perform an application file detection process on the target application file based on the multi-level deduplication detection results.

[0044] In one feasible implementation, the deduplication detection module is used to: parse the attribute combination hash value and / or application resource hash value of the target application file, perform a second-level deduplication detection process based on the attribute combination hash value and / or the application resource hash value, and obtain a second deduplication detection result;

[0045] If the second deduplication detection result is of the same type, then the target application file is subjected to deduplication filtering processing. If the second deduplication detection result is of different types, then the file code fingerprint of the target application file is determined, and a third-level deduplication detection processing is performed based on the file code fingerprint to obtain a third deduplication detection result.

[0046] If the third deduplication detection result is of the same type, then the target application file is subjected to deduplication filtering. If the third deduplication detection result is of different types, then the target application file is subjected to the application file detection process.

[0047] The attribute combination hash value and / or application resource hash value of the target application file are parsed, and a second-level deduplication detection process is performed based on the attribute combination hash value and / or application resource hash value to obtain a second deduplication detection result.

[0048] The deduplication detection module is used to perform deduplication filtering on the target application file if the second deduplication detection result is of the same type.

[0049] The deduplication detection module is used to determine the file code fingerprint of the target application file if the second deduplication detection result is of a different type, and perform a third-level deduplication detection process based on the file code fingerprint to obtain a third deduplication detection result.

[0050] The deduplication detection module is used to perform deduplication filtering on the target application file if the third deduplication detection result is of the same type, and to perform an application file detection process on the target application file if the third deduplication detection result is of different types.

[0051] In one feasible implementation, the deduplication detection module is used to:

[0052] Obtain the target original file hash value and / or target list file hash value of the target application file, and perform historical file deduplication comparison processing based on the target original file hash value and / or target list file hash value to obtain the first deduplication detection result.

[0053] In one feasible implementation, the deduplication detection module is used to:

[0054] The hash value of the target original file is compared with the hash value of the target file in the historical detection file hash set to obtain the historical file hash comparison result.

[0055] If the hash comparison results of the historical files are of the same type, then a first deduplication detection result of the same type is generated;

[0056] If the historical file hash comparison result is a different result type, then the target decompressed file list information of the target application file is determined, the target decompressed file list information is combined with information characters to obtain the target list string, the target list file hash value corresponding to the target list string is calculated, and the target list file hash value is compared with the historical file list in the historical list file detection record to obtain the historical list hash comparison result;

[0057] If the hash comparison results of the historical list are of the same type, then a first deduplication detection result of the same type is generated; if the hash comparison results of the historical list are of different types, then a first deduplication detection result of different types is generated.

[0058] In one feasible implementation, the deduplication detection module is used to:

[0059] Extract key metadata from the target application file, extract preset attribute character combinations from the key metadata, calculate the attribute combination hash value of the preset attribute character combinations, perform historical attribute deduplication comparison processing based on the attribute combination hash value, and obtain the attribute deduplication detection result; and / or,

[0060] Extract target application resource data from the target application file, calculate the application resource hash value of the target application resource data, perform historical resource deduplication comparison processing based on the application resource hash value, and obtain resource deduplication detection results.

[0061] A second deduplication result is generated based on the attribute deduplication result and the resource deduplication result.

[0062] In one feasible implementation, the step of performing historical attribute deduplication comparison processing based on the attribute combination hash value to obtain the attribute deduplication detection result includes: performing historical attribute deduplication comparison processing on the attribute combination hash value in the historical attribute file detection record to obtain the historical attribute hash comparison result; and / or,

[0063] The step of performing historical resource deduplication comparison based on the application resource hash value to obtain resource deduplication detection results includes: performing historical resource deduplication comparison on the application resource hash value in the historical resource file detection record to obtain historical resource hash comparison results; and / or,

[0064] The device further includes:

[0065] If the historical attribute hash comparison results are of the same type, then a second deduplication detection result of the same type is generated;

[0066] If the historical attribute hash comparison result is a different result type, then the step of extracting target application resource data from the target application file is executed;

[0067] If the historical resource hash comparison results are of the same type, a second deduplication detection result of the same type is generated; if the historical resource hash comparison results are of different types, a second deduplication detection result of different types is generated.

[0068] In one feasible implementation, the deduplication detection module is used to:

[0069] Extract a preset core code segment from the target application file, calculate the file code fingerprint corresponding to the preset core code segment, and perform historical code fingerprint deduplication and comparison processing on the file code fingerprint in the historical file code detection record to obtain the historical code fingerprint comparison result.

[0070] If the historical code fingerprint comparison results are of the same type, then a third deduplication detection result of the same type is generated;

[0071] If the historical code fingerprint comparison results are of different types, then a third deduplication detection result of different types is generated.

[0072] In one feasible implementation, the deduplication detection module is used to:

[0073] Based on the deduplication detection results, similar application files corresponding to the target application file are determined, and a similar file association relationship is established between the target application file and the similar application files;

[0074] The target application files are sampled and detected based on the aforementioned similar file association relationships;

[0075] The deduplication detection results include at least one of the first deduplication detection results, the second deduplication detection results, and the third deduplication detection results.

[0076] Thirdly, embodiments of this specification provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.

[0077] Fourthly, embodiments of this specification provide an electronic device that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0078] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0079] In one or more embodiments of this specification, the electronic device obtains the target file hash value of the target application file to perform a first-level deduplication detection process to obtain a first deduplication detection result. If the results are of the same type, deduplication filtering is performed. If the results are of different types, the file metadata features of the target application file are determined. Based on the file metadata features, multi-level deduplication detection is performed to obtain multi-level deduplication detection results. Based on the multi-level deduplication detection results, the application file detection process is executed on the target application file, which can efficiently filter and accurately distinguish the type of the target application file. By using the hierarchical comparison of file hash value and metadata features, duplicate files are quickly filtered. At the same time, the multi-level deduplication detection further analyzes potential differences in depth, avoiding misjudgment of new files or omission of important information, thereby improving detection efficiency, reducing resource waste, and ensuring that a comprehensive detection process is performed on key files, achieving a beneficial effect that balances performance and accuracy. Attached Figure Description

[0080] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0081] Figure 1 This is a schematic diagram of a scenario of an application processing system provided in the embodiments of this specification;

[0082] Figure 2 This is a flowchart illustrating an application processing method provided in an embodiment of this specification;

[0083] Figure 3a This is a flowchart illustrating an application processing method provided in an embodiment of this specification;

[0084] Figure 3b This is a schematic diagram of a first-level deduplication detection process provided in the embodiments of this specification;

[0085] Figure 4 This is a schematic diagram of a second-level deduplication detection process provided in the embodiments of this specification;

[0086] Figure 5This is a schematic diagram of a third-level deduplication detection process provided in the embodiments of this specification;

[0087] Figure 6 This is a schematic diagram of the structure of an application processing device provided in the embodiments of this specification;

[0088] Figure 7 This is a schematic diagram of the structure of an electronic device provided in the embodiments of this specification;

[0089] Figure 8 This is a schematic diagram of the operating system and user space structure provided in the embodiments of this specification;

[0090] Figure 9 yes Figure 8 Architecture diagram of the Android operating system in China;

[0091] Figure 10 yes Figure 8 Architecture diagram of the iOS operating system. Detailed Implementation

[0092] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0093] In the description of this specification, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this specification, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this specification based on the specific circumstances. Furthermore, in the description of this specification, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0094] In related technologies, malicious developers can use online packaging techniques to quickly generate multiple variants, each with a different digest value (MD5 value). This makes simple deduplication techniques ineffective in filtering out these essentially duplicated risky applications. Therefore, detecting massive amounts of application files requires enormous server resources to parse file characteristics. Both static and dynamic analysis require high-performance computing resources. Dynamic analysis, in particular, typically requires running applications in a sandbox environment to observe their behavior. Dynamic analysis relies heavily on virtual machine resources, especially ARM architecture emulators. The deployment and maintenance costs of these emulators are high, further increasing the burden on the application detection system. Because unknown applications need to be detected one by one, the burden on the application detection system increases significantly. This not only leads to resource waste but also risks false positives and false negatives due to the backlog of applications being discarded indiscriminately. False positives may cause legitimate applications to be incorrectly flagged as fraudulent, affecting user experience and developer interests. False negatives may allow fraudulent apps to escape detection, causing actual harm to users.

[0095] Therefore, it is evident that an effective deduplication filtering technique for massive application detection is urgently needed. This deduplication filtering application detection method can identify application files with simple changes in a short time and ensure that homogeneous application files do not need to be detected multiple times within a period of time, thereby helping the application detection system reduce redundant analysis and improve detection efficiency.

[0096] Please see Figure 1 This is a schematic diagram of a scenario for an application detection system provided in an embodiment of this application. Figure 1 As shown, an image creation and processing system can include at least a client cluster and a service platform 100.

[0097] In some embodiments, the client cluster may include at least one client, such as Figure 1 As shown, it specifically includes client 1 corresponding to user 1, client 2 corresponding to user 2, ..., client n corresponding to user n, where n is an integer greater than 0.

[0098] Each client in a client cluster can be an electronic device with communication capabilities, including but not limited to: wearable devices, handheld devices, personal computers, tablets, in-vehicle devices, smartphones, computing devices, or other processing devices connected to a wireless modem. Electronic devices may have different names in different networks, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, cellular phone, cordless phone, personal digital assistant (PDA), and electronic devices in 5G networks or future evolved networks.

[0099] In some embodiments, the service platform 100 may be a single server device, such as a rack-mounted, blade, tower, or cabinet-type server device, or a workstation, mainframe, or other hardware device with strong computing power; or it may be a server cluster composed of multiple servers, wherein the servers in the service cluster may be composed in a symmetrical manner, wherein each server is functionally and hierarchically equivalent in the transaction link, and each server can provide services to the outside world independently, wherein providing services independently can be understood as not requiring the assistance of other servers.

[0100] In one or more embodiments of this specification, the service platform 100 may establish a communication connection with at least one client in the client cluster, and complete the data interaction during the application detection process based on the communication connection.

[0101] For example:

[0102] Service Platform 100 provides application testing services to external users, who can submit target application files to be tested to Service Platform 100 through a client.

[0103] Service platform 100 can identify the target application file to be detected;

[0104] Service platform 100 obtains the target file hash value of the target application file, performs a first-level deduplication detection process based on the target file hash value, and obtains a first deduplication detection result.

[0105] If the first deduplication detection result is of the same type, the service platform 100 performs deduplication filtering on the target application file;

[0106] If the first deduplication detection result is of a different type, the service platform 100 parses the attribute combination hash value and / or application resource hash value of the target application file, and performs a second-level deduplication detection process based on the attribute combination hash value and / or the application resource hash value to obtain a second deduplication detection result.

[0107] If the second deduplication detection result is of the same type, the service platform 100 performs deduplication filtering on the target application file;

[0108] If the second deduplication detection result is of a different type, the service platform 100 determines the file code fingerprint of the target application file, performs a third-level deduplication detection process based on the file code fingerprint, and obtains a third deduplication detection result.

[0109] If the third deduplication detection result is of the same type, the service platform 100 performs deduplication filtering on the target application file; if the third deduplication detection result is of different types, the application file detection process is executed on the target application file.

[0110] It should be noted that the service platform 100 establishes a communication connection with at least one client in the client cluster via a network for interactive communication. This network can be a wireless network or a wired network. Wireless networks include, but are not limited to, cellular networks, wireless LANs, infrared networks, or Bluetooth networks. Wired networks include, but are not limited to, Ethernet, universal serial bus (USB), or controller area networks. In one or more embodiments of the specification, technologies and / or formats including HyperText Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network (such as target compressed packets). Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0111] The application detection system embodiments provided in this specification and the application processing methods described in one or more embodiments belong to the same concept. The execution entity corresponding to the application processing methods involved in one or more embodiments of this specification can be an electronic device, which can be the aforementioned service platform 100 or the aforementioned client, depending on the actual application environment. The specific implementation process of the application detection system embodiments can be found in the following method embodiments, and will not be repeated here.

[0112] The present specification will now be described in detail with reference to specific embodiments.

[0113] In one embodiment, such as Figure 2 As shown, an application processing method is proposed, which can be implemented using a computer program and can run on an application processing device based on the von Neumann architecture. This computer program can be integrated into an application or run as a standalone utility application. The application processing device can be an electronic device.

[0114] Specifically, the application's processing methods include:

[0115] S002: Determine the target application file to be detected;

[0116] Target application file: refers to the mobile application file for which the application file detection process needs to be executed. In this specification, before determining whether to execute the application file detection process, the application detection method described in one or more embodiments of this specification is first executed for deduplication filtering. This deduplication filtering application detection method can identify application files with simple changes in a short time and ensure that homogeneous application files do not need to be detected multiple times within a certain period, thereby helping the application detection system reduce redundant analysis and improve detection efficiency. Target application files are usually in application file formats such as APK and IPA.

[0117] The target application file to be detected can be understood as the target application file that needs to go through the deduplication detection process corresponding to "the application detection described in one or more embodiments of this specification";

[0118] Through tracking and analysis of various rapidly iterating app family files, creative work revealed that most app files typically undergo only minor modifications during repackaging and release. These modifications include slight changes to the application name, package name, resources, re-signing, and even minor adjustments to some dex or .so files. To address these different changes, this specification proposes a multi-layered deduplication strategy and corresponding application processing method.

[0119] As an example, the target application files to be detected can be obtained through app stores, user uploads, or developer submissions. There can be one or more target application files. The target application files are placed in the detection queue and marked as pending detection.

[0120] Example: Suppose a developer submits an APK file named "Application Assistant" to the app store, and this file is identified as the target application file to be detected.

[0121] S004: Obtain the target file hash value of the target application file, and perform a first-level deduplication detection process based on the target file hash value to obtain a first deduplication detection result;

[0122] Target file hash value: A fixed-length identifier calculated using a hash algorithm for the target application file. Common hash algorithms include MD5 and SHA256. The target file hash value can be calculated based on the entire content of the file or on a list of files within the file.

[0123] The first level of deduplication detection can be understood as quickly determining whether the content of a file is completely consistent with the content of a historical file that has already been detected by comparing the file's hash value.

[0124] In one feasible implementation, a hash algorithm is used to calculate the target application file's hash value. The generated target file hash value is then matched against an existing hash value database to determine whether the current application file has actually undergone the application file detection process.

[0125] For example, by calculating the MD5, SHA-1, and other hash values ​​of the target application file, identical APP files can be quickly excluded. This can significantly reduce the application analysis burden of subsequent application file detection processes.

[0126] S006: If the first deduplication detection result is of the same type, then the target application file is subjected to deduplication filtering processing;

[0127] Deduplication filtering: Duplicate application files (such as target application files that are determined to be of the same type) are marked as "duplicate application". Optionally, they can be removed from the detection queue that performs the application file detection process.

[0128] Same type: Can be directly identified as the type of repeated application files.

[0129] In one alternative implementation, for files detected as "same type," the subsequent application file detection process is skipped. The target application file is marked as processed, and relevant logs are recorded.

[0130] Example: "Application Assistant.apk" was determined to be of "same type", the system marked it as duplicate, and stopped the subsequent application file detection process.

[0131] S008: If the first deduplication detection result is of a different type, then determine the file metadata characteristics of the target application file, perform multi-level deduplication detection processing based on the file metadata characteristics to obtain multi-level deduplication detection results, and perform application file detection process on the target application file based on the multi-level deduplication detection results.

[0132] File metadata features refer to descriptive information about the target application file itself, including but not limited to filename, size, creation time, modification time, type identifier, developer information, file resource information, file code information, permissions, version number, etc. These metadata features are used to more comprehensively characterize the file's properties.

[0133] Multi-level deduplication detection refers to a progressively deeper detection and comparison based on multiple metadata feature dimensions (such as attribute feature dimension, file resource feature dimension, and file code feature dimension) within the file's metadata features. It verifies whether files are duplicated from simple to complex and from coarse-grained to fine-grained, ensuring the accuracy of the detection. This includes, but is not limited to, attribute data comparison, file content feature comparison (such as fingerprint detection), and behavioral analysis.

[0134] The application file inspection process can be understood as an inspection process for formal application review, such as static analysis inspection process and dynamic analysis inspection process.

[0135] Optionally, if the multi-level deduplication detection results are determined to be of different types, an application file detection process is performed on the target application file; if the multi-level deduplication detection results are determined to be of the same type, an application file detection process is performed on the target application file.

[0136] To illustrate, the process first extracts file metadata features from the target application file, then performs hierarchical feature comparison based on these features: Secondary comparison: A simple comparison is made with metadata attribute features in the historical database, such as whether file size and creation time are consistent, quickly filtering out obviously different files. Tertiary comparison: More complex file code features are compared for similarity. Deep analysis methods, such as code fingerprinting and decompiling key logic, are used to achieve finer-grained file detection, generating multi-level deduplication results. Based on the comparison results, the final judgment of the multi-level detection is made, marking the files as "same type" or "different type".

[0137] If the multi-level deduplication detection results indicate that a file is a duplicate, then the file will be filtered.

[0138] If the multi-level deduplication test results indicate that the file is a non-duplicate type, then the application file detection process will proceed to conduct more in-depth security or functional tests.

[0139] In one or more embodiments of this specification, the electronic device obtains the target file hash value of the target application file to perform a first-level deduplication detection process to obtain a first deduplication detection result. If the results are of the same type, deduplication filtering is performed. If the results are of different types, the file metadata features of the target application file are determined. Based on the file metadata features, multi-level deduplication detection is performed to obtain multi-level deduplication detection results. Based on the multi-level deduplication detection results, the application file detection process is executed on the target application file, which can efficiently filter and accurately distinguish the type of the target application file. By using the hierarchical comparison of file hash value and metadata features, duplicate files are quickly filtered. At the same time, the multi-level deduplication detection further analyzes potential differences in depth, avoiding misjudgment of new files or omission of important information, thereby improving detection efficiency, reducing resource waste, and ensuring that a comprehensive detection process is performed on key files, achieving a beneficial effect that balances performance and accuracy.

[0140] Optional, such as Figure 3a As shown, an application processing method is proposed. Specifically, the application processing method includes:

[0141] S102: Determine the target application file to be detected;

[0142] S104: Obtain the target file hash value of the target application file, and perform a first-level deduplication detection process based on the target file hash value to obtain a first deduplication detection result;

[0143] S106: If the first deduplication detection result is of the same type, then the target application file is subjected to deduplication filtering processing;

[0144] S108: If the first deduplication detection result is of different types, then parse the attribute combination hash value and / or application resource hash value of the target application file, and perform a second-level deduplication detection process based on the attribute combination hash value and / or the application resource hash value to obtain a second deduplication detection result;

[0145] Attribute combination hash value: The hash value is calculated by combining specific attributes of the file (such as version number, package name, permission list, version list).

[0146] Application resource hash value: A hash value calculated based on application resource files, used to identify applications with similar resources. Resources (files) include, but are not limited to, icons, images, audio, etc.

[0147] Second-level deduplication detection: A more detailed similarity check is performed on application files that were determined to be "different types" by the first deduplication detection.

[0148] To illustrate, the attributes of the target application file, such as package name, version number, and permission declaration, are parsed, and the attributes are combined to obtain attribute combinations. Then, the attribute combination hash value is calculated, and the attribute combination hash value is compared with the set of attribute combination hash values ​​of historically detected applications in the database to obtain the attribute deduplication detection result.

[0149] Extract resource files such as icon images from the application file, calculate the application resource hash value based on the resource files, compare the application resource hash value with the historical resource hash value set of previously detected applications in the database to obtain the resource deduplication detection result. If the resource deduplication detection result is the same, it can be considered that the appearance characteristics of the application are similar to those of previously detected applications, and they may belong to the same template. Such similar application files can be exempted from the application file detection process in the short term.

[0150] A second deduplication result can be generated based on the attribute deduplication result and the resource deduplication result.

[0151] S112: If the second deduplication detection result is of the same type, then the target application file is subjected to deduplication filtering processing;

[0152] Please refer to the explanations of other methods and steps.

[0153] S114: If the second deduplication detection result is of a different type, then determine the file code fingerprint of the target application file, and perform a third-level deduplication detection process based on the file code fingerprint to obtain a third deduplication detection result;

[0154] File code fingerprint: A feature identifier generated based on the core code logic of an application file, used to detect the similarity of application functions. For example, SimHash can be used to calculate the result value corresponding to the core code logic segment as the file code fingerprint. Even if other parts of the application change, the core code fingerprint can remain consistent. In some embodiments, for APP files with similar file code fingerprints, random sampling can be used for detection to ensure the accuracy and comprehensiveness of the detection.

[0155] The third level of deduplication detection: Through code-level analysis, it deeply judges the similarity of the substantive code logic between files.

[0156] This example illustrates how code logic extraction algorithms (such as function call sequence analysis and bytecode feature extraction) are used to generate core code logic fragments for a file. Then, an algorithm (such as a digest algorithm) is applied to these core code logic fragments to calculate a code fingerprint. This fingerprint is compared with the code fingerprints of previously detected applications in a database to generate a third deduplication detection result, determining whether the file duplicates the application's functionality.

[0157] S116: If the third deduplication detection result is of the same type, then the target application file is subjected to deduplication filtering processing;

[0158] Please refer to the explanations of other methods and steps.

[0159] S118: If the third deduplication detection result is of a different type, then the application file detection process is executed on the target application file.

[0160] Application file inspection process: This can be understood as the inspection process for formal application review, such as static analysis inspection process and dynamic analysis inspection process.

[0161] This manual does not specify the application file detection process; the specific settings should be based on the actual application environment.

[0162] In one or more embodiments of this specification, the electronic device obtains the target file hash value of the target application file to perform a first-level deduplication detection process to obtain a first deduplication detection result. If the results are of the same type, deduplication filtering is performed. If the results are of different types, the attribute combination hash value and / or application resource hash value of the target application file are parsed to perform a second-level deduplication detection process to obtain a second deduplication detection result. If the results are of the same type, deduplication filtering is performed. If the results are of different types, the file code fingerprint of the target application file is determined. Based on the file code fingerprint, a third-level deduplication detection process is performed to obtain a third deduplication detection result. If the results are of the same type, deduplication filtering is performed. If the results are of different types, the application file detection process is executed. This achieves multi-level efficient filtering from the file level, attribute level to the code level, significantly reducing the burden of detecting duplicate files. By combining hash value fast matching with deep fingerprint comparison and other multi-level detection processes, it can not only efficiently process massive application files, but also effectively deal with the disguised behavior of application package variants and minor modifications. At the same time, by concentrating resources on application documents that truly require in-depth analysis, the overall efficiency and accuracy of the detection system have been significantly improved, the risk of false alarms and false negatives has been reduced, and a strong guarantee has been provided for the healthy ecosystem of the application market.

[0163] In one or more embodiments of this specification, the deduplication filtering process for the target application file can be performed in the following manner:

[0164] S1. Based on the deduplication detection results, determine the similar application files corresponding to the target application file, and establish the similar file association relationship between the target application file and the similar application files;

[0165] The deduplication detection results include at least one of the first deduplication detection results, the second deduplication detection results, and the third deduplication detection results.

[0166] Determining similar application files: Based on the results of deduplication detection (at least one of the first, second, and third deduplication detection results), find other historically detected application files that are similar to or have similar functions to the target application file.

[0167] Understandably, the first deduplication check identifies files with completely identical content. The second deduplication check identifies slightly modified or repackaged files based on attribute combination hash values ​​or icon hash values. The third deduplication check uses code fingerprinting to identify files with the same core functionality but which may have undergone code obfuscation.

[0168] As an illustration, the target application file is associated with similar application files to obtain similar file associations, which are then stored in a database to form a traceable set of similar files, facilitating subsequent analysis and management.

[0169] Understandably, by establishing associations, the connection between the target file and similar files can be quickly identified, avoiding repeated analysis of similar files, providing traceability information for application variants, and assisting in analyzing the propagation chain of malicious behavior.

[0170] S2. Sampling and detection processing is performed on the target application file based on the similar file association relationship;

[0171] Sampling inspection: Based on the association relationship of similar files, sample files are selected from the set of associated files for detailed inspection, rather than performing repeated analysis on all files.

[0172] Indicatively, one or more files are randomly or strategically selected from a set of similar files of the target file for application file detection processes, such as dynamic analysis and static analysis, and the detection results are used to infer the possible behavior of undetected files.

[0173] In the embodiments described in this specification, by combining methods S1 and S2, the target application file can not only be quickly classified into corresponding similar file groups, but also reduce duplicate detections using sampling strategies, while retaining the ability to flexibly adapt to new variants. This processing method further enhances the intelligence level of the detection system, providing an efficient and accurate solution for large-scale application detection scenarios.

[0174] Optionally, the process of obtaining the target file hash value of the target application file and performing a first-level deduplication detection based on the target file hash value can be described in the following implementation:

[0175] A2: Obtain the target source file hash value and / or target list file hash value of the target application file;

[0176] Hash value of the target source file: A hash value generated based on the overall file structure or content of the target application file, such as calculated using algorithms like MD5 or SHA-256.

[0177] Target list file hash value: Extracts the contents of specific list files (such as configuration files, resource files, manifest files, etc.) from the target application files and generates list file hash values.

[0178] A4: Based on the hash value of the target original file and / or the hash value of the target list file, perform historical file deduplication comparison processing to obtain the first deduplication detection result.

[0179] Historical file deduplication comparison: By matching the hash value of the target file with the existing historical file hash value database, duplicate files can be quickly identified.

[0180] In the initial deduplication stage, firstly, the MD5 and SHA-1 hash values ​​of the target application file are calculated as the hash value of the target source file. This target source file hash value is then used for historical file deduplication comparison to quickly eliminate identical application files, significantly reducing the burden of subsequent analysis. For example, the complete file of each application file is obtained, and its hash value is calculated using hash algorithms such as MD5 and SHA-1. The calculated target source file hash value is then compared with historical detection records. The processing logic is as follows:

[0181] If a historical record is matched, the first deduplication detection result is output as "same type". If no historical record is matched, the first deduplication detection result is "different type", and the next stage of detection is required. If the same hash value exists, the APP file is considered to have been detected, and no further analysis is performed.

[0182] Additionally, the target application file is decompressed using a decompression tool (such as zip, tar, etc.). The decompressed file paths and names are listed, and the file list is concatenated into a string in sequence. The hash value of this string (i.e., the hash value of the target list files) is calculated using hash algorithms such as MD5 and SHA-1. The calculated hash value of the target list files is then compared with historical detection records. The processing logic is as follows:

[0183] If a historical record is matched, the first deduplication detection result is output as "same type". If no historical record is matched, the first deduplication detection result is "different type", and the next stage of detection is required. If the same hash value exists, the APP file is considered to have been detected, and no further analysis is performed.

[0184] This specification describes a method for rapidly screening completely identical or highly similar files by using a dual matching of the original file's hash value and the list file's hash value. This avoids redundant testing of completely identical or slightly modified files, saving system resources. Furthermore, the introduction of the list file's hash value effectively identifies lightly modified variant files, providing a reliable basis for subsequent detection processes. Comparison with historical files helps analyze whether the target file is associated with known malicious files, improving the intelligence of the detection system.

[0185] In one possible implementation, please refer to Figure 3b , Figure 3b This is a flowchart illustrating a first-level deduplication detection process as described in this specification. Specifically, the process of comparing historical files based on the hash value of the target original file and / or the hash value of the target list file to obtain the first deduplication detection result can be implemented as follows:

[0186] S202: Perform historical file hash comparison processing on the hash value of the target original file in the historical detection file hash set to obtain the historical file hash comparison result;

[0187] Historical file hash set: A collection of original file hash values ​​of all files that have been detected in the storage system, used to compare whether the target file has been detected.

[0188] Hash value of the target original file: A unique identifier generated from the entire content of the target application file, calculated using hash algorithms such as SHA-256 and MD5.

[0189] Indicatively, the hash value of the target original file is matched with each data record in the historical detection file hash set to generate historical file hash comparison results; the historical file hash comparison results include at least the same result type and different result types;

[0190] S204: If the hash comparison results of the historical files are of the same type, then generate a first deduplication detection result of the same type;

[0191] S206: If the historical file hash comparison result is a different result type, then determine the target decompressed file list information of the target application file, combine the information characters of the target decompressed file list information to obtain the target list string, calculate the target list file hash value corresponding to the target list string, and perform historical file list comparison processing on the target list file hash value in the historical list file detection record to obtain the historical list hash comparison result;

[0192] Target decompressed file list information: Extracts information about the main sub-files within the target file by decompressing it, including filenames and content summaries.

[0193] Target list string: Combines the list of decompressed files into a string according to certain rules, which is used to generate a hash value.

[0194] Hash value of target list files: A unique identifier obtained by hashing the target list string, used to compare the similarity of file lists.

[0195] Use a decompression tool (such as zip, tar, etc.) to decompress the target application file, list all file paths and names after decompression to obtain the target decompressed file list information. Concatenate the file list information in order into a target list string, and calculate the hash value of this target list string (i.e., the target list file hash value) using hash algorithms such as MD5 and SHA-1. Compare the calculated target list file hash value with historical detection records. Processing logic:

[0196] If a historical record is matched, the output hash comparison result of the historical list is of the same type. If no historical record is matched, the hash comparison result of the historical list is of a different type, and the process needs to proceed to the next stage of detection.

[0197] S208: If the hash comparison results of the historical list are of the same type, then generate a first deduplication detection result of the same type;

[0198] S210: If the hash comparison result of the historical list is of a different type, then generate a first deduplication detection result of a different type.

[0199] In the embodiments described in this specification, by introducing a two-layer deduplication mechanism using the original file hash value and the file list hash value, this process can efficiently identify files that are completely identical or partially similar, reducing redundant detection. Simultaneously, the accurate identification of unmatched files provides a basis for subsequent in-depth analysis, significantly improving the efficiency and accuracy of the detection system.

[0200] Optional, such as Figure 4 As shown, Figure 4 This is a flowchart illustrating a second-level deduplication process. The process involves parsing the attribute combination hash value and / or application resource hash value of the target application file, and then performing a second-level deduplication process based on the attribute combination hash value and / or application resource hash value to obtain a second deduplication result. The following implementation method can be referenced:

[0201] S3002: Extract key metadata of the target application file, extract preset attribute character combinations from the key metadata, calculate the attribute combination hash value of the preset attribute character combinations, perform historical attribute deduplication comparison processing based on the attribute combination hash value, and obtain attribute deduplication detection results; and / or,

[0202] Key metadata: A collection of data in the target application file that describes its basic information, including package name, version number, permission declaration, etc.

[0203] Preset attribute character combinations: Some attribute fields extracted from key metadata are combined according to fixed rules to generate attribute combination hash values.

[0204] Attribute combination hash value: A unique identifier calculated by a hash algorithm from a preset combination of attribute characters, used to identify whether the basic attributes of the application are consistent.

[0205] Historical attribute deduplication comparison: Match the attribute combination hash value with the hash value in the historical record to determine whether the attribute already exists.

[0206] For example, metadata fields such as package name, version number, and permission list are extracted from the target application file. A hash calculation is performed on the combined string to generate an attribute combination hash value. The attribute combination hash value is then matched with the attribute hash set in the history to obtain the attribute deduplication detection result.

[0207] For example, the step of performing historical attribute deduplication comparison processing based on the attribute combination hash value to obtain the attribute deduplication detection result includes: performing historical attribute deduplication comparison processing on the attribute combination hash value in the historical attribute file detection record to obtain the attribute deduplication detection result.

[0208] S3004: Extract target application resource data from the target application file, calculate the application resource hash value of the target application resource data, perform historical resource deduplication comparison processing based on the application resource hash value, and obtain resource deduplication detection results;

[0209] Target application resource data: resource files in the application files that are related to visuals, audio, or interaction, such as icons, images, audio, etc.

[0210] Application resource hash value: Calculates a hash value for the contents of the application resource file to determine whether the resource files are identical.

[0211] Historical resource deduplication comparison: Match the application resource hash value with the resource hash set in the historical records.

[0212] For example, the step of performing historical resource deduplication comparison processing based on the application resource hash value to obtain resource deduplication detection results includes: performing historical resource deduplication comparison processing on the application resource hash value in the historical resource file detection record to obtain resource deduplication detection results.

[0213] This process illustrates how to decompress the target application file, extract resource-related files such as icon files, UI images, and audio files, and calculate hash values ​​for the content of each resource file. This can be done individually or by combining all files. The application resource hash values ​​are then matched against the hash sets in historical resource records to obtain the resource deduplication detection results. For example, if a match is found, it means the resource files are identical.

[0214] S3006: Generate a second deduplication detection result based on the attribute deduplication detection result and the resource deduplication detection result.

[0215] Attribute deduplication detection results: Based on the comparison results of attribute combination hash values, it is determined whether the attribute is consistent with the historical records.

[0216] Resource deduplication detection results: Based on the comparison results of application resource hash values, determine whether the resource file is consistent with the historical records.

[0217] The second deduplication detection result can be the comparison result of comprehensive attributes and resources to determine the deduplication result of the target file in the second-level detection.

[0218] In one feasible implementation, if both the attribute and resource detection results are "same type", then the second deduplication detection result is "same type". If either result is "different type", then the second deduplication detection result is "different type".

[0219] In one feasible implementation, the following implementation method may also be referred to:

[0220] B2: If the attribute deduplication detection results are of the same type, then generate a second deduplication detection result of the same type;

[0221] B4: If the attribute deduplication detection result is a different result type, then execute the step of extracting target application resource data from the target application file;

[0222] B6: If the resource deduplication detection results are of the same type, then generate a second deduplication detection result of the same type; if the resource deduplication detection results are of different types, then generate a second deduplication detection result of different types.

[0223] In one or more embodiments of this specification, the second-level deduplication detection can further refine the judgment through a dual comparison mechanism of attribute combination hash value and application resource hash value, achieving the following effects:

[0224] High efficiency: Quickly identifies the basic attributes of applications and the similarity of resource files, reducing invalid detections.

[0225] Accuracy: It considers both the logical attributes and resource characteristics of the file to improve the comprehensiveness of the detection.

[0226] Reliability: Multi-level detection ensures that similarity can be effectively identified even if application files have been partially modified.

[0227] This provides a reliable preliminary screening for subsequent in-depth detection, saving system resources while improving detection accuracy.

[0228] Optional, such as Figure 5 As shown, Figure 5 This is a flowchart illustrating a third-level deduplication detection process. The process involves determining the file code fingerprint of the target application file, performing third-level deduplication detection based on the file code fingerprint, and obtaining the third deduplication detection result, including:

[0229] S4002: Extract a preset core code segment from the target application file, calculate the file code fingerprint corresponding to the preset core code segment, perform historical code fingerprint deduplication and comparison processing on the file code fingerprint in the historical file code detection record, and obtain the historical code fingerprint comparison result.

[0230] Preset core code snippets: These are the core or logically crucial code sections in the target application file. These snippets typically reflect the application's main functions or behavioral patterns. Examples include: dynamic function call sequences, class and method names, and API call patterns.

[0231] File code fingerprinting: A feature identifier generated from core code segments using algorithms, used to identify whether the code logic of files is the same or similar. Fingerprint generation methods used for file code fingerprinting may include: hash algorithms (such as SHA256), feature vector generation (such as n-gram analysis).

[0232] Historical code fingerprint deduplication comparison: The file code fingerprint of the target file is matched with the historical code fingerprint records stored in the database to determine whether the target file has been detected.

[0233] S4004: If the historical code fingerprint comparison results are of the same type, then generate a third deduplication detection result of the same type;

[0234] Historical code fingerprint comparison results are of the same type: This means that the code logic of the target file is completely consistent with a certain file in the history, and there is no difference in the core function.

[0235] Third-party deduplication results of the same type: Code-level deduplication results of the marked files, used to determine whether to perform more in-depth deduplication.

[0236] S4006: If the historical code fingerprint comparison result is of a different result type, then a third deduplication detection result of a different type is generated.

[0237] Different result types: This refers to cases where the code logic of the target file is not significantly consistent with any file in the history.

[0238] Third-party deduplication results of different types: This indicates that the file needs to enter a more in-depth application file detection process for dynamic analysis or further static analysis.

[0239] In one or more embodiments of this specification, fingerprints are generated by core code snippets, which can identify whether the core functional logic is consistent, ensuring accuracy even if the file as a whole or resources change; and, fingerprint comparison can quickly screen existing files, reducing resource consumption on duplicate files and ensuring efficiency; and, even if the application file is obfuscated or repackaged, as long as the core logic remains unchanged, the fingerprint can still match, ensuring the reliability of the detection.

[0240] The following will combine Figure 6 This specification provides a detailed description of the application processing apparatus provided in the embodiments. It should be noted that... Figure 6 The application processing device shown is used to execute this specification. Figures 1-5 The methods shown in the embodiments are illustrated for ease of explanation, showing only the parts related to the embodiments of this specification. For specific technical details not disclosed, please refer to this specification. Figures 1-5 The example shown.

[0241] Please see Figure 6 This diagram illustrates the structure of an application processing device according to an embodiment of this specification. The application processing device 1 can be implemented as all or part of a device through software, hardware, or a combination of both. According to some embodiments, the application processing device 1 includes a file determination module 11 and a deduplication detection module 12, specifically used for:

[0242] File determination module 11 is used to determine the target application file to be detected;

[0243] The deduplication detection module 12 is used to obtain the target file hash value of the target application file, perform a first-level deduplication detection process based on the target file hash value, and obtain a first deduplication detection result.

[0244] The deduplication detection module 12 is used to perform deduplication filtering on the target application file if the first deduplication detection result is of the same type.

[0245] The deduplication detection module 12 is used to determine the file metadata features of the target application file if the first deduplication detection result is of different types, perform multi-level deduplication detection processing based on the file metadata features to obtain multi-level deduplication detection results, and perform an application file detection process on the target application file based on the multi-level deduplication detection results.

[0246] In one feasible implementation, the deduplication detection module 12 is used to: parse the attribute combination hash value and / or application resource hash value of the target application file, perform a second-level deduplication detection process based on the attribute combination hash value and / or the application resource hash value, and obtain a second deduplication detection result;

[0247] If the second deduplication detection result is of the same type, then the target application file is subjected to deduplication filtering processing. If the second deduplication detection result is of different types, then the file code fingerprint of the target application file is determined, and a third-level deduplication detection processing is performed based on the file code fingerprint to obtain a third deduplication detection result.

[0248] If the third deduplication detection result is of the same type, then the target application file is subjected to deduplication filtering. If the third deduplication detection result is of different types, then the target application file is subjected to the application file detection process.

[0249] In one feasible implementation, the deduplication detection module 12 is used for:

[0250] Obtain the target original file hash value and / or target list file hash value of the target application file, and perform historical file deduplication comparison processing based on the target original file hash value and / or target list file hash value to obtain the first deduplication detection result.

[0251] In one feasible implementation, the deduplication detection module 12 is used for:

[0252] The hash value of the target original file is compared with the hash value of the target file in the historical detection file hash set to obtain the historical file hash comparison result.

[0253] If the hash comparison results of the historical files are of the same type, then a first deduplication detection result of the same type is generated;

[0254] If the historical file hash comparison result is a different result type, then the target decompressed file list information of the target application file is determined, the target decompressed file list information is combined with information characters to obtain the target list string, the target list file hash value corresponding to the target list string is calculated, and the target list file hash value is compared with the historical file list in the historical list file detection record to obtain the historical list hash comparison result;

[0255] If the hash comparison results of the historical list are of the same type, then a first deduplication detection result of the same type is generated; if the hash comparison results of the historical list are of different types, then a first deduplication detection result of different types is generated.

[0256] In one feasible implementation, the deduplication detection module 12 is used for:

[0257] Extract key metadata from the target application file, extract preset attribute character combinations from the key metadata, calculate the attribute combination hash value of the preset attribute character combinations, perform historical attribute deduplication comparison processing based on the attribute combination hash value, and obtain the attribute deduplication detection result; and / or,

[0258] Extract target application resource data from the target application file, calculate the application resource hash value of the target application resource data, perform historical resource deduplication comparison processing based on the application resource hash value, and obtain resource deduplication detection results.

[0259] A second deduplication result is generated based on the attribute deduplication result and the resource deduplication result.

[0260] In one feasible implementation, the step of performing historical attribute deduplication comparison processing based on the attribute combination hash value to obtain the attribute deduplication detection result includes: performing historical attribute deduplication comparison processing on the attribute combination hash value in the historical attribute file detection record to obtain the historical attribute hash comparison result; and / or,

[0261] The step of performing historical resource deduplication comparison based on the application resource hash value to obtain resource deduplication detection results includes: performing historical resource deduplication comparison on the application resource hash value in the historical resource file detection record to obtain historical resource hash comparison results; and / or,

[0262] The device 1 further includes:

[0263] If the historical attribute hash comparison results are of the same type, then a second deduplication detection result of the same type is generated;

[0264] If the historical attribute hash comparison result is a different result type, then the step of extracting target application resource data from the target application file is executed;

[0265] If the historical resource hash comparison results are of the same type, a second deduplication detection result of the same type is generated; if the historical resource hash comparison results are of different types, a second deduplication detection result of different types is generated.

[0266] In one feasible implementation, the deduplication detection module 12 is used for:

[0267] Extract a preset core code segment from the target application file, calculate the file code fingerprint corresponding to the preset core code segment, and perform historical code fingerprint deduplication and comparison processing on the file code fingerprint in the historical file code detection record to obtain the historical code fingerprint comparison result.

[0268] If the historical code fingerprint comparison results are of the same type, then a third deduplication detection result of the same type is generated;

[0269] If the historical code fingerprint comparison results are of different types, then a third deduplication detection result of different types is generated.

[0270] In one feasible implementation, the deduplication detection module 12 is used for:

[0271] Based on the deduplication detection results, similar application files corresponding to the target application file are determined, and a similar file association relationship is established between the target application file and the similar application files;

[0272] The target application files are sampled and detected based on the aforementioned similar file association relationships;

[0273] The deduplication detection results include at least one of the first deduplication detection results, the second deduplication detection results, and the third deduplication detection results.

[0274] It should be noted that the application processing device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the application processing method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the application processing device and the application processing method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0275] The example numbers in this specification are for descriptive purposes only and do not represent the superiority or inferiority of the examples.

[0276] In the embodiments of this specification, the electronic device obtains the target file hash value of the target application file to perform a first-level deduplication detection process to obtain a first deduplication detection result. If the results are of the same type, deduplication filtering is performed. If the results are of different types, the attribute combination hash value and / or application resource hash value of the target application file are parsed to perform a second-level deduplication detection process to obtain a second deduplication detection result. If the results are of the same type, deduplication filtering is performed. If the results are of different types, the file code fingerprint of the target application file is determined. Based on the file code fingerprint, a third-level deduplication detection process is performed to obtain a third deduplication detection result. If the results are of the same type, deduplication filtering is performed. If the results are of different types, the application file detection process is executed. This achieves multi-level efficient filtering from the file level, attribute level to the code level, significantly reducing the burden of detecting duplicate files. By combining hash value fast matching with deep fingerprint comparison and other multi-level detection processes, it can not only efficiently process massive application files, but also effectively deal with the disguised behavior of application package variants and minor modifications. At the same time, by concentrating resources on application documents that truly require in-depth analysis, the overall efficiency and accuracy of the detection system have been significantly improved, the risk of false alarms and false negatives has been reduced, and a strong guarantee has been provided for the healthy ecosystem of the application market.

[0277] This specification also provides a computer storage medium that can store multiple instructions adapted to be loaded and executed by a processor as described above. Figures 1-5 The application processing method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-5 The specific details of the illustrated embodiments will not be elaborated here.

[0278] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1-5 The application processing method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1-5 The specific details of the illustrated embodiments will not be elaborated here.

[0279] Please refer to Figure 7 This diagram illustrates a structural block diagram of an electronic device provided in an exemplary embodiment of this specification. The electronic device in this specification may include one or more components such as a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, memory 120, input device 130, and output device 140 may be connected via the bus 150.

[0280] Processor 110 may include one or more processing cores. Processor 110 connects to various parts of the electronic device using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). Processor 110 may integrate one or more of the following: central processing unit (CPU), graphics processing unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately using a communication chip.

[0281] The memory 120 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 120 may include a non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described below, etc. The operating system may be the Android system, including systems deeply developed based on the Android system, the iOS system developed by Apple Inc., including systems deeply developed based on the iOS system, or other systems. The data storage area may also store data created by the electronic device during use, such as phonebook data, audio and video data, chat log data, etc.

[0282] See Figure 8As shown, the memory 120 can be divided into operating system space and user space. The operating system runs in the operating system space, while native and third-party applications run in the user space. To ensure that different third-party applications can achieve good running performance, the operating system allocates corresponding system resources for each application. However, different application scenarios within the same third-party application have different requirements for system resources. For example, in local resource loading scenarios, third-party applications have high requirements for disk read speed; in animation rendering scenarios, third-party applications have high requirements for GPU performance. Since the operating system and third-party applications are independent of each other, the operating system often cannot promptly perceive the current application scenario of a third-party application, resulting in the operating system's inability to adapt system resources accordingly to the specific application scenario of the third-party application.

[0283] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to establish data communication between the third-party applications and the operating system. This would allow the operating system to obtain the current scenario information of the third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.

[0284] Taking the Android operating system as an example, the programs and data stored in memory 120 are as follows: Figure 9As shown, the memory 120 can store the Linux kernel layer 320, the system runtime library layer 340, the application framework layer 360, and the application layer 380. The Linux kernel layer 320, system runtime library layer 340, and application framework layer 360 belong to the operating system space, while the application layer 380 belongs to the user space. The Linux kernel layer 320 provides low-level drivers for various hardware components of the electronic device, such as display drivers, audio drivers, camera drivers, Bluetooth drivers, Wi-Fi drivers, and power management. The system runtime library layer 340 provides support for key features of the Android system through several C / C++ libraries. For example, the SQLite library provides database support, the OpenGL / ES library provides 3D graphics support, and the Webkit library provides browser kernel support. The system runtime library layer 340 also provides the Android runtime library, which mainly provides core libraries that allow developers to write Android applications using the Java language. The Application Framework Layer 360 provides various APIs that may be used when building applications. Developers can also use these APIs to build their own applications, such as activity management, window management, view management, notification management, content provider, package management, call management, resource management, and location management. At least one application runs in the Application Layer 380. These applications can be native applications that come with the operating system, such as contacts, SMS, clock, and camera apps; or third-party applications developed by third-party developers, such as games, instant messaging, and photo editing apps.

[0285] Taking the operating system as an example (iOS), the programs and data stored in memory 120 are as follows: Figure 10As shown, the iOS system includes: Core OS layer 420, Core Services layer 440, Media layer 460, and Cocoa Touch layer 480. Core OS layer 420 includes the operating system kernel, drivers, and low-level program frameworks. These low-level program frameworks provide hardware-level functionality for use by the program frameworks located in Core Services layer 440. Core Services layer 440 provides system services and / or program frameworks required by applications, such as Foundation framework, account framework, advertising framework, data storage framework, network connectivity framework, geolocation framework, motion framework, etc. Media layer 460 provides applications with audiovisual interfaces, such as interfaces related to graphics and images, audio technology, video technology, and AirPlay (wireless playback of audio and video transmission technologies). Cocoa Touch layer 480 provides various commonly used interface-related frameworks for application development and is responsible for user touch interaction on electronic devices. Examples include local notification services, remote push services, advertising frameworks, game tool frameworks, message user interface (UI) frameworks, UIKit frameworks, map frameworks, and so on.

[0286] exist Figure 10 The framework shown includes, but is not limited to, the base framework in the core service layer 440 and the UIKit framework in the touchable layer 480. The base framework provides many basic object classes and data types, offering the most basic system services to all applications, and is independent of the UI. The UIKit framework, on the other hand, provides a basic UI class library for creating touch-based user interfaces. iOS applications can use the UIKit framework to provide their UI, thus providing the application's infrastructure for building user interfaces, drawing, handling user interaction events, responding to gestures, and so on.

[0287] The methods and principles for implementing data communication between third-party applications and the operating system in the iOS system can be found in the Android system, and will not be repeated here.

[0288] The input device 130 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 140 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined into a touch screen, which is used to receive touch operations from the user using a finger, stylus, or any suitable object on or near it, and to display the user interface of various applications. The touch screen is usually located on the front panel of the electronic device. The touch screen can be designed as a full-screen, curved screen, or irregularly shaped screen. The touch screen can also be designed as a combination of a full-screen and a curved screen, or a combination of an irregularly shaped screen and a curved screen; this specification does not limit this aspect.

[0289] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device may also include radio frequency circuits, input units, sensors, audio circuits, wireless fidelity (WiFi) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.

[0290] In the embodiments of this specification, the executing entity for each step can be the electronic device described above. Optionally, the executing entity for each step can be the operating system of the electronic device. The operating system can be Android, iOS, or other operating systems; this specification does not limit this.

[0291] The electronic device described in this specification can also be equipped with a display device. This display device can be any device capable of displaying information, such as a cathode ray tube display (CR), a light-emitting diode display (LED), an e-ink screen, a liquid crystal display (LCD), or a plasma display panel (PDP). Users can use the display device on the electronic device to view displayed text, images, videos, and other information. The electronic device can be a smartphone, tablet computer, gaming device, AR (Augmented Reality) device, automobile, data storage device, audio playback device, video playback device, laptop, desktop computing device, or wearable device such as a smartwatch, smart glasses, smart helmet, smart bracelet, smart necklace, or smart clothing.

[0292] exist Figure 7 In the illustrated electronic device, the processor 110 can be used to call the application program stored in the memory 120 and specifically perform the following operations:

[0293] Identify the target application file to be tested;

[0294] Obtain the target file hash value of the target application file, and perform a first-level deduplication detection process based on the target file hash value to obtain the first deduplication detection result;

[0295] If the first deduplication detection result is of the same type, then the target application file is subjected to deduplication filtering processing;

[0296] If the first deduplication detection result is of a different type, then the file metadata characteristics of the target application file are determined, and multi-level deduplication detection processing is performed based on the file metadata characteristics to obtain multi-level deduplication detection results. Based on the multi-level deduplication detection results, the application file detection process is executed on the target application file.

[0297] In one embodiment, when the processor 110 executes the process of determining the file metadata features of the target application file, performing multi-level deduplication detection processing based on the file metadata features to obtain multi-level deduplication detection results, and performing an application file detection process on the target application file based on the multi-level deduplication detection results, the processor 110 may perform the following steps: parsing the attribute combination hash value and / or application resource hash value of the target application file, performing a second-level deduplication detection process based on the attribute combination hash value and / or application resource hash value, and obtaining a second deduplication detection result;

[0298] If the second deduplication detection result is of the same type, then the target application file is subjected to deduplication filtering processing. If the second deduplication detection result is of different types, then the file code fingerprint of the target application file is determined, and a third-level deduplication detection processing is performed based on the file code fingerprint to obtain a third deduplication detection result.

[0299] If the third deduplication detection result is of the same type, then the target application file is subjected to deduplication filtering. If the third deduplication detection result is of different types, then the target application file is subjected to the application file detection process.

[0300] In one embodiment, when the processor 110 performs the process of obtaining the target file hash value of the target application file and performing a first-level deduplication detection process based on the target file hash value to obtain a first deduplication detection result, it may execute the following steps:

[0301] Obtain the target original file hash value and / or target list file hash value of the target application file, and perform historical file deduplication comparison processing based on the target original file hash value and / or target list file hash value to obtain the first deduplication detection result.

[0302] In one embodiment, the processor 110, when performing the historical file deduplication comparison processing based on the hash value of the target original file and / or the hash value of the target list file to obtain a first deduplication detection result, may perform the following steps:

[0303] The hash value of the target original file is compared with the hash value of the target file in the historical detection file hash set to obtain the historical file hash comparison result.

[0304] If the hash comparison results of the historical files are of the same type, then a first deduplication detection result of the same type is generated;

[0305] If the historical file hash comparison result is a different result type, then the target decompressed file list information of the target application file is determined, the target decompressed file list information is combined with information characters to obtain the target list string, the target list file hash value corresponding to the target list string is calculated, and the target list file hash value is compared with the historical file list in the historical list file detection record to obtain the historical list hash comparison result;

[0306] If the hash comparison results of the historical list are of the same type, then a first deduplication detection result of the same type is generated; if the hash comparison results of the historical list are of different types, then a first deduplication detection result of different types is generated.

[0307] In one embodiment, when the processor 110 performs the parsing of the attribute combination hash value and / or application resource hash value of the target application file, and performs a second-level deduplication detection process based on the attribute combination hash value and / or application resource hash value to obtain a second deduplication detection result, it may perform the following steps:

[0308] Extract key metadata from the target application file, extract preset attribute character combinations from the key metadata, calculate the attribute combination hash value of the preset attribute character combinations, perform historical attribute deduplication comparison processing based on the attribute combination hash value, and obtain the attribute deduplication detection result; and / or,

[0309] Extract target application resource data from the target application file, calculate the application resource hash value of the target application resource data, perform historical resource deduplication comparison processing based on the application resource hash value, and obtain resource deduplication detection results.

[0310] A second deduplication result is generated based on the attribute deduplication result and the resource deduplication result.

[0311] In one embodiment, when the processor 110 performs the historical attribute deduplication comparison processing based on the attribute combination hash value to obtain the attribute deduplication detection result, it may perform the following steps: perform historical attribute deduplication comparison processing on the attribute combination hash value in the historical attribute file detection record to obtain the attribute deduplication detection result; and / or,

[0312] In one embodiment, the processor 110, when performing the historical resource deduplication comparison processing based on the application resource hash value to obtain the resource deduplication detection result, includes: performing historical resource deduplication comparison processing on the application resource hash value in the historical resource file detection record to obtain the resource deduplication detection result; and / or,

[0313] In one embodiment, the processor 110 may further perform the following steps while executing the method:

[0314] If the attribute deduplication detection results are of the same type, then a second deduplication detection result of the same type is generated; if the attribute deduplication detection results are of different types, then the step of extracting target application resource data from the target application file is executed.

[0315] If the resource deduplication detection results are of the same type, a second deduplication detection result of the same type is generated; if the resource deduplication detection results are of different types, a second deduplication detection result of different types is generated.

[0316] In one embodiment, when the processor 110 executes the process of determining the file code fingerprint of the target application file and performing a third-level deduplication detection process based on the file code fingerprint to obtain a third deduplication detection result, it may perform the following steps: extracting a preset core code segment from the target application file, calculating the file code fingerprint corresponding to the preset core code segment, performing historical code fingerprint deduplication comparison processing on the file code fingerprint in historical file code detection records to obtain historical code fingerprint comparison results; if the historical code fingerprint comparison results are of the same result type, then generating a third deduplication detection result of the same type; if the historical code fingerprint comparison results are of different result types, then generating a third deduplication detection result of a different type.

[0317] In one embodiment, when the processor 110 performs the deduplication filtering process on the target application file, it may perform the following steps: determine similar application files corresponding to the target application file based on the deduplication detection results, and establish a similar file association relationship between the target application file and the similar application files; perform sampling detection processing on the target application file based on the similar file association relationship; wherein, the deduplication detection results include at least one of a first deduplication detection result, a second deduplication detection result, and a third deduplication detection result.

[0318] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0319] The above-disclosed embodiments are merely preferred embodiments of this specification and should not be construed as limiting the scope of this specification. Therefore, any equivalent variations made in accordance with the claims of this specification shall still fall within the scope of this specification.

Claims

1. An application processing method, characterized in that, The method includes: Identify the target application file to be tested; Obtain the target file hash value of the target application file, and perform a first-level deduplication detection process based on the target file hash value to obtain the first deduplication detection result; If the first deduplication detection result is of the same type, then the target application file is subjected to deduplication filtering processing; If the first deduplication detection result is of a different type, the file metadata characteristics of the target application file are determined, and multi-level deduplication detection processing is performed based on the file metadata characteristics to obtain multi-level deduplication detection results. Based on the multi-level deduplication detection results, the application file detection process is executed on the target application file.

2. The method according to claim 1, characterized in that, The process involves determining the file metadata features of the target application file, performing multi-level deduplication detection based on the file metadata features to obtain multi-level deduplication detection results, and executing an application file detection process on the target application file based on the multi-level deduplication detection results, including: The attribute combination hash value and / or application resource hash value of the target application file are parsed, and a second-level deduplication detection process is performed based on the attribute combination hash value and / or application resource hash value to obtain a second deduplication detection result. If the second deduplication detection result is of the same type, then the target application file is subjected to deduplication filtering processing. If the second deduplication detection result is of different types, then the file code fingerprint of the target application file is determined, and a third-level deduplication detection processing is performed based on the file code fingerprint to obtain a third deduplication detection result. If the third deduplication detection result is of the same type, then the target application file is subjected to deduplication filtering. If the third deduplication detection result is of different types, then the target application file is subjected to the application file detection process.

3. The method according to claim 2, characterized in that, The process of obtaining the target file hash value of the target application file and performing a first-level deduplication detection based on the target file hash value to obtain a first deduplication detection result includes: Obtain the target original file hash value and / or target list file hash value of the target application file, and perform historical file deduplication comparison processing based on the target original file hash value and / or target list file hash value to obtain the first deduplication detection result.

4. The method according to claim 3, characterized in that, The step of performing historical file deduplication comparison based on the hash value of the target original file and / or the hash value of the target list file to obtain the first deduplication detection result includes: The hash value of the target original file is compared with the hash value of the target file in the historical detection file hash set to obtain the historical file hash comparison result. If the hash comparison results of the historical files are of the same type, then a first deduplication detection result of the same type is generated; If the historical file hash comparison result is a different result type, then the target decompressed file list information of the target application file is determined, the target decompressed file list information is combined with information characters to obtain the target list string, the target list file hash value corresponding to the target list string is calculated, and the target list file hash value is compared with the historical file list in the historical list file detection record to obtain the historical list hash comparison result; If the hash comparison results of the historical list are of the same type, then a first deduplication detection result of the same type is generated; if the hash comparison results of the historical list are of different types, then a first deduplication detection result of different types is generated.

5. The method according to claim 2, characterized in that, The process involves parsing the attribute combination hash value and / or application resource hash value of the target application file, and performing a second-level deduplication detection process based on the attribute combination hash value and / or application resource hash value to obtain a second deduplication detection result, including: Extract key metadata from the target application file, extract preset attribute character combinations from the key metadata, calculate the attribute combination hash value of the preset attribute character combinations, perform historical attribute deduplication comparison processing based on the attribute combination hash value, and obtain the attribute deduplication detection result; and / or, Extract target application resource data from the target application file, calculate the application resource hash value of the target application resource data, perform historical resource deduplication comparison processing based on the application resource hash value, and obtain resource deduplication detection results. A second deduplication result is generated based on the attribute deduplication result and the resource deduplication result.

6. The method according to claim 5, characterized in that, The step of performing historical attribute deduplication comparison based on the attribute combination hash value to obtain the attribute deduplication detection result includes: performing historical attribute deduplication comparison on the attribute combination hash value in the historical attribute file detection record to obtain the attribute deduplication detection result; and / or, The step of performing historical resource deduplication comparison based on the application resource hash value to obtain resource deduplication detection results includes: performing historical resource deduplication comparison on the application resource hash value in the historical resource file detection record to obtain resource deduplication detection results; and / or, The method further includes: If the attribute deduplication detection results are of the same type, then a second deduplication detection result of the same type is generated; If the attribute deduplication detection result is a different result type, then the step of extracting target application resource data from the target application file is executed; If the resource deduplication detection results are of the same type, a second deduplication detection result of the same type is generated; if the resource deduplication detection results are of different types, a second deduplication detection result of different types is generated.

7. The method according to claim 2, characterized in that, The process of determining the file code fingerprint of the target application file, and performing a third-level deduplication detection process based on the file code fingerprint to obtain a third deduplication detection result includes: Extract a preset core code segment from the target application file, calculate the file code fingerprint corresponding to the preset core code segment, and perform historical code fingerprint deduplication and comparison processing on the file code fingerprint in the historical file code detection record to obtain the historical code fingerprint comparison result. If the historical code fingerprint comparison results are of the same type, then a third deduplication detection result of the same type is generated; If the historical code fingerprint comparison results are of different types, then a third deduplication detection result of different types is generated.

8. The method according to any one of claims 1-7, characterized in that, The deduplication filtering process for the target application file includes: Based on the deduplication detection results, similar application files corresponding to the target application file are determined, and a similar file association relationship is established between the target application file and the similar application files; The target application files are sampled and detected based on the aforementioned similar file association relationships; The deduplication detection results include at least one of the first deduplication detection results, the second deduplication detection results, and the third deduplication detection results.

9. An application processing device, characterized in that, The device includes: The file determination module is used to determine the target application file to be detected. The deduplication detection module is used to obtain the target file hash value of the target application file, perform a first-level deduplication detection process based on the target file hash value, and obtain a first deduplication detection result. The deduplication detection module is used to perform deduplication filtering on the target application file if the first deduplication detection result is of the same type. The deduplication detection module is used to determine the file metadata characteristics of the target application file if the first deduplication detection result is of a different type, perform multi-level deduplication detection processing based on the file metadata characteristics to obtain multi-level deduplication detection results, and perform an application file detection process on the target application file based on the multi-level deduplication detection results.

10. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions, which are adapted to be loaded by a processor and executed as method steps as claimed in any one of claims 1 to 8.

Images