Data processing method and apparatus, storage medium, device, and program product

Abstract

The application discloses a data processing method and device, a storage medium, equipment and a program product. The method comprises the following steps: obtaining to-be-processed data; performing multi-level deduplication on the to-be-processed data to obtain a repetition determination result, wherein the multi-level deduplication comprises: performing multi-level deduplication processing based on different dimension feature information of the to-be-processed data in sequence, and when any level deduplication processing determines that the to-be-processed data is repeated data, terminating subsequent level deduplication processing; and determining a processing mode corresponding to the to-be-processed data based on the repetition determination result and a historical processing result cache library. The scheme provided in the application embodiment provides accurate repetition determination and audit reuse capability by constructing a multi-level deduplication processing mechanism and combining a historical processing result cache library to determine a differentiated data processing mode, reduces the redundancy audit cost of data, and optimizes the data audit processing efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a data processing method, apparatus, storage medium, device, and program product. Background Technology

[0002] In multimedia content moderation scenarios, relevant technologies typically perform single-dimensional duplicate checks based solely on the metadata of the content. For content with different metadata but identical or highly similar content, existing solutions lack effective duplicate identification capabilities, resulting in a large amount of content data being repeatedly submitted to the moderation server. This repeated submission leads to the server processor performing numerous redundant moderation calculations, continuously consuming network bandwidth due to repeated content transmissions, and causing high-concurrency read / write pressure on the database, severely reducing the overall throughput of the moderation system. Furthermore, existing solutions lack a systematic caching and retrieval mechanism for reusing historical moderation conclusions, making it difficult to quickly match and reuse similar content that has already been moderated, further exacerbating resource consumption in the moderation process. Summary of the Invention

[0003] This application provides a data processing method, apparatus, storage medium, device, and program product. By constructing a multi-level deduplication mechanism and combining it with a historical processing result cache library to determine differentiated audit processing methods, it provides accurate duplicate judgment and audit reuse capabilities, reduces the cost of redundant data audit, and optimizes data audit processing efficiency.

[0004] On one hand, embodiments of this application provide a data processing method, the method comprising: Obtain the data to be processed; Multi-level deduplication is performed on the data to be processed to obtain a duplicate determination result. The multi-level deduplication includes: performing multi-level deduplication processing sequentially based on different dimensional feature information of the data to be processed. When any level of deduplication processing determines that the data to be processed is duplicate data, the subsequent deduplication processing at each level is terminated. Based on the duplicate determination result and the historical processing result cache, the processing method corresponding to the data to be processed is determined.

[0005] On the other hand, embodiments of this application provide a data processing apparatus, the apparatus comprising: The acquisition unit is used to acquire data to be processed. The deduplication unit is used to perform multi-level deduplication on the data to be processed to obtain a duplicate determination result. The multi-level deduplication includes: performing multi-level deduplication processing based on different dimensional feature information of the data to be processed in sequence. When any level of deduplication processing determines that the data to be processed is duplicate data, the subsequent deduplication processing at each level is terminated. The determining unit is used to determine the processing method corresponding to the data to be processed based on the duplicate determination result and the historical processing result cache library.

[0006] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program adapted for loading by a processor to execute the data processing method as described in any of the above embodiments.

[0007] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory, the memory storing a computer program, the processor executing the data processing method as described in any of the above embodiments by calling the computer program stored in the memory.

[0008] On the other hand, embodiments of this application provide a computer program product, including computer instructions, which, when executed by a processor, implement the data processing method as described in any of the above embodiments.

[0009] This application provides a data processing method for obtaining data to be processed; performing multi-level deduplication on the data to be processed to obtain a duplicate determination result. The multi-level deduplication includes: sequentially performing multi-level deduplication processing based on different dimensional feature information of the data to be processed; terminating subsequent deduplication processing at each level when any level of deduplication processing determines that the data to be processed is duplicate data; and determining the review processing method corresponding to the data to be processed based on the duplicate determination result and a historical processing result cache. By constructing a multi-level deduplication mechanism covering different dimensional feature information of the data to be processed and combining the obtained duplicate determination result with the historical processing result cache, the method intelligently determines the differentiated processing method corresponding to the data to be processed. This solves the problems in related technologies where a large number of nearly duplicate materials are repeatedly processed due to a single deduplication dimension, and the high data processing cost caused by the lack of historical processing conclusion reuse capability. It significantly reduces the number of redundant processing requests and improves the efficiency and resource utilization of data processing. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of the overall architecture of the data processing system provided in the embodiments of this application.

[0012] Figure 2This is a first flowchart illustrating the data processing method provided in an embodiment of this application.

[0013] Figure 3 This is a second flowchart illustrating the data processing method provided in an embodiment of this application.

[0014] Figure 4 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application.

[0015] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] This application provides a data processing method, apparatus, storage medium, device, and program product. Specifically, the data processing method of this application can be executed by a computer device, which can be a terminal or a server. The terminal can be a smartphone, tablet, laptop, smart TV, wearable smart device, smart vehicle terminal, etc. The terminal can also include a client, which can be an advertising client, a media platform management client, a content review client, or a mini-program, etc. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0018] It should be noted that, in this embodiment, the entity executing the data processing method can be a terminal device or a server. The terminal device can be a local terminal device or a client device in the aforementioned cloud gaming. This embodiment does not limit the type of the entity executing the method.

[0019] Please see Figure 1 , Figure 1This is a schematic diagram of the overall architecture of the data processing system provided in this application embodiment. The data processing system is divided into three layers: client, decision layer, and processing server, and relies on historical processing result cache 109, processing rule base 110, and preprocessing model base 111 to provide data persistence support.

[0020] The client includes a data creation module 101, a pre-scoring module 102, and a data reuse recommendation module 103. The data creation module 101 generates data to be processed and outputs it to the processing decision layer via data stream S1. The pre-scoring module 102 performs pre-scoring processing on the data before submission. This includes determining a source reputation score based on pass rate, rejection rate, and historical data submission volume; determining a content risk score by scanning the data for keywords based on a preset sensitive word library; and determining a structural risk score based on the data structure characteristics of the data to be processed, thus obtaining a comprehensive risk score for the data. The data reuse recommendation module 103 retrieves historical processed data similar to the current data from the historical processing result cache 109 based on the pre-scoring results to provide users with data reuse suggestions, thereby mitigating redundant processing risks in advance during the data submission stage. A feedback update relationship is established between the pre-scoring module 102 and the data reuse recommendation module 103 to continuously optimize the recommendation strategy.

[0021] The decision-making layer is the core processing layer of this system, including the differentiated incremental review module 104, the processing queue awareness and scheduling module 105, and the processing strategy engine 106. The differential incremental review module 104 is used to perform multi-level deduplication processing on the data to be processed. That is, it performs multi-level deduplication processing based on different dimensional feature information of the data to be processed in sequence. When any level of deduplication processing determines that the data to be processed is duplicate data, the subsequent deduplication processing at each level is terminated. Furthermore, based on the duplicate judgment result and the historical processing result cache library 109, it determines the processing method corresponding to the data to be processed, so as to realize the reuse of historical review results for duplicate or highly similar data and avoid redundant review interface calls. The processing queue perception and scheduling module 105 is used to perceive the processing queue status of the processing server in real time, and dynamically adjust the processing rate and processing scale of the current batch accordingly to reduce the risk of processing timeout and failure in high-concurrency scenarios. The processing strategy engine 106 integrates the outputs of the above two modules, comprehensively determines the final processing method of each piece of data to be processed, sends the data to be submitted for processing to the processing interface gateway 107 of the processing server through data stream S2, and receives the processing conclusion returned by the processing result callback module 108 through data stream S3.

[0022] The processing server includes a processing interface gateway 107 and a processing result callback module 108. The processing interface gateway 107 is used to receive processing requests submitted by the processing decision layer and forward them to the processing flow inside the processing server for execution. The processing result callback module 108 is used to asynchronously callback the processing conclusion to the processing decision layer after processing is completed, so that the processing strategy engine 106 can write the processing result into the historical processing result cache 109 to support the reuse of subsequent processing results.

[0023] At the data storage level, the historical processing result cache 109 receives data writes via data stream S4 and stores metadata fingerprint information, perception fingerprint information, video sampling fingerprint information, and their corresponding historical processing results for historical data to be processed. This provides a comparison data source and a basis for reusing historical processing conclusions for multi-level deduplication. The processing rule library receives rule configuration writes via data stream S5 and stores dynamic threshold parameters corresponding to processing strategy configurations and comprehensive risk scores. The preprocessing model library 111 receives processing conclusion writes via data stream S6 and stores and continuously updates the scoring model parameters on which the pre-scoring module 102 depends. All three databases provide model updates and strategy feedback to the processing decision layer to support the system's adaptive optimization iteration.

[0024] Please see Figure 2 , Figure 2 This is a first flowchart illustrating a data processing method provided in an embodiment of this application. The method may include the following steps: Step 110: Obtain the data to be processed.

[0025] In some embodiments, the data to be processed may be data to be reviewed, such as data objects for which content compliance checks, duplicate checks, or security analyses need to be performed.

[0026] In some embodiments, the data to be processed may include at least one of text content, image content, video content, and resource location information.

[0027] Specifically, the data to be processed can be, for example, advertising creative units, media placement creatives, short video content objects, image and text content objects, or live stream slice data.

[0028] In some embodiments, the data to be processed may originate from external media platforms, advertising platforms, content distribution platforms, or third-party data interfaces. For example, the media-side server may receive advertising material data pushed in real-time by an external media platform via an asynchronous message queue. This asynchronous message queue may be, for example, a Kafka message queue, a RocketMQ message queue, or a RabbitMQ message queue. This application does not limit this specific implementation.

[0029] In some embodiments, after acquiring the data to be processed, material unit encapsulation processing can also be performed on the data to be processed.

[0030] For example, the received raw data to be processed can be encapsulated into a unified data structure object to form a corresponding data entity to be processed. This data entity to be processed may include, for example, the following fields: unitId: A unique identifier corresponding to the data to be processed; mediaSourceId: Data source identifier; textContent: Text content; imageUrls: A collection of image resource locators; videoUrls: A collection of video resource locators; receiveTimestamp: Data received timestamp.

[0031] Step 120: Perform deduplication on the data to be processed to obtain the duplicate determination result. The multi-level deduplication includes: performing multi-level deduplication based on different dimensional feature information of the data to be processed in sequence. When any level of deduplication determines that the data to be processed is duplicate data, the subsequent deduplication at each level is terminated.

[0032] Step 120 is used to perform multi-dimensional duplicate detection on the data to be processed before it enters the review service, in order to identify duplicate content or near-duplicate content that has been processed in the past, thereby reducing the resource consumption caused by duplicate processing.

[0033] The duplication determination result refers to the determination information used to characterize whether the data to be processed has a duplication relationship with historically processed data. Specifically, the duplication determination result may include exact duplication result, approximate duplication result, video duplication result, and non-duplication result, etc.

[0034] In some embodiments, the deduplication process corresponding to step 120 can adopt a progressive processing mechanism. For example, the first level of deduplication, which has lower computational overhead, can be performed first, based on the metadata information of the data to be processed; if no duplicate results are found, the second level of deduplication, based on the image content of the data to be processed, can be performed; if no duplicate results are found, the third level of deduplication, based on the video content of the data to be reviewed, can be performed. Through the above progressive processing mechanism, a large amount of duplicate data can be filtered out using low-cost detection methods, thereby reducing the number of times the subsequent high-cost content analysis process is executed.

[0035] Step 130: Based on the duplicate determination results and the historical processing result cache, determine the processing method corresponding to the data to be processed.

[0036] The historical processing result cache library is used to store historical processing data and its corresponding processing results.

[0037] More specifically, the cached data in the historical processing result cache library can include fingerprint information, review result information, and association management information corresponding to the data to be processed. Fingerprint information is used to characterize the content features of the historically processed data; review result information is used to characterize the review conclusions corresponding to the historically processed data; and association management information is used to implement cache data validity period control, hit frequency statistics, and cache eviction management. Specifically, validity period control sets a lifespan for each cached record and automatically cleans it up after expiration to reduce the risk of outdated review conclusions being mistakenly reused; hit frequency statistics record the number of times each cached record is hit within a preset time period, serving as a basis for prioritizing the retention of high-value data; and cache eviction management removes low-value records based on a least-used or least recently used strategy when the cache capacity reaches its limit, thereby maintaining a high hit rate and low false positive rate in the cache library within limited storage space.

[0038] In some embodiments, the processing method may be to directly reuse the results of historical processing, or to re-invoke the third-party processing interface to perform processing.

[0039] Through the multi-level deduplication and diversion mechanism constructed in steps 110 to 130 above, effective filtering of duplicate processing requests can be achieved without reducing processing quality, and the number of processing interface calls, video processing bandwidth consumption and processing latency can be reduced, thereby improving the overall resource utilization efficiency, system throughput and operational stability of the processing system.

[0040] The first-level deduplication process primarily performs precise matching based on the metadata information of the data to be processed, identifying duplicate data objects that are completely identical to historically processed data at the metadata level. Compared to deep analysis based on content features, the first-level deduplication process performs lightweight duplicate detection on the structured metadata of the data to be processed, thus featuring low computational complexity, fast processing speed, and low resource consumption. By prioritizing the execution of the first-level deduplication process, a large amount of completely duplicate data can be quickly filtered out before the data to be processed enters the high-cost image or video analysis process, thereby reducing the computational resource consumption and the number of audit interface calls in the overall processing chain.

[0041] In some embodiments, multi-level deduplication is performed on the data to be processed to obtain a duplicate determination result, including: Step 121: Obtain the metadata information corresponding to the data to be processed.

[0042] Metadata information refers to structured descriptive information used to describe the basic attributes, resource location, or content organization of the data to be processed.

[0043] In some embodiments, metadata information includes at least one of text data, image resource locators, and video resource locators.

[0044] Text data refers to data objects used to describe the text content of the data to be processed. Text data may include, for example, title text, descriptive text, tag information, and keyword information.

[0045] Image resource locators are resource index information used to locate the storage location of image resources. They may include, for example, network resource addresses, object storage paths, or file index paths.

[0046] A video resource locator is a resource identifier used to locate the storage location of a video resource. It may include, for example, a network video address, a streaming media playback address, or an object storage path.

[0047] Step 122: Generate the first fingerprint information of the data to be processed based on the metadata information.

[0048] The first fingerprint information refers to the summary information generated based on the metadata information of the data to be processed, which is used to uniquely represent the metadata characteristics of the data to be processed.

[0049] In some embodiments, the first fingerprint information can be generated using a hash function, a digest algorithm, or an encoding mapping algorithm. In a specific embodiment provided in this application, the MD5 (Message-Digest Algorithm 5) algorithm is used to generate the first fingerprint information.

[0050] In some embodiments, generating first fingerprint information based on metadata information includes: Step 1221: Combine at least one of the text data, image resource locators, and video resource locators to obtain the combined target text; Step 1222: Generate first fingerprint information based on the combined target text.

[0051] In some embodiments, if the metadata information corresponding to the data to be processed includes an image resource locator sequence consisting of multiple image resource locators and a video resource locator sequence consisting of multiple video resource locators, then to avoid generating different first fingerprint information for the same data to be processed due to changes in resource order, sorting processing can be performed on the image resource locator sequence and the video resource locator sequence before combined processing.

[0052] For example, combined target text can be generated in the following form: CombinedText = textContent + sort(imageUrls) + sort(videoUrls), Here, textContent represents text data, sort(imageUrls) represents a sorted sequence of image resource locators, and sort(videoUrls) represents a sorted sequence of video resource locators.

[0053] Step 123: Perform first-level deduplication processing on the data to be processed based on the first fingerprint information to obtain the first-level deduplication result.

[0054] Step 123 is used to determine whether the data to be processed belongs to duplicate data at the metadata level based on the matching relationship between the first fingerprint information and the historical fingerprint information corresponding to the historical processing data.

[0055] The first-level deduplication result refers to the duplicate detection result obtained based on precise matching of metadata. The first-level deduplication result belongs to the duplicate detection status at the metadata level. The first-level deduplication result can include both exact duplicate results and non-duplicate results.

[0056] In some embodiments, based on the duplicate determination result and the historical processing result cache library, the processing method corresponding to the data to be processed is determined, including: Step 131: In response to the first-level deduplication result indicating that the data to be processed and the target historical processing data are matched at the metadata level, obtain the target historical processing result corresponding to the target historical processing data from the historical processing result cache library.

[0057] Step 132: Reuse the target's historical processing results as the processing results corresponding to the data to be processed.

[0058] Specifically, if the metadata fingerprint information of the target historical processing data is completely consistent with the first fingerprint information, it can be determined that the data to be processed and the target historical processing data match at the metadata level, and the data to be processed belongs to exact duplicate data. At this time, the target historical processing result corresponding to the target historical processing data can be obtained from the historical processing result cache library, and the target historical processing result can be reused as the processing result corresponding to the data to be processed.

[0059] By using the above-mentioned result reuse mechanism, we can avoid repeatedly calling the processing service interface for completely duplicate materials, thereby reducing the number of API calls and the consumption of processing computing resources.

[0060] In some embodiments, if the first-level deduplication process has already found an exact duplicate result, the subsequent second-level and third-level deduplication processes can be terminated, thus forming a short-circuit processing mechanism. This short-circuit processing mechanism allows for the priority use of computationally inefficient metadata matching methods to filter large amounts of duplicate data, thereby reducing resource consumption during subsequent image and video feature analysis.

[0061] In some embodiments, the second-level deduplication process is primarily used to perform near-duplicate detection based on the image content in the data to be processed, in order to identify data that, although the metadata information is different, still have the same or highly similar image content. Compared to the exact matching method performed based on metadata in the first-level deduplication process, the second-level deduplication process mainly performs content-level approximate matching based on image content features. Therefore, it can identify near-duplicate image data caused by operations such as changes in resource addresses, image cropping, watermarking, compression, resolution changes, or brightness adjustments.

[0062] In some embodiments, performing multi-level deduplication on the data to be processed to obtain a duplicate determination result further includes: Step 124: Obtain all image data corresponding to the data to be processed.

[0063] In some embodiments, after acquiring image data, standardized preprocessing can be performed on the image data. For example, image size can be standardized, color space can be standardized, alpha channels can be removed, or image compression and decoding can be performed to improve the consistency of subsequent feature extraction.

[0064] Step 125: For each image in the total image data, determine the corresponding second fingerprint information.

[0065] Specifically, corresponding visual features are extracted from each image data, and second fingerprint information is generated for performing approximate matching.

[0066] The second fingerprint information refers to the feature summary information generated based on the image content features to characterize the visual similarity of images. The second fingerprint information belongs to the perceptual feature representation at the image content level.

[0067] In some embodiments, the second fingerprint information can be generated using a perceptual hashing algorithm, a frequency domain analysis algorithm, a pixel difference analysis algorithm, or a deep feature encoding algorithm. The second fingerprint information may include, for example, a perceptual hash fingerprint, a difference hash fingerprint, an average hash fingerprint, or an image feature vector generated based on a convolutional neural network.

[0068] In some embodiments, determining the second fingerprint information corresponding to the image data includes: Step 1251: Extract features from the image data to obtain the hash fingerprint corresponding to the image data.

[0069] In some embodiments, a hash fingerprint includes at least one of the following: The first hash fingerprint is obtained by performing frequency domain transformation on the image data; The second hash fingerprint is obtained by calculating the brightness difference information between adjacent pixels in the image data.

[0070] In some embodiments, a first hash fingerprint can be obtained by using Discrete Cosine Transform (DCT), which is then called a pHash fingerprint.

[0071] By calculating the brightness difference information between adjacent pixels in the image data, the second hash fingerprint of the image data, namely the dHash fingerprint, can be obtained.

[0072] Step 1252: Based on the hash fingerprint, determine the corresponding second fingerprint information of the image data.

[0073] After obtaining the hash fingerprint of the image data, the hash fingerprint can be used as the corresponding second fingerprint information of the image data. For example, the first hash fingerprint and the second hash fingerprint can be combined as the second fingerprint information.

[0074] For example: FP_perceptual = {pHash, dHash} Wherein: FP_perceptual represents the second fingerprint information; pHash represents the first hash fingerprint; dHash represents the second hash fingerprint.

[0075] By combining multiple hash features, the stability and accuracy of image approximation matching can be improved.

[0076] Step 126: Perform second-level deduplication processing on the data to be processed based on the second fingerprint information to obtain the second-level deduplication result.

[0077] Specifically, based on the degree of difference between the second fingerprint information and the historical image features in the historical processing data, it is determined whether the data to be processed belongs to the near-repeated type.

[0078] In some embodiments, a second-level deduplication process is performed on the data to be processed based on the second fingerprint information to obtain a second-level deduplication result, including: Step 1261: For each image in the total image data, compare the second fingerprint information with the image fingerprint information in the historical processed data to obtain the feature difference value.

[0079] Feature difference value refers to a numerical value used to measure the degree of difference between the second fingerprint information and the image fingerprint information in the historical processed data. It can be generated based on Hamming distance, Euclidean distance or cosine similarity.

[0080] In one specific embodiment provided in this application, Hamming distance is used to generate feature difference values.

[0081] Hamming distance can refer to the number of corresponding bits that differ between two binary sequences of equal length.

[0082] For example, the number of corresponding bits that differ between the binary sequences 10100101 and 11100111 is 2, therefore the Hamming distance is 2.

[0083] In some embodiments, a first feature difference value can be obtained by comparing the first hash in the second fingerprint information with the perceived hash information of the historical processed data, and a second feature difference value can be obtained by comparing the second hash in the second fingerprint information with the difference hash information of the historical processed data.

[0084] Step 1262: Based on the comparison result between the feature difference value and the first difference threshold, determine the image duplication determination result corresponding to the image data.

[0085] The first difference threshold can be used to determine whether image data is approximately repetitive.

[0086] In some embodiments, if the feature difference value is less than or equal to a first difference threshold, the image data is determined to be an approximately duplicate image; if the feature difference value is greater than the first difference threshold, the image data is determined to be a non-duplicate image.

[0087] In some embodiments, the feature difference value includes the first feature difference value and the second feature difference value, and the first difference threshold includes the perceptual hash distance threshold and the differential hash distance threshold. Based on the comparison result of the feature difference value and the first difference threshold, the image duplication determination result corresponding to the image data is determined, including: if the first feature difference value ≤ the perceptual hash distance threshold and the second feature difference value ≤ the differential hash distance threshold, then the image data is considered to be an approximately duplicate image; otherwise, the image data is considered to be a non-duplicate image.

[0088] Step 1263: Calculate the image duplication results for all image data to determine the duplication rate.

[0089] The duplication rate refers to the percentage of images in the total image data that are identified as duplicates.

[0090] Step 1264: In response to the repetition ratio meeting the preset ratio condition, generate a second-level deduplication result indicating that the data to be processed belongs to the near-repetition type.

[0091] The preset ratio condition can be used to determine whether the entire data to be processed belongs to the ratio determination condition of approximately duplicate material.

[0092] In some embodiments, if the repetition ratio is greater than or equal to a preset ratio condition, an approximate repetition result is generated; if the repetition ratio is less than the preset ratio condition, a non-repetition result is generated.

[0093] For example, the preset ratio conditions can be 55%, 60%, 70%, etc.

[0094] By employing a two-stage determination mechanism that combines image-level duplicate detection and material-level proportional aggregation, the entire data to be processed can be incorrectly identified as duplicate material due to the repetition of a single common image or a partial image, thereby improving the accuracy and stability of the overall near-duplicate detection.

[0095] In some embodiments, after the second-level deduplication process determines that the data to be processed belongs to the near-duplicate type, a cache hit confirmation process can be further performed based on the historical image perception features in the historical processing result cache library to determine whether the corresponding historical review results can be reused. Compared with the processing method of directly reusing the review conclusion based only on the preliminary near-duplicate result, the embodiments of this application further introduce a secondary cache difference verification mechanism, thereby reducing the problem of incorrect reuse of results due to false hits caused by local image similarity or weakly related images.

[0096] In some embodiments, based on the duplicate determination result and the historical processing result cache library, the processing method corresponding to the data to be processed is determined, including: Step 133: In response to the second-level deduplication result indicating that the data to be processed belongs to the near-repeated type, the second fingerprint information is compared with the perceptual features of each historical image stored in the historical processing result cache library to obtain multiple candidate difference values. The minimum value among the multiple candidate difference values ​​is determined as the target cache difference value.

[0097] Step 134: In response to the target cache difference value satisfying the second difference threshold, retrieve the target historical processing result corresponding to the target cache difference value from the historical processing result cache library, wherein the second difference threshold is a positive number less than the first difference threshold.

[0098] Specifically, step 134 is used to determine whether to allow the reuse of historical processing results based on stricter cache reuse conditions.

[0099] The second difference threshold is used to further improve the accuracy and security of reusing historical audit results.

[0100] The first difference threshold is mainly used in the approximate repetition detection stage, while the second difference threshold is mainly used in the cached result reuse stage. Since the reuse of processing results directly affects the final processing result, the second difference threshold, which is more stringent than the first difference threshold, is used in this embodiment.

[0101] For example, the first difference threshold can be 8; the second difference threshold can be 4.

[0102] The above-mentioned dual-threshold mechanism can avoid misjudgment caused by weakly similar content hitting cached results.

[0103] Step 135: If the target historical processing result is a pass result, reuse the target historical processing result as the processing result corresponding to the data to be processed.

[0104] Step 136: If the target historical processing result is a rejection result, the data to be processed is determined as data to be reprocessed, and the processing operation is performed on the data to be reprocessed.

[0105] For rejection results, this application embodiment does not directly reuse the corresponding rejection conclusion, but instead re-enters the processing flow.

[0106] This is because some historical rejection results may have expired, some processing rules may have been updated, and some nearly identical data may only be locally similar.

[0107] Therefore, in order to avoid erroneously rejecting normal data, in this embodiment of the application, when the target historical processing result is a rejection result, only the corresponding data to be processed is marked as data to be reprocessed, and the processing flow is re-executed.

[0108] The above mechanism can improve the efficiency of review cache reuse while reducing the risk of erroneous rejection, thus balancing the processing performance and review accuracy of the review system.

[0109] In some embodiments, to improve the accuracy and adaptability of near-duplicate judgment in the second-level deduplication process, the first difference threshold, the preset ratio condition, and the second difference threshold can be dynamically adjusted according to the risk level of the data to be processed. Compared with a uniform judgment method using fixed thresholds, the dynamic threshold adjustment mechanism in this embodiment can adaptively adjust the near-duplicate judgment criteria based on the credibility of the data source, the risk level of the content, and the complexity of the data structure of the data to be processed, thereby reducing the false positive rate while improving the duplicate detection hit rate.

[0110] In some embodiments, the step of determining the first difference threshold, the second difference threshold, or the preset ratio condition includes: Based on at least one of the data source, data content, and data structure of the data to be processed, determine a first difference threshold and / or a second difference threshold and / or a preset ratio condition.

[0111] The data source refers to the data provider or content source channel corresponding to the data to be processed, such as media source identifier, channel identifier, advertising account identifier, or developer platform identifier.

[0112] Data content refers to the semantic information or visual content contained in the data to be processed.

[0113] Data structure refers to the internal organization and structural complexity of the data to be processed, which may include the number of fields, the number of images, the number of videos, and the external link structure.

[0114] In some embodiments, determining a first difference threshold and / or a second difference threshold and / or a preset ratio condition based on at least one of the data source, data content, and data structure of the data to be processed includes: Step 01: Based on the processing pass rate, processing rejection rate, and number of historical data submissions of the data source to be processed, determine the source reputation score corresponding to the data source, and / or, based on a preset sensitive word library, perform keyword scanning on the data to be processed to determine the content risk score corresponding to the data to be processed, and / or, based on the data structure characteristics of the data to be processed, determine the structure risk score corresponding to the data to be processed.

[0115] Among them, the pass rate P_pass can be the historical pass rate, the rejection rate R_reject can be the rejection rate within a recent preset time period, and the number of historical data submissions N_total indicates the scale of historical submissions.

[0116] For example, a source reputation score can be generated as follows: S_media = α × P_pass + β × (1 - R_reject) + γ × log(N_total + 1) Where: S_media represents the source reputation score, α, β and γ represent weight parameters, and log represents the logarithmic function.

[0117] In one specific embodiment provided in this application, the weight parameters α = 0.5, β = 0.3, γ = 0.2, and the range of the source reputation score is normalized to [0, 100].

[0118] Content risk scoring refers to scoring information used to characterize the degree of risk of content violations in the data to be processed.

[0119] In some embodiments, it can be generated based on sensitive word matching, risk industry identification, or illegal semantic analysis. More specifically, for example, keyword scanning processing can be performed on the text content through a preset sensitive word library.

[0120] A pre-defined sensitive word library can refer to a set of keywords used to identify high-risk text content.

[0121] Structural risk scoring is a rating system used to characterize the complexity of the data structure to be processed and the degree of potential risk. It can be generated based on the number of images, videos, external links, and structural nesting complexity.

[0122] For example, data with multiple video clips corresponds to higher structural risk, data containing external links corresponds to higher structural risk, and data containing a large number of images corresponds to higher structural complexity.

[0123] Step 02: Determine the comprehensive risk score S_risk corresponding to the data to be processed based on at least one of the source reputation score, content risk score, and structural risk score.

[0124] In some embodiments, the comprehensive risk score S_risk can be generated, for example, as follows: S_risk = w1 × (100 - S_media) + w2 × S_text + w3 × S_struct, Where: S_media represents the source reputation score, S_text represents the content risk score, S_struct represents the structure risk score, and w1, w2, and w3 represent the weight parameters.

[0125] In one specific embodiment provided in this application, w1 = 0.4, w2 = 0.4, and w3 = 0.2.

[0126] Step 03: Based on the comprehensive risk score, determine the first difference threshold and / or the second difference threshold and / or the preset ratio condition, wherein the first difference threshold and the second difference threshold are negatively correlated with the comprehensive risk score, and the preset ratio condition is positively correlated with the comprehensive risk score.

[0127] Among them, the first difference threshold is negatively correlated with the comprehensive risk score. That is, the higher the comprehensive risk score, the smaller the first difference threshold and the second difference threshold are; the lower the comprehensive risk score, the larger the first difference threshold and the second difference threshold are.

[0128] That is, for high-risk data, stricter approximate matching conditions are used; for low-risk data, more lenient approximate matching conditions are used.

[0129] Furthermore, the preset ratio condition is positively correlated with the comprehensive risk score, which can be expressed as follows: the higher the comprehensive risk score, the higher the required repetition ratio; the lower the comprehensive risk score, the lower the required repetition ratio.

[0130] In other words, for high-risk data, more images are needed to meet the duplication criteria before the entire material is allowed to be judged as approximately duplicated.

[0131] In some embodiments, the first difference threshold includes a perceptual hash distance threshold and a difference hash distance threshold. The first difference threshold, the second difference threshold, and the preset ratio condition can be dynamically generated, for example, as follows: Perceived hash distance threshold (pHash hamming distance threshold): Base value T_pHash_base = 5, adjusted T_pHash = T_pHash_base + delta_p, where delta_p = round(2 × (1 - S_risk / 100)). When S_risk = 0 (very low risk), T_pHash = 7 (lenient); when S_risk = 100 (very high risk), T_pHash = 5 (strict, back to base value).

[0132] Differential hash distance threshold (dHash Hamming distance threshold): Base value T_dHash_base = 8, adjusted T_dHash = T_dHash_base + delta_d, where delta_d = round(3 × (1 - S_risk / 100)). When S_risk = 0, T_dHash = 11; when S_risk = 100, T_dHash = 8.

[0133] The second difference threshold (cache approximation matching Hamming distance threshold): Base value T_cache_base = 3, adjusted T_cache = T_cache_base + delta_c, where delta_c = round(2 × (1 - S_risk / 100)). When S_risk = 0, T_cache = 5, and when S_risk = 100, T_cache = 3.

[0134] Preset scaling conditions (approximate judgment scaling threshold at the material unit level): Base value theta_img_base = 70%, after adjustment theta_img = theta_img_base - delta_theta, where delta_theta = round(15 ×(1 - S_risk / 100))%. When S_risk = 0, theta_img = 55% (easier to judge as approximate repetition), and when S_risk = 100, theta_img = 70%.

[0135] The aforementioned dynamic threshold adjustment mechanism makes it easier for data from high-reputation, low-risk sources to be subject to near-duplicate judgment, thereby improving cache reuse rate and audit resource utilization. Meanwhile, for data from high-risk sources, the risk of erroneously reusing historical audit results is reduced by raising the near-duplicate judgment standard, thereby improving the security and stability of the audit system.

[0136] The third-level deduplication process is mainly used to perform duplicate detection based on the video content in the data to be processed, in order to identify duplicate video data that has been transcoded, repackaged, partially modified, or lightly edited. Compared with precise matching methods based on metadata and static visual matching methods based on image content, video content has the characteristics of large data volume, strong temporal continuity, and complex encoding structure. Therefore, this application embodiment achieves high-efficiency duplicate detection of video content by generating video fingerprints based on video sampling data segments.

[0137] In some embodiments, performing multi-level deduplication on the data to be processed to obtain a duplicate determination result further includes: Based on the video data to be processed, a third level of deduplication processing is performed on the data to be processed. More specifically, this includes: Step 127: Obtain multiple sampled data segments corresponding to the data to be processed.

[0138] A sampled data segment refers to a local data content extracted from the data to be processed.

[0139] In some embodiments, obtaining multiple sampled data segments corresponding to the data to be processed includes: Step 1271: Obtain the data segment corresponding to the target byte range from the data to be processed, and use it as the first round of sampled data segment.

[0140] In some embodiments, a data segment of a fixed byte range may be extracted preferentially from the header, middle or tail region of the data to be processed.

[0141] For example, the following byte ranges can be extracted: the data segments corresponding to 0KB to 64KB; the data segments corresponding to 512KB to 576KB; and the data segments corresponding to the last 64KB, as the first round of sampling data segments.

[0142] Using the above method, temporary feature fingerprints can be quickly generated from relatively stable data regions in the data to be processed.

[0143] In some embodiments, a first-round temporary fingerprint can be generated based on a first-round sampled data fragment.

[0144] For example: TempFP_1 = MD5(Segment_1 + Segment_2 + Segment_3), Where: TempFP_1 represents the first round of temporary fingerprints; Segment_1 to Segment_3 represent the first round of sampled data segments.

[0145] In some embodiments, the first-round temporary fingerprint can be used to preferentially query the historical processing result cache. If the first-round temporary fingerprint has already hit the historical processing result cache, it means that the corresponding video data is highly consistent with the historical video data, and there is no need to continue performing depth sampling processing.

[0146] Step 1272: If the first-round temporary fingerprint generated based on the first-round sampled data fragment does not hit the historical processing result cache library, then the number of target sampling points is dynamically determined based on the total amount of data to be processed, and the number of target sampling points is obtained from the data to be processed at a preset interval as the second-round sampled data fragment.

[0147] In some embodiments, the number of target sampling points can be dynamically determined based on the total amount of data to be processed.

[0148] For example, when the data to be processed is less than 10MB, 3 sampling points can be set; when the data to be processed is between 10MB and 100MB, 5 sampling points can be set; when the data to be processed is greater than 100MB, 8 sampling points can be set.

[0149] In some embodiments, uniform sampling can be performed according to a preset time interval, a preset byte interval, or based on the timestamp interval of video keyframes. For example, the first method performs equal-interval sampling at intervals of several fixed byte lengths; the second method performs equal-duration sampling at intervals of several milliseconds; and the third method first parses the keyframe timestamp sequence in the video index and then performs interval sampling using the equal intervals between adjacent keyframes as the sampling times.

[0150] For example: Interval = FileSize / SampleCount, where: FileSize represents the total amount of data to be processed; SampleCount represents the number of target sampling points; Interval represents the sampling interval.

[0151] In some embodiments, multiple second-round sampling data segments can be acquired sequentially according to the sampling interval.

[0152] For example: Segment_i = VideoData[offset_i : offset_i + SampleLength], where: offset_i represents the offset position of the i-th sample; SampleLength represents the length of a single sample segment.

[0153] The dynamic sampling mechanism described above enables large-volume videos to acquire more sampling features, thereby improving the stability of video repetition detection.

[0154] Step 1273: Determine the first round of sampled data segments and / or the second round of sampled data segments as multiple sampled data segments corresponding to the data to be processed.

[0155] Step 128: Generate third fingerprint information based on multiple sampled data fragments.

[0156] In some embodiments, a third fingerprint information can be generated by splicing multiple sampled data fragments.

[0157] For example: FP_video = MD5(Segment_1 + Segment_2 + ... + Segment_n), where: FP_video represents the third fingerprint information; Segment_1 to Segment_n represent multiple sampled data segments.

[0158] Furthermore, in some embodiments, local video fingerprints can be generated for each sampled data segment separately, and multiple local video fingerprints can be combined into a third fingerprint information.

[0159] For example: FP_video = {FP_1, FP_2, ..., FP_n}.

[0160] By combining multiple segments, the video feature representation ability and robustness of repeated detection can be improved.

[0161] Step 129: Perform third-level deduplication based on the third fingerprint information to obtain the third-level deduplication result.

[0162] Step 129 is used to determine whether the data to be processed belongs to video duplicate data based on the matching relationship between the third fingerprint information and the historical video fingerprint.

[0163] In some embodiments, based on the duplicate determination result and the historical processing result cache library, the processing method corresponding to the data to be processed is determined, including: Step 137: In response to the third fingerprint information corresponding to the data to be processed and the video fingerprint information of the target historical processing data in the historical processing data satisfying the preset matching conditions, the target historical processing result corresponding to the target historical processing data is obtained from the historical processing result cache library.

[0164] Among them, the preset matching conditions can be matching rules used to determine whether video content meets the conditions for historical cache reuse.

[0165] In some embodiments, the preset matching conditions may include at least one of the following: video fingerprints are completely identical; the video sample fingerprint matching ratio is greater than a preset ratio; the number of video partial fingerprint matches reaches a preset number.

[0166] Step 138: If the target historical processing result is a pass result, then reuse the target historical processing result as the processing result corresponding to the data to be processed.

[0167] Step 138 is used to directly reuse historical processing results for already processed video duplicate data.

[0168] By using the above methods, repeated calls to the video review model can be avoided, thereby reducing the consumption of video processing resources.

[0169] Step 139: If the target historical processing result is a rejection result, the data to be processed is determined as data to be reprocessed, and the reprocessing operation is performed on the data to be reprocessed.

[0170] Step 139 is used to avoid the risk of false rejections caused by directly reusing historical rejection results.

[0171] Since video processing rules may change dynamically and some video content may only have local similarities, in this embodiment of the application, for rejection results, the corresponding rejection conclusion is not directly reused, but the process is re-entered.

[0172] In some embodiments, the method further includes: After the data to be reprocessed is processed, the third fingerprint information of the data to be reprocessed and the processing result are written into the historical processing result cache.

[0173] This embodiment is used to update the cache of new processing results so that the results can be reused in subsequent video duplication detection processes.

[0174] In some embodiments, fingerprint information of other levels or types, such as first fingerprint information and second fingerprint information, may also be written into the historical processing result cache library in addition to the third fingerprint information.

[0175] All of the above technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0176] This application provides a data processing method for obtaining data to be processed; performing multi-level deduplication on the data to be processed to obtain a duplicate determination result. The multi-level deduplication includes: sequentially performing multi-level deduplication processing based on different dimensional feature information of the data to be processed; terminating subsequent deduplication processing at each level when any level of deduplication processing determines that the data to be processed is duplicate data; and determining the processing method corresponding to the data to be processed based on the duplicate determination result and a historical processing result cache. By constructing a multi-level deduplication processing mechanism covering metadata information, image content, and video content, and combining the obtained duplicate determination result with the historical processing result cache, the method intelligently determines the differentiated processing method corresponding to the data to be processed. This solves the problems in related technologies where a large number of nearly duplicate materials are repeatedly submitted for review due to a single deduplication dimension, and the high processing cost caused by the lack of historical processing conclusion reuse capability. It significantly reduces the number of redundant processing requests and improves the efficiency and resource utilization of data processing.

[0177] To better illustrate the data processing method provided in the embodiments of this application, please refer to... Figure 3 , Figure 3 This is a second flowchart illustrating the data processing method provided in an embodiment of this application.

[0178] S31, Perform data feature extraction processing on the newly generated data to be processed.

[0179] Specifically, metadata information, image content features, and video content features of the data to be processed are extracted as inputs for subsequent multi-level deduplication and comprehensive risk scoring.

[0180] S32, Perform comprehensive risk pre-screening and scoring on the data to be processed.

[0181] Specifically, a source reputation score is determined based on the historical approval rate, rejection rate, and number of historical data submissions of the data source. A content risk score is determined by scanning the data to be processed using a preset sensitive word library. A structural risk score is determined based on the data structure characteristics of the data to be processed. These factors combined result in a comprehensive risk score for the data to be processed, which is used to predict the approval rate of the data to be processed on the client side. Next, it is determined whether the comprehensive risk score reaches a preset threshold: if it does not reach the preset threshold, step S32a is executed, providing the user with modification suggestions for the data to be processed. After the user adjusts the data to be processed according to the suggestions, the process returns to step S31 to perform data feature extraction processing; if the comprehensive risk score reaches the preset threshold, the subsequent step S33 is executed.

[0182] S33, Perform differentiated incremental audit processing on the data to be processed.

[0183] Specifically, the data to be processed undergoes a first-level deduplication process based on metadata information, a second-level deduplication process based on image content, and a third-level deduplication process based on video content in sequence to obtain a duplicate determination result. Furthermore, based on the duplicate determination result and the historical processing result cache, the corresponding review processing method for the data to be processed is determined. For data to be processed that is determined to be duplicate or highly similar, the corresponding historical review conclusion in the historical processing result cache is directly reused, and only the data to be processed that has substantial differences is submitted to the subsequent review process, thereby avoiding redundant review interface calls for duplicate data.

[0184] S34 performs audit queue awareness scheduling for pending data that needs to be submitted for review.

[0185] Specifically, the system monitors the current queue status of the review server in real time and dynamically adjusts the submission timing and scale of approval requests based on queue load. This allows for the selection of the optimal submission time for pending data, thereby reducing the risk of review timeouts and failures in high-concurrency scenarios.

[0186] S35 submits the data to be processed, which has undergone differentiated incremental review processing and review queue awareness scheduling, to the review interface gateway of the review server to perform review processing and wait for the review result callback.

[0187] Determine whether the review is approved: If the review is not approved, proceed to step S35a and record the reason for the failure of this review process for subsequent analysis and model updates; if the review is approved, continue to proceed to the next step S36.

[0188] S36, Perform historical review result reuse recommendation processing on the approved pending data.

[0189] Specifically, the metadata fingerprint information, perception fingerprint information, and video sampling fingerprint information of the data to be processed that has passed the review are written into the historical processing result cache library along with the review conclusion, so as to update the cache library and the pre-review model library; at the same time, based on the updated model, historical processing data similar to the currently approved data are recommended to users to support the reuse decision of similar data to be processed in the future.

[0190] To facilitate better implementation of the data processing method of this application embodiment, this application embodiment also provides a data processing apparatus. Please refer to... Figure 4 , Figure 4 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application. The data processing apparatus 200 may include: Acquisition unit 210 is used to acquire data to be processed; The deduplication unit 220 is used to perform multi-level deduplication on the data to be processed to obtain the duplicate determination result. The multi-level deduplication includes: performing multi-level deduplication processing based on different dimensional feature information of the data to be processed in sequence. When any level of deduplication processing determines that the data to be processed is duplicate data, the subsequent deduplication processing at each level is terminated. The determining unit 230 is used to determine the processing method corresponding to the data to be processed based on the duplicate judgment result and the historical processing result cache.

[0191] In some embodiments, when the apparatus performs multi-level deduplication on the data to be processed to obtain a duplicate determination result, it is specifically used for: Obtain the metadata information corresponding to the data to be processed; Generate the first fingerprint information of the data to be processed based on the metadata information; Based on the first fingerprint information, the first-level deduplication process is performed on the data to be processed to obtain the first-level deduplication result.

[0192] In some embodiments, when the device determines the processing method corresponding to the data to be processed based on the duplicate determination result and the historical processing result cache, it is specifically used for: In response to the first-level deduplication result indicating that the data to be processed and the target historical processing data are matched at the metadata level, the target historical processing result corresponding to the target historical processing data is obtained from the historical processing result cache library. The historical processing results of the target are reused as the processing results corresponding to the data to be processed.

[0193] In some embodiments, the metadata information includes at least one of text data, image resource locators, and video resource locators. When the apparatus generates first fingerprint information based on the metadata information, it is specifically used for: Combine at least one of the text data, image resource locators, and video resource locators to obtain the combined target text; The first fingerprint information is generated based on the combined target text.

[0194] In some embodiments, when the apparatus performs multi-level deduplication on the data to be processed to obtain a duplicate determination result, it is specifically used for: Retrieve all image data corresponding to the data to be processed; For each image in the total image data, determine the corresponding second fingerprint information for that image. Based on the second fingerprint information, a second-level deduplication process is performed on the data to be processed to obtain the second-level deduplication result.

[0195] In some embodiments, when determining the second fingerprint information corresponding to the image data, the device is specifically used for: Feature extraction is performed on the image data to obtain the hash fingerprint corresponding to the image data; Based on hash fingerprinting, the corresponding second fingerprint information of the image data is determined.

[0196] In some embodiments, a hash fingerprint includes at least one of the following: The first hash fingerprint is obtained by performing frequency domain transformation on the image data; The second hash fingerprint is obtained by calculating the brightness difference information between adjacent pixels in the image data.

[0197] In some embodiments, when the device performs a second-level deduplication process on the data to be processed based on the second fingerprint information to obtain a second-level deduplication result, it is specifically used for: For each image in the entire image data, the second fingerprint information is compared with the image fingerprint information in the historical processed data to obtain the feature difference value; Based on the comparison results between the feature difference value and the first difference threshold, the image duplication determination result corresponding to the image data is determined; Analyze the image duplication results of all image data to determine the duplication rate; In response to the repetition ratio meeting the preset ratio condition, a second-level deduplication result is generated, indicating that the data to be processed belongs to the near-repetition type.

[0198] In some embodiments, when the device determines the processing method corresponding to the data to be processed based on the duplicate determination result and the historical processing result cache, it is specifically used for: In response to the second-level deduplication result indicating that the data to be processed belongs to the near-repeated type, the second fingerprint information is compared with the perceptual features of each historical image stored in the historical processing result cache to obtain multiple candidate difference values. The minimum value among the multiple candidate difference values ​​is determined as the target cache difference value. In response to the target cache difference value satisfying the second difference threshold, the target historical processing result corresponding to the target cache difference value is obtained from the historical processing result cache library, wherein the second difference threshold is a positive number less than the first difference threshold; If the target historical processing result is a pass result, the target historical processing result is reused as the processing result corresponding to the data to be processed. If the target historical processing result is a rejection result, the data to be processed is identified as data to be reprocessed, and processing operations are performed on the data to be reprocessed.

[0199] In some embodiments, the apparatus is further configured to determine a first difference threshold and / or a preset ratio condition, specifically configured to: Based on at least one of the data source, data content, and data structure of the data to be processed, determine a first difference threshold and / or a preset ratio condition.

[0200] In some embodiments, when the device determines a first difference threshold and / or a preset ratio condition based on at least one of the data source, data content, and data structure of the data to be processed, it is specifically used for: Based on the data source's processing pass rate, processing rejection rate, and historical data submission quantity, determine the source reputation score corresponding to the data source, and / or, based on a preset sensitive word library, perform keyword scanning on the data to be processed to determine the content risk score corresponding to the data to be processed, and / or, based on the data structure characteristics of the data to be processed, determine the structure risk score corresponding to the data to be processed. Determine the comprehensive risk score for the data to be processed based on at least one of the source reputation score, content risk score, and structural risk score. Based on the comprehensive risk score, a first difference threshold and / or a preset ratio condition are determined, wherein the first difference threshold is negatively correlated with the comprehensive risk score, and the preset ratio condition is positively correlated with the comprehensive risk score.

[0201] In some embodiments, when the apparatus performs multi-level deduplication on the data to be processed to obtain a duplicate determination result, it is specifically used for: Obtain multiple sampled data segments corresponding to the data to be processed; Third fingerprint information is generated based on multiple sampled data fragments; The third-level deduplication process is performed based on the third fingerprint information to obtain the third-level deduplication result.

[0202] In some embodiments, when the device determines the audit processing method corresponding to the data to be processed based on the duplicate determination result and the historical processing result cache, it is specifically used for: In response to the third-level deduplication result indicating that the third fingerprint information corresponding to the data to be processed and the video fingerprint information of the target historical data in the historical processing data meet the preset matching conditions, the target historical processing result corresponding to the target historical processing data is obtained from the historical processing result cache library. If the target historical processing result is a pass result, then the target historical processing result is reused as the processing result corresponding to the data to be processed; If the target historical processing result is a rejection result, the data to be processed is determined as data to be reprocessed, and processing operations are performed on the data to be reprocessed.

[0203] In some embodiments, when the apparatus is used to acquire multiple sampled data segments corresponding to the data to be processed, it is specifically used for: Obtain the data segment corresponding to the target byte range from the data to be processed, and use it as the first round of sampled data segment; If the first-round temporary fingerprint generated based on the first-round sampled data fragments does not hit the historical processing result cache, the number of target sampling points is dynamically determined based on the total amount of data to be processed, and the number of target sampling points is obtained from the video data at a preset interval as the second-round sampled data fragments. The first round of sampled data segments and / or the second round of sampled data segments are identified as multiple sampled data segments corresponding to the data to be processed.

[0204] In some embodiments, the device is further configured to: After the data to be reprocessed is processed, the third fingerprint information of the data to be reprocessed and the processing result are written into the historical processing result cache.

[0205] Each unit in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each unit.

[0206] The data processing device 200 can be integrated into a terminal or server that has storage and a processor and thus computing power, or the data processing device 200 can be the terminal or server.

[0207] Optionally, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0208] Figure 5This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device may include: a communication interface 301, a memory 302, a processor 303, and a communication bus 304. The communication interface 301, memory 302, and processor 303 communicate with each other via the communication bus 304. The communication interface 301 is used for data communication between the device 300 and external devices. The memory 302 can be used to store software programs and modules, and the processor 303 runs the software programs and modules stored in the memory 302, such as the software programs for corresponding operations in the aforementioned method embodiments.

[0209] In some embodiments, the processor 303 may invoke software programs and modules stored in the memory 302 to perform the following operations: acquire data to be processed; perform multi-level deduplication on the data to be processed to obtain a duplicate determination result, wherein the multi-level deduplication includes: sequentially performing multi-level deduplication processing based on different dimensional feature information of the data to be processed, terminating subsequent deduplication processing at each level when any level of deduplication processing determines that the data to be processed is duplicate data; and determining the audit processing method corresponding to the data to be processed based on the duplicate determination result and the historical processing result cache.

[0210] In some embodiments, the computer device 300 may be integrated into a terminal or server that has storage and a processor, thus possessing computing capabilities; or the computer device 300 may be the terminal or server. The terminal may be a smartphone, tablet, laptop, smart TV, smart speaker, wearable smart device, personal computer, or other similar device. The server may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0211] This application also provides a computer-readable storage medium for storing a computer program. This computer-readable storage medium can be applied to a computer device, and the computer program causes the computer device to execute the corresponding flow in the data processing method of the embodiments of this application; for brevity, further details are omitted here.

[0212] This application also provides a computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding flow in the data processing method of the embodiments of this application. For simplicity, further details are omitted here.

[0213] This application also provides a computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding flow in the data processing method of this application. For brevity, further details are omitted here.

[0214] It should be understood that the processor in this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0215] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0216] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0217] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0218] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0219] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0220] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0221] In addition, the functional units in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0222] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer or a server) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0223] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the data to be processed; Multi-level deduplication is performed on the data to be processed to obtain a duplicate determination result. The multi-level deduplication includes: performing multi-level deduplication processing sequentially based on different dimensional feature information of the data to be processed. When any level of deduplication processing determines that the data to be processed is duplicate data, the subsequent deduplication processing at each level is terminated. Based on the duplicate determination result and the historical processing result cache, the processing method corresponding to the data to be processed is determined.

2. The data processing method as described in claim 1, characterized in that, The process of performing multi-level deduplication on the data to be processed to obtain the duplicate determination result includes: Obtain the metadata information corresponding to the data to be processed; Generate the first fingerprint information of the data to be processed based on the metadata information; Based on the first fingerprint information, the data to be processed is subjected to a first-level deduplication process to obtain a first-level deduplication result.

3. The data processing method as described in claim 2, characterized in that, Based on the duplicate detection result and the historical processing result cache, the processing method corresponding to the data to be processed is determined, including: In response to the first-level deduplication result indicating that the data to be processed is matched with the target historical processing data at the metadata level, the target historical processing result corresponding to the target historical processing data is obtained from the historical processing result cache library; The target historical processing result is reused as the processing result corresponding to the data to be processed.

4. The data processing method as described in claim 2, characterized in that, The metadata information includes at least one of text data, image resource locators, and video resource locators. Generating the first fingerprint information of the data to be processed based on the metadata information includes: The text data, the image resource locator, and the video resource locator are combined to obtain the combined target text. The first fingerprint information of the data to be processed is generated based on the combined target text.

5. The data processing method as described in claim 1, characterized in that, The process of performing multi-level deduplication on the data to be processed to obtain the duplicate determination result includes: Obtain all image data corresponding to the data to be processed; For each image in the total image data, determine the corresponding second fingerprint information for that image data; Based on the second fingerprint information, a second-level deduplication process is performed on the data to be processed to obtain the second-level deduplication result.

6. The data processing method as described in claim 5, characterized in that, Determining the second fingerprint information corresponding to the image data includes: Feature extraction is performed on the image data to obtain the hash fingerprint corresponding to the image data; Based on the hash fingerprint, the corresponding second fingerprint information of the image data is determined.

7. A data processing apparatus, characterized in that, The device includes: The acquisition unit is used to acquire data to be processed. The deduplication unit is used to perform multi-level deduplication on the data to be processed to obtain a duplicate determination result. The multi-level deduplication includes: performing multi-level deduplication processing based on different dimensional feature information of the data to be processed in sequence. When any level of deduplication processing determines that the data to be processed is duplicate data, the subsequent deduplication processing at each level is terminated. The determining unit is used to determine the processing method corresponding to the data to be processed based on the duplicate determination result and the historical processing result cache library.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the data processing method as described in any one of claims 1-6.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program, and the processor executing the data processing method according to any one of claims 1-6 by calling the computer program stored in the memory.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the data processing method according to any one of claims 1-6.

Images