Resource object processing method, device and computer equipment

Abstract

The present application proposes a resource object processing method, apparatus and computer equipment. The present application performs different levels of bucketing processing on a large number of historical resource objects in a resource library based on a multi-layer nested bucketing mechanism. In this way, after the computer equipment obtains the first resource object, it can determine the first resource length and the first encoding parameter of the first resource object according to the bucketing dimension based on the multi-layer bucketing mechanism, and preliminarily reduce the similarity query scope of the historical resource objects in the resource library through the first resource length, that is, determine the historical resource objects contained in the target resource length bucket, and then further reduce the query scope of similar resource objects based on the comparison result of the first encoding parameter and the index information of each resource object sub-bucket contained in the target resource length bucket, thereby quickly and accurately obtaining the similarity query result of the first resource object, which greatly improves the resource object processing efficiency compared with directly performing similarity queries on a large number of historical resource objects in the resource library.

Description

Technical Field

[0001] The present application mainly relates to the field of data processing technology, and more specifically to a resource object processing method, device and computer equipment. Background Art

[0002] In the era of big data, in order to improve data query efficiency, when obtaining resource objects such as articles and images and storing them in a resource library, it is usually necessary to first check whether the corresponding similar resource objects have been stored in the resource library to avoid repeated storage of resource objects, resulting in a waste of storage resources and an adverse impact on data query efficiency.

[0003] Among them, in order to realize the detection of similar resource objects, similarity algorithms such as Hamming distance are usually used at present, and the obtained resource objects are calculated pairwise with each historical resource object stored in the resource library to determine the similar resource objects of the obtained resource object. This is time-consuming and labor-intensive, greatly reduces the processing efficiency, and cannot meet the real-time requirements. Summary of the invention

[0004] In view of this, the present invention provides a resource object processing method, the method comprising:

[0005] Obtaining a first resource object, and determining a first resource length and a first encoding parameter of the first resource object;

[0006] Determine a target resource length bucket corresponding to the first resource length from a plurality of resource length buckets in a resource library; each of the plurality of resource length buckets contains a historical resource object of a corresponding resource length;

[0007] Compare the first encoding parameter with the index information of each resource object sub-bucket in the target resource length bucket; the resource object sub-bucket is obtained based on the secondary bucket processing corresponding to the dimension to which the first encoding parameter belongs;

[0008] Based on the obtained first comparison result, a similarity query result between the first resource object and the historical resource object is obtained.

[0009] Optionally, the method for obtaining the resource length bucket and the resource object sub-bucket includes:

[0010] Determining a plurality of resource length bucketing points according to the resource length of each historical resource object in the resource library;

[0011] According to the multiple resource length bucketing points, the historical resource objects are bucketed to obtain different resource length buckets containing historical resource objects of corresponding resource lengths;

[0012] According to the encoding parameters of the historical resource objects in at least one dimension, the historical resource objects contained in each resource length bucket are subjected to secondary bucketing processing to obtain a plurality of resource object sub-buckets in the corresponding dimension under the resource length bucket, and index information of each of the plurality of resource object sub-buckets;

[0013] The encoding parameters of the corresponding dimensions of at least part of the contents of the historical resource objects contained in the same resource object sub-bucket are the same or have a similarity reaching a first similarity threshold or are located in the same parameter range.

[0014] Optionally, determining a plurality of resource length bucketing points according to the resource length of each historical resource object in the resource library includes:

[0015] Obtaining the resource length of each historical resource object in the resource library;

[0016] Performing normal distribution statistics on a plurality of resource lengths to obtain normal distribution quantiles of the resource lengths;

[0017] Using the normal distribution quantile, multiple resource length bucketing points are determined.

[0018] Optionally, the first encoding parameter includes a first title encoding parameter, and comparing the first encoding parameter with the index information of each resource object sub-bucket in the target resource length bucket, and obtaining a similarity query result between the first resource object and the historical resource object based on the first comparison result, includes:

[0019] Compare the first title encoding parameter with the title encoding parameters of each resource object sub-bucket in the target resource length bucket for similarity;

[0020] The resource object sub-bucket corresponding to the first similarity reaching the first similarity threshold is determined as the target resource object sub-bucket, and the historical resource objects contained in the target resource object sub-bucket are determined as similar resource objects of the first resource object.

[0021] Optionally, performing secondary bucketing processing on the historical resource objects contained in each resource length bucket according to at least one dimension parameter of the historical resource object to obtain multiple resource object sub-buckets under the corresponding dimension under the resource length bucket, and index information of each of the multiple resource object sub-buckets, including:

[0022] Obtaining the title encoding parameter of the historical resource object;

[0023] Based on the title encoding parameter, performing similarity comparison on the historical resource objects in the same resource length bucket;

[0024] Based on the obtained first similarity comparison result, allocating the historical resource objects in the resource length bucket to corresponding resource object sub-buckets;

[0025] Based on the title encoding parameter, the index information of the corresponding resource object sub-bucket is determined.

[0026] Optionally, the first encoding parameter includes a first hash encoding parameter, and comparing the first encoding parameter with the index information of each resource object sub-bucket in the target resource length bucket, and obtaining a similarity query result between the first resource object and the historical resource object based on the first comparison result, includes:

[0027] Based on the first hash coding parameter, segmenting the first resource object to obtain first segmented hash codes of different first resource segment objects;

[0028] Determine, from a plurality of resource object sub-buckets under the target resource length bucket, a target resource object sub-bucket that matches the first segment hash code;

[0029] Compare the first hash coding parameter with each hash coding parameter included in the index information of the target resource object sub-bucket for similarity;

[0030] The historical resource object whose second similarity reaches the second similarity threshold is determined as a similar resource object of the first resource object.

[0031] Optionally, performing secondary bucketing processing on the historical resource objects contained in each resource length bucket according to at least one dimension parameter of the historical resource object to obtain multiple resource object sub-buckets under the corresponding dimension under the resource length bucket, and index information of each of the multiple resource object sub-buckets, including:

[0032] Obtaining hash code parameters of the historical resource object;

[0033] Based on the hash coding parameter, the historical resource objects in the same resource length bucket are segmented and bucketed to obtain multiple resource object sub-buckets;

[0034] The segmented hash codes of each resource segment object in the resource object sub-bucket and the hash code parameters of the historical resource objects to which each resource segment object belongs are determined as index information of the resource object sub-bucket.

[0035] Optionally, if there are multiple resource object sub-buckets corresponding to multiple dimensions under the same resource length bucket, the query priorities of the multiple dimensions are obtained;

[0036] The comparing the first encoding parameter with the index information of each resource object sub-bucket in the target resource length bucket includes:

[0037] In descending order of the query priority, the first encoding parameter under the higher query priority is compared with the index information of each resource object sub-bucket in the target resource length bucket for similarity;

[0038] Determine that the third similarities between the encoding parameters of each resource object sub-bucket and the first encoding parameter do not reach a third similarity threshold, obtain the second encoding parameter corresponding to the next query priority of the first resource object, and continue to compare the similarity with the index information of each resource object sub-bucket.

[0039] The present application proposes a resource object processing device, the device comprising:

[0040] A first data acquisition module, used to obtain a first resource object and determine a first resource length and a first encoding parameter of the first resource object;

[0041] A target resource length bucket determination module, used to determine a target resource length bucket corresponding to the first resource length from a plurality of resource length buckets in a resource library; each of the plurality of resource length buckets contains a historical resource object of a corresponding resource length;

[0042] A comparison module, configured to compare the first encoding parameter with index information of each resource object sub-bucket in the target resource length bucket; the resource object sub-bucket is obtained based on a secondary bucketing processing method corresponding to the dimension to which the first encoding parameter belongs;

[0043] The similarity query result obtaining module is used to obtain similarity query results between the first resource object and the historical resource object based on the obtained first comparison result.

[0044] The present application proposes a computer device, the computer device comprising at least one communication interface, at least one memory and at least one processor, wherein:

[0045] The memory is used to store a program for implementing the resource object processing method as described above;

[0046] The processor is used to load and execute the program stored in the memory to implement the resource object processing method as described above.

[0047] It can be seen that the present application provides a resource object processing method, device and computer equipment. The present application performs different levels of bucket processing on a large number of historical resource objects in a resource library based on a multi-layer nested bucketing mechanism. In this way, after the computer device obtains the first resource object, it can determine the first resource length and the first encoding parameter of the first resource object according to the bucketing dimension based on the multi-layer bucketing mechanism, and preliminarily reduce the similarity query range of the historical resource objects in the resource library through the first resource length, that is, determine the historical resource objects contained in the target resource length bucket, and then further reduce the query range of similar resource objects based on the comparison result of the first encoding parameter and the index information of each resource object sub-bucket contained in the target resource length bucket. Based on this, the similarity query result of the first resource object is obtained quickly and accurately, which greatly improves the resource object processing efficiency compared to directly performing similarity queries on a large number of historical resource objects in the resource library. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0049] Figure 1 A schematic diagram of the hardware structure of an optional example of a computer device applicable to the resource object processing method proposed in this application;

[0050] Figure 2 A schematic diagram of a system architecture applicable to the resource object processing method proposed in this application;

[0051] Figure 3 A flowchart of an optional example of the resource object processing method proposed in this application;

[0052] Figure 4 A flowchart diagram of another optional example of the resource object processing method proposed in this application;

[0053] Figure 5 This is a schematic diagram of a multi-layer nested bucketing method for historical resource objects in a resource library in the resource object processing method proposed in this application;

[0054] Figure 6 A flowchart diagram of another optional example of the resource object processing method proposed in this application;

[0055] Figure 7 A flowchart diagram of another optional example of the resource object processing method proposed in this application;

[0056] Figure 8 A flowchart diagram of another optional example of the resource object processing method proposed in this application;

[0057] Fig. 9 A flowchart diagram of another optional example of the resource object processing method proposed in this application;

[0058] Fig.10 A schematic diagram of the structure of an optional example of the resource object processing device proposed in this application;

[0059] Fig.11 This is a structural diagram of another optional example of the resource object processing device proposed in this application. DETAILED DESCRIPTION

[0060] With reference to the description in the background technology section, for the first resource object to be written into a resource library containing a large number of resource objects, in order to solve the technical problem of excessive computational complexity of pairwise similarity queries, it is proposed to use the title of the resource object as an index, and have an external search engine perform a similarity query on the first resource object based on the index, so as to quickly cluster similar resource objects that are consistent with the title of the first resource object. However, this similarity query method has certain limitations, and the reliability and comprehensiveness of the similarity query results are low, which may not meet the application requirements based on the similarity query results.

[0061] In order to further improve the efficiency and reliability of similarity queries of resource objects based on resource libraries, the present application proposes a bucketing mechanism, which pre-buckets a large number of historical resource objects based on the resource length of the historical resource objects currently contained in the resource library, and then performs secondary bucketing on the historical resource objects in each resource length bucket according to one or more dimensions to determine the multiple resource object sub-buckets contained in each resource length bucket. In this way, for any first resource object to be stored in the library, after determining its first resource length and the first encoding parameter of the corresponding dimension, the first encoding information can be compared with the index information of each resource object sub-bucket contained in the target resource length bucket corresponding to the first resource object length, so as to perform a similarity query between the first resource object and the historical resource objects in one or more resource object sub-buckets based on the first comparison result obtained. Compared with performing a similarity query between the first resource object and all historical resource objects in the resource library, this processing method of the present application greatly improves the efficiency of similarity queries and ensures the reliability and accuracy of similarity query results.

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] Reference Figure 1 , is a schematic diagram of the hardware structure of an optional example of a computer device applicable to the resource object processing method proposed in this application. The computer device may be a server, such as an independent physical server, a server cluster consisting of multiple physical servers, or a cloud server capable of cloud computing. In some other embodiments, the computer device may also be a terminal with certain data processing capabilities, such as a desktop computer, a robot, smart transportation equipment, smart medical equipment, smart home equipment, a vehicle-mounted terminal, etc. The product type of the computer device may be determined according to the application scenario of the resource object.

[0064] In practical applications, the computer device can be a stand-alone device, such as Figure 2 As shown, after the computer device obtains any first resource object that needs to be written into the resource library, it can execute the resource processing method proposed in this application to determine whether to send the first resource object to the resource library for storage. Among them, the first resource object can be a resource object directly input into the computer device, or it can be a resource object from a third-party device (such as various resource servers, terminals, storage devices, etc.). In some other embodiments, the above-mentioned computer device can also be integrated into the above-mentioned resource library, that is, the computer device and the resource library are integrated into the same device, which can be determined according to the application scenario requirements. This application takes the computer device as a server as an example to illustrate the composition structure of the computer device, such as Figure 1 As shown, the computer device may include: at least one communication interface 11 , at least one memory 12 , and at least one processor 13 .

[0065] Among them, the communication interface 11, memory 12 and processor 13 can be connected to a communication bus, through which data communication between each other can be achieved, and data interaction between other structural components of the computer device can also be achieved. The specific details can be determined according to actual needs and will not be described in detail in this application.

[0066] The communication interface 11 may include a data interface for realizing data interaction with an external device by wireless and / or wired means, such as a communication interface of a communication module such as a WIFI module, a 5G / 6G (fifth generation mobile communication network / sixth generation mobile communication network) module, a GPRS module, a near field communication module, etc. that can realize data interaction using a wireless communication network; a communication interface such as a USB interface and a multimedia interface that support wired connection; of course, the communication interface 11 may also include a data interface such as a serial port / parallel port that realizes data interaction between components inside a computer device, etc. This application does not limit the type and quantity of the communication interface 11 included in the computer device, which may be determined as appropriate.

[0067] The memory 12 can be used to store a program for implementing the resource object processing method described in each method embodiment below; the processor 13 can load and execute the program stored in the memory 12 to implement the various steps of the resource object processing method described in the corresponding method embodiment below. The specific implementation process can refer to the description of the corresponding part of the method embodiment below and will not be described in detail here.

[0068] In the embodiment of the present application, the memory 12 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device. The processor 13 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, etc. The present application does not limit the structure and model of the above-mentioned memory 12 and processor 13, and can be flexibly adjusted according to actual needs.

[0069] It should be understood that Figure 1 The structure of the computer device shown does not constitute a limitation on the computer device in the embodiment of the present application. In practical applications, the computer device may include Figure 1 More components as shown, or some components may be combined. For example, when the computer device is any of the above-mentioned terminal devices, the computer device may also include various input devices, various output devices, various sensors, power management modules, antennas, etc.; of course, when the computer device is a server, it may also include monitoring devices, databases, etc., which are not listed one by one in this application.

[0070] Reference Figure 3 , is a flowchart of an optional example of a resource object processing method proposed in this application, which can be executed by a computer device, such as Figure 3 As shown, the method may include:

[0071] Step S31, obtaining a first resource object, and determining a first resource length and a first encoding parameter of the first resource object;

[0072] In an embodiment of the present application, the first resource object may be any resource object to be written into the corresponding resource library, so as to avoid storing duplicate resource objects and wasting storage resources; of course, the first resource object may also be any resource object in the corresponding resource library, so as to achieve deduplication of similar resource objects in the resource library, etc. The source of the first resource object may be determined based on the resource object processing scenario requirements. The present application does not impose any restrictions on the source and type of the first resource object.

[0073] In combination with the above description of the technical solution of the present application, since the present application adopts a multi-layer nested bucketing mechanism, a large number of historical resource objects in the resource library are bucketed in advance, that is, bucketing is first performed according to the length of the resource object to obtain multiple resource length buckets, and then the historical resource objects contained in each resource length bucket are bucketed again according to one or more dimensions, so as to reduce the amount of data for resource object similarity calculation and comparison through the bucketing filtering mechanism, thereby improving processing efficiency and accuracy.

[0074] Therefore, after the computer device obtains the first resource object, it can calculate the resource length of the first resource object and the encoding parameters of the corresponding dimension according to the preset multi-layer nested bucketing mechanism to obtain the first resource length and the first encoding parameter. Exemplarily, if the secondary bucketing includes implementation according to the dimension of title, the first encoding parameter may include the first title encoding parameter of the first resource object; if the secondary bucketing includes segmented bucketing, the first encoding parameter may include the first hash encoding parameter after encoding and segmenting the first resource object according to the segmented hash algorithm based on the segmented bucketing, etc.; for other dimensions based on the secondary bucketing (such as the release time of the resource object, etc.), the first resource object can be calculated according to the encoding method of the corresponding dimension to obtain the encoding parameter of the corresponding dimension. The implementation process is not listed one by one in this application.

[0075] Step S32, determining a target resource length bucket corresponding to the first resource length from a plurality of resource length buckets in the resource library;

[0076] Following the above analysis, this application uses a multi-layer nested bucketing mechanism to process a large number of historical resource objects in the resource library. The multiple resource length buckets obtained by the first bucketing process can each contain historical resource objects of corresponding resource lengths. Among them, as the historical resource objects contained in the resource library change, the resource length based on which the bucketing process is based may change accordingly. In other words, this application dynamically adjusts according to the resource length distribution of historical resource objects in the resource library, so that the historical resource objects contained in the resource length bucket obtained by a bucketing can be as balanced as possible. This application does not elaborate on the implementation process of dynamic resource length bucketing.

[0077] Exemplarily, the resource length distribution of historical resource objects in the resource library can obey the normal distribution. Therefore, this embodiment can determine the bucket points of different resource length buckets, that is, the resource length thresholds, by calculating the normal distribution quantiles. The resource length range composed of two adjacent resource length thresholds can compare the resource length of each historical resource object with each resource length threshold, and determine the historical resource objects belonging to the same resource length range as historical resource objects of a resource length bucket. This application does not elaborate on the calculation implementation method of the normal distribution quantile, and does not limit the numerical value of the resource length threshold corresponding to each quantile, which can be determined according to the situation.

[0078] Based on this, the present application can compare the first resource length of the first resource object with the resource length range corresponding to each resource length bucket, determine the resource length range to which the first resource length belongs, and determine the resource length bucket corresponding to the resource length range as the target resource length bucket, that is, the resource length bucket containing similar resource objects of the first resource object, so as to narrow the similarity comparison range of the resource objects, and subsequently perform further similarity filtering processing on the historical resource objects contained therein.

[0079] Step S33, comparing the first encoding parameter with the index information of each resource object sub-bucket in the target resource length bucket;

[0080] In the embodiment of the present application, the resource object sub-bucket can be obtained based on the secondary bucket processing method corresponding to the dimension to which the first coding parameter belongs, that is, according to the dimension to which the first coding parameter belongs, the historical resource objects contained in the target resource length bucket are secondary bucket processed to obtain multiple resource object sub-buckets. For example, according to the secondary bucket processing implemented by the title coding, this embodiment can divide multiple historical resource objects in the same resource length bucket into historical resource objects belonging to different titles, and determine the historical resource objects of the same title as a resource object sub-bucket. The implementation process of the secondary bucket processing in other dimensions is similar, and this embodiment will not be described in detail here.

[0081] Based on this, in order to facilitate the similarity comparison of each resource object in the resource object sub-bucket, the present application can use the encoding information of the corresponding dimension of each resource object to constitute the index information of the resource object sub-bucket. In this way, the computer device can compare the first encoding parameter of the first resource object in the corresponding dimension with the index information of each resource object sub-bucket in the target resource length bucket, in order to further determine similar resource objects among the resource objects contained in each resource object sub-bucket that are similar to the first resource object.

[0082] It should be noted that the present application does not limit the comparison method of coding parameters of different dimensions. It can be a one-to-one comparison of the contents of the coding parameters, or a similarity algorithm can be used to achieve the similarity between the resource objects represented by different coding parameters, etc. The embodiments of the present application are not described in detail here.

[0083] Step S34: obtaining similarity query results between the first resource object and the historical resource object based on the obtained first comparison result.

[0084] According to the relevant description of the implementation steps above, the first comparison result can characterize the similarity between the resource objects represented by the two encoding parameters involved in the comparison. Therefore, by analyzing the multiple first comparison results obtained, the similarity query range of the historical resource objects can be further shortened for the target resource length bucket initially screened out, and the resource object sub-bucket where the historical resource objects that may be similar to the first resource object are located can be determined. After that, the similarity algorithm can be used to calculate the similarity between the first resource object and each historical resource object contained in the screened resource object sub-bucket, so as to accurately determine the similar resource objects of the first resource object.

[0085] It can be seen that the present application performs different levels of bucketing processing on a large number of historical resource objects in the resource library based on a multi-layer nested bucketing mechanism. In this way, after the computer device obtains the first resource object, it can determine the first resource length and the first encoding parameter of the first resource object according to the bucketing dimension based on the multi-layer bucketing mechanism, and preliminarily reduce the similarity query scope of the historical resource objects in the resource library through the first resource length, that is, determine the historical resource objects contained in the target resource length bucket, and then further reduce the query scope of similar resource objects based on the comparison result of the first encoding parameter and the index information of each resource object sub-bucket contained in the target resource length bucket, thereby quickly and accurately obtaining the similarity query result of the first resource object, which greatly improves the resource object processing efficiency compared to directly performing similarity queries on a large number of historical resource objects in the resource library.

[0086] Reference Figure 4 , which is a flowchart of another optional example of the resource object processing method proposed in the present application. This embodiment can be a detailed description of the implementation process of the multi-layer nested bucketing mechanism for bucketing the historical resource objects contained in the resource library proposed in the above embodiment, and obtaining multiple resource length buckets, and each resource length bucket can include a plurality of resource object sub-buckets corresponding to the secondary bucketing according to at least one dimension, but is not limited to the implementation method of the multi-layer nested bucketing mechanism described in this embodiment, and the bucketing implementation method can still be executed by a computer device, such as Figure 4 As shown, the method may include:

[0087] Step S41, obtaining the resource length of each historical resource object in the resource library;

[0088] Step S42, performing normal distribution statistics on multiple resource lengths to obtain normal distribution quantiles of the resource lengths;

[0089] Step S43, using the normal distribution quantiles to determine multiple resource length bucketing points;

[0090] The similarity between resource objects can be similar in content or in other dimensional features of the resource objects, such as similar resource lengths, the same titles, similar release times, etc. Compared with directly calculating the similarity of resource object contents, the resources spent on similarity calculations of other dimensional features of resource objects are much less. Therefore, in order to improve the efficiency of similarity queries, the present application can determine multiple resource length bucketing points based on the resource length of each historical resource object in the resource library, and realize first-level bucketing processing of a large number of historical resource objects.

[0091] In order to ensure the accuracy of the first-level bucketing results of historical resource objects, this application adopts a dynamic resource length bucketing method to determine the resource length bucketing points of a large number of historical resource objects contained in the current resource library, so that the number of historical resource objects contained in multiple resource length buckets obtained by bucketing is basically the same. For the resource length bucketing point, the normal distribution quantile calculation method described in the above steps can be used to implement it. The implementation process is not described in detail, and it is not limited to the implementation method described in this embodiment.

[0092] Step S44, bucketing each historical resource object in the resource library according to multiple resource length bucketing points to obtain different resource length buckets containing historical resource objects of corresponding resource lengths;

[0093] Following the above description, after determining each historical resource object in the current resource library and determining the resource length bucket point according to the resource length balance grouping, multiple continuous resource length ranges can be obtained. i Indicates different resource lengths. The obtained resource length range can be: l1-l2, l2-l3, l3-l4, ..., l n-1 -l n ,like Figure 5 As shown, the historical resource objects in the resource library represented by the entire box can be divided into multiple resource length buckets within the corresponding resource length range. In this way, the number of historical resource objects contained in each resource length bucket will be greatly reduced relative to the number of historical resource objects contained in the entire resource library.

[0094] In some embodiments, in the process of performing a similarity query on a first resource object for a historical resource object, the first resource length of the first resource object can be first compared with the resource length range corresponding to each pre-stored resource length bucket, and after determining the target resource length bucket corresponding to the resource length range to which the first resource length belongs, the first resource object can also be directly similarly calculated with each historical resource object contained in the target resource length bucket to determine similar historical resource objects of the first resource object, that is, to obtain a similarity query result for the first resource object. This processing method can also significantly improve processing efficiency compared to a processing method in which the first resource object is similarly calculated with a large number of historical resource objects contained in a resource library, and ensures the accuracy of similarity query results compared to a similarity clustering processing method directly based on the same title.

[0095] Step S45, according to the encoding parameters of the historical resource objects in at least one dimension, the historical resource objects contained in each resource length bucket are subjected to secondary bucketing processing to obtain multiple resource object sub-buckets in the corresponding dimension under the resource length bucket, and the index information of each of the multiple resource object sub-buckets.

[0096] In order to further reduce the scope of similarity queries and improve the efficiency of similarity queries, the embodiment of the present application will perform secondary bucketing processing on the historical resource objects in each resource length bucket according to other dimensional features of the historical resource objects, and determine the multiple resource object sub-buckets contained in each resource length bucket. Among them, the secondary bucketing processing of historical resource objects in different resource length buckets can be implemented according to the same dimension or according to different dimensions, and this application does not limit this.

[0097] In some embodiments, during the above-mentioned secondary bucketing process, if the dimension based on it includes the title of the resource object, such as the title of an article, etc., for each historical resource object contained in a resource length bucket, the title of the historical resource object can be determined, and the title of each historical resource object can be encoded to obtain the corresponding title encoding parameters, thereby constituting the index information of the resource length bucket, and the historical resource objects contained in the resource length bucket can be further bucketed accordingly.

[0098] Among them, regarding the implementation method of resource object title encoding processing, the MD5 message digest algorithm (Message-Digest Algorithm) can be used, that is, a cryptographic hash function to process the title of the historical resource object contained in each resource length bucket, and obtain a fixed-length hash value (i.e., hash value) corresponding to the historical resource object title, such as a 128-bit (16-byte) hash value, which is used as the title encoding parameter of the corresponding historical resource object, but is not limited to the resource object title encoding processing implementation method described in this embodiment.

[0099] For example, Figure 5 As shown, for the historical resource objects within the resource length range corresponding to any resource length bucket, the titles they have are often not the same, so that multiple historical resource objects contained in the resource length bucket may apply multiple title encoding parameters, such as title MD5 encoding, and each title MD5 encoding may correspond to one or more historical resource objects. Optionally, in order to reduce the number of similarity comparisons, when performing secondary bucketing based on the title encoding parameters, the titles can be classified, and historical resource objects with the same type of titles can be divided into a resource object sub-bucket. The implementation process is not described in detail in this application.

[0100] From the above analysis, it can be seen that the encoding parameters of the corresponding dimensions of at least part of the content of each historical resource object contained in the same resource object sub-bucket are the same or the similarity reaches the first similarity threshold or is in the same parameter interval. Therefore, when performing secondary bucketing based on the title encoding parameter, the similarity between the title encoding parameters of each historical resource object contained in the same resource object sub-bucket reaches the corresponding first similarity threshold or is in the same parameter interval, and belongs to the same type of title. Of course, when performing secondary bucketing based on the title encoding parameter, each title encoding parameter can also be used as a resource object sub-bucket, so that each resource object sub-bucket corresponds to a title encoding parameter, and corresponds to one or more historical resource objects with the title encoding parameter.

[0101] Based on the above analysis, if Figure 6 As shown, it is a flowchart of another optional example of the resource object processing method proposed in this application. This embodiment can describe an optional refinement of the storage process of a large number of historical resource objects contained in the resource library based on the multi-layer nested bucketing mechanism described above. This embodiment can describe the secondary bucketing process of the historical resource objects contained in each resource length bucket. Regarding the implementation process of dynamic bucketing of resource length, please refer to the description of the corresponding part of the above embodiment, which will not be described in detail in this embodiment. Figure 6 As shown, the secondary bucket processing method for historical resource objects proposed in this embodiment may include but is not limited to:

[0102] Step S61, obtaining the title encoding parameter of the historical resource object;

[0103] Step S62, based on the title encoding parameter, performing similarity comparison on the historical resource objects in the same resource length bucket;

[0104] Step S63, based on the obtained first similarity comparison result, allocating the historical resource objects in the resource length bucket to corresponding resource object sub-buckets;

[0105] Step S64: Determine the index information of the corresponding resource object sub-bucket based on the title encoding parameter.

[0106] Combined with the above description of bucketing based on title encoding parameters, for the historical resource objects contained in each resource length bucket, after obtaining the corresponding title encoding parameters, the historical resource objects corresponding to each title encoding parameter can be divided into a resource object sub-bucket, and the corresponding relationship between the title encoding parameters and the historical resource objects constitutes the index information of the resource object sub-bucket.

[0107] Optionally, as analyzed above, a group of historical resource objects with title similarity in the same resource length bucket can be divided into a resource object sub-bucket. Similarly, the correspondence between the title encoding parameters of similar titles and the historical resource objects constitutes the index information of the resource object sub-bucket. The implementation process is not described in detail in the embodiment of this application.

[0108] In some other embodiments, the present application may adopt a segmentation and bucketing mechanism to perform secondary bucketing processing on the historical resource objects contained in each resource length bucket. Therefore, for the historical resource objects contained in each resource length bucket, encoding processing is performed according to a preset encoding method to obtain the hash coding parameters of the historical resource object. Then, the hash coding parameters are divided into n+1 parts in order, that is, the historical resource object is divided into n+1 segments, and the segmented hash codes corresponding to the content of the n+1 segments of the resource object are stored in n+1 resource object sub-buckets respectively. Then, the inverted index method can be used to determine the index information of each resource object sub-bucket. For example, the key-value method is used to construct the index information. The corresponding segmented hash code in each resource object sub-bucket can be used as the key, and the hash coding parameter of the historical resource object to which it belongs can be used as the corresponding value, but it is not limited to this implementation method.

[0109] Exemplarily, the present application may use the simhash algorithm to implement segmentation and bucketing. According to the drawer principle, it is assumed that the Hamming distance threshold for determining whether resource objects are similar based on the Hamming distance similarity algorithm is n (i.e., a positive integer greater than 1, and the present application does not impose any restriction on its value). For each historical resource object in any resource length bucket, after obtaining the corresponding simhash encoding parameter (i.e., the above-mentioned hash encoding parameter, which may be a 64-bit binary encoding value), the 64-bit simhash of each historical resource object may be divided into n+1 parts in order and evenly, and the segmented hash code corresponding to each historical resource object segment is obtained, such as Figure 5 Shown are simhash segment 1, simhash segment 2, simhash segment 3, ..., simhash segment n, and simhash segment n+1.

[0110] Combined with the above description of the method for obtaining the index information of a resource object sub-bucket, the content of the index information of any resource object sub-bucket is as follows: Figure 5 As shown in the figure, after the historical resource objects in the same resource length bucket are processed according to the above simhash segmentation and bucketing, the segmented hash code of the corresponding segment in the complete simhash of the historical resource objects belonging to the same resource object sub-bucket, such as 101001010, 100001010, 001010101, etc., is used as the key of the key value index, and the complete simhash corresponding to each historical resource object with the key (referred to as s i , such as s1, s2, s3, s4, s5...) as the value of the key, thereby constituting the index information of the resource object sub-bucket in key-value mode, but is not limited to Figure 5 The index information is shown.

[0111] Based on the above analysis, if Figure 7 The flowchart of another optional example of the resource object processing method proposed in the present application is shown. This embodiment can describe another optional detailed implementation method of the secondary bucket processing process for the historical resource objects contained in each resource length bucket. Combined with the description of the corresponding part of the above embodiment, as shown in FIG. Figure 7 As shown, the detailed implementation method may include but is not limited to:

[0112] Step S71, obtaining hash code parameters of the historical resource object;

[0113] Step S72, based on the hash coding parameters, segment and bucket the historical resource objects in the same resource length bucket to obtain multiple resource object sub-buckets;

[0114] Step S73: determine the segmented hash code of each resource segment object in the resource object sub-bucket and the hash code parameter of the historical resource object to which each resource segment object belongs as the index information of the resource object sub-bucket.

[0115] In an embodiment of the present application, the simhash algorithm (or other similar algorithms, which are not limited in the present application) can be used to process each historical resource object to obtain the hash coding parameters of the entire historical resource object, that is, the simhash value of the entire historical resource object. Afterwards, based on the simhash segmentation and bucketing mechanism, the historical resource objects in each resource length bucket can be secondary bucketed to obtain resource object sub-buckets corresponding to different segmentation coding parameters. The key value composed of the segmentation hash code and the hash coding parameters of the entire historical resource object is determined as the index information of the corresponding resource object sub-bucket. The implementation process can refer to the description of the corresponding part above, and this embodiment will not be repeated here.

[0116] In some other embodiments proposed in the present application, since resource objects with similar release times have a higher probability of being similar, in the bucket storage process of historical resource objects, each value contained in the index information of a resource object sub-bucket determined above can be stored in reverse order according to the release time of the corresponding historical resource object, that is, the earlier the release time, the later the ranking in the index information, thereby further reducing the number of similar queries, but it is not limited to the index information acquisition method described in this embodiment.

[0117] It can be understood that for different types of resource objects, this application can also implement secondary bucketing based on other dimensions of historical resource objects, such as the type of image / video, publisher, etc. The implementation process can refer to the secondary bucketing implementation process based on title encoding above, and this application does not give detailed examples one by one. In addition, in actual applications, for the secondary bucketing of the same resource length bucket, the secondary bucketing can be implemented based on one dimension to obtain multiple resource object sub-buckets under the dimension; the secondary bucketing can also be implemented based on multiple dimensions to obtain multiple resource object sub-buckets corresponding to each of the multiple dimensions, and even the secondary bucketing can be implemented based on a combination of multiple dimensions to obtain multiple resource object sub-buckets, etc. It can be determined according to the actual needs of the application scenario, and this application will not be described in detail here.

[0118] Reference Figure 8 , which is a flowchart of another optional example of the resource object processing method proposed in this application. This embodiment can be a description of an optional detailed implementation method of the resource object processing method proposed in the above embodiment, but is not limited to the detailed implementation method described in this embodiment. Figure 8 As shown, the refinement implementation method may include:

[0119] Step S81, obtaining a first resource object, and determining a first resource length and a first title encoding parameter of the first resource object;

[0120] Step S82, determining a target resource length bucket corresponding to the first resource length from a plurality of resource length buckets in the resource library;

[0121] Regarding the implementation process of step S81 and step S82, reference may be made to the description of the corresponding parts of the above embodiment, which will not be elaborated in this embodiment.

[0122] In an embodiment of the present application, for the newly obtained first resource object, after determining the first resource length l, the first resource length l is compared with the resource length range corresponding to each resource length bucket ((1-a)l, (1+a)l), where a is a threshold parameter, which can be determined based on the dynamic adjustment of the resource length, such as based on the above-mentioned normal distribution quantile, and the first resource length l is compared with each resource length range to determine the target resource length bucket, thereby reducing the similar query range relative to the entire resource library.

[0123] Step S83, comparing the first title encoding parameter with the title encoding parameters of each resource object sub-bucket in the target resource length bucket for similarity;

[0124] Step S84, determining the resource object sub-bucket corresponding to the first similarity reaching the first similarity threshold as the target resource object sub-bucket;

[0125] Step S85: Determine similar resource objects to the first resource object from the historical resource objects contained in the target resource object sub-bucket.

[0126] In order to further query similar historical resource objects of the first resource object from the multiple historical resource objects contained in the target resource length bucket that narrows the similarity query scope, the computer device can compare the first title encoding parameter of the first resource object with the title encoding parameters of the multiple historical resource objects contained in the target resource length bucket. If it is determined that there is a historical resource object with the same title encoding parameter as the first title encoding parameter in the target resource length bucket, the historical resource object can be determined as a similar resource object of the first resource object, that is, a target resource object sub-bucket is determined in which the first similarity between the first title encoding parameter and the title encoding parameters of each resource object sub-bucket in the target resource length bucket reaches 100%, and the historical resource object contained in the target resource object sub-bucket is determined as a similar resource object of the first resource object. There is no need to perform full-text hash similarity calculation on the historical resource object and the first resource object, which greatly reduces the calculation amount of similarity query and improves the efficiency of similarity query.

[0127] Based on the above analysis, it can be known that the similarity comparison of the title encoding parameters can include a direct comparison between the first title encoding parameters and the title encoding parameters of the historical resource object; in some embodiments, if the resource library is pre-processed with multi-layer nested bucketing, the historical resource objects with similar titles in the same resource length bucket are divided into a resource object sub-bucket, then, in the actual similarity query process for the first resource object, when matching the title encoding in the comparison mechanism, the first similarity between the title encoding parameters of each resource object sub-bucket and the first title encoding parameters can be obtained, and the target resource object sub-bucket whose first similarity reaches the first similarity threshold (this application does not limit this value, which can be the above 100% or other higher values, such as 90%) is determined, that is, the historical resource objects contained in the target resource object sub-bucket with higher title similarity with the first resource object in the target resource length bucket. This processing method is used to further reduce the scope of the similarity query from the first data amount of the historical resource objects contained in the target resource length bucket to the second data amount of the historical resource objects contained in the target resource object sub-bucket, and the second data amount is much smaller than the first data amount.

[0128] Afterwards, in order to improve the accuracy of similarity queries, after determining the multiple historical resource objects contained in the target resource object sub-bucket, since the number of these historical resource objects is often small, the similarity between the first resource object and the full-text hash coding parameters of these historical resource objects (such as the simhash of the entire resource object) can be calculated to obtain the corresponding second similarity, and the historical resource objects whose second similarity reaches a second similarity threshold (the value of which can be unlimited and can be determined depending on the situation) are determined as similar resource objects of the first resource object.

[0129] It can be seen that this embodiment reduces the similarity comparison range of resource objects by successively comparing the historical resource objects in the secondary buckets, thereby reducing the number of subsequent full-text hash similarity calculations of resource objects, greatly reducing the amount of hash similarity calculations, and improving processing efficiency.

[0130] In actual applications, for the secondary bucketing of multiple historical resource objects under the same resource length bucket, if it can be implemented based on multiple dimensions to obtain multiple resource object sub-buckets corresponding to different dimensions, the present application can obtain the query priority of these multiple dimensions, so that in the subsequent similarity query process for the first resource object, further similarity query can be performed on the target resource length bucket according to the query priority. The present application does not restrict the configuration method and content of the query priority, which can be determined according to the actual situation.

[0131] Based on this, the implementation process of the present application for comparing the first encoding parameter with the index information of each resource object sub-bucket in the target resource length bucket can be: in order of query priority from high to low, the first encoding parameter under the higher query priority is compared with the index information of each resource object sub-bucket in the target resource length bucket for similarity; it is determined that the third similarity between the encoding parameters of each resource object sub-bucket and the first encoding parameter does not reach the third similarity threshold, that is, according to the encoding parameters corresponding to the dimension of the higher query priority, no matching historical resource object is queried, the second encoding parameter corresponding to the next query priority of the first resource object can be obtained, and the similarity comparison with the index information of each resource object sub-bucket is continued until it is determined that the third similarity between the encoding parameters corresponding to a certain query priority and the encoding parameters of the first resource object in the corresponding dimension reaches the third similarity threshold, and the resource object sub-bucket with the encoding parameter can be determined as the target resource object sub-bucket.

[0132] The following will take the secondary bucketing of the resource object's title and segment as an example to explain. Since the amount of computation based on title code matching is less than that based on segment hash code similarity query, the query priority based on title code matching can be configured to be higher than the query priority based on segment hash code similarity query. Fig. 9, which is a flowchart of another optional example of the resource object processing method proposed in this application. This embodiment can be a description of another optional detailed implementation method of the resource object processing method proposed in the above embodiment, such as Fig. 9 As shown, the method may include:

[0133] Step S91, obtaining a first resource object, determining a first resource length, a first title encoding parameter, and a first hash encoding parameter of the first resource object;

[0134] Step S92, determining a target resource length bucket corresponding to the first resource length from a plurality of resource length buckets in the resource library;

[0135] Regarding the implementation process of step S91 and step S92, reference may be made to the description of the corresponding parts of the above embodiment, which will not be elaborated in this embodiment.

[0136] Step S93, comparing the first header encoding parameter with the header encoding parameters of each resource object sub-bucket in the target resource length bucket;

[0137] Step S94, if the comparison results are all different, segment the first resource object based on the first hash code parameter to obtain first segment hash codes of different first resource segment objects;

[0138] In an embodiment of the present application, according to the above comparison mechanism, it is determined through the title code matching method that there is no historical resource object with the same title as the first resource object. Another comparison mechanism can be used for similarity query, that is, a secondary comparison query is performed on the resource object sub-bucket obtained by the secondary bucketing mechanism based on another dimension. This embodiment is explained by taking the segmented bucketing method as an example. Therefore, in combination with the segmented bucketing principle described above, the first hash coding parameter can be segmented, such as dividing the first hash coding parameter into n+1 parts in order, that is, dividing the content of the first resource object into n+1 first resource segment objects, and obtaining n+1 corresponding first segmented hash codes. The implementation process is not described in detail in this application.

[0139] Step S95, determining a target resource object sub-bucket matching the first segment hash code from a plurality of resource object sub-buckets under the target resource length bucket;

[0140] Step S96, comparing the first hash code parameter with each hash code parameter of the index information of the target resource object sub-bucket for similarity;

[0141] Combined with the above description of the segmentation and bucketing mechanism, taking the resource object as an example, for two similar resource objects, there will usually be at least one segment (i.e., resource segment object) that is exactly the same. In this way, the segmented hash code can be used for similarity comparison to determine the target resource object sub-bucket that has the same resource segment object as the first resource object. The target resource object sub-bucket here can be one or more resource object sub-buckets, that is, in the resource object sub-buckets of different resource segment objects contained in the target resource length bucket, there may be one or more resource segment objects that are the same as the corresponding first resource segment object of the first resource object.

[0142] And because in this embodiment, the index information of the target resource object sub-bucket is composed of the segmented hash code of the corresponding resource segment object and the hash code parameters of the historical resource object having the resource segment object, the present application can compare the first segmented hash code of different segments of the first resource object with the key, i.e., the segmented hash code, of each resource object sub-bucket in the target resource length bucket, and determine the resource object sub-bucket with the same segmented hash code as the target resource object sub-bucket, or determine the resource object sub-bucket whose segmented hash code similarity reaches a third similarity threshold as the target resource object sub-bucket, that is, the resource object sub-bucket where the resource segment object of the historical resource object having one or more identical first resource segment object contents as the first resource object is located.

[0143] Afterwards, in order to further determine the similar resource objects of the first resource object in the target resource object sub-bucket corresponding to each resource segment object, the first hash coding parameter of the first resource object can be compared with each hash coding parameter in the index information of the target resource object sub-bucket, that is, the complete hash coding parameters of the first resource object and the historical resource object with the same segment content are calculated to accurately determine the similarity between the historical resource object and the first resource object, and then determine whether the historical resource object is a similar resource object of the first resource object. This application does not limit the similarity algorithm based on which the similarity calculation of two resource objects is based, including but not limited to the simhash algorithm described above.

[0144] Step S97: determine the historical resource object whose second similarity reaches the corresponding second similarity threshold as a similar resource object to the first resource object.

[0145] Following the above analysis, by calculating the content similarity between the first resource object and the historical resource objects in the sub-bucket of the target resource object, similar resource objects of the first resource object can be accurately determined. This application does not limit the value of the second similarity threshold, which can be determined based on experience, business needs, etc.

[0146] It should be noted that the method for determining the sub-bucket of the target resource object and the method for obtaining the corresponding similar resource object can be determined in combination with the principle of secondary bucketing of historical resource objects, including but not limited to the implementation method described in the above embodiment.

[0147] In some other embodiments proposed in the present application, if the present application performs secondary bucketing processing on each resource length bucket of the historical resource objects in the resource library, a segmented bucketing mechanism is adopted. After the embodiment of the present application determines the target resource length bucket of the first resource object, the first hash coding parameter of the first resource object can be directly segmented to determine the first segmented hash code corresponding to the different first resource segments contained in the first resource object. After determining the target resource object sub-bucket based on this, similar resource objects of the first resource object are determined by similarity comparison of the complete hash coding parameters of the resource object. The implementation process can refer to the description of the corresponding part of the above embodiment, and this embodiment will not be repeated here.

[0148] Reference Fig.10 , is a schematic diagram of an optional example of a resource object processing device proposed in this application, and the device may include:

[0149] A first data acquisition module 101 is used to obtain a first resource object and determine a first resource length and a first encoding parameter of the first resource object;

[0150] The target resource length bucket determination module 102 is used to determine a target resource length bucket corresponding to the first resource length from a plurality of resource length buckets in a resource library; each of the plurality of resource length buckets contains a historical resource object of a corresponding resource length;

[0151] A comparison module 103 is used to compare the first encoding parameter with the index information of each resource object sub-bucket in the target resource length bucket; the resource object sub-bucket is obtained based on the secondary bucket processing corresponding to the dimension to which the first encoding parameter belongs;

[0152] The similarity query result obtaining module 104 is used to obtain similarity query results between the first resource object and the historical resource object based on the obtained first comparison result.

[0153] In some embodiments, to obtain the resource length bucket and resource object sub-bucket, such as Fig.11 As shown, the resource object processing device may include:

[0154] A resource length bucket processing module 105 is used to determine a plurality of resource length bucket points according to the resource length of each historical resource object in the resource library;

[0155] A resource length bucket obtaining module 106, configured to perform bucketing processing on the historical resource objects according to the multiple resource length bucketing points, and obtain different resource length buckets containing historical resource objects of corresponding resource lengths;

[0156] The secondary bucketing module 107 is used to perform secondary bucketing processing on the historical resource objects contained in each resource length bucket according to the encoding parameters of the historical resource objects in at least one dimension, so as to obtain multiple resource object sub-buckets in the corresponding dimension under the resource length bucket, and index information of each of the multiple resource object sub-buckets;

[0157] The encoding parameters of the corresponding dimensions of at least part of the contents of the historical resource objects contained in the same resource object sub-bucket are the same or have a similarity reaching a first similarity threshold or are located in the same parameter range.

[0158] Optionally, the resource length bucket processing module 105 may include:

[0159] A resource length acquisition unit, used to acquire the resource length of each historical resource object in the resource library;

[0160] A normal distribution quantile obtaining unit, used for performing normal distribution statistics on a plurality of resource lengths to obtain normal distribution quantiles of the resource lengths;

[0161] The resource length bucket point determination unit is used to determine multiple resource length bucket points using the normal distribution quantile.

[0162] In some other embodiments proposed in the present application, if the first encoding parameter includes a first title encoding parameter, the comparison module 103 may include:

[0163] A similarity comparison unit, configured to compare the first title encoding parameter with the title encoding parameter of each resource object sub-bucket in the target resource length bucket for similarity;

[0164] Accordingly, the similar query result obtaining module 104 may include:

[0165] The first similar resource object determining unit is configured to determine a resource object sub-bucket corresponding to a first similarity reaching a first similarity threshold as a target resource object sub-bucket, and determine historical resource objects contained in the target resource object sub-bucket as similar resource objects to the first resource object.

[0166] Optionally, the secondary bucketing module 107 may include:

[0167] A title encoding parameter acquisition unit, used to acquire the title encoding parameter of the historical resource object;

[0168] A first similarity comparison unit, configured to perform similarity comparison on the historical resource objects in the same resource length bucket based on the title encoding parameter;

[0169] A first resource object sub-bucket determining unit, configured to allocate the historical resource objects in the resource length bucket to corresponding resource object sub-buckets based on the obtained first similarity comparison result;

[0170] The first index information determining unit is used to determine the corresponding index information of the resource object sub-bucket based on the title encoding parameter.

[0171] In some other embodiments proposed in the present application, if the first encoding parameter includes a first hash encoding parameter, the comparison module 103 may include:

[0172] A segment processing unit, configured to perform segment processing on the first resource object based on the first hash coding parameter to obtain first segment hash codes of different first resource segment objects;

[0173] a target resource object sub-bucket determining unit, configured to determine a target resource object sub-bucket matching the first segment hash code from a plurality of resource object sub-buckets under the target resource length bucket;

[0174] A second similarity comparison unit, configured to compare the first hash coding parameter with each hash coding parameter included in the index information of the target resource object sub-bucket for similarity;

[0175] Accordingly, the similar query result obtaining module 104 may include:

[0176] The second similar resource object determining unit is used to determine the historical resource object corresponding to the second similarity reaching the second similarity threshold as the similar resource object of the first resource object.

[0177] Optionally, the secondary bucketing module 107 may include:

[0178] A hash code parameter acquisition unit, used to acquire the hash code parameters of the historical resource object;

[0179] A segmentation and bucketing unit, configured to perform segmentation and bucketing processing on the historical resource objects in the same resource length bucket based on the hash coding parameter to obtain a plurality of resource object sub-buckets;

[0180] The second index information determining unit is used to determine the segmented hash code of each resource segment object in the resource object sub-bucket and the hash code parameter of the historical resource object to which each resource segment object belongs as the index information of the resource object sub-bucket.

[0181] Based on the resource object processing device described in the above embodiments, the device may further include:

[0182] A query priority acquisition module, used to acquire the query priorities of the multiple dimensions when there are multiple resource object sub-buckets corresponding to the multiple dimensions under the same resource length bucket;

[0183] Based on this, the comparison module 103 may include:

[0184] A third similarity comparison unit is used to compare the first encoding parameter under a higher query priority with the index information of each resource object sub-bucket in the target resource length bucket in a similarity manner according to the query priority from high to low;

[0185] The fourth similarity comparison unit is used to determine that the third similarities between the encoding parameters of each resource object sub-bucket and the first encoding parameter do not reach a third similarity threshold, obtain the second encoding parameter corresponding to the next query priority of the first resource object, and continue to perform similarity comparison with the index information of each resource object sub-bucket.

[0186] It should be noted that the various modules, units, etc. in the above-mentioned device embodiments can be stored in the memory as program modules, and the processor executes the above-mentioned program modules stored in the memory to implement the corresponding functions. For the functions implemented by each program module and its combination, as well as the technical effects achieved, please refer to the description of the corresponding parts of the above-mentioned method embodiments, which will not be repeated in this embodiment.

[0187] The present application also provides a computer-readable storage medium on which a computer program can be stored. The computer program can be called and loaded by a processor to implement the various steps of the resource object processing method described in the above embodiment. The implementation process of the present application embodiment will not be described in detail here.

[0188] Finally, it should be noted that, with respect to the above embodiments, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not specifically refer to the singular, but may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list, and the method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

[0189] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0190] The terms such as "first", "second", etc. used in this application are only used for descriptive purposes to distinguish one operation, unit or module from another operation, unit or module, and do not necessarily require or imply any actual relationship or order between these units, operations or modules. They cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0191] In addition, the various embodiments in this specification are described in a progressive or parallel manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the devices and computer equipment disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0192] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A resource object processing method, the method comprising: Obtaining a first resource object, and determining a first resource length and a first encoding parameter of the first resource object; Determine a target resource length bucket corresponding to the first resource length from a plurality of resource length buckets in a resource library; each of the plurality of resource length buckets contains a historical resource object of a corresponding resource length; Compare the first encoding parameter with the index information of each resource object sub-bucket in the target resource length bucket; the resource object sub-bucket is obtained based on the secondary bucket processing corresponding to the dimension to which the first encoding parameter belongs; Based on the obtained first comparison result, a similarity query result between the first resource object and the historical resource object is obtained.

2. According to the method of claim 1, the method for obtaining the resource length bucket and the resource object sub-bucket comprises: Determining a plurality of resource length bucketing points according to the resource length of each historical resource object in the resource library; According to the multiple resource length bucketing points, the historical resource objects are bucketed to obtain different resource length buckets containing historical resource objects of corresponding resource lengths; According to the encoding parameters of the historical resource objects in at least one dimension, the historical resource objects contained in each resource length bucket are subjected to secondary bucketing processing to obtain a plurality of resource object sub-buckets in the corresponding dimension under the resource length bucket, and index information of each of the plurality of resource object sub-buckets; The encoding parameters of the corresponding dimensions of at least part of the contents of the historical resource objects contained in the same resource object sub-bucket are the same or have a similarity reaching a first similarity threshold or are located in the same parameter range.

3. According to the method of claim 2, the step of determining a plurality of resource length bucketing points based on the resource length of each historical resource object in the resource library comprises: Obtaining the resource length of each historical resource object in the resource library; Performing normal distribution statistics on a plurality of resource lengths to obtain normal distribution quantiles of the resource lengths; Using the normal distribution quantile, multiple resource length bucketing points are determined.

4. The method according to claim 1, wherein the first encoding parameter comprises a first title encoding parameter, and the step of comparing the first encoding parameter with the index information of each resource object sub-bucket in the target resource length bucket, and obtaining a similarity query result between the first resource object and the historical resource object based on a first comparison result, comprises: Compare the first title encoding parameter with the title encoding parameters of each resource object sub-bucket in the target resource length bucket for similarity; The resource object sub-bucket corresponding to the first similarity reaching the first similarity threshold is determined as the target resource object sub-bucket, and the historical resource objects contained in the target resource object sub-bucket are determined as similar resource objects of the first resource object.

5. The method according to claim 2, wherein the historical resource objects contained in each resource length bucket are subjected to secondary bucketing processing according to at least one dimension parameter of the historical resource objects to obtain a plurality of resource object sub-buckets under the corresponding dimension under the resource length bucket, and respective index information of the plurality of resource object sub-buckets, including: Obtaining the title encoding parameter of the historical resource object; Based on the title encoding parameter, performing similarity comparison on the historical resource objects in the same resource length bucket; Based on the obtained first similarity comparison result, allocating the historical resource objects in the resource length bucket to corresponding resource object sub-buckets; Based on the title encoding parameter, the index information of the corresponding resource object sub-bucket is determined.

6. The method according to claim 1, wherein the first encoding parameter comprises a first hash encoding parameter, and the comparing the first encoding parameter with the index information of each resource object sub-bucket in the target resource length bucket, and obtaining a similarity query result between the first resource object and the historical resource object based on the first comparison result, comprises: Based on the first hash coding parameter, segmenting the first resource object to obtain first segmented hash codes of different first resource segment objects; Determine, from a plurality of resource object sub-buckets under the target resource length bucket, a target resource object sub-bucket that matches the first segment hash code; Compare the first hash coding parameter with each hash coding parameter included in the index information of the target resource object sub-bucket for similarity; The historical resource object whose second similarity reaches the second similarity threshold is determined as a similar resource object of the first resource object.

7. According to the method of claim 2, the historical resource objects contained in each resource length bucket are subjected to secondary bucketing processing according to at least one dimension parameter of the historical resource objects to obtain multiple resource object sub-buckets under the corresponding dimension under the resource length bucket, and the index information of each of the multiple resource object sub-buckets, including: Obtaining hash code parameters of the historical resource object; Based on the hash coding parameter, the historical resource objects in the same resource length bucket are segmented and bucketed to obtain multiple resource object sub-buckets; The segmented hash codes of each resource segment object in the resource object sub-bucket and the hash code parameters of the historical resource objects to which each resource segment object belongs are determined as index information of the resource object sub-bucket.

8. According to the method according to any one of claims 1 to 7, if there are multiple resource object sub-buckets corresponding to multiple dimensions under the same resource length bucket, the query priority of the multiple dimensions is obtained; The comparing the first encoding parameter with the index information of each resource object sub-bucket in the target resource length bucket includes: In descending order of the query priority, the first encoding parameter under the higher query priority is compared with the index information of each resource object sub-bucket in the target resource length bucket for similarity; Determine that the third similarities between the encoding parameters of each resource object sub-bucket and the first encoding parameter do not reach a third similarity threshold, obtain the second encoding parameter corresponding to the next query priority of the first resource object, and continue to compare the similarity with the index information of each resource object sub-bucket.

9. A resource object processing device, the device comprising: A first data acquisition module, used to obtain a first resource object and determine a first resource length and a first encoding parameter of the first resource object; A target resource length bucket determination module, used to determine a target resource length bucket corresponding to the first resource length from a plurality of resource length buckets in a resource library; each of the plurality of resource length buckets contains a historical resource object of a corresponding resource length; A comparison module, configured to compare the first encoding parameter with index information of each resource object sub-bucket in the target resource length bucket; the resource object sub-bucket is obtained based on a secondary bucketing processing method corresponding to the dimension to which the first encoding parameter belongs; The similarity query result obtaining module is used to obtain similarity query results between the first resource object and the historical resource object based on the obtained first comparison result.

10. A computer device comprising at least one communication interface, at least one memory and at least one processor, wherein: The memory is used to store a program for implementing the resource object processing method according to any one of claims 1 to 8; The processor is used to load and execute the program stored in the memory to implement the resource object processing method according to any one of claims 1 to 8.

Images