Method, computer device, and computer program for eliminating duplicate content in large-scale recommendation system

Abstract

To provide a method, computer device, and computer program for eliminating duplicate content in a large-scale recommendation system.SOLUTION: A method for eliminating duplicate content in a large-scale recommendation system includes steps of: generating, for each piece of video content, a video hash using multiple segmented images generated from the video content; and identifying, based on the video hash, duplicate video content and eliminating duplication.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The following description relates to techniques for solving the overlapping content recommendation problem. [Background technology]

[0002] Instant messenger, a common communication tool, is software for sending and receiving messages and data in real time. Users can register people they want to chat with on the messenger and exchange messages with people on their chat list in real time.

[0003] The use of such messenger functions has become common not only on PCs but also in mobile environments on mobile communication devices.

[0004] For example, Korean Patent Publication No. 10-2002-0074304 (published on September 30, 2002) discloses a mobile messenger service system and method for a mobile terminal using a wireless communication network, which can provide a messenger service between mobile messengers installed on the mobile terminal.

[0005] As instant messengers become more popular, the functions they offer are becoming increasingly diverse, and they now offer video platforms that allow users to upload, watch, and share videos. Summary of the Invention [Problem to be solved by the invention]

[0006] To provide a technology for identifying and eliminating duplicate content with higher accuracy.

[0007] To provide a technique for efficiently performing deduplication in a large-scale recommendation environment.

[0008] Provides deduplication technology suitable for online serving (scalable real-time serving system). [Means for solving the problem]

[0009] A content deduplication method for a computer device including at least one processor is provided, the method including a step of generating, by the at least one processor, a video hash for each video content using multiple divided images generated from the video content, and a step of identifying, by the at least one processor, duplicate video content based on the video hash and performing deduplication.

[0010] According to one aspect, the generating step may include generating a certain number of divided images from the video content, calculating a hash value for each of the divided images, and concatenating the certain number of hash values ​​to calculate the video hash.

[0011] According to another aspect, the step of generating the certain number of divided images may include the steps of: extracting frames of the video content one by one at a certain time interval to obtain primary condensed frames of the video content; extracting frames at equal intervals from the primary condensed frames to obtain secondary condensed frames of the video content; horizontally connecting the secondary condensed frames to obtain a summary image of the video content; and equally dividing the summary image into the certain number of divided images to generate the certain number of divided images.

[0012] According to another aspect, the step of generating the certain number of divided images may include the steps of: extracting frames of the video content one by one at a fixed time interval to obtain condensed frames of the video content; horizontally connecting the condensed frames to obtain a summary image of the video content; and equally dividing the summary image into the certain number of divided images to generate the certain number of divided images.

[0013] According to another aspect, the number of divided images may be determined based on the length of the video content.

[0014] According to another aspect, the step of obtaining a hash value for each of the divided images may obtain a hash value for each of the divided images by perceptual hashing or wavelet hashing.

[0015] According to another aspect, the performing step may perform deduplication using the video hash on a set of content candidates for recommendation in an online serving (real-time serving) environment.

[0016] According to another aspect, the performing step may include constructing a set of candidate content for recommendation, determining a video hash distance between video content included in the candidate content set, identifying video content in the candidate content set whose video hash distance is less than a threshold as duplicate content, and eliminating one of the duplicate content.

[0017] According to another aspect, the step of constructing the content candidate group may construct a content candidate group personalized for a user of the service based on information associated with the user.

[0018] According to yet another aspect, the performing step may further include a step of determining and serving a ranking of the deduplicated content candidates to a user for the service based on information associated with the user.

[0019] A computer program is provided that is recorded on a computer-readable recording medium to cause the computer device to execute the content deduplication method.

[0020] A computer device is provided, comprising at least one processor implemented to execute computer-readable instructions, wherein the at least one processor generates a video hash for each video content using multiple split images generated from the video content, and identifies and deduplicates duplicate video content based on the video hash. [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 illustrates an example of a network environment in accordance with an embodiment of the present invention. [Figure 2] FIG. 1 is a block diagram illustrating an example of a computing device according to an embodiment of the present invention. [Figure 3] 1 is a flowchart illustrating an example of a method that can be performed by a computer device in accordance with an embodiment of the present invention. [Figure 4] FIG. 1 is a diagram illustrating the overall architecture of a video recommendation system using video hashing in one embodiment of the present invention. [Figure 5] 1 is a flowchart illustrating a process for generating a video hash in accordance with one embodiment of the present invention. [Figure 6] FIG. 2 is an exemplary diagram illustrating a first type of video hash in an embodiment of the present invention. [Figure 7] FIG. 2 is an exemplary diagram illustrating a first type of video hash in an embodiment of the present invention. [Figure 8] FIG. 2 is an exemplary diagram illustrating a first type of video hash in an embodiment of the present invention. [Figure 9] FIG. 2 is an exemplary diagram illustrating a first type of video hash in an embodiment of the present invention. [Figure 10] FIG. 2 is an exemplary diagram illustrating a first type of video hash in an embodiment of the present invention. [Figure 11] FIG. 10 is an exemplary diagram illustrating a second type of video hash in an embodiment of the present invention. [Figure 12] FIG. 10 is an exemplary diagram illustrating a second type of video hash in an embodiment of the present invention. [Figure 13] FIG. 10 is an exemplary diagram illustrating a second type of video hash in an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0023] SUMMARY OF THE INVENTION An embodiment of the present invention relates to a technique for solving the overlap problem of content recommendations.

[0024] Embodiments including those specifically disclosed in this specification can provide deduplication logic suitable for large-scale online serving environments, thereby improving the user experience in the recommendation system and increasing the performance and efficiency of recommendations.

[0025] A content deduplication system according to an embodiment of the present invention may be implemented by at least one computer device, and a content deduplication method according to an embodiment of the present invention may be executed by at least one computer device included in the content deduplication system. In this case, a computer program according to an embodiment of the present invention may be installed and executed in the computer device, and the computer device may execute the content deduplication method according to an embodiment of the present invention under the control of the executed computer program. The computer program may be recorded on a computer-readable recording medium so as to be combined with the computer device and cause the computer to execute the content deduplication method.

[0026] FIG. 1 is a diagram showing an example of a network environment in one embodiment of the present invention. The network environment in FIG. 1 shows an example including multiple electronic devices 110, 120, 130, and 140, multiple servers 150 and 160, and a network 170. FIG. 1 is merely an example for explaining the invention, and the number of electronic devices and the number of servers are not limited to those shown in FIG. 1. Furthermore, the network environment in FIG. 1 is merely an example of an environment applicable to this embodiment, and environments applicable to this embodiment are not limited to the network environment in FIG. 1.

[0027] The electronic devices 110, 120, 130, and 140 may be fixed or mobile terminals implemented by computers. Examples of the electronic devices 110, 120, 130, and 140 include smartphones, mobile phones, navigation systems, personal computers (PCs), notebook PCs, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), and tablets. While FIG. 1 illustrates a smartphone as an example of the electronic device 110, in embodiments of the present invention, the electronic device 110 may represent one of a variety of physical computer devices capable of communicating with the other electronic devices 120, 130, and 140 and / or the servers 150 and 160 via the network 170 using a substantially wireless or wired communication method.

[0028] The communication method is not limited, and may include not only communication methods using communication networks (for example, a mobile communication network, a wired Internet, a wireless Internet, and a broadcast network) that can be included in network 170, but also short-range wireless communication between devices. For example, network 170 may include any one or more of networks such as a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), and the Internet. Furthermore, network 170 may include any one or more of network topologies including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or hierarchical network, etc.

[0029] Each of the servers 150, 160 may be implemented by one or more computing devices that communicate with the plurality of electronic devices 110, 120, 130, 140 via the network 170 to provide instructions, code, files, content, services, etc. For example, the server 150 may be a system that provides services (such as video services, social media services, etc.) to the plurality of electronic devices 110, 120, 130, 140 connected via the network 170.

[0030] 2 is a block diagram showing an example of a computer device according to an embodiment of the present invention. Each of the electronic devices 110, 120, 130, and 140 and each of the servers 150 and 160 described above may be realized by a computer device 200 shown in FIG.

[0031] As shown in FIG. 2 , such a computer device 200 may include a memory 210, a processor 220, a communication interface 230, and an input / output interface 240. The memory 210 is a computer-readable recording medium and may include random access memory (RAM), read-only memory (ROM), and a persistent mass storage device such as a disk drive. Here, a persistent mass storage device such as a ROM or a disk drive may be included in the computer device 200 as a separate persistent storage device distinct from the memory 210. The memory 210 may also store an operating system and at least one program code. Such software components may be loaded into the memory 210 from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, or a memory card. In another embodiment, the software components may be loaded into the memory 210 through a communication interface 230, which is not a computer-readable recording medium. For example, the software components may be loaded into the memory 210 of the computing device 200 based on a computer program installed by a file received over the network 170 .

[0032] Processor 220 may be configured to process computer program instructions by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to processor 220 by memory 210 or by communication interface 230. For example, processor 220 may be configured to execute instructions received according to program code stored in a storage device such as memory 210.

[0033] The communication interface 230 may provide a function for the computer device 200 to communicate with other devices (e.g., the above-mentioned storage device) via the network 170. For example, requests, instructions, data, files, etc. generated by the processor 220 of the computer device 200 in accordance with program code stored in a storage device such as the memory 210 may be transmitted to other devices via the network 170 under the control of the communication interface 230. Conversely, signals, instructions, data, files, etc. from other devices may be received by the computer device 200 via the communication interface 230 of the computer device 200 via the network 170. The signals, instructions, data, etc. received via the communication interface 230 may be transmitted to the processor 220 or the memory 210, and files, etc. may be recorded on a storage medium (e.g., the above-mentioned permanent storage device) that the computer device 200 may further include.

[0034] The input / output interface 240 may be a means for interfacing with the input / output device 250. For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface 240 may be a means for interfacing with a device that integrates input and output functions into one, such as a touch screen. The input / output device 250 may be configured as a single device together with the computer device 200.

[0035] Also, in other embodiments, computing device 200 may include fewer or more components than those shown in Figure 2. However, most conventional components need not be explicitly shown in the figures. For example, computing device 200 may be implemented to include at least some of the input / output devices 250 described above, and may further include other components such as a transceiver, a database, etc.

[0036] Specific embodiments of a method and apparatus for eliminating content duplication in a large-scale recommender system are described below.

[0037] In this specification, social media may refer to an integrated media platform that provides various services utilizing resources such as user profile information and friendships within social network services, as well as social network services such as messengers and various communities.

[0038] Short-form video content is one of the fastest growing areas in the digital media industry, including social media. Currently, video platforms that share various types of videos, including short-form videos, serve recommended content from a content pool, which is a collection of content that meets the service conditions.

[0039] Previously, embedding cosine similarity between pieces of content within a content pool was used to eliminate one of the overlapping pieces of content, and the remaining piece was used for the service.

[0040] To improve user experience and enhance recommendation performance and efficiency, video platform recommendation systems are evolving into scalable real-time serving systems that target all content, rather than using content pools.

[0041] The present embodiment aims to provide a deduplication logic suitable for a large-scale recommendation system in an online serving environment that does not use a content pool.

[0042] The computer device 200 according to the present embodiment may provide video services to a client through a dedicated application installed on the client or through connection to a web / mobile site associated with the computer device 200. A computer-implemented content deduplication system may be configured in the computer device 200. For example, the content deduplication system may be implemented as an independently operating program or may be implemented as an in-app for a specific application so as to be operable on the specific application.

[0043] The processor 220 of the computing device 200 may be implemented with components for performing the following content deduplication method. Depending on the embodiment, the components of the processor 220 may be selectively included or excluded from the processor 220. Also, depending on the embodiment, the components of the processor 220 may be separated or combined to represent the functionality of the processor 220.

[0044] Such processor 220 and components of processor 220 may control computer device 200 to perform steps included in the following content deduplication method. For example, processor 220 and components of processor 220 may be implemented to execute instructions from operating system code and at least one program code contained in memory 210.

[0045] Here, the components of the processor 220 may represent different functions that are performed by the processor 220 according to instructions provided by program code stored on the computer device 200 .

[0046] The processor 220 may read the necessary instructions from the memory 210, which is loaded with instructions related to the control of the computing device 200. In this case, the read instructions may include instructions for controlling the processor 220 to perform the steps described below.

[0047] The steps involved in the content deduplication method described below may be performed in a different order than that shown in the figures, some steps may be omitted, or additional processes may be included.

[0048] The steps involved in the content deduplication method may be performed on the server 150, although in some embodiments at least some of the steps may be performed on the client.

[0049] FIG. 3 is a flowchart illustrating an example method that may be performed by a computing device in accordance with an embodiment of the present invention.

[0050] 3, in step 310, processor 220 may generate a video hash for each piece of video content within the video platform of server 150. In this embodiment, multiple divided images are generated from the video content, and a final hash value obtained by concatenating the hash values ​​of the divided images is defined as the video hash. When video content is registered on the video platform, processor 220 may generate a video hash for the video content and then store the generated video hash in a separate database for storing the video hash, together with information for identifying the video content.

[0051] In step 320, processor 220 may perform deduplication using the video hash of each video content. Processor 220 may select a group of recommendation candidates associated with user information (e.g., gender, age, region, interests, past content consumption history, etc.) from among the video content registered on the video platform for users of the service, and then remove duplicate content from the group of recommendation candidates using the video hash. Video hashes may be compared between video contents to identify content that overlaps with each other, and one of two pieces of content that overlap with each other may be randomly removed.

[0052] In step S330, the processor 220 may provide recommendations using the deduplicated recommendation candidate set. When providing recommendations to the user, the processor 220 may first perform deduplication and then recommend video content included in the deduplicated recommendation candidate set to the user.

[0053] FIG. 4 is a diagram showing the overall architecture of a video recommendation system using video hashing in one embodiment of the present invention.

[0054] Referring to FIG. 4, the video recommendation system may be implemented as a system in an online serving environment that recommends all content without using a content pool, and may be composed of a hash generation pipeline 410 and a recommendation pipeline 420.

[0055] The hash generation pipeline 410 may include a hash generation unit 411 and a hash storage 412. When a post is registered on the video platform, the hash generation unit 411 may generate a video hash of the post and then store the generated video hash in the hash storage 412 together with identification information of the post.

[0056] The recommendation pipeline 420 includes a recommendation search unit 421 and a recommendation ranking unit 422, and may include a unit that accesses hash values ​​stored in the hash storage 412 to eliminate duplicates. The recommendation search unit 421 is responsible for creating a set of recommendation candidates personalized for a user based on user information from posts registered on the video platform, i.e., those likely to be liked by the user. The recommendation search unit 421 imports recommendation candidates from multiple sources from various tasks (retrieve candidates) and imports video hashes from the hash storage 412 for the recommendation candidates (request N posts' hash values). In this case, the recommendation search unit 421 uses the video hashes imported from the hash storage 412 to find duplicate posts among the recommendation candidates. The hash distance between two posts among the recommendation candidates is calculated, and if the hash distance between the two posts is less than a threshold (e.g., 30), the posts are determined to be duplicates, and one of them is randomly eliminated (apply deduplication). The recommendation search unit 421 applies necessary filtering, such as blacklist filtering, to the group of recommendation candidates from which duplicates have been removed, and then transmits the final group of recommendation candidates to the recommendation ranking unit 422. The recommendation ranking unit 422 ranks the posts included in the final group of recommendation candidates in order of the user's likelihood of preference based on user information, and provides the rankings to the user.

[0057] FIG. 5 is a flow chart illustrating a process for generating a video hash in accordance with one embodiment of the present invention.

[0058] Referring to FIG. 5, in step 501, the processor 220 may generate N divided images from each video content. Here, the number of divided images, N, may be determined in proportion to the overall length of the video content. As an example, the processor 220 may extract frames of the video content one by one at regular time intervals (e.g., 3 seconds), and then horizontally connect the extracted frames and divide the result into N equal parts to generate N divided images. As another example, the processor 220 may extract frames of the video content one by one at regular time intervals, and then extract frames at regular intervals from the extracted frames. For example, the processor 220 may extract frames of the video content one by one every 3 seconds, and then select only the 0th, 4th, 8th, and 12th frames from the extracted frames. Then, the processor 220 may horizontally connect the 0th, 4th, 8th, and 12th frames and divide the result into N equal parts to generate N divided images.

[0059] In step 502, the processor 220 may calculate a hash value for each of the N divided images generated from the video content, and then combine the N hash values ​​to calculate a final hash value, which may be defined as the video hash of the video content. As an example, the processor 220 may calculate an image hash value for each divided image using perceptual hashing or wavelet hashing. In addition to perceptual hashing and wavelet hashing, various image hashing methods such as average hashing, difference hashing, and HSV color hashing may also be used.

[0060] 6 to 10 are exemplary diagrams illustrating a first type of video hash in one embodiment of the present invention.

[0061] The process of generating the first type of video hash is performed in the following order:

[0062] (1) Referring to FIG. 6, the processor 220 may obtain a first-order condensed frame 601 by extracting frames of the video content 600 at regular intervals, for example, by extracting one frame of the video content 600 every three seconds.

[0063] (2) Referring to Figure 7, the processor 220 may obtain secondary condensed frames 702 for the video content 600 by extracting frames at equal intervals from the primary condensed frames 601 of the video content 600. For example, frames 4, 8, 12, and 16 may be extracted from the primary condensed frames 601 at intervals of four frames from the first frame (frame 0) and used as the secondary condensed frames 702.

[0064] (3) Referring to FIG. 8, the processor 220 may stitch together the secondary condensed frames 702 of the video content 600 horizontally side-by-side to generate a summary image 803 for the video content 600 .

[0065] (4) Referring to Figure 9, the processor 220 may divide the summary image 803 of the video content 600 into N equal parts to generate the video content 600 into N divided images 904. For example, for the video content 600, the summary image 803 may be divided into four divided images 904. Because the number of frames extracted from the video content 600 varies depending on the length of the video content 600, the number of divided images may vary in proportion to the length of the video.

[0066] (5) Referring to Figure 10, the processor 220 may determine a hash value 1005 for each of the divided images 904 generated from the video content 600, and then combine the N hash values ​​1005 to determine a first type video hash 1006 of the video content 600. For example, if the video content 600 is generated using four divided images 904, the final 256-bit hash value obtained by combining the four 64-bit hash values ​​may be defined as the first type video hash 1006 of the video content 600.

[0067] 11 to 13 are exemplary diagrams illustrating a second type of video hash in one embodiment of the present invention.

[0068] The process of generating the second type of video hash is as follows:

[0069] (1) The processor 220 may obtain the first-order condensed frame 601 by extracting frames of the video content 600 at regular intervals, for example, by extracting one frame of the video content 600 every three seconds.

[0070] (2) Referring to FIG. 11, the processor 220 may horizontally stitch together the primary condensed frames 601 of the video content 600 to generate a summary image 1103 for the video content 600 .

[0071] (3) Referring to Figure 12, the processor 220 may divide the summary image 1103 of the video content 600 into N equal parts to generate the video content 600 into N divided images 1204. For example, for the video content 600, the summary image 1103 may be divided into four divided images 1204. Because the number of frames extracted from the video content 600 varies depending on the length of the video content 600, the number of divided images may vary in proportion to the length of the video.

[0072] (4) Referring to Figure 13, the processor 220 may calculate a hash value 1305 for each of the divided images 1204 generated from the video content 600, and then combine the N hash values ​​1305 to calculate a second type video hash 1306 for the video content 600. For example, if the video content 600 is generated using four divided images 1204, the final 256-bit hash value obtained by combining the four 64-bit hash values ​​may be defined as the second type video hash 1306 for the video content 600.

[0073] The first type of video hash generation process can achieve faster calculation speeds than the second type of video hash generation process by using two rounds of reduction, and the second type of video hash generation process can apply divided images with richer image information by using a shorter reduction than the first type of video hash generation process.

[0074] According to an embodiment of the present invention, the accuracy of duplicate identification can be improved by obtaining a video hash using multiple divided images generated from video content and comparing the video hashes between video content to identify duplicate content. The newly defined video hash can provide a deduplication logic suitable for online serving, improving the user experience in large-scale recommendation systems and increasing the efficiency and quality of deduplication.

[0075] The above-described devices may be implemented using hardware components, software components, and / or a combination of hardware and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or various devices capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the OS. The processing device may also access, record, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, a single processing device may be described. However, those skilled in the art will understand that a processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing device may include multiple processors or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0076] Software may include computer programs, codes, instructions, or a combination of one or more of these, which may configure a processing device to operate as desired or may independently or collectively instruct the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed and stored and executed in a distributed manner on computer systems connected by a network. The software and data may be stored on one or more computer-readable storage media.

[0077] Methods according to embodiments may be implemented in the form of program instructions executable by various computer means and recorded on a computer-readable medium. In this case, the medium may continuously record a computer-executable program or may temporarily record the program for execution or download. The medium may be various recording or storage means in the form of a single piece of hardware or multiple pieces of hardware combined together. It may be a medium directly connected to a computer system or distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to record program instructions, such as ROMs, RAMs, and flash memories. Other examples of media include recording media or storage media managed by app stores that distribute applications, or by websites or servers that provide or distribute various software.

[0078] Although the embodiments have been described above based on limited examples and drawings, those skilled in the art will appreciate that various modifications and variations may be made from the above description. For example, the described techniques may be performed in an order different from that described, and / or the described system, structure, device, circuit, or other element may be coupled or combined in a manner different from that described, or may be substituted or replaced by other elements or equivalents, and still achieve suitable results.

[0079] Therefore, different embodiments are within the scope of the appended claims, provided that they are equivalent to the claims. [Explanation of symbols]

[0080] 110 Electronic equipment 120 Electronic equipment 130 Electronic equipment 140 Electronic equipment 150 servers 160 servers 170 Network 200 Computer Equipment 210 memory 220 processors 230 Communication Interface 240 Input / Output Interface 250 Input / Output Devices 410 Hash Generation Pipeline 411 Hash Generation Unit 412 Hash Storage 420 Recommendation Pipeline 421 Recommendation Search Department 422 Recommendation Ranking Section 600 video contents 601 First-order contracted frame 702 Quadratic Contraction Frame 803 Summary Image 904 Split Image 1005 hash value 1006 Video Hash 1103 Summary Image 1204 Split Image 1305 hash value 1306 Video Hash

Claims

1. 1. A content deduplication method for a computer device including at least one processor, comprising: generating, by the at least one processor, for each video content, a video hash using a plurality of segmented images generated from the video content; and and identifying, by the at least one processor, duplicate video content based on the video hashes and performing de-duplication. Content deduplication methods, including:

2. The generating step includes: generating a certain number of divided images from the video content; calculating a hash value for each of the divided images; and Combining the number of hash values ​​to obtain the video hash. The content deduplication method of claim 1 , comprising:

3. The step of generating a certain number of divided images includes: extracting frames of the video content one by one at regular intervals to obtain a first-order condensed frame of the video content; obtaining secondary condensed frames of the video content by extracting equally spaced frames from the primary condensed frames; horizontally stitching the secondary condensed frames to obtain a summary image for the video content; and Dividing the abstract image into the predetermined number of equal parts to generate the predetermined number of divided images. The content deduplication method of claim 2 , comprising:

4. The step of generating a certain number of divided images includes: extracting frames of the video content one by one at regular intervals to obtain condensed frames of the video content; horizontally stitching the condensed frames together to obtain a summary image for the video content; and Dividing the abstract image into the predetermined number of equal parts to generate the predetermined number of divided images. The content deduplication method of claim 2 , comprising:

5. The number of the divided images is determined based on the length of the video content. The content deduplication method according to claim 2 .

6. The step of obtaining a hash value for each of the divided images includes: determining a hash value for each of the divided images by perceptual hashing or wavelet hashing; The content deduplication method according to claim 2 .

7. The performing step includes: Utilizing the video hash to perform deduplication on a set of candidate content for recommendation in an online serving environment. The content deduplication method according to claim 1 .

8. The performing step includes: constructing a set of content candidates for recommendation; calculating a video hash distance between video contents included in the content candidate group; identifying video content in the candidate content set whose video hash distance is equal to or less than a threshold as duplicate content; and Removing one video content from the duplicate content. The content deduplication method of claim 1 , comprising:

9. The step of forming the content candidate group includes: To create a personalized set of content candidates for a user of a service based on information related to that user. The content deduplication method according to claim 8 .

10. The performing step includes: determining a ranking of the duplicate-eliminated content candidates based on information related to the user for the service and serving the ranking to the user; The content deduplication method of claim 8 , further comprising:

11. A computer program recorded on a computer-readable recording medium for causing the computer device to execute the content deduplication method according to any one of claims 1 to 10.

12. At least one processor implemented to execute computer-readable instructions Including, The at least one processor generating a video hash for each video content using a plurality of segmented images generated from the video content; Identifying and de-duplication duplicate video content based on the video hashes. A computer device comprising:

13. The at least one processor generating a number of divided images from the video content; calculating a hash value for each of the divided images; Combining the number of hash values ​​to determine the video hash.

13. The computer device of claim 12.

14. The at least one processor extracting frames of the video content one by one at regular intervals to obtain a first-order condensed frame of the video content; obtaining secondary condensed frames of the video content by extracting equally spaced frames from the primary condensed frames; horizontally stitching the secondary condensed frames to obtain a summary image of the video content; Dividing the abstract image into the predetermined number of equal parts to generate the predetermined number of divided images.

14. The computer device of claim 13.

15. The at least one processor extracting frames of the video content one by one at regular intervals to obtain condensed frames of the video content; horizontally stitching the condensed frames together to obtain a summary image of the video content; Dividing the abstract image into the predetermined number of equal parts to generate the predetermined number of divided images.

14. The computer device of claim 13.

16. The number of the divided images is determined based on the length of the video content.

14. The computer device of claim 13.

17. The at least one processor determining a hash value for each of the divided images by perceptual hashing or wavelet hashing; 14. The computer device of claim 13.

18. The at least one processor Utilizing the video hash to perform deduplication on a set of candidate content for recommendation in an online serving environment.

13. The computer device of claim 12.

19. The at least one processor Constructing a set of content candidates for recommendation; calculating a video hash distance between video contents included in the content candidate group; Identifying video content from the content candidate group whose video hash distance is equal to or less than a threshold as duplicate content; Removing one video content from the duplicate content.

13. The computer device of claim 12.

20. The at least one processor To create a personalized set of content candidates for a user of a service based on information related to that user.

20. The computer device of claim 19.

Images