Method and apparatus for providing similar content in a content streaming system

Abstract

This disclosure relates to a method and apparatus for providing similarity content in a content streaming system, the operation method of the server in the content streaming system may include the steps of: acquiring first sequence-type text data including information contained in first metadata of a first content item; acquiring second sequence-type text data including information contained in second metadata of a second content item; determining a first vector corresponding to the first sequence-type text data and a second vector corresponding to the second sequence-type text data using a language model learned based on synopsis information contained in the metadata of the content item; determining the similarity between the first content item and the second content item using the first vector and the second vector; and providing a content list including at least one content item, including the second content item selected based on the similarity.

Description

Technical Field

[0001] The present disclosure relates to a content streaming system, and particularly to a method and apparatus for providing similar content in a content streaming system.

Background Art

[0002] With the development of various technologies and changes in consumption trends, significant changes have occurred in the supply and consumption methods of content. The development of digital technology, computer technology, Internet / communication technology, etc. has blurred the boundaries for the types of content and the main producers. This has caused major changes in the production and consumption patterns of content. Platforms have emerged that enable the general public to create and distribute content. Also, the ease of access to various content has been ensured, and various options have begun to be provided for consumption methods.

[0003] Among many changes in such a content industry, there is the OTT (over the top) service. The OTT service is a media platform based on the Internet and mobile communication, and provides various content to consumers without equipment such as a separate set-top box beyond the framework of existing broadcast services. The concept of the OTT service started from providing movies, TV programs, etc. in a VOD (video on demand) manner, but the OTT service is still an expanding service that not only provides self-produced content by OTT service providers but also expands its scope to mobile platforms.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure can provide a method and apparatus for effectively providing similar content in a content streaming system.

[0005] This disclosure can provide a method and apparatus for recommending content similar to specific content in a content streaming system.

[0006] This disclosure can provide a method and apparatus for determining similar content using a language model in a content streaming system.

[0007] This disclosure can provide a method and apparatus for recommending content in a content streaming system based on text metadata that describes the content.

[0008] This disclosure can provide a method and apparatus for training a language model based on content hashtags.

[0009] This disclosure can provide a method and apparatus for training a language model based on the genre of content.

[0010] This disclosure can provide a method and apparatus for training a language model based on a content synopsis.

[0011] This disclosure can provide a method and apparatus for performing two-step training on a language model based on two different types of information from the text metadata of content.

[0012] This disclosure can provide a method and apparatus for determining the similarity between contents using a language model trained on the text metadata of the contents.

[0013] The technical challenges to be addressed in this disclosure are not limited to those described above, and other technical challenges not described above may be considered by persons with ordinary skill in the art to which the technical configuration of this disclosure applies, based on the embodiments of this disclosure described below. [Means for solving the problem]

[0014] An example of how a server operates in a content streaming system according to this disclosure may include the steps of: acquiring first sequence-type text data containing information contained in first metadata of a first content item; acquiring second sequence-type text data containing information contained in second metadata of a second content item; determining a first vector corresponding to the first sequence-type text data and a second vector corresponding to the second sequence-type text data using a language model learned based on synopsis information contained in the metadata of the content item; determining the similarity between the first content item and the second content item using the first and second vectors; and providing a content list containing at least one content item including the second content item selected based on the similarity.

[0015] According to one example of this disclosure, the language model can be trained to predict synopsis information of the content item based on a masked language model (MLM).

[0016] According to one example of this disclosure, the language model can be primarily trained through training to predict hashtag information of the content item based on the MLM, and secondarily trained through training to predict synopsis information of the content item based on the MLM.

[0017] According to one example of this disclosure, the language model can be primarily trained through training to predict synopsis information of the content item based on the MLM, and secondarily trained through training to predict hashtag information of the content item based on the MLM.

[0018] According to one example of this disclosure, the language model can be trained to predict masked tokens located between tokens that indicate a synopsis region among a plurality of tokens contained in the input sequence-type text data.

[0019] According to one example of the present disclosure, the token indicating the synopsis region may include at least one of isolation tokens for separating different types of features from each other, and special tokens for other types of features that are not synopsis.

[0020] According to one example of the present disclosure, the steps include: converting text metadata describing the content of a content item into sequence text data; masking synopsis tokens among a plurality of tokens contained in the sequence text data that are located between tokens indicating the synopsis region; and training the language model through training to predict the masked synopsis tokens, wherein the text metadata may include at least one of title, synopsis, genre, director, actor, or hashtag information.

[0021] According to one example of the present disclosure, the step of converting the text metadata into sequence text data includes the steps of dividing the text metadata into a plurality of tokens and generating the sequence text data by inserting at least one delimiter between the tokens, wherein the at least one delimiter may further include at least one of a token indicating the synopsis region, a separation token for separating different types of features from each other, and a special token for indicating a region of a particular type of feature.

[0022] According to an example of the present disclosure, the step of masking the synopsis token includes selecting a non-dependent token from the synopsis tokens located between the tokens indicating the synopsis region, and masking the selected non-dependent token, where the non-dependent token may be a token that does not start with a specified symbol.

[0023] According to an example of the present disclosure, the training is performed using a prediction model, where the prediction model takes as input sequence-type text data including masked synopsis tokens, and includes the language model that outputs vector values corresponding to the sequence-type text data, and an MLM (masked language model) head layer configured to predict at least one input token corresponding to at least one vector value output from the language model.

[0024] According to an example of the present disclosure, the step of determining the similarity between the first content item and the second content item includes calculating the similarity between the first vector and the second vector using a cosine similarity algorithm, and each of the first vector and the second vector can be obtained by performing average pooling on the output vector values of the final hidden layer of the learned language model.

[0025] According to an example of the present disclosure, each of the first vector and the second vector can be determined by weighting the vector values corresponding to the positions of the specified features among the output vector values of the final hidden layer of the learned language model.

[0026] According to an example of the present disclosure, a step of obtaining third sequence type text data including information included in third metadata of a third content item, a step of determining a third vector corresponding to the third sequence type text data using the learned language model, and a step of determining a similarity between the first content item and the third content item using the first vector and the third vector, and the step of providing the content list includes a step of selecting the second content item from the second content item and the third content item based on a similarity between the first content item and the second content item and a similarity between the first content item and the third content item.

[0027] A server in a content streaming system according to an embodiment of the present disclosure includes a communication unit that transmits and receives signals to and from at least one client device, and a processor electrically connected to the communication unit. The processor obtains first sequence type text data including information included in first metadata of a first content item, obtains second sequence type text data including information included in second metadata of a second content item, and uses a language model learned based on synopsis information included in metadata of the content item to determine a first vector corresponding to the first sequence type text data and a second vector corresponding to the second sequence type text data, determines a similarity between the first content item and the second content item using the first vector and the second vector, and can be controlled to provide a content list including at least one content item including the second content item selected based on the similarity.

[0028] A program stored in a recording medium according to an embodiment of the present disclosure can execute any of the methods described above when operated by a processor. [[Effect of the Invention]]

[0029] According to this disclosure, content similar to the standard content can be recommended.

[0030] The effects obtained by this disclosure are not limited to those described above, and other effects not mentioned above will be clearly understood by a person with ordinary skill in the art to which this disclosure pertains from the following description. [Brief explanation of the drawing]

[0031] [Figure 1] This figure shows a content streaming system according to one embodiment of the present disclosure.

[0032] [Figure 2] This figure shows the structure of a client device according to one embodiment of the present disclosure.

[0033] [Figure 3] This figure shows the structure of a server according to one embodiment of the present disclosure.

[0034] [Figure 4] This figure shows a concept of a content streaming service according to one embodiment of the present disclosure.

[0035] [Figure 5] This figure shows an example of the relative relationship between vectors.

[0036] [Figure 6] This figure shows an example of a server structure according to one embodiment of the present disclosure.

[0037] [Figures 7A-7B] This figure shows an example of the structure of a model learning unit according to one embodiment of the present disclosure.

[0038] [Figure 8] This figure shows an example of converting text metadata of content into sequenced text data according to one embodiment of the present disclosure.

[0039] [Figure 9A-9B] This figure shows an example of language model training according to one embodiment of the present disclosure.

[0040] [Figure 9C] This figure shows an example of the structure of a predictive model according to one embodiment of the present disclosure.

[0041] [Figure 10A] This figure shows an example of language model training according to one embodiment of the present disclosure.

[0042] [Figure 10B] This figure shows an example of the input / output structure of a prediction model according to one embodiment of the present disclosure.

[0043] [Figure 10C] This disclosure presents concepts of multi-class and multi-label prediction models applicable to this disclosure.

[0044] [Figure 11A-11E] This figure shows an example of the relationship between content-specific predictions and similarity according to one embodiment of the present disclosure.

[0045] [Figure 12] This figure shows an example of calculating content similarity using a trained language model according to one embodiment of the present disclosure.

[0046] [Figure 13] This figure shows an example of a procedure for recommending content using a trained language model according to one embodiment of the present disclosure.

[0047] [Figure 14A] This figure shows an example of a procedure for training a language model according to one embodiment of the present disclosure.

[0048] [Figure 14B]This figure shows an example of training a language model using hashtag prediction according to one embodiment of the present disclosure.

[0049] [Figure 15A] This figure shows an example of a procedure for training a language model according to one embodiment of the present disclosure.

[0050] [Figure 15B] This figure shows an example of training a language model using genre prediction according to one embodiment of the present disclosure.

[0051] [Figure 16A] This figure shows an example of a procedure for training a language model according to one embodiment of the present disclosure.

[0052] [Figure 16B] This figure shows an example of a procedure for training a language model according to one embodiment of the present disclosure.

[0053] [Figure 16C] This figure shows an example of training a language model using hashtags and synopses according to one embodiment of the present disclosure.

[0054] [Figure 17] This figure shows an example of a procedure for determining content similarity using a trained language model according to one embodiment of the present disclosure.

[0055] [Figure 18A] This figure shows an example of a transformer structure applicable to the embodiments of this disclosure.

[0056] [Figure 18B] This figure shows an example of the detailed structure of an encoder and decoder block of a transformer applicable to embodiments of this disclosure.

[0057] [Figure 19]This figure shows an example of the structure of a BERT model applicable to the embodiments of this disclosure.

[0058] [Figure 20] This figure shows an example of a test set according to one embodiment of the present disclosure.

[0059] [Figure 21] This figure shows an example of how similar content can be used, as determined by one embodiment of this disclosure. [Modes for carrying out the invention]

[0060] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that they can be easily implemented by a person with ordinary skill in the art to which the present invention pertains. However, the present invention can be realized in various different forms and is not limited to the embodiments described herein.

[0061] In describing embodiments of the present invention, detailed descriptions of known configurations or functions will be omitted if they would obscure the gist of the invention. In the drawings, parts unrelated to the description of the present invention will be omitted, and similar parts will be given the same reference numerals.

[0062] The functional blocks shown in the drawings and described below are merely possible examples. Other functional blocks may be used in other examples, provided they do not deviate from the ideas and scope of the detailed description. Furthermore, while one or more functional blocks of the present invention are shown as individual blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software configurations that perform the same function.

[0063] Furthermore, the expression "contains a certain component" is an "open" expression, merely indicating the existence of that component and should not be understood as excluding additional components. Consequently, when it is mentioned that a component is "connected" or "linked" to another component, it should be understood that it may be directly connected or linked to the other component, but there may also be other components between them.

[0064] Furthermore, a singular expression for an object may be understood as a plural expression unless the context clearly indicates otherwise. In this disclosure, expressions such as "A or B" or "at least one of A and / or B" may be understood to include all possible combinations of the items listed together. Expressions such as "first," "second," and "third" may modify the object regardless of order or importance and are used solely to distinguish one object from other objects of the same kind.

[0065] Furthermore, in this disclosure, “configured to” can be understood, depending on the context, to be technically equivalent to any one of the following expressions: “suitable for,” “capable of,” “modified to,” “made to,” “capable of,” and “designed to,” and they can be interchanged with each other, either in hardware or software terms.

[0066] This disclosure relates to recommending content in a content streaming system, and specifically describes a technology for recommending content based on text metadata of the content. In particular, this disclosure presents various embodiments of training a language model based on text metadata of the content and using the trained language model to determine the similarity between content.

[0067] Figure 1 shows a content streaming system according to one embodiment of the present disclosure. Figure 1 illustrates a system for providing content-related services, such as content streaming and content-related information provision, and entities belonging to that system. Hereinafter, in the present disclosure, various content-related services may be referred to as "content services" or other terms with equivalent technical meaning.

[0068] Referring to Figure 1, the content streaming system can include client devices 110 and servers 120. Here, client devices 110 are exemplified as a set of three client devices 110-1 to 110-3, but the content streaming system can include two or fewer client devices, or four or more. Similarly, although server 120 is exemplified as one, the content streaming system can include multiple servers that share and interact with various functions.

[0069] The client device 110 receives and displays content. After accessing the server 120 via the network, the client device 110 can receive content streamed from the server 120. In other words, the client device 110 is hardware on which client software or applications designed to use the content services provided by the server 120 are installed, and can interact with the server 120 through the installed software or applications. The client device 110 can be embodied as various types of devices. For example, the client device 110 may be one of the following: a portable device that is movable, a device that is movable but is usually fixed in place during use, or a device that is permanently installed in a specific location.

[0070] Specifically, the client device 110 can be embodied in at least one form from among a smartphone 110-1, a desktop computer 110-2, a tablet PC, a laptop PC, a netbook computer, a workstation, a server, a personal data assistant (PDA), a portable multimedia player (PMP), a camera, or a wearable device. Here, the wearable device may be embodied in at least one form from among accessory type (e.g., a watch, ring, bracelet, necklace, glasses, contact lenses, head-mounted device (HMD)), clothing type, body-attached type (e.g., skin pad or tattoo), or bio-implantable circuit. The client device 110 can also be a consumer electronics product, and can be embodied in at least one form from among a television 110-3, a digital video disc (DVD) player, an audio system, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washing machine, or an air purifier.

[0071] Server 120 performs various functions for providing content services. In other words, Server 120 can use various functions to provide content streaming and various content-related services to client devices 110. Specifically, Server 120 can convert content into data for streaming and transmit the content to client devices 110 via the network. To this end, Server 120 can perform at least one of the following functions: content encoding, data segmentation, transmission scheduling, and streaming transmission. Furthermore, for the convenience of content use, Server 120 can further perform at least one of the following functions: providing content guides, user account management, user preference analysis, and content recommendations based on preferences. Multiple of the aforementioned functions can be provided, and for this reason, Server 120 can be embodied as multiple servers.

[0072] The client device 110 and the server 120 exchange information over the network, and content services may be provided to the client device 110 based on the exchanged information. In this case, the network may be a single network or a combination of various types of networks. The network can be understood as a configuration in which different types of networks are connected depending on the section. For example, the network may include at least one of a wireless network or a wired network. Specifically, the network may include a cellular network based on at least one of the following: 6th generation mobile communication system (6G), 5th generation mobile communication system (5G), LTE (Long Term Evolution), LTE-A (LTE Advance), CDMA (code division multiple access), WCDMA (wideband CDMA), and UMTS (universal mobile telecommunications system), WiMAX (Wireless Broadband), or GSM (Global System for Mobile Communications). Furthermore, the network may include a local area network based on at least one of the following: wireless local area network (WLAN), Bluetooth, Zigbee, near field communication (NFC), or ultra-wideband (UWB). In addition, the network may include wired networks such as the Internet or Ethernet.

[0073] Figure 2 shows the structure of a client device according to one embodiment of the present disclosure. Figure 2 illustrates the block structure of a client device (for example, client device 110 in Figure 1).

[0074] Referring to Figure 2, the client device includes a display 202, an input unit 204, a communication unit 206, a sensing unit 208, an audio input / output unit 210, a camera module 212, a memory 214, a power supply unit 216, an external connection terminal 218, and a processor 220. However, depending on the type of device, at least one of the components exemplified in Figure 2 may be omitted.

[0075] The display 202 outputs information such as visually recognizable images and graphics. For this purpose, the display 202 may include a panel and circuitry for controlling the panel. For example, the panel may include at least one of the following: a liquid crystal display (LCD), a light-emitting diode (LED), a light-emitting polymer display (LPD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode (AMOLED), or a flexible LED (FLED).

[0076] The input unit 204 receives input generated by the user. The input unit 204 can include various types of input sensing means. For example, the input unit 204 can include at least one of a physical button, a keypad, or a touchpad. Alternatively, the input unit 204 can include a touch panel. If the input unit 204 includes a touch panel, the input unit 204 and the display 202 can be realized as a single module.

[0077] The communication unit 206 provides an interface for client devices to form a network with other devices and to transmit or receive data over the network. To this end, the communication unit 206 may include circuits for physically processing signals (e.g., encoders / decoders, modulators / demodulators, radio frequency (RF) front-ends, etc.) and a protocol stack for processing data according to a communication standard (e.g., a modem). In various embodiments, the communication unit 206 may include multiple modules to support multiple different communication standards.

[0078] The sensing unit 208 collects sensing data, including data relating to the state of the client device or the surrounding environment. For example, the sensing unit 208 can measure physical values ​​or changes in values ​​related to the operating state or orientation of the client device and generate an electrical signal representing the measurement result. The sensing unit 208 can also measure physical values ​​or changes in values ​​related to the surrounding environment of the client device and generate an electrical signal representing the measurement result. To this end, the sensing unit 208 may include at least one sensor and a circuit for controlling at least one sensor. Specifically, the sensing unit 208 may include at least one of the following: a gyro sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, a biosensor, a barometric pressure sensor, a temperature sensor, a humidity sensor, an illuminance sensor, or an ultraviolet (UV) sensor, an e-nose sensor, a gesture sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocadiogram (ECG) sensor, an infrared (IR) sensor, an iris sensor, or a fingerprint sensor.

[0079] The audio input / output unit 210 outputs sound in accordance with electrical signals generated based on audio data and detects external sounds. In other words, the audio input / output unit 210 can convert sound and electrical signals to each other. For this purpose, the audio input / output unit 210 may include at least one of a speaker, a microphone, or a circuit to control them.

[0080] The camera module 212 collects data for generating images and videos. To this end, the camera module 212 may include at least one of a lens, a lens drive circuit, an image sensor, a flash, or an image processing circuit. The camera module 212 can focus light through the lens and use the image sensor to generate data representing the color and brightness values ​​of the light.

[0081] Memory 214 stores the operating system, programs, applications, commands, configuration information, etc., necessary for the client device to operate. Memory 214 can store data temporarily or permanently. Memory 214 may include volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory.

[0082] The power supply unit 216 supplies the power necessary for the operation of the client device's components. For this purpose, the power supply unit 216 may include a converter circuit that converts the power into the amount of power required by each component. The power supply unit 216 may rely on an external power source or include a battery. If a battery is included, the power supply unit 216 may further include a charging circuit. The charging circuit may support wired or wireless charging.

[0083] The external connection terminal 218 is a physical connection means for connecting the client device to other devices. For example, the external connection terminal 218 may include at least one of various standard terminals such as a universal serial bus (USB) terminal, an audio terminal, an HDMI (high definition multimedia interface) terminal, an RS-232 (recommended standard-232) terminal, an infrared terminal, an optical terminal, or a power terminal.

[0084] The processor 220 controls the overall operation of the client device. The processor 220 controls the operation of other components and can perform various functions using those components. For example, the processor 220 can request content data from the server via the communication unit 206 and receive content data. The processor 220 can also reconstruct the content by decoding the received content data. Furthermore, the processor 220 can output the content received from the server via the display 202 and the audio input / output unit 210. In addition, the processor 220 can control the state related to content playback based on information input or sensed by at least one of the input unit 204, communication unit 206, sensing unit 208, audio input / output unit 210, camera module 212, power supply unit 216, and external connection terminal 218. For this purpose, the processor 220 may include at least one of at least one processor, at least one microprocessor, or at least one DSP (digital signal processor). In particular, the processor 220 can control other components and perform necessary operations so that the client device operates according to the various embodiments described later.

[0085] In the client device structure described with reference to Figure 2, it is illustrated that all components are connected to the processor 220. Although not shown in Figure 2, at least some of the components can be connected via a bus. In this case, data can be directly exchanged between some of the components based on the control of the processor 220.

[0086] Figure 3 shows the structure of a server according to one embodiment of the present disclosure. Figure 3 illustrates the block structure of the server (server 120 in Figure 1).

[0087] Referring to Figure 3, the server includes a communication unit 302, a memory 304, and a processor 308. However, in various embodiments, at least one of the components illustrated in Figure 3 may be omitted. Also, in various embodiments, at least one additional component may be included besides the components illustrated in Figure 3.

[0088] The communication unit 302 provides an interface for communication between the server and other devices. To this end, the communication unit 302 may include circuits for generating and analyzing physical signals for communication. The interface provided by the communication unit 302 can support wired or wireless communication.

[0089] Memory 304 stores various information, orders, and / or data, and loads computer programs, instructions, etc., stored in storage 306. Memory 304 can temporarily store data and instructions for server calculations and may include random access memory (RAM). Alternatively, memory 304 may include various storage media.

[0090] The storage 306 can non-temporarily store an operating system for the server's operation, programs for performing the server's functions, configuration information for the server's operation, and so on. For example, the storage 306 may include at least one of the following: non-volatile memory such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory; hard disk; removable disk; SSD (solid state drive); or any form of computer-readable recording medium widely known in the art to which this disclosure belongs.

[0091] The processor 308 controls the overall operation of the server. The processor 308 controls the operation of other components and can perform various functions using those components. The processor 308 may include at least one of a central processing unit (CPU), a microprocessor unit (MPU), a microcontroller unit (MCU), or a processor of a form widely known in the art to which this disclosure belongs. In particular, the processor 308 can control other components and perform necessary calculations so that the server operates according to the various embodiments described below.

[0092] In the server structure described with reference to Figure 3, it is illustrated that all components are connected to the processor 308. Although not shown in Figure 3, at least some of the components can be connected via a bus. In this case, data can be directly exchanged between some of the components under the control of the processor 308.

[0093] Figure 4 is a diagram illustrating the concept of a content streaming service according to one embodiment of the present disclosure. Figure 4 diagrams some of the functions related to content streaming, and content streaming services according to various embodiments may have various other functions in addition to those illustrated in Figure 4.

[0094] Referring to Figure 4, control data and content data can be transmitted and received between client 410 and server 420. Specifically, control data can be transmitted from client 410 to server 420, control data can be transmitted from server 420 to client 410, and content data can be transmitted from server 420 to client 410.

[0095] Server 420 stores user information 422a, content information 422b, and content database (DB) 422c. User information 422a may include user account information, information about the user's service usage history, and information about the user's preferences. Content information 422b may include a list of available content, content guide information, content metadata, and information about content consumption history. Content DB 422c may contain content stored in data format. In addition, server 420 may store other information necessary to provide services.

[0096] Control data from client 410 to server 420 may include information about user login, information about user content selection, and information about user content control. To this end, client 410 can generate and transmit control data from user input via user input processing operation 401. The control data from client 410 is processed by control / management operation 403 and used for content provision. For example, control / management operation 403 may select control data and / or content based on the control data from client 401. Furthermore, control / management operation 403 may analyze the user's consumption history and behavior to determine their preferences and select recommended content according to the determined preferences.

[0097] The procedure for providing content to the user will be explained below, with reference to Figure 4. First, client 410 performs user input processing operations. 401 The server generates control data containing login information (e.g., ID and password) entered by the user via the server and transmits that control data. The server 420 can determine whether the user is valid by searching for the login information contained in the control data from the client 410 in the user information 422a, and can determine the scope of content and services permitted according to the user's privileges. However, if login is not required, or if limited services that can be provided without login are supported, the transmission and processing of login information may be omitted.

[0098] Next, the server 410 extracts content guide information from content information 422b via control / management operation 403 and sends control data containing the content guide information to the client 410. The client 410 outputs the content guide information contained in the control data to confirm the user's selection. The user's selection is sent to the server 410 as control data via user input processing operation 401. Information regarding the user's selection is processed by control / management operation 403 and used to select the content to be streamed. The server 420 searches the content DB 422 for the selected content, compresses and segments the retrieved content by encoding operation 407, and then sends the content data. The content data can be pre-compressed and stored by encoding operation 407. Here, encoding operation 407 may include not only the operation of compressing the original content image, but also the operation of decrypting the content data generated by the compression and then compressing it again. In this case, compression can be performed based on the resolution, bitrate, and frames per second of the content image. If the content is pre-compressed and stored, the compression operation is omitted, and the server 420 can perform segmentation of the content data. The content data can be restored by the decoding operation 409 and provided to the user by the playback operation 411. At this time, at least one of various video codecs or various audio codecs can be used for compression. For example, the various video codecs may include at least one of MPEG-2 (Moving Picture Experts Group-2), H.264 / AVC (Advanced Video Coding), H.265 HEVC (High Efficiency Video Coding), H.266 / VVC (Versatile Video Coding), VP8 (Video Processor 8), VP9 (Video Processor 9), AV1 (AOMedia Video 1), DivX, Xvid, VC-1, Theora, or Daala.

[0099] Audio codecs can include MP3 (MPEG 1 Audio Layer 3), AC3 (Dolby Digital AC-3), E-AC3 (Enhanced AC-3), AAC (Advanced Audio Coding, MPEG 2 Audio), FLAC (Free Lossless Audio Codec), HE-AAC (High Efficiency Advanced Audio Coding), OGG Vorbis, and OPUS, among others.

[0100] Multiple content data sets can be pre-generated by compressing content images according to various image resolutions, bitrates, and frames per second. Client 310 can measure throughput (or bandwidth) and determine the bitrate based on the measured throughput (or bandwidth).

[0101] Client 410 can receive information about multiple content data from server 420. The received information may include information representing the bitrate, resolution, frames per second, and location of the multiple content data.

[0102] Client 410 can determine at least one of several content data based on the bitrate, and based on the capability information of Client 410, it can determine the playback content data corresponding to the playable resolution and frames per second, as well as its position, from among the at least one content data. In this case, the capability information may include, but is not limited to, the client's maximum supported resolution and maximum supported frames per second.

[0103] The client 410 can send a content request to the server 420 based on the position of the playback content data. Based on the received content request, the server 420 can send content data corresponding to the content request to the client 410.

[0104] In another embodiment, the client 410 can receive user input regarding at least one of the image resolution or the number of frames per second, determine the playback content data and its position in accordance with the user input, and send the content request to the server 420.

[0105] This disclosure relates to a technology for recommending content based on text metadata (hereinafter, "text metadata") that describes the content itself in a content streaming system. In particular, this disclosure relates to a method and apparatus for training a language model based on the text metadata of content, and for recommending content by determining the similarity between content based on the trained language model. The text metadata may include at least one of the following: title, synopsis, genre, director, actors, or hashtags.

[0106] Content recommendation techniques can be broadly categorized into two methodologies: one using a collaborative filtering model, and the other using a content-based filtering (CBF) model. The collaborative filtering methodology recommends content based on user-content interaction data. On the other hand, the CBF methodology recommends content similar to the user's preferred content. User preferred content can include content that the user has viewed, consumed, purchased, and / or selected. The CBF methodology has the advantage of being able to recommend content even without user-content interaction data, because it determines recommended content based on the characteristics of the content itself. In other words, using the CBF model can help resolve the cold start problem, where it is difficult to recommend content to new users or content that have no evaluation or purchase history. As a result, various content recommendation techniques based on the CBF model are currently available. For example, techniques can be used to recommend content via a circular neural network-based model that uses movie and rating metadata, to recommend content based on a matrix obtained by the inner product of user movie ratings and movie genre matrices, or to recommend content using a nearest neighbor model based on metadata such as images, audio, tags, and genres of the content. However, most of these existing CBF model-based techniques only utilize metadata that indicates the features of the content, and do not utilize text metadata that describes the content itself.

[0107] Accordingly, the following disclosure will describe various embodiments of recommending content by leveraging text metadata that describes the content itself, based on a methodology using a CBF model. For example, the text metadata that describes the content itself may include at least one of the following: title, synopsis, combined genre, director, actors, or hashtags. In embodiments of the disclosure, a language model may also be used for content recommendation, and the language model may be trained on the text metadata. The language model may be a transformer-based model, which is a natural language processing model for quantifying, i.e., embedding, the text metadata of the content so that it can be understood by a computer. For example, transformer-based models may include, but are not limited to, the BERT, ELECTRA, RoBERTa, BART, GPT3, DeBERTa, and KLUE-RoBERTa-large models.

[0108] Before describing specific methods for recommending content using language models, this disclosure explains the basic concepts of natural language processing and the RoBERTa model to aid in understanding the CBF model.

[0109] To determine content similarity based on the CBF model, metadata consisting of natural language, i.e., unstructured data, needs to be quantified as data that a computer can understand. The technique of quantifying, or vectorizing, unstructured data in natural language as data that a computer can understand is called embedding. Unstructured data in natural language can be represented as vectors through embedding, and these vectors can be mapped to a vector space, as shown in Figure 5. In this case, the distance and / or direction between vectors can be interpreted as relative relationship information between vectors. Figure 5 shows an example of the relative relationship between vectors. For example, in Figure 5, if we let vector 501 representing king be v1, vector 502 representing queen be v2, vector 505 representing man be v3, and vector 504 representing woman be v4, then since king and queen, and man and woman, have similar meanings related to gender, the distances v1, v2 and distances v3, v4 can be similar, and the directions v1, v2 and directions v3, v4 can be similar. In contrast, although not shown in Figure 5, if we consider the vector v5 to represent the computer, then the distances v1 and v5 will be greater than the distances v1 and v2, and the directions v1 and v5 will be different from the directions v1 and v2. In this way, the relative similarity between vectors can be determined. In the example in Figure 5, the embedding size, which is the length of the vector, was set in three dimensions, but in actual CBF models, the embedding size can be set to a higher multidimensional value. This is because if a vector has a multidimensional embedding size, it is possible to imbue the vector with more complex meaning.

[0110] In a CBF model that represents content as vectors, it is crucial that the vectors accurately represent the semantic information of the content. This is because accurate similarity between content can only be determined if the vectors accurately represent the semantic information of the content. Therefore, according to embodiments of this disclosure, in order to represent content with vectors that have accurate semantic information, the system will fine-tune the language model of the CBF model by training it. Specifically, in various embodiments of this disclosure, the language model can be trained to convert an input text sequence containing metadata such as the title and synopsis of each piece of content into vectors that have accurate semantic information.

[0111] Language models are models that have the ability to vectorize input text and can be divided into word-level embedding models and sentence or document-level embedding models. Word-level embedding models assign the same vector to words of the same form, such as the word2vec model. Sentence-level embedding models distinguish each word by taking contextual information into account, such as the BERT model.

[0112] JPEG2026509732000045.jpg52161

[0113] The RoBERTa model is an evolution of the BERT model. The BERT model, the predecessor of the RoBERTa model, is a language model pre-trained on large amounts of text data through unsupervised learning. The BERT model has a structure in which transformer-structured encoder blocks are stacked in multiple layers, and is pre-trained using the MLM (masked language model) and NSP (next sentence prediction) methods. A detailed explanation of the transformer structure and the BERT model structure can be found in the diagram. 18A ,figure 18B and figure 19 See below for further details.

[0114] The MLM method predicts randomly masked words, while the NSP method predicts whether two sentences can appear consecutively in context. The BERT model has the advantage of obtaining better semantic representation information compared to one-way models because it has a bidirectional text learning structure.

[0115] RoBERTa is a model trained after adding training data and adjusting hyperparameters and training techniques to enhance the performance of the BERT model. The RoBERTa model can be trained using only MLM methods, excluding NSP methods. Compared to the BERT model, the RoBERTa model has undergone longer training with larger training data and longer sequences, and has been improved to obtain more sophisticated semantic representation information by applying dynamic masking. In other words, RoBERTa has been improved to have better performance on the GLUE (general language understanding evaluation) benchmark than previous models, including BERT.

[0116] Therefore, the system according to the embodiments of this disclosure can use the RoBERTa model, a natural language processing model pre-trained on a Korean corpus for content recommendation. However, the language model in the embodiments described below is not necessarily limited to the RoBERTa model, and the system can also be applied to other language models besides RoBERTa.

[0117] Figure 6 shows an example of the structure of a server that recommends content according to one embodiment of the present disclosure. At least some of the components of the server shown in Figure 6 (for example, server 120 in Figure 1) can be understood as components included in the processor 308 in Figure 3. The following description of at least some of the components of Figure 6 will be given with reference to Figures 7A to 13.

[0118] Referring to Figure 6, the server 120 may include a content storage unit 610, a model learning unit 620, a similarity determination unit 630, and a content determination unit 640.

[0119] The content storage unit 610 stores content items that can be provided to the client. Content items include movie content, drama content, and program content that can be streamed, with each content item corresponding to one movie, one drama, or one program. For example, the first content item and the second content item may correspond to different movies. However, in other embodiments, the content storage unit 610 may reside outside the server 120, in which case the server 120 can connect to the external content storage unit 610 and retrieve content items.

[0120] According to one embodiment, the content storage unit 610 may include a content vector DB 612. The content vector DB 612 stores the vector value of each content item stored in the content storage unit 610. The vector value of each content item can be obtained using a language model learned by the model learning unit 620. The content vector DB 612 can be updated by the updated language model when the language model is updated. For example, the language model can be updated by being retrained when a new content item is stored in the content storage unit 610 or when an already stored content item is deleted. That is, when the language model is retrained and updated, the content vector DB 612 can obtain and store the vector value of each content item using the updated language model. In this case, the vector values ​​of previously stored content items can be deleted.

[0121] According to one embodiment, the content vector DB 612 may be updated periodically or automatically when a specified event occurs, or it may be updated under the control of the operator and / or administrator. For example, when a new content item is stored in the content storage unit 610, the content vector DB 612 may be updated to add the vector value of the new content item. As another example, when a content item already stored in the content storage unit 610 is deleted, the content vector DB 612 may be updated to delete the vector value of the deleted content item.

[0122] The model learning unit 620 can train a language model based on text metadata that describes the content of a content item. Text metadata refers to text features that describe the content of a content item. Text metadata can include at least one of the following: title, synopsis, composite genre, director, actors, and hashtag information of the content item. Here, composite genre can include at least one of a major genre and a minor genre. For example, the minor genres of the major genre "Action / SF" are divided into "Action," "Fantasy," "SF," "Adventure," "War," "Wuxia," etc. Hashtag information refers to tag information that represents at least one of the subject, sentiment, or purpose of the content item. Synopsis refers to summary information that represents at least one of the subject, intention, or plot of the content item.

[0123] According to one embodiment, the model learning unit 620 may include a preprocessing unit 710 and a learning unit 720, as shown in Figure 7A, or it may include a preprocessing unit 750, a first learning unit 760, and a second learning unit 770, as shown in Figure 7B. Figures 7A and 7B show an example of the structure of a model learning unit according to one embodiment of the present disclosure.

[0124] First, referring to Figure 7A, the preprocessing unit 710 of the model learning unit 620 acquires text metadata of content items for training the language model and converts the acquired text metadata into sequence-type text data. Sequence-type text data means data in a string format in which text data is connected sequentially. The reason the preprocessing unit 710 converts text metadata into sequence-type text data is that text data that is divided into unstructured data, such as the metadata of content items, cannot be directly input into the language model. Therefore, the preprocessing unit 710 can convert text metadata into sequence-type text data by dividing the text metadata of content items into token units and then inserting at least one delimiter. Here, a token means an input unit of the language model that is replaced by a unique embedding value, and the at least one delimiter that is inserted can also be treated as a token. The at least one delimiter may include at least one of the following: a separation token for distinguishing different types of features from each other (e.g., [SEP]) and a special token representing a specific feature. A special token may include, for example, at least one of the following: [GENRE] and [ / GENRE], which represent genres; [DIR] and [ / DIR], which represent directors; [ATR] and [ / ATR], which represent actors; and [TAG] and [ / TAG], which represent hashtags. The listed special tokens are merely examples to aid understanding, and embodiments of this disclosure are not limited thereto. Each special token can be inserted before or after the text corresponding to the feature. The use of special tokens in this disclosure is due to the fact that the text metadata of content items contains various types of features; that is, it may be difficult for a language model to recognize various types of features solely from the isolated tokens and / or the order of isolated tokens contained in the input sequence. Special tokens can be added to the lexical dictionary of a language model.

[0125] According to one embodiment, the preprocessing unit 710 can convert text metadata, including the content item's identification code, title, genre, director, actor, hashtags, and synopsis, into sequence-type text data including delimiters, as shown in Table 1 below.

[0126] [Table 1]

[0127] In Table 1, Synopsis Token 1, Synopsis Token 2, and Synopsis Token N each represent distinct tokens included in the synopsis of the content item. As a specific example, the preprocessor 710 can generate sequenced text data as shown in Figure 8. Figure 8 shows an example of converting the text metadata of content into sequenced text data according to one embodiment of the present disclosure. Referring to Figure 8, the preprocessor 710 can convert the text metadata 810 of the content item into sequenced text data 820 by adding isolated tokens and special tokens. In this case, if there are multiple directors and / or actors for the content item, the preprocessor 710 can limit the number of directors and / or actors included in the sequenced text data. For example, the number of directors and / or actors can be limited to a maximum of 5 or fewer, but is not limited thereto. The preprocessor 710 provides the generated sequenced text data to the first learning unit 720.

[0128] The learning unit 720 of the model learning unit 620 performs training on the language model based on sequence-type text data. That is, the learning unit 720 can perform training on the language model by training the predictive model based on a specific type of information from the sequence-type text data acquired by the preprocessing unit 710. The specific type of information may include hashtag information, genre information, or synopsis information. Specifically, the learning unit 720 can perform any of the following first to third embodiments.

[0129] First Embodiment

[0130] According to the first embodiment, the learning unit 720 can train a language model by training a prediction model based on hashtag information in sequence-type text data. Here, the prediction model may include a hashtag prediction model, which is an MLM-type prediction model configured to predict or infer masked hashtag tokens based on the language model. For example, the learning unit 720 can train a language model as shown in Figure 9A. Figure 9A shows an example of language model training according to one embodiment of the present disclosure.

[0131] Referring to Figure 9A, the learning unit 720 can mask one token (e.g., "tag 2") from the sequence-type text data that corresponds to a hashtag, and define the value of the masked token as a label. The learning unit 720 can input the text data 910 containing the masked token 901 into the hashtag prediction model 920, determine a loss value using the output value and label, and perform backpropagation based on the loss value to train and / or learn the hashtag prediction model 920. This allows the hashtag prediction model 920 to be trained and / or learned to predict 930 and / or infer the value of the masked token 901. In this case, the hashtag prediction model 920 can acquire contextual information from other unmasked tokens and be trained or learned to infer the masked token, i.e., the token corresponding to the hashtag, based on the acquired contextual information. For example, the hashtag prediction model 920 can learn based on contextual information acquired from unmasked tokens such as titles and synopses. Thus, the inputs and targets for the language model-based hashtag prediction model 920's training task can be shown in Table 2 below.

[0132] [Table 2]

[0133] Table 2 shows that when the "Tag 2" token among multiple tokens located in the hashtag region is masked and input to the hashtag prediction model 920, the hashtag prediction model 920 learns to infer the "Tag 2" token. Here, even though multiple tokens exist in the hashtag region, only one token is masked because it is not easy for the language model to grasp the positional relationship between the masked token and the target token included in the input when two or more tokens are masked. Therefore, the learning unit 720 according to the first embodiment can operate in a manner that masks and infers one token in the hashtag region, and then masks and infers other tokens in the hashtag region. For example, the tokens masked in the hashtag region may differ from epoch to epoch. According to the first embodiment, the learning unit 720 can mask tokens located in the hashtag region that do not start with "#", i.e., independent tokens. The hashtag region can be determined based on the special tokens [TAG] and [ / TAG] that indicate hashtags.

[0134] Second Embodiment

[0135] According to the second embodiment, the learning unit 720 can train a language model by training a predictive model based on synopsis information in sequence-type text data. Here, the predictive model may include a synopsis prediction model, which is an MLM-type predictive model configured to predict or infer masked synopsis tokens based on the language model. For example, the learning unit 720 can train a language model as shown in Figure 9B. Figure 9B shows an example of language model training according to one embodiment of the present disclosure.

[0136] Referring to Figure 9B, the learning unit 720 can mask one token in the sequence-type text data that corresponds to the synopsis (e.g., "Synopsis Token 1") and define the value of the masked token as a label. The learning unit 720 can input the text data 950 containing the masked token 951 into the synopsis prediction model 960, determine a loss value using the output value and label, and perform backpropagation based on the loss value to train and / or learn the synopsis prediction model 960. As a result, the synopsis prediction model 960 can be trained and / or learned to predict and / or infer the value of the masked token 951. In this case, the synopsis prediction model 960 can be trained or learned to obtain contextual information from other unmasked tokens and infer the masked token, i.e., the token corresponding to the synopsis, based on the obtained contextual information. For example, the synopsis prediction model 960 can learn based on contextual information obtained from unmasked tokens such as titles, genres, and hashtags. Thus, the inputs and targets for the training task of a language model-based synopsis prediction model can be shown in Table 3 below.

[0137] [Table 3]

[0138] Table 3 shows that when the "Synopsis Token 1" token, one of several tokens located in the synopsis domain, is masked and input to the synopsis prediction model 960, the synopsis prediction model 960 is trained to infer the "Synopsis Token 1" token. Here, even though multiple tokens exist in the synopsis domain, only one token is masked because it is not easy for the language model to grasp the positional relationship between the masked token and the target token included in the input when two or more tokens are masked. Therefore, the learning unit 720 according to the second embodiment can operate in a manner that masks and infers one token in the synopsis domain, and then masks and infers other tokens in the synopsis domain. For example, the token to be masked in the synopsis domain may change from epoch to epoch. The learning unit 720 according to the second embodiment is not limited to masking and inferring tokens in the synopsis domain, but can also mask and infer tokens in the title domain. For example, the learning unit 730 can mask and infer tokens in the title region in addition to the synopsis region. Alternatively, the learning unit 730 can mask and infer tokens in the title region instead of the synopsis region.

[0139] According to the second embodiment, the learning unit 720 can mask tokens located in the synopsis region that do not begin with "#", i.e., independent tokens. The synopsis region can be determined based on isolated tokens and / or special tokens. For example, the synopsis region can be determined as the region between the isolated token [SEP] and the special token for the genre [GENRE]. However, this is merely an example for cases where the text metadata of a content item is converted into sequence-type text data as shown in Table 1, and the method of determining the synopsis region is not limited to this. For example, if the sequence-type text data consists of "Title [SEP] Director [SYNOPSIS] Synopsis Token 1 Synopsis Token 2...Synopsis Token N [ / SYNOPSIS] [GENRE] Genre 1 Genre 2 [ / GENRE] [ATR] Actor 1 Actor 2 [ / ATR] [TAG] Tag 1 Tag 2 [ / TAG]", the synopsis region can be determined as the region between the special tokens [SYNOPSIS] and [ / SYNOPSIS] that indicate the synopsis. In other words, the synopsis domain can vary depending on the structure of the sequence-type text data.

[0140] In the first and second embodiments described above, the reason for masking tokens that do not begin with "#" is that, due to the BPE (Byte Pair Encoding) tokenizer characteristics of the RoBERTa model, tokens that begin with "#" are either dependent on the preceding token or are tokens with grammatical meaning. That is, since tokens containing core meaning, such as nouns and verbs, do not begin with "#", the learning unit 720 can mask tokens that do not begin with "#" among the tokens located in the synopsis domain. For example, when the BPE tokenizer divides a text sentence into token units, it can divide "Mr. / Ms. XX is employed in the interesting OTT field of Teabing." into "Mr. / Ms. XX is employed in the interesting OTT field of Teabing." As illustrated above, the tokenizer can indicate that a token is dependent on the preceding token by adding "#" to the dependent token.

[0141] The method for representing dependent tokens is not limited to adding a "#" to the token. For example, other tokenizers can add "##" or "_" to dependent tokens, and various other methods can be used to indicate that a token is a dependent token. Therefore, the form of dependent tokens is not limited to a specific form, and the learning unit 720 can mask tokens that are not dependent tokens.

[0142] According to one embodiment, the hashtag prediction model 920 and / or synopsis prediction model 960 may include, as shown in Figure 9C, a masking block 921 that masks at least one of a plurality of input tokens (e.g., [W1, W2, W3, W4, W5]), a language model 922 that outputs vector values ​​(e.g., [O1, O2, O3, O4, O5]) corresponding to a plurality of input tokens (e.g., [W1, W2, W3, [MASK], W5]) including the masked token, a classification layer 923 that infers the vector value of the masked token from the vector values ​​output from the language model, and an embedding to vocabulary layer 924 that converts the vector values ​​into tokens. Here, the language model 922 may include a RoBERTa model. The classification layer 923 may also include a fully connected layer, a GELU (Gaussian Error Linear Unit), and a norm, and may be called an MLM head layer. The classification layer 923 can output prediction tokens (e.g., [W'1, W'2, W'3, W'4, W'5]) corresponding to multiple input vector values ​​(e.g., [O1, O2, O3, O4, O5]). 。 The predictive model 920 can be trained to predict and / or infer the target, a content-appropriate masked token (e.g., W4), based on contextual information from unmasked tokens (e.g., [W1, W2, W3, W5]), without overlapping with the unmasked tokens.

[0143] Third Embodiment

[0144] According to the third embodiment, the learning unit 720 can train a language model by training a prediction model based on genre information in sequence-type text data. In this case, the prediction model may include a genre prediction model, which is a prediction model of a text classification scheme configured to predict or infer the genre of a content item based on the language model. For example, the learning unit 720 can train a language model as shown in Figure 10A. Figure 10A shows an example of language model training according to one embodiment of the present disclosure.

[0145] Referring to Figure 10A, the learning unit 720 can predict the genre to which a content item with input sequence-type text data belongs by acquiring input sequence-type text data that does not contain genre-related tokens and performing a text classification task using the genre prediction model 1020. The text classification task means classifying which class the text input to the prediction model belongs to. Here, input sequence-type text data can be generated by removing genre-related tokens from sequence-type text data. Genre-related tokens can include special tokens that represent genres, such as [GENRE] and [ / GENRE], and tokens that correspond to genre information (hereinafter, "genre tokens"). Genre tokens are located in the genre area between the special tokens that represent genres, [GENRE] and [ / GENRE], and can contain at least one token that represents the genre. For example, a genre token that represents the genre "horror / thriller" can contain three tokens, "horror", " / ", and "thriller", and a genre token that represents the genre "drama" can contain one token, "drama". Input sequence-type text data can be generated by the preprocessing unit 710 or the learning unit 720.

[0146] Specifically, the learning unit 720 can obtain at least one token representing at least one genre from sequence-type text data and set a class label based on the obtained at least one token. Here, one genre can be represented by one or more tokens. For example, the genre "horror / thriller" can be represented by three tokens: "horror," " / ," and "thriller," and the genre "drama" can be represented by one token: "drama." Therefore, if the learning unit 720 obtains one or more tokens representing one genre from sequence-type text data, it can set a class label to predict one genre based on the obtained one or more tokens. Furthermore, if the learning unit 720 obtains multiple tokens representing multiple genres from sequence-type text data, it can set a class label to predict multiple genres based on the obtained multiple tokens. Depending on the number of genres to be predicted, the learning unit 720 can use a multi-class classification model or a multi-label classification model, which will be described later in Figure 10c.

[0147] The learning unit 720 can train and / or learn the genre prediction model 1020 by inputting the input sequence-type text data 1010, which does not contain genre-related tokens, into the genre prediction model 1020, determining a loss value (e.g., cross-entropy) using the output value of the genre prediction model 1020 and pre-set class labels, and performing backpropagation based on the loss value. As a result, the genre prediction model 1020 can be trained and / or learned to predict 1030 and / or infer at least one genre set as a class label from the input sequence-type text data 1010.

[0148] According to the third embodiment, the genre prediction model 1020, as shown in Figure 10B, uses vector values ​​(e.g., [C, T1, T2, ..., T) corresponding to input tokens (e.g., [CLS, Tok1, Tok2, ..., TokN]). N The system may include a language model 1021 that outputs ]), and a classification layer 1027 that outputs probability values ​​for class labels based on at least one vector value output from the language model 1021. Here, the language model 1021 may include a RoBERTa model. The classification layer 1027 may also be called a text classification layer and / or text classification head layer.

[0149] As shown in Figure 10B, the learning unit 720 can obtain a genre prediction result for the content from the prediction model 1020 by inputting input sequence-type text data 1010 that does not contain genre-related tokens into the prediction model 1020. At this time, the input sequence-type text data 1010 may contain multiple tokens Tok1, Tok2, ..., TokN1010-1, 1010-2, ..., 1010-N. The learning unit 720 can add a start token [CLS]1011 to the start position of the input sequence-type text data 1010 and input it into the language model 1021. The language model 1021 inputs a final hidden vector C1023 corresponding to the start token [CLS]1011, as well as final hidden vectors T1, T2, ..., T1010-N corresponding to the multiple tokens Tok1, Tok2, ..., TokN1010-1, 1010-2, ..., 1010-N. NThe final hidden vector C1023 can output 1025-1, 1025-2, ..., 1025-N. The final hidden vector C1023 may be an output vector that reflects the contextual information of all the tokens Tok1, Tok2, ..., TokN1010-1, 1010-2, ..., 1010-N contained in the input sequence-type text data 1010. The final hidden vector C1023 is input to the classification layer 1027, which can output probability values ​​of class labels based on the final hidden vector C1023. The learning unit 1020 can predict the class, i.e., genre, to which the content belongs based on the output probability values ​​of class labels. According to one embodiment, the classification layer 1027 can also use only the final hidden vector C1023 as input, and the final hidden vector C2023 and other final hidden vectors T1, T2, ..., T N 1025-1, 1025-2, ..., 1025-N can also be used together as input. For example, the classification layer 1027 uses the final hidden vectors T1, T2, ..., T output from the language model 1021. N It can accept mean pooling inputs for 1025-1, 1025-2, ..., 1025-N and output probability values ​​for class labels based on this input.

[0150] As mentioned above, the genre prediction model 1020 can be trained or learned to extract contextual information from all tokens contained in the input sequence-type text data 1010 and to infer the genre based on the extracted contextual information. For example, the genre prediction model 1020 can be learned based on contextual information extracted from tokens such as title, synopsis, and hashtags. Thus, the inputs and targets for the training task of a language model-based genre prediction model can be shown in Table 4 below.

[0151] [Table 4]

[0152] Table 4 shows that when sequential text data is input to the prediction model, the genre prediction model is trained to infer tokens for "Genre 1" and "Genre 2". Here, the target means the class label, and the reason there are multiple targets for "Genre 1" and "Genre 2" is that the content item can belong to more than one genre. For example, a particular content item may belong to the "Action / SF" genre in the major category genres while also belonging to the "Fantasy" genre in the minor category genres. In general, genres for a content item can be divided into major category genres and / or minor category genres. Major category genres may include drama, romance / melodrama, comedy, action / SF, horror / thriller, etc. Minor category genres may include drama, action, thriller, romance, comedy, horror, fantasy, SF, crime, period drama, war, martial arts, etc. The genres listed are merely examples to aid understanding, and embodiments of this disclosure are not limited thereto. As mentioned above, genres for a content item can be divided in various ways, and a single content item can belong to one or more genres. Therefore, the genre prediction model according to the third embodiment can be trained to infer only one genre to which a content item belongs, or to infer one or more genres to which a content item belongs. For example, the prediction model 1020 can be trained to infer one or more genres to which a content item belongs by including a supervised learning algorithm-based multi-class classification model or a multi-label classification model, as shown in Figure 10C.

[0153] Figure 10C illustrates the concepts of multi-class and multi-label classification models applicable to this disclosure. In Figure 10C, C can represent the number of classes. That is, Figure 10C assumes the existence of three classes 1001, 1003, and 1005.

[0154] The multi-class classification model 1040 is a model for inferring one class among multiple classes to which an input sample belongs. Therefore, the label of the multi-class classification model 1040, i.e., the target vector t, can be set to a one-hot vector having one positive class and C-1 negative classes. For example, the label for the first input sample 1041 of the multi-class classification model 1040 can be set to

[0001] , the label for the second input sample 1043 can be set to

[0100] , and the label for the third input sample 1043 can be set to

[0010] . Here, the label is the expected output vector value for the input sample and can be set based on the class to which the input sample actually belongs. For example, setting the label to

[0100] could mean that the input sample belongs to the actual first class 1001 but not to the second class 1003 or third class 1005, and setting the label to

[0010] could mean that the input sample belongs to the actual second class 1003 but not to the first class 1001 or third class 1005. Also, setting the label to

[0001] could mean that the input sample belongs to the actual third class 1005 but not to the first class 1001 or second class 1003.

[0155] The multi-label classification model 1050 is a model for inferring multiple classes to which an input sample belongs among multiple classes. The labels of the multi-label classification model, i.e., the target vector t, can be set to a vector having multiple positive classes. For example, the label for the fourth input sample 1051 of the multi-label classification model can be set to

[0101] , the label for the fifth input sample 1053 can be set to

[0010] , and the label for the sixth input sample 1055 can be set to

[0111] . Here, the labels are expected output vector values ​​for the input sample and can be set based on one or more classes to which the input sample actually belongs. For example, setting the label to

[0101] means that the input sample belongs to the actual first class 1001 and third class 1005, setting the label to

[0010] means that the input sample belongs to the actual second class 1003, and setting the label to

[0111] means that the input sample belongs to the actual first class 1001, second class 1003, and third class 1005.

[0156] The learning unit 720 can be trained to infer one or more genres to which each content item belongs via a genre prediction model 1020 configured based on a multi-class classification model 1040 or a multi-label classification model 1050, as shown in Figure 10C. With the structure described above, the more accurately the genre prediction model infers the target, the more sophisticated the semantic representation of the language model can become.

[0157] Next, referring to Figure 7B, the preprocessing unit 760 of the model learning unit 620 acquires text metadata of content items for training the language model and converts the acquired text metadata into sequence-type text data. That is, the preprocessing unit 760 can convert text metadata, including the content item's identification code, title, genre, director, actors, hashtags, and synopsis, into sequence-type text data including delimiters, as shown in Table 1. In other words, the preprocessing unit 760 in Figure 7B can perform at least one operation that can be performed by the preprocessing unit 710 in Figure 7A.

[0158] The first learning unit 770 performs primary training on the language model using a predictive model configured to predict or infer masked tokens. The first learning unit 770 can perform primary training on the language model by training the predictive model based on a specific type of information from the sequence-type text data of content items acquired by the preprocessing unit 760. According to one embodiment, the first learning unit 770 can train the predictive model based on hashtag information in the sequence-type text data of content items. For example, as shown in Figure 9a, the first learning unit 770 can train the language model based on hashtag information. That is, the first learning unit 770 can perform primary training on the language model based on hashtag information using a hashtag predictive model 920, which is an MLM predictive model. As another example, as shown in Figure 9B, the first learning unit 770 can train the language model based on synopsis information. That is, the first learning unit 770 can train the language model based on synopsis information using a synopsis predictive model 960, which is an MLM predictive model. The second learning unit 730 performs secondary learning on the language model using a predictive model configured to predict or infer masked tokens. That is, the second learning unit 730 performs secondary learning, which is additional learning to the language model that has been primarily learned by the first learning unit 720. The second learning unit 730 can perform secondary learning on the language model by using an MLM predictive model to perform additional training on the primarily learned language model based on other types of information from the sequence-type text data of content items acquired by the preprocessing unit 720 that were not used in the primary learning. According to one embodiment, if the primary learning was performed based on hashtag information, the secondary learning can be performed based on synopsis information in the sequence-type text data of content items. For example, as shown in Figure 9B, the second learning unit 780 can perform secondary learning on the language model based on synopsis information using a synopsis prediction model 960, which is an MLM predictive model.According to one embodiment, if primary learning is performed based on synopsis information, secondary learning can be performed based on hashtag information within the sequence-type text data of content items. For example, as shown in Figure 9a, the second learning unit 780 can perform secondary learning on the language model based on hashtag information using a hashtag prediction model 920, which is an MLM-based prediction model.

[0159] According to one embodiment, the second learning unit 780 can perform secondary learning using the text metadata of the content items used for learning by the first learning unit 770. According to one embodiment, the second learning unit 780 can select at least some of the content items used for learning by the first learning unit 770 that have information of the type used for secondary learning, and perform secondary learning using the text metadata of the selected at least some of the content items. For example, if hashtag information is used for secondary learning of a language model, the second learning unit 780 can select only the content items that have hashtag information from the content items used for learning by the first learning unit 770, and perform secondary learning using the text metadata of the selected content items as a training dataset for a prediction model. As another example, if synopsis information is used for secondary learning of a language model, the second learning unit 780 can select only the content items that have synopsis information from the content items used for learning by the first learning unit 770, and perform secondary learning using the text metadata of the selected content items as a training dataset for a prediction model. However, this is merely an example, and the training dataset used for training the second learning unit 780 is not limited to this.

[0160] In the explanation with reference to Figure 7B, the model learning unit 620 performs either primary learning based on hashtags using an MLM predictive model followed by secondary learning based on synopsis, or primary learning based on synopsis followed by secondary learning based on hashtags. However, the disclosure is not limited thereto. That is, the model learning unit 620 can perform Nth-order learning using at least two types of information from among the various types of information contained in the sequence-type text data of the acquired content items. For example, the model learning unit 620 can perform primary learning based on synopsis or hashtags using an MLM predictive model, followed by secondary learning based on genres using a text classification predictive model. As another example, the model learning unit 620 can perform primary learning based on genres using a text classification predictive model, followed by secondary learning based on hashtags or synopsis using an MLM predictive model. As another example, the model learning unit 620 can perform synopsis-based primary learning using an MLM predictive model, hashtag-based secondary learning using an MLM predictive model, and then genre-based tertiary learning using a text classification predictive model.

[0161] In the structure described above, the more accurately the predictive model infers the target, the more sophisticated the semantic representation of the language model becomes, thereby enabling more accurate calculation of the similarity between content items. For example, as shown in Figure 11A, when training or learning to predict hashtags, content items with identical predicted values ​​in terms of hashtags can be embedded in similarity. Figure 11A shows an example of the relationship between predicted values ​​and similarity for each content item according to one embodiment of the present disclosure. Referring to Figure 11A, the similarity 1106a between content 1 and content 2, which have the same hashtag, may be greater than the similarity 1106b between content 2 and content 3, which have different hashtags.

[0162] To give another example, as shown in Figures 11B and 11C, when training or learning to predict genres, content items with identical predicted values ​​in terms of genre can be embedded as similar. Figure 11B shows an example of the relationship between predicted values ​​and similarity for each content according to one embodiment of the present disclosure. Figure 11B shows the case where the predicted genres of Content 1 and Content 2 are "Genre 1" and the predicted genre of Content 3 is "Genre 2". Referring to Figure 11B, the similarity 1116a between Content 1 and Content 2, which have the same predicted genre, may be greater than the similarity 1116b between Content 2 and Content 3, which have different predicted genres. Figure 11C shows another example of the relationship between predicted values ​​and similarity for each content according to one embodiment of the present disclosure. Figure 11C shows the case where the predicted genres of Content 1 and Content 2 are "Genre 1" and "Genre 2" and the predicted genre of Content 3 is "Genre 2" and "Genre 4". 11C Referring to the data, the predicted genre of Content 1 is the same as the predicted genre of Content 2. In contrast, the predicted genre of Content 3 is partially the same as that of Content 2, and partially different. Therefore, the similarity score 1126a between Content 1 and Content 2, which share the same predicted genre, may be greater than the similarity score 1126b between Content 2 and Content 3, which share some other predicted genres.

[0163] To give another example, when training or learning to predict hashtags and synopsis, as shown in Figures 11D and 11E, content items that have the same predicted values ​​in terms of hashtags and / or synopsis can be embedded in similarity. Figures 11D and 11E show an example of the relationship between content-specific predicted values ​​and similarity according to one embodiment of the present disclosure. Referring to Figure 11D, the similarity 1136a between content 1 and content 2 having the same hashtag may be greater than the similarity 1136b between content 2 and content 3 having different hashtags. However, if the hashtags of content 2 and content 3 have similar meanings, the similarity between content 2 and content 3 may be determined to be as high as the similarity between content 1 and content 2 having the same hashtag. For example, if the hashtag for Content 2 is "exorcism" and the hashtag for Content 3 is "occult," even though "exorcism" and "occult" are not composed of the same tag token, their meanings are similar. Therefore, the similarity between Content 2 and Content 3 can be determined to be higher than the similarity when they have different hashtags with different meanings. This is because the predictive model incorporates context information based on the "interrelationships" between tokens into its spatial learning. In other words, the predictive model can learn to judge content as similar even if the tags of each piece of content are different, as long as their meanings are similar. That is, even if two pieces of content have different hashtags, if those hashtags are semantically similar, the similarity between the two pieces of content may be even higher than the similarity between content with completely different hashtags. Furthermore, the similarity between two pieces of content can be calculated to be a high value, comparable to the similarity between content with the same tag.

[0164] Referring to Figure 11E, if Content 1, Content 2, and Content 3 have the same hashtag, the similarity 1146a between Content 1 and Content 2, which have the same synopsis token, may be greater than the similarity 1146b between Content 2 and Content 3, which have different synopsis tokens.

[0165] In the above description, the model learning unit 620 trained the language model by training an MLM predictive model based on hashtag information or synopsis information, or by training a text classification predictive model based on genre information. However, the disclosure is not limited thereto. According to one embodiment, the model learning unit 620 can also train the language model by training an MLM predictive model based on other types of information other than hashtag information and synopsis information, or by training a text classification predictive model based on other types of information other than genre information. For example, the model learning unit 620 can train the language model using other information that can reflect the user's content preferences. Table 5 below shows examples of expressions for user preferred content.

[0166] [Table 5]

[0167] Table 5 shows that user content preferences can be reflected in content genre, hashtags, director, or actor information. 5As shown, director or actor information reflects the user's content preferences. However, there is a vast amount of information that corresponds to director or actor information, and it is rare for content to have the same director or actor information, making it difficult to learn a generalized semantic representation for director or actor information. In contrast, hashtag or genre information, while reflecting the user's content preferences, has relatively less information compared to other features (e.g., director, actor), and content often shares the same genre and / or hashtags. Furthermore, genre information appears for each individual data within a defined category, and the main nouns corresponding to hashtag information are largely learned in the pre-training step. Therefore, it can be said that it is easy to learn a generalized semantic representation for genre or hashtag information.

[0168] The similarity determination unit 630 can determine the similarity between content items using the language model learned by the model learning unit 620. The similarity determination unit 630 can acquire text metadata for each content item and convert the acquired text metadata into sequence-type text data. The similarity determination unit 630 can use the learned language model to acquire content item-specific vector values ​​from the sequence-type text data acquired for each content item. Then, the similarity determination unit 630 can determine the similarity between content items by comparing the content item-specific vector values.

[0169] For example, the similarity determination unit 630 can determine similarity as shown in Figure 12. Figure 12 shows an example of calculating similarity between contents using a language model trained according to one embodiment of the present disclosure. Referring to Figure 12, the similarity determination unit 630 can use RoBERTa model 1220-1 to obtain vector 1204a of content 1 from the sequenced text data of content 1, <content1data>1202a, and use RoBERTa model 1220-2 to obtain vector 1204b of content 2 from the sequenced text data of content 2, <content2data>1202b. Here, it is stated that two RoBERTa models 1220-1 and 1220-2 are used, but this is to emphasize that one vector is obtained for each content data, and the similarity determination unit 630 can use one RoBERTa model repeatedly or process in parallel. That is, the two RoBERTa models 1220-1 and 1220-2 can be the same one model. The similarity determination unit 630 can calculate the similarity between vector 1204a of content 1 and vector 1204b of content 2 using the similarity calculation block 1240, which calculates the similarity of vectors. For example, the similarity calculation block 1240 can calculate the similarity based on the cosine similarity algorithm. The similarity between vector 1204a of content 1 and vector 1204b of content 2 can be interpreted as the similarity between content 1 and content 2, which is 1206.

[0170] According to one embodiment, the similarity determination unit 630 can remove the MLM head layer from the prediction model used in the secondary learning unit 730 of the model learning unit 620 and determine the vector values ​​for the sequence-type text data of the content using the embedding values ​​of the final hidden layer of the language model. In other words, the model used to determine similarity and the model used for fine-tuning can have different structures from each other. That is, the model in the learning step for fine-tuning includes an MLM head layer for predicting masked tokens, while the model in the step for determining similarity does not include an MLM head layer and may further include a similarity calculation block.

[0171] The similarity determination unit 630 can obtain a vector for each content item, i.e., an input text vector used for calculating similarity, through various embodiments. Embodiments for determining the input text vector are as follows:

[0172] In one embodiment, a method using pooler output can be applied. Specifically, when using pooler output, the output vector of the final hidden layer of the [CLS] token in the language model is used as the input text vector.

[0173] In one embodiment, a method using the average of the last hidden state values ​​can be applied. When using the average of the last hidden state values, the vector obtained by average pooling for the output vectors of the last hidden layer of all words in the language model is used as the input text vector.

[0174] In one embodiment, a method using the maximum value of the final hidden state can be applied. When using the maximum value of the final hidden state, the vector obtained by max pooling for the output vectors of the final hidden layer of all words in the language model is used as the input text vector.

[0175] Among the various embodiments described above, the similarity determination unit 630 can obtain an input text vector for similarity calculation according to a method that uses the average of the final hidden state values. This is the similarity test set classification criterion described later, and experimental results against the above-mentioned methods confirmed that the method that uses the average of the final hidden state values ​​has the highest performance. Specifically, when comparing the method that uses the maximum value of the final hidden state and the method that uses the puller output with the method that uses the average of the final hidden state values, both the first-kind precision and the second-kind precision decreased. Here, first-kind precision refers to the precision calculated by determining that a correct judgment was made if the similarity between a reference content item and a similar content item is higher than the similarity between a reference content item and a content item that is not very similar to the reference content item, and second-kind precision refers to the precision calculated by determining that a correct judgment was made if the similarity between a reference content item and a similar content item is higher than the similarity between a reference content item and a dissimilar content item.

[0176] Furthermore, the similarity determination unit 630 can assign weights to the positions of specific features among the final hidden state values ​​of the language model. Embodiments of weighting are as follows. In the following description, we assume the application of a weight of 2 (e.g., 2x) as an example to aid understanding, but the weight is not limited to 2. For example, the weight may be k, and k may be a real number greater than 1.

[0177] In one embodiment, a method of assigning weights to hashtag values ​​can be applied. In this case, among the vector values ​​of the final hidden layer, the vector values ​​corresponding to the tokens located between [TAG] and [ / TAG], which are special tokens representing the hashtag region, can be weighted twice.

[0178] In one embodiment, a method of weighting genre values ​​can be applied. In this case, among the vector values ​​of the final hidden layer, the vector values ​​corresponding to tokens located between [GENRE] and [ / GENRE], which are special tokens representing genre regions, can be weighted twice. For example, after averaging the vector values ​​corresponding to the tokens, the average can be re-averaged only for the vectors at genre locations, and then this average can be added to the average pooling result. However, embodiments of this disclosure are not limited thereto. For example, weights can be applied to each feature's location during average pooling, and the weighted average can be calculated.

[0179] In one embodiment, a method can be applied to weight the values ​​of different types of features (e.g., title and hashtag values, or synopsis and hashtag values). For example, a method can be applied to weight the title and synopsis values. In this case, the vector values ​​of the final hidden layer that correspond to tokens located before and after [SEP] can be weighted twice. As another example, a method can be applied to weight the genre and hashtag values. In this case, the vector values ​​of the final hidden layer that correspond to tokens located between [TAG] and [ / TAG], and tokens located between [GENRE] and [ / GENRE] can be weighted twice.

[0180] As in the various embodiments described above, the similarity determination unit 630 can weight the vector values ​​corresponding to the locations of at least one type of feature among the vector values ​​of the final hidden layer. After weighting, the similarity determination unit 630 can obtain an input text vector for similarity calculation by determining the average of the vector values ​​of the final hidden layer.

[0181] According to one embodiment, if a language model is trained based on a genre prediction model, which is a text classification prediction model, genre values ​​may not be present in the vector values ​​of the final hidden layer. This is because genre-related tokens do not exist in the input sequence-type text data input to the language model. In this case, the weighting method that assigns weight to genre values, as described above, would not be applicable.

[0182] The content determination unit 640 can determine content items similar to a reference content item based on the similarity between content items determined by the similarity determination unit 630. The content determination unit 640 can check the similarity of each content item to the reference content item and generate a content item list based on the similarity. For example, the content determination unit 640 can select a specified number of content items from the content items stored in the server 120 in descending order of similarity to the reference content item and generate a content list containing the selected content items. That is, the content items included in the content list can be enumerated according to their similarity.

[0183] In the above description, the model learning unit 620 can add frequently occurring words from the content's text metadata to the vocabulary dictionary of the base language model and learn using the vocabulary dictionary to which the frequently occurring words have been added. When frequently occurring words are added to the vocabulary dictionary, they can be recognized as a single token in the language model without being split. For example, frequently occurring words representing major categories can be added to the vocabulary dictionary. When frequently occurring words representing major categories are added to the vocabulary dictionary, the recognition of these words as a single token in the language model increases the sequence length that the language model can recognize, resulting in even higher performance.

[0184] The reason we performed MLM based on hashtag information in the explanation above is that the experimental results showed the highest performance when MLM was applied to hashtag information. In other words, MLM can also be performed on other information within the content's text metadata, but the performance may be lower than when MLM is performed based on hashtag information.

[0185] In the embodiments described above, the model learning unit 620 was described as being included in the server 120. That is, the server 120, using the learned language model, can perform learning on the language model. However, in other embodiments, learning on the language model can be performed by an entity other than the server 120. In this case, the model learning unit 620 does not have to be included in the server, and the server 120 can receive information about the learned language model from the third device, build the learned language model, and then use the learned language model to determine the similarity between content items.

[0186] Figure 13 shows an example of a procedure for recommending content using a trained language model according to one embodiment of the present disclosure. The operating entity in Figure 13 may be the server 120 in Figure 1.

[0187] Referring to Figure 13, in step S1301, the server obtains sequential text data for the content items. The server can obtain text metadata for each content item and convert the obtained text metadata into sequential text data. That is, the server can obtain sequential text data by concatenating the features contained in the metadata of the content items together with a delimiter. For example, the delimiter can include at least one of the following: a separation token or a special token (e.g., a genre token, a director token, an actor token, a hashtag token).

[0188] In step S1303, the server determines the similarity between content items using the trained language model. The server can obtain content item-specific vector values ​​from sequenced text data obtained for each content item using the language model trained based on the content item's hashtag and synopsis. The server can determine the similarity between content items based on the content item-specific vector values. For example, the server can obtain the vector for the first content by inputting the sequenced text data of the first content item into the trained language model, and obtain the vector for the second content by inputting the sequenced text data of the second content item into the trained language model. The server can calculate the similarity between the two vectors using a similarity algorithm (e.g., the cosine similarity algorithm). The server can determine the calculated similarity as the similarity between the first content item and the second content item. This allows the server to calculate the similarity between a reference content item and each of the other content items.

[0189] In step S1305, the server provides at least one similar content item. The server can determine at least one content item similar to the reference content item based on the similarity determined in step S1303. That is, the server can provide at least one content item similar to the reference content item based on the similarity between the content items. For example, the server can select a specified number of content items from its owned content items in descending order of similarity to the reference content item, or content items with a similarity above a threshold. For example, the server can select a specified number of content items from candidate content items specified according to different criteria, in descending order of similarity to the content item, or content items with a similarity above a threshold. The server can then generate a content list containing information about the selected content items and provide the generated content list to the client device. In other words, the server can send the content list to the client device. At this time, the specific format of the content list may vary depending on the environment, service, etc., that provides similar content.

[0190] According to one embodiment, the server can pre-calculate and store the similarity between all content items it possesses. When a specified event occurs, the server can determine a reference content item and generate and provide a content list based on the pre-calculated similarity between the reference content item and other content items. For example, the specified event may include at least one of a client device's content list request event or a client device's recommended content request event. The listed events are merely examples to aid understanding, and the specified event is not limited to these. Here, the reference content item may be determined based on the client device's and / or user's consumption history and preferred content. The content list may include some of the content items it possesses that have a relatively high similarity to the reference content item.

[0191] According to one embodiment, the server can pre-calculate vector values ​​for all content items it possesses using a pre-trained language model and store the calculated vector values ​​in a content vector DB612. When a specified event occurs, the server can retrieve the vector values ​​of a reference content item and other content items from the content vector DB612 and determine the similarity of content items based on the vector values ​​of the reference content item and other content items.

[0192] Figure 14A shows an example of a procedure for training a language model according to one embodiment of the present disclosure. At least some of the operations in Figure 14A may be performed sequentially or in parallel. For example, some of the operations in Figure 14A may be performed at least temporarily at the same time. In the following, at least some of the operations in Figure 14A will be described with reference to Figure 14B. Figure 14B shows an example of training a language model using a language model according to one embodiment of the present disclosure.

[0193] Referring to Figure 14A, in step S1401, the server retrieves text metadata for the content. For example, as shown in Figure 14B, the server may retrieve text metadata 1410 including the content's title, genre, director, actors, hashtags, and synopsis.

[0194] In step S1403, the server performs tokenization of the text metadata. For example, the server can divide the text metadata into tokens using a byte-pair encoding (BPE) algorithm or a morphological analyzer. A byte-pair encoding algorithm is an information compression algorithm that compresses data by merging the most frequently occurring strings in the target data, and can consist of a vocabulary building step and a tokenization step. Specifically, a byte-pair encoding algorithm merges frequently occurring strings in the data, adds the merged strings to a vocabulary set to build a vocabulary set, and then, if each phrase in the target data contains a subword from the vocabulary set, it separates that subword from the phrase. A morphological analyzer is a technique that divides the target data into morphemes, which are the smallest semantic units.

[0195] In step S1405, the server obtains sequenced text data. For example, sequenced text data can be obtained by adding at least one delimiter to data separated into tokens. For example, sequenced text data can be determined as shown in Figure 14B. The server can obtain sequenced text data 1420 by separating metadata 1410 into tokens and inserting at least one isolation token and at least special tokens (e.g., genre token, director token, actor token, hashtag token, etc.) into the tokens.

[0196] In step S1407, the server masks the hashtag. The server can mask any one token from among several tokens located in the hashtag area. At this time, the hashtag area can be identified based on the special tokens [TAG] and [ / TAG] that represent the hashtag. For example, referring to Figure 14, the server recognizes that in the sequence text data 1420, there are tokens "emotional" and "warm" between [TAG] and [ / TAG], and can replace the "warm" token with [MASK]1431 or the "emotional" token with [MASK]1432. In one embodiment, the server can mask tokens that do not begin with "#" from among several tokens located in the hashtag area. Masking tokens that do not begin with "#" is necessary because tokens containing core meaning, such as nouns and verbs, do not begin with "#".

[0197] In step S1409, the server learns to infer masked hashtags using a language model-based predictive model. For example, as shown in Figure 14B, the server learns to infer if the “warm” token is masked. 1440 Using this method, the predictive model 1440 can be trained to infer the masked hashtag "warm," and if the "emotional" token is masked, it can be trained to infer the masked hashtag "emotional." In this case, the predictive model 1440 can learn to infer the masked hashtag by backpropagating the loss value. This allows the parameters of the language model that derives each token vector from the predictive model 1440 to be updated so that the token vectors of the title and synopsis reflect the semantic information of the masked hashtag.

[0198] The server can repeatedly perform steps S1407 and S1409 described above for multiple content items. Furthermore, the server can repeatedly perform steps S1407 and S1409 for multiple tokens within a hashtag area. In this way, by repeatedly applying the random masking training method to multiple hashtag information, the parameters of the language model can be updated so that the semantic information of multiple hashtags is reflected in the vectors of other tokens in the sequence-type text data. Therefore, the language model can learn to provide more sophisticated semantic representations through tasks such as inferring masked tokens as shown in Figure 14A, thereby enabling better identification of similarity relationships between content.

[0199] Furthermore, as mentioned above, even if hashtags are missing or absent from the sequenced text data, the trained language model can return a compressed vector of hashtag feature information from other types of features in the sequenced text data (e.g., title, synopsis).

[0200] Figure 15A shows an example of a procedure for training a language model according to one embodiment of the present disclosure. Hereinafter, at least some of the operations in Figure 15A may be performed sequentially or in parallel. For example, some of the operations in Figure 15A may be performed at least temporarily at the same time. Hereinafter, at least some of the operations in Figure 15A will be described with reference to Figure 15B. Figure 15B shows an example of training a language model using genre prediction according to one embodiment of the present disclosure.

[0201] Referring to Figure 15A, in step S1501, the server retrieves text metadata for the content. For example, as shown in Figure 15B, the server may retrieve text metadata 1510 including the content's title, genre, director, actors, hashtags, and synopsis.

[0202] In step S1503, the server performs tokenization of the text metadata. Tokenization of the text metadata can be performed in the manner described in step S1403 of Figure 14A.

[0203] In step S1505, the server obtains sequenced text data. For example, sequenced text data can be obtained by adding at least one delimiter to data separated into tokens. For example, sequenced text data can be determined as shown in Figure 15B. The server can obtain sequenced text data 1520 by separating metadata 1510 into tokens and inserting at least one isolation token and at least one special token (e.g., genre token, director token, actor token, hashtag token, etc.) into each token.

[0204] In step S1507, the server sets the input and target for the prediction model. The server can obtain the input sequence text data by removing genre-related tokens from the sequence text data and set the target label based on the genre information tokens. For example, referring to Figure 15B, the server can recognize that in the sequence text data 1520, there are "drama" tokens and "music" tokens between [GENRE] and [ / GENRE], and can set the input sequence text data 1520 from which they have been removed as the input for the prediction model. Furthermore, the server can set the target label based on the "drama" tokens and "music" tokens located between [GENRE] and [ / GENRE] in the sequence text data 1520.

[0205] In step S1509, the server trains a language model-based predictive model to infer genres for input sequence-type text data. For example, as shown in Figure 15B, the server can train the predictive model 1540 so that the genres for the input sequence-type text data 1530 are inferred as "drama" and "music". At this time, the predictive model 1540 can be trained by backpropagating loss values ​​to infer the target genres. This allows the parameters of the language model that derive each token vector from the predictive model 1540 to be updated so that the title, synopsis, and hashtag token vectors reflect the semantic information of the genre tokens.

[0206] As described above, even if genre information is not present in the sequenced text data, the trained language model can return a vector in which genre feature information is compressed from other types of features in the sequenced text data (e.g., title, synopsis).

[0207] Figure 16A shows an example of a procedure for training a language model according to one embodiment of the present disclosure. At least some of the operations in Figure 16A may be performed sequentially or in parallel. For example, some of the operations in Figure 16A may be performed at least temporarily at the same time. In the following, at least some of the operations in Figure 16A will be described with reference to Figure 16C. 16C This embodiment of the present disclosure illustrates an example of training a language model using hashtags and synopses.

[0208] Referring to Figure 16A, step S1601 The server then retrieves text metadata for the content. For example, as shown in Figure 16C, the server may retrieve text metadata 1610 that includes the content's title, genre, director, actors, hashtags, and synopsis.

[0209] In step S1603, the server performs tokenization on the text metadata. Tokenization of the text metadata can be performed in the same manner as described in step S1403 in Figure 14A.

[0210] In step S1605, the server obtains sequenced text data. For example, sequenced text data can be obtained by adding at least one delimiter to data separated into tokens. For example, sequenced text data can be determined as shown in Figure 16C. The server can obtain sequenced text data 1620 by separating metadata 1610 into tokens and inserting at least one isolation token and at least one special token (e.g., genre token, director token, actor token, hashtag token, etc.) into each token.

[0211] In step S1607, the server performs MLM-based primary learning using hashtags. The server masks any one hashtag token from among multiple hashtag tokens located in the hashtag region of the sequence-type text data, and performs primary learning to infer the masked hashtag token using a language model-based prediction model. At this time, the hashtag region can be identified based on the special tokens [TAG] and [ / TAG] that represent hashtags. For example, referring to Figure 16C, the server recognizes that in the sequence-type text data 1620, there are "emotional" tokens and "warm" tokens between [TAG] and [ / TAG], and can replace the "warm" token with [MASK]1631, or replace the "emotional" token with [MASK]1632. In one embodiment, the server can mask tokens that are not dependent tokens from among multiple tokens located in the hashtag region. As shown in Figure 16C, the server can train the prediction model 1640 to infer the masked hashtag token "warm" if the "warm" token is masked, and to infer the masked hashtag token "emotional" if the "emotional" token is masked. In this case, the prediction model 1640 can learn by backpropagating the loss value to infer the masked hashtag tokens. This allows the parameters of the language model that derives each token vector in the prediction model 1640 to be updated so that the token vectors of the title and synopsis can reflect the semantic information of the masked hashtags. The server can obtain a first-order trained language model by repeatedly performing the hashtag masking and inference operations described above multiple times for multiple content items.

[0212] In step S1609, the server performs MLM-based secondary learning using the synopsis. The server masks any one synopsis token from among multiple synopsis tokens located in the synopsis region of the sequenced text data, and performs secondary learning to infer the masked synopsis token using a language model-based predictive model. At this time, the synopsis region can be identified based on the isolation token [SEP] and the special token [GENRE] for the genre region. For example, referring to Figure 16B, the server recognizes that in the sequenced text data 1520, there are "female" tokens and "prison" tokens between [SEP] and [GENRE], and can replace the "female" token with [MASK] 1651 or the "prison" token with "MASK" 1652. In one embodiment, the server can mask tokens that are not dependent tokens from among multiple tokens located in the synopsis region. As shown in Figure 16C, the server can train the predictive model 1650 to infer the masked synopsis token "woman" if the "woman" token is masked, and to infer the masked synopsis token "prison" if the "prison" token is masked. In this case, the predictive model 1650 may include the language model that was first trained in step S1607, i.e., the language model trained based on hashtags. The predictive model 1650 can be trained by backpropagating the loss value to infer the masked synopsis token. This allows the parameters of the language model that derive each token vector from the predictive model 1640 to be updated so that the token vectors of title, hashtag, or genre can reflect the semantic information of the masked synopsis token. The server can obtain a second-order trained language model by repeating the synopsis masking and inference operations described above multiple times for multiple content items.

[0213] Figure 16B shows an example of a procedure for training a language model according to one embodiment of the present disclosure. Hereinafter, at least some of the operations in Figure 16B may be performed sequentially or in parallel. For example, some of the operations in Figure 16B may be performed at least temporarily at the same time. Hereinafter, at least some of the operations in Figure 16B will be described with reference to Figure 16C.

[0214] Referring to Figure 16B, in step S1651, the server retrieves text metadata for the content. For example, as shown in Figure 16C, the server may retrieve text metadata 1610 that includes the content's title, genre, director, actors, hashtags, and synopsis.

[0215] In step S1653, the server performs tokenization of the text metadata. Tokenization of the text metadata can be performed in the manner described in step S1403 of Figure 14A.

[0216] In step S1655, the server obtains sequence-type text data. For example, sequence-type text data can be obtained by adding at least one delimiter to data separated into token units. For example, the server can obtain sequence-type text data 1620 as shown in Figure 16C.

[0217] In step S1657, the server performs MLM-based learning using the synopsis. The server masks any one of several synopsis tokens located in the synopsis region of the sequence-type text data and performs primary learning by inferring the masked synopsis token using a language model-based predictive model. For example, as shown in Figure 16C, the server can first perform synopsis learning by inferring the masked synopsis using the predictive model 1650.

[0218] In step S1659, the server performs MLM-based learning using hashtags. The server masks any one of several hashtag tokens located in the hashtag region of the sequence-type text data and performs secondary learning to infer the masked hashtag token using a language model-based prediction model. For example, as shown in Figure 16C, the server can perform hashtag learning to infer the masked hashtag token using the prediction model 1645. In this case, the prediction model 1645 may include a language model that has been primary trained by synopsis learning.

[0219] As shown in Figures 16A and 16B above, when a random masking training method is repeatedly applied to multiple hashtag tokens and multiple synopsis tokens, the parameters of the language model can be updated so that the semantic information of the multiple hashtag tokens and the semantic information of the multiple synopsis tokens are reflected in the vectors of other tokens in the sequenced text data. Therefore, the language model Figures 16A and 16B Tasks involving inferring masked tokens, as shown, can be used to learn to provide more sophisticated semantic representations, thereby enabling better identification of similarities between content.

[0220] Furthermore, as explained with reference to Figures 16A and 16B, the trained language model can return a compressed vector of hashtag feature information from other types of features in the sequenced text data (e.g., title, genre), even if hashtags or synopses are missing or absent in the sequenced text data.

[0221] Figure 16A illustrates a procedure in which a server performs primary training on a language model based on hashtag information using MLM, followed by secondary training on the language model based on synopsis. Figure 16B illustrates a procedure in which a server performs primary training on a language model based on synopsis information using MLM, followed by secondary training on the language model based on hashtags. Generally, hashtag information for content items includes information related to the user's content preferences, or information that can reflect the user's content preferences. In contrast, synopsis information can include not only information related to the user's content preferences, but also information unrelated to the user's content preferences. Therefore, the performance of the language model can change depending on which information—hashtag information or synopsis information—is used first during language model training. Specifically, as shown in Figure 16A, when a language model is trained using hashtag information and then trained using synopsis information, the parameters of the language model can be quickly converged to values ​​close to the optimal value based on hashtag information, and then fine-tuned based on synopsis information. On the other hand, as shown in Figure 16B, if the synopsis information is used first for training, rather than hashtag information, overfitting of the trained language model can be prevented. Overfitting refers to a state where the language model is too well-adapted to the training data, resulting in decreased performance on other data. In other words, because synopsis information includes information unrelated to the user's content preferences, it can suppress the overfitting phenomenon of the language model.

[0222] In the explanation with reference to Figures 16A and 16B, the language model was trained based on hashtag information and synopsis information within the metadata of the content items, but other information can also be used to train the language model. For example, the language model may be first trained using hashtag information based on MLM, and then second trained using genre information. Another example is that the language model may be first trained using synopsis information based on MLM, and then second trained using genre information.

[0223] Furthermore, language models can also be trained using only synopsis information of content items based on MLM.

[0224] Figure 17 shows an example of a procedure for determining content similarity using a learning language model according to one embodiment of the present disclosure. The operation in Figure 17 is an example of operation S1303 in Figure 13 and can be understood as a procedure for determining the similarity between two content items. Hereinafter, at least some of the operations in Figure 17 may be performed sequentially or in parallel. For example, some of the operations in Figure 17 may be performed at least temporarily at the same time.

[0225] Referring to Figure 17, in step S1701, the server determines the vector for the reference content item. Here, the vector can be determined based on sequenced text data determined using text metadata. For example, the server can obtain sequenced text data by obtaining the text metadata for the reference content item, tokenizing the obtained text metadata, and then inserting at least one delimiter. The server can then obtain the vector corresponding to the sequenced text data for the reference content item using a trained language model. Specifically, the server can determine the vector, i.e., the embedding value, by inputting the sequenced text data into the trained language model and obtaining the output data of the language model. The trained language model may be one as described in Figures 14A, 15A, 16A, or 16B. However, except for the head layer used in the language model to infer tokens (e.g., hashtag tokens or synopsis tokens) or predict classes for similarity calculations, the final hidden layer embedding value of the language model itself can be used as the embedding value for the content's text metadata. In this case, according to one embodiment, the server can determine the content vector for similarity calculation using one of the following methods: a method using puller output, a method using the average of the final hidden state values, or a method using the maximum value of the final hidden state values. Furthermore, according to one embodiment, when determining the content vector for similarity calculation, the server can weight the values ​​of the final hidden state values ​​that correspond to the positions of specific features.

[0226] In step S1703, the server determines a vector for the content item to be compared. Here, the vector can be determined based on sequenced text data determined using text metadata. For example, the server can obtain sequenced text data by acquiring the text metadata of the content item to be compared, tokenizing the acquired text metadata, and then inserting at least one delimiter. The server can then use a trained language model to obtain a vector corresponding to the sequenced text data of the content item to be compared. Specifically, the server can determine the vector, or embedding value, by inputting the sequenced text data into the trained language model and obtaining the output data of the language model. The trained language model may be one as described in Figures 14A, 15A, 16A, or 16B. However, except for the head layer used in the language model to infer tokens (e.g., hashtag tokens or synopsis tokens) or predict classes for similarity calculation, the final hidden layer embedding value of the language model itself can be used as the embedding value for the content's text metadata. In this embodiment, the server can determine the content vector for similarity calculation using one of the following methods: a method using puller output, a method using the average of the final hidden state values, or a method using the maximum value of the final hidden state values. Furthermore, in one embodiment, when determining the content vector for similarity calculation, the server can weight the values ​​corresponding to the positions of specific features among the final hidden state values.

[0227] In step S1705, the server can calculate the similarity between content items. For example, the server can determine the similarity between a reference content item and a comparison content item based on a cosine similarity algorithm. For instance, the server can calculate the similarity between the vector of the reference content item and the vector of the comparison content item, and then determine the calculated similarity as the similarity between the reference content item and the comparison content item.

[0228] In the above description, when the language model is trained based on a genre prediction model of a text classification scheme, the input sequence-type text data from which genre-related tokens have been removed was used as input to the trained language model to obtain content item-specific vector values ​​for determining the similarity between content items. However, embodiments of the present disclosure are not limited thereto. For example, the server or server similarity determination unit 630 according to embodiments of the present disclosure may also use sequence-type text data including genre-related tokens as input to the trained language model. For example, the server or server similarity determination unit 630 may input sequence-type text data including genre-related tokens, as shown in Table 1, into the trained language model to obtain content item-specific vector values ​​and then determine the similarity between content items.

[0229] Figure 18A shows an example of a transformer structure applicable to embodiments of the present disclosure, and Figure 18B shows an example of a detailed structure of the encoder and decoder blocks of a transformer applicable to embodiments of the present disclosure.

[0230] Referring to Figures 18A and 18B, the transformer 1800 can include N encoder blocks 1810-1 to 1810-N and N decoder blocks 1820-1 to 1820-N. Each of the N encoder blocks 1810-1 to 1810-N can include a self-attention block 1811 and a feedforward block (or neural network) 1813. Each of the N decoder blocks 1820-1 to 1820-N can include a self-attention block 1821, an encoder-decoder attention block 1823, and a feedforward block 1825.

[0231] The input to transformer 1800 is tokenized and embedded, added to a positional encoding vector, and then input to the first encoder block 1810-1, located at the bottom of the N encoder blocks 1810-1 to 1810-N. Each of the N encoder blocks 1810-1 to 1810-N has a self-attention block 1811 that can determine which word to focus on from among multiple input words. The self-attention block 1811 can generate a query vector, a key vector, and a value vector by multiplying the input embedding vector by three learnable matrices, respectively. The self-attention block 1811 may be a multi-headed attention block having multiple attention heads, representing each vector as a different representation space for each purpose using multiple query vectors, key vectors, and value vectors. The output of the self-attention block 1811 can pass through the neural network of feedforward block 1813 and be input to the next encoder block (e.g., the second encoder block 1810-2).

[0232] Of the N encoder blocks 1810-1 to 1810-N, the output of the uppermost Nth encoder block 1810-N may be an attention vector consisting of a key vector and a value vector. These vectors can be input to the encoder-decoder attention block 1823 of each of the N decoder blocks 1820-1 to 1820-N.

[0233] The previous output of transformer 1800 can be used as input to the first decoder block 1820-1, which is located at the bottom of the N decoder blocks 1820-1 to 1820-N. For example, the previous output of transformer 1800 can be tokenized, embedded, added to a position coding vector, and then input to the first decoder block 1820-1.

[0234] Each of the N decoder blocks 1820-1 to 1820-N has a self-attention block 1821 that is similar to each of the N encoder blocks 1810-1 to 1810-N. However, each of the N decoder blocks 1820-1 to 1820-N has a difference from each of the N encoder blocks 1810-1 to 1810-N's self-attention block 1821 in that it performs masking so that attention can only be paid to the previous position of the current position within the output sequence.

[0235] Each of the N decoder blocks 1820-1 to 1820-N has an encoder-decoder attention block 1823 that can generate an output using the query vector output from the self-attention block 1821 and the key vector and value vector output from the Nth encoder block 1810-N as input.

[0236] Of the N decoder blocks 1820-1 to 1820-N, the output vector of the topmost decoder block 1820-N can be input to the linear layer 1830 and the softmax layer 1840. The linear layer 1830 and the softmax layer 1840 can convert the output vector of the Nth decoder block 1820-N into a single word. The linear layer 1830 consists of a fully-connected neural network and can project the output vector of the Nth decoder block 1820-N into a larger vector called a logit vector. Each cell in the projected logit vector can have a score for each corresponding word. The softmax layer 1840 can convert the score of each cell into a probability. All converted probability values ​​of each cell are positive numbers, and the sum of the probability values ​​can be 1. At this point, the word corresponding to the cell with the highest probability value can be output as the final result of the softmax layer 1840. The output of the softmax layer 1840 can be re-embedded and added to the position coding vector, and then input to the first decoder block 1820-1 located at the bottom.

[0237] The subblocks contained in each of the N encoder blocks 1810-1 to 1810-N and the N decoder blocks 1820-1 to 1820-N may be connected using a residual connection method, and layer-normalization (or Add&normalize) blocks may be included between each subblock. The layer-normalization blocks prevent excessive data modification in a single layer by matching the inputs and outputs of the self-attention blocks 1811 and 1821.

[0238] Transformer 1800 is a neural network that tracks the relationships between words in a text to learn the context and meaning of the text, and can mathematically find patterns between elements without a labeled dataset. Therefore, Transformer 1800 does not require the process of generating a dataset and is suitable for parallel processing, thus achieving high speed.

[0239] Recurrent Neural Networks (RNNs) have been widely used in the field of natural language processing because they process words sequentially as input according to their position, allowing them to retain positional information for each word. However, RNNs have problems such as difficulty in parallel processing and long-term dependencies. In contrast, transformers can capture dependencies between inputs and outputs using an attention mechanism instead of an RNN. Furthermore, transformers apply attention to the position of each word in the encoder block during training, that is, they emphasize the value that has the closest relationship to the query, and they use masking techniques in the decoder block, thus enabling parallel processing.

[0240] The magnitude of the transformer's encoder / decoder inputs and outputs, the number of encoders / decoders, the number of attention heads, and / or the size of the hidden layers in the feedforward neural network are hyperparameters that can be modified by the user.

[0241] The BERT model is a transformer-based language model as described above, and can be used by substituting or deleting some of the components of the transformer. Figure 19 shows an example of the structure of a BERT model applicable to embodiments of this disclosure. For example, the BERT model may be a model that uses encoder blocks 1810-1 to 1810-N by removing decoder blocks 1820-1 to 1820-N from the transformer, as shown in Figure 19.

[0242] In the BERT model, the [CLS] token can be placed at the beginning of an input sentence, and the [SEP] token can be used at the end of a sentence to separate it. The output embedding after the BERT operation can be an embedding that takes into account the entire context of the sentence. For example, [CLS] is a simple embedding vector that has gone through the embedding layer when it is input to BERT, but after passing through the BERT model, it can become a vector that has contextual information that takes into account all the word vectors in the sentence.

[0243] Natural language processing using transformer-based models such as the BERT model can be carried out in two steps. These two steps may include a pre-training step in which a large encoder embeds input text to model the language, and a fine-tuning step in which the pre-trained model is fine-tuned to perform multiple natural language processing tasks.

[0244] The BERT model is a pre-trained model that performs pre-training embeddings before performing a specific task, and is attracting attention as a model that can further improve task performance compared to existing embedding techniques. When considering the modeling process using the BERT model, pre-training is performed using an unsupervised learning method, where the encoder embeds a large corpus, and then transfers and fine-tunes it to achieve the desired learning, enabling it to perform the task. Another feature of the BERT model is that it applies a bidirectional model to consider the context before and after sentences, which can show even higher accuracy than before.

[0245] As described above, the language model trained by the embodiments of this disclosure obtains a content vector by comprehensively considering not only hashtag information and synopsis information, but also semantic and / or contextual information of other types of features, and calculates the similarity between content based on this vector. Therefore, the method of determining the similarity between content based on the language model by the embodiments of this disclosure is different from simply filtering content that has similar hashtags and synopses.

[0246] In this disclosure, similarity is a relative concept, not an absolute one. Therefore, the test set for verifying the performance of the language model must accurately reflect this relative concept. The fact that it is not an absolute concept means that comparing only two pieces of content cannot determine whether they are similar or not. Therefore, in this disclosure, to reflect the relative concept of similarity, a method was designed to classify content similarity by comparing three pieces of content. For example, if the similarity of the content that is more similar to the reference content is higher than the similarity of the content that is less similar to the reference content, then the similarity is judged to have been accurately determined. In this disclosure, to improve the accuracy of the test set while complementing the relative concept of similarity, only objective cases judged identically by three or more multiple reviewers were used as the test set. This is to minimize the influence of the reviewer's subjectivity in weighting specific features when judging content similarity. In the embodiments of this disclosure, the similarity comparison criteria were constructed into easy and difficult criteria. This is because, depending on the model, improvements in performance based on difficult criteria and improvements in performance based on easy criteria do not always correlate, requiring consideration from multiple perspectives.

[0247] Specifically, a test set like the one shown in Figure 20 was designed based on the following criteria for similarity test sets. Figure 20 shows an example of a test set according to one embodiment of the present disclosure.

[0248] 1. Criteria for the similarity test set

[0249] a. Similar content: Content that shares genre and hashtags with the reference content will be considered similar content.

[0250] b. Content that is not very similar: Content that shares a genre with the reference content but does not share hashtags with the reference content is considered to be not very similar.

[0251] c. Dissimilar content: Content whose genre and hashtags do not have anything in common with the genre and hashtags of the reference content will be judged as dissimilar content.

[0252] In addition to the genres and hashtags mentioned above, we primarily considered the title and synopsis content to determine the similarity of the content.

[0253] 2. Classification of Similarity Test Sets

[0254] a. Classification accuracy 1: If the similarity between the reference content and similar content is higher than the similarity between the reference content and content that is not very similar, it is determined that the similarity has been accurately determined.

[0255] b. Classification accuracy 2: If the similarity between the reference content and similar content is higher than the similarity between the reference content and dissimilar content, it is determined that the similarity has been accurately determined.

[0256] In embodiments of this disclosure, the data splitting ratio was adjusted and test set performance was obtained and analyzed multiple times to understand the generalized performance of the predictive model. During training of the predictive model, the size of the test set was determined based on the number of new content input to the predictive model within the training cycle (e.g., about one week) for the training set, validation set, and test set.

[0257] Although it depends on the actual system, assume a situation where it is expected that there are more than 100 pieces of new content input into the prediction model within the training cycle. Under such an assumption, 100 pieces were assigned to each of the test set and the validation set, and all the remaining data was assigned to the training set. As described above, the data was randomly divided, and the process of analyzing the test set performance was repeated multiple times. This is because more refined generalization performance can be grasped during multiple performance analyses using different test sets than during a single performance analysis. Table 6 below shows the results of comparing the learning performance of each hash tag by dividing the training set, validation set, and test set through three random samplings.

[0258]

Table 6

[0259] Referring to Table 6, the prediction model showed performance of 90.23% - 91.38% and 97.7% - 98.28% respectively based on the criteria of similarity test set accuracy 1 and accuracy 2. Compared with 78.16% and 88.51% which are the performances of the basic language model before learning through hash tag prediction, it improved by about 12% and 9% respectively. Table 7 below shows the results of comparing the learning performance of each genre by dividing the training set, validation set, and test set through three random samplings.

[0260]

Table 7

[0261] Referring to Table 7, the prediction model showed performance of 83.50% and 96.12% respectively based on the criteria of similarity test set accuracy 1 and accuracy 2. Compared with 78.16% and 88.51% which are the basic language model performances before learning through genre prediction, it improved by about 5% and 8% respectively. Referring to Table 6 and Table 7, it can be seen that the learning method of the prediction model significantly improves the ability to distinguish similar content based on content text metadata.

[0262] As described above, the similar content determination technology using the language model learned according to the embodiments of the present disclosure can be utilized in various ways. For example, when a reference content item is specified, a list of similar content as shown in FIG. 21 can be provided. FIG. 21 shows an example of the use of similar content determined according to an embodiment of the present disclosure. Referring to FIG. 21, when a reference content item 2102 is specified, a plurality of content items 2104a, 2104b, 2104c, 2104d similar to the reference content item 2102 are determined, and a list including the reference content item 2102 and the plurality of similar content items 2104a, 2104b, 2104c, 2104d can be provided. The provided list can be displayed on a client device. At this time, the plurality of similar content items 2104a, 2104b, 2104c, 2104d can be arranged in descending order of similarity to the reference content item 2102. That is, the first similar content item 2104a displayed closest to the reference content item 2102 can have a higher similarity than the other content items 2104b, 2104c, 2104d. For example, the reference content item 2102 may be a content item that the user has recently viewed, or a content item specified by the user.

[0263] The exemplary method of the present invention is represented by a series of operations for clarity of explanation, but this is not intended to limit the order in which the steps are performed, and each step can be performed simultaneously or in a different order as necessary. To implement the method according to the present invention, the exemplary steps may further include other steps, or may include the remaining steps except for some steps, or may include still other steps except for some steps.

[0264] The various embodiments of the present invention are not intended to enumerate all possible combinations, but rather to illustrate representative aspects of the invention, and the matters described in the various embodiments may be applied independently or in combination of two or more.

[0265] Furthermore, various embodiments of the present invention can be realized by hardware, firmware, software, or a combination thereof. In the case of hardware implementation, embodiments can be realized by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general processors, controllers, microcontrollers, microprocessors, and the like.

[0266] The scope of the present invention includes software or machine-executable commands (e.g., operating systems, applications, firmware, programs, etc.) that enable the operation of various embodiments to be performed on a device or computer, and non-transitory computer-readable medium on which such software or commands etc. are stored and can be executed on a device or computer.

Claims

1. A method for operating a server in a content streaming system, A step of obtaining first sequence-type text data containing information contained in the first metadata of the first content item, The steps include obtaining second sequence-type text data containing information included in the second metadata of the second content item, A step of determining a first vector corresponding to the first sequence-type text data and a second vector corresponding to the second sequence-type text data using a language model learned based on synopsis information contained in the metadata of the content item, A step of determining the similarity between the first content item and the second content item using the first vector and the second vector, A method comprising the step of providing a content list that includes at least one content item, which includes the second content item selected based on the similarity.

2. The method according to claim 1, wherein the language model is trained to predict synopsis information of the content item based on a masked language model (MLM).

3. The method according to claim 2, wherein the language model is first trained through training to predict hashtag information of the content item based on the MLM, and second trained through training to predict synopsis information of the content item based on the MLM.

4. The method according to claim 2, wherein the language model is first trained through training to predict synopsis information of the content item based on the MLM, and second trained through training to predict hashtag information of the content item based on the MLM.

5. The method according to claim 1, wherein the language model is trained to predict masked tokens located between tokens that indicate a synopsis region among a plurality of tokens included in the input sequence-type text data.

6. The method according to claim 5, wherein the token indicating the synopsis region includes at least one of isolation tokens for distinguishing different types of features from each other, and special tokens for other types of features that are not synopsis.

7. A step of converting text metadata describing the content of the content item into sequence-type text data, The steps include: masking synopsis tokens located between tokens that indicate the synopsis region among a plurality of tokens contained in the sequence-type text data; The step further includes training the language model through training to predict the masked synopsis tokens, The method according to claim 5, wherein the text metadata includes at least one of the following: title, synopsis, genre, director, actor, or hashtag information.

8. The step of converting the text metadata into sequence-type text data is: The steps include dividing the aforementioned text metadata into multiple tokens, The step of generating the sequence-type text data by inserting at least one delimiter between the aforementioned tokens, The method according to claim 7, wherein the at least one delimiter further includes at least one of a token indicating the synopsis region, a delimiter for separating different types of features from each other, and a special token for indicating a region of a particular type of feature.

9. The step of masking the aforementioned synopsis token is: The steps include selecting an independent token from the synopsis tokens located between the tokens that indicate the aforementioned synopsis area, The step of masking the selected independent tokens is included, The method according to claim 7, wherein the independent token is a token that does not begin with the specified symbol.

10. The aforementioned training is performed using a predictive model. The method according to claim 7, wherein the prediction model includes a language model that takes sequence-type text data containing masked synopsis tokens as input and outputs vector values ​​corresponding to the sequence-type text data, and an MLM (masked language model) head layer configured to predict at least one input token corresponding to at least one vector value output from the language model.

11. The step of determining the similarity between the first content item and the second content item is: The step includes calculating the similarity between the first vector and the second vector using a cosine similarity algorithm, The method according to claim 1, wherein each of the first vector and the second vector is obtained by performing average pooling on the output vector values ​​of the final hidden layer of the learned language model.

12. The method according to claim 11, wherein each of the first vector and the second vector is determined by weighting the vector values ​​corresponding to the positions of specified features among the output vector values ​​of the final hidden layer of the learned language model.

13. The steps include obtaining third sequence-type text data containing information included in the third metadata of the third content item, The steps include determining a third vector corresponding to the third sequence-type text data using the learned language model, The method further includes the step of determining the similarity between the first content item and the third content item using the first vector and the third vector, The step of providing the aforementioned content list is: The method according to claim 11, further comprising the step of selecting the second content item from the second content item and the third content item based on the similarity between the first content item and the second content item, and the similarity between the first content item and the third content item.

14. A server in a content streaming system, A communication unit that transmits and receives signals from at least one client device, The communication unit includes a processor electrically connected to the communication unit, The aforementioned processor, Obtain the first sequence-type text data containing the information contained in the first metadata of the first content item, Retrieve the second sequence-type text data containing the information included in the second metadata of the second content item, Using a language model trained based on synopsis information contained in the metadata of the content item, a first vector corresponding to the first sequence-type text data and a second vector corresponding to the second sequence-type text data are determined. The similarity between the first content item and the second content item is determined using the first vector and the second vector. A server controlled to provide a content list containing at least one content item, which includes the second content item selected based on the similarity.

15. A program stored on a recording medium for performing the method according to any one of claims 1 to 13, when operated by a processor.

Images