Method, apparatus, and computer program for providing additional information related to content

Abstract

A method, an apparatus, and a non-transitory computer-readable recording medium for providing additional information related to content are disclosed. The additional information providing method includes a character recognizing step of recognizing a character from an image frame during reproduction of the image by a video player, a user preference degree judging step of judging a user preference degree related to the character, a recognizing step of recognizing an object related to a specific character among the characters from the image frame based on the user preference degree, and a providing step of providing additional information of the object related to the specific character.

Description

Technical Field

[0001] This invention discloses a technique for providing additional information during image regeneration. Background Technology

[0002] Product placement (PPL) advertising, which involves advertising products featured in videos, is a commonly used advertising method. However, PPL advertising is a one-way delivery of advertising and has limitations in providing users with the product information they need.

[0003] As one of the technologies for solving this problem, Korean Patent Publication No. 10-2000-0012639 (publication date: March 6, 2000) discloses a system and its operation method that may dynamically provide users with product-related information about various products appearing in the video (movies, TV series, music videos, advertisements) during the process of watching videos (movies, TV series, music videos, advertisements) on the Internet.

[0004] Previously, it was possible to pre-register the time when a product appeared in a video or use separate tools or manual work, and then display product-related information in a pop-up window based on the time when the product appeared or the product tag. Summary of the Invention

[0005] The purpose of this invention is to automatically record the timestamps of people and objects appearing during the image regeneration process by utilizing person recognition technology and object recognition technology.

[0006] The purpose of this invention is to provide timestamp tags related to people and objects within a video player.

[0007] The purpose of this invention is to provide additional information based on people and objects identified in frames with specific timestamps during video regeneration.

[0008] The present invention provides a method for providing additional information operating in a computer device, the computer device including at least one processor executing a plurality of computer-readable instructions contained in a memory, the method comprising: a person recognition step, wherein the at least one processor identifies a person from an image frame during the playback of an image by means of the at least one processor; a user favorability determination step, wherein the at least one processor determines a user favorability related to the person; a recognition step in the image frame, wherein the at least one processor identifies an object related to a specific person from the image frame based on the user favorability; and a provision step, wherein the at least one processor provides additional information about the object related to the specific person.

[0009] According to one embodiment, the person recognition step may include: using person recognition technology to identify the person in the candidate regions segmented after segmenting the candidate regions from the image frame.

[0010] According to another embodiment, the person recognition step may include: generating a person-recognition-based timestamp tag by storing the timestamp of the person's appearance according to the person.

[0011] According to another embodiment, the step of recognizing in an image frame may include: using object recognition technology to identify the object in the segmented candidate regions after segmenting the candidate regions from the image frame.

[0012] According to another embodiment, the step of identification in the image frame may include: identifying at least one person based on the user's favorability among the people identified from the image frame; and identifying objects related to the specific person from the image frame.

[0013] According to another embodiment, the step of recognizing in the image frame may include: generating a timestamp label based on object recognition by storing timestamps of object appearance according to the object.

[0014] According to another embodiment, the step of determining user favorability may include: determining user favorability related to the person based on the user's resume information on the computer device.

[0015] According to another embodiment, the step of determining user favorability may include: determining user favorability related to the person based on the resume information of the person that the user of the computer device shows interest in.

[0016] According to another embodiment, the providing step may include: providing the additional information in association with a portion of an image frame that identifies the object.

[0017] According to another embodiment, the providing step may include providing the additional information at the time point in the first or last frame of the image frames in which the object appears.

[0018] According to another embodiment, the providing step may include providing the additional information at a time point prior to the appearance of an image frame based on a preset number of image frames in which the object appears.

[0019] According to another embodiment, the providing step may include: providing a directory of people identified from the image frame and a directory of objects appearing with each person according to the individuals included in the directory of people.

[0020] According to another embodiment, the providing step may include: providing shopping information for goods that match the object as additional information for the object appearing with the specific person.

[0021] The present invention provides a non-transitory computer-readable recording medium storing a computer program, according to which a computer system runs the additional information providing method.

[0022] The present invention provides a computer device including at least one processor that executes computer-readable instructions contained in a memory. The at least one processor includes: a person recognition unit for recognizing people from image frames during the playback of images by a video player; a favorability judgment unit for judging user favorability related to the people; an object recognition unit for recognizing objects related to a specific person from the image frames based on the user favorability; and an additional information providing unit for providing additional information about the objects related to the specific person. Attached Figure Description

[0023] Figure 1 A diagram illustrating an example of a network environment according to an embodiment of the present invention.

[0024] Figure 2 This is a block diagram illustrating the internal structure of an electronic device and a server in one embodiment of the present invention.

[0025] Figure 3 The figure illustrates an example of the structural elements that a processor of an electronic device according to an embodiment of the present invention may include.

[0026] Figure 4 A flowchart illustrating an example of a method for providing additional information executable by an electronic device according to an embodiment of the present invention.

[0027] Figure 5 This is an example diagram illustrating the person recognition steps of an embodiment of the present invention.

[0028] Figure 6 This is a diagram illustrating a timestamp tag based on person recognition, according to an embodiment of the present invention.

[0029] Figure 7 This is an example diagram illustrating the object recognition steps of an embodiment of the present invention.

[0030] Figures 8 to 9 This is an example diagram illustrating a timestamp tag based on object recognition, according to an embodiment of the present invention.

[0031] Figures 10 to 14 This is an example diagram of a service screen that provides additional information according to an embodiment of the present invention. Detailed Implementation

[0032] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0033] Several embodiments of the present invention relate to techniques for providing additional information during image regeneration.

[0034] According to several embodiments specifically disclosed in this specification, person recognition technology and object recognition technology can be used to automatically record the timestamps of the appearance of people and objects during video playback, and additional information can be provided based on the people and objects that appear at a specific timestamp.

[0035] Figure 1 A diagram illustrating an example of a network environment according to an embodiment of the present invention. Figure 1 The network environment includes multiple electronic devices 110, 120, 130, and 140, multiple servers 150 and 160, and a network 170. The above... Figure 1 For the purpose of illustrating one example of the invention, the number of electronic devices or servers is not limited. Figure 1 As shown.

[0036] Multiple electronic devices 110, 120, 130, and 140 can be fixed or mobile terminals manifested through a computer system. Examples of multiple electronic devices 110, 120, 130, and 140 include smartphones, mobile phones, navigators, computers, laptops, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), tablet computers, game consoles, wearable devices, Internet of Things (IoT) devices, virtual reality (VR) devices, and augmented reality (AR) devices. As an example, Figure 1 In the example shown, electronic device 110 is in the shape of a smartphone. In embodiments of the invention, electronic device 110 may be one of a variety of physical computer systems that communicate with other electronic devices 120, 130, 140 and / or servers 150, 160 via network 170 using wireless or wired communication methods.

[0037] The communication method is not limited and may include communication methods using communication networks that network 170 may include (e.g., mobile communication networks, wired networks, wireless networks, broadcast networks, satellite networks, etc.) and wireless communication between multiple devices. For example, network 170 may include any one or more of the following networks: personal area network (PAN), local area network (LAN), campus network (CAN), metropolitan area network (MAN), wide area network (WAN), broadband network (BBN), and the Internet. Furthermore, network 170 may include one or more of the following network topologies, but is not limited to: bus network, star network, ring network, mesh network, star-bus network, tree network, hierarchical network, etc.

[0038] Servers 150 and 160 can be computer devices or multiple computer devices that communicate with multiple electronic devices 110, 120, 130, and 140 via network 170 to provide instructions, code, files, content, services, etc. For example, server 150 can be a system that provides a first service to multiple electronic devices 110, 120, 130, and 140 connected via network 170, and server 160 can be a system that provides a second service to multiple electronic devices 110, 120, 130, and 140 connected via network 170. As a more specific example, server 150 can provide services required by an application (for example, image services, etc.) as a first service to multiple electronic devices 110, 120, 130, and 140 as a first service, which is a computer program driven by an application set on multiple electronic devices 110, 120, 130, and 140. As another example, server 160 can provide a service as a second service, which involves allocating files to multiple electronic devices 110, 120, 130, and 140 for setting up and driving the aforementioned application.

[0039] Figure 2 This is a block diagram illustrating the internal structure of an electronic device and a server according to an embodiment of the present invention. Figure 2 The internal structures of electronic device 110 and server 150 are described below as examples of electronic devices. Furthermore, other electronic devices 120, 130, 140, or server 160 may also have the same or similar internal structures as electronic device 110 or server 150.

[0040] Electronic device 110 and server 150 may include memories 211 and 221, processors 212 and 222, communication modules 213 and 223, and input / output interfaces 214 and 224. Memories 211 and 221, as non-transitory computer-readable recording media, may include permanent mass storage devices such as random access memory (RAM), read-only memory (ROM), hard disk drives (HDDs), solid-state drives (SSDs), and flash memory. The permanent mass storage devices, such as read-only memory, SSDs, flash memory, and hard disk drives, are separate permanent storage devices distinct from memories 211 and 221 and may be incorporated into electronic device 110 or server 150. Furthermore, memories 211 and 221 may store an operating system and at least one program code (for example, code for a browser installed on electronic device 110 to drive it, or code for an application installed on electronic device 110 to provide specific services). These software structural elements may be loaded from a separate computer-readable recording medium from memories 211 and 221. The aforementioned individual computer-readable recording media may include floppy disk drives, magnetic disks, magnetic tapes, DVD / CD-ROM drives, memory cards, and other computer-readable recording media. In another embodiment, software structural elements may be loaded into memories 211 and 221 via communication modules 213 and 223, rather than via computer-readable recording media. For example, at least one program may be loaded into memories 211 and 221 based on a computer program (as an example, the application described above) configured via a file provided by the developer or a file allocation system (e.g., the server 160 described above) that distributes the application's configuration files via network 170.

[0041] Processors 212 and 222 can perform basic arithmetic, logical, and input / output calculations, thereby processing instructions from computer programs. Instructions can be provided to processors 212 and 222 via memories 211 and 221 or communication modules 213 and 223. For example, processors 212 and 222 can execute received instructions based on program code stored in recording devices such as memories 211 and 221.

[0042] Communication modules 213 and 223 provide the function of enabling electronic device 110 to communicate with server 150 via network 170, and also provide the function of enabling electronic device 110 and / or server 150 to communicate with other electronic devices (e.g., electronic device 120) or other servers (e.g., server 160). As an example, processor 212 of electronic device 110 can, under the control of communication module 213, transmit requests generated according to program code stored in a recording device such as memory 211 to server 150 via network 170. Conversely, control signals, instructions, content, files, etc., provided under the control of processor 222 of server 150 can be provided to electronic device 110 via communication module 223 and network 170 through communication module 213. For example, control signals, instructions, content, files, etc., received from server 150 via communication module 213 can be transmitted to processor 212 or memory 211, and the content or files can be stored in a storage device (the aforementioned permanent storage device) that may also be included in electronic device 110.

[0043] Input / output interface 214 can be a unit for interaction with input / output device 215. For example, input devices may include keyboards, mice, microphones, cameras, etc., and output devices may include displays, speakers, haptic feedback devices, etc. As another example, input / output interface 214 can also be a unit for interaction with devices that combine input and output functions, such as touchscreens. Input / output device 215 can be composed of electronic device 110 and another device. Furthermore, input / output interface 224 of server 150 can be a unit for connection to server 150 or for interaction with input or output devices (not shown) that may be included in server 150. As a more specific example, during the processing of instructions for a computer program loaded into memory 211, the processor 212 of electronic device 110 may display a service screen or content constructed using data provided by server 150 or electronic device 120 on a display via input / output interface 214.

[0044] Furthermore, in another embodiment, the electronic device 110 and the server 150 may include a ratio Figure 2The electronic device 110 includes more structural elements. However, it is not necessary to explicitly show most of the structural elements of the prior art. For example, the electronic device 110 includes at least a portion of the input / output devices 215 described above, or may also include other structural elements such as a transceiver, a Global Positioning System (GPS) module, a camera, various sensors, a database, etc. As a more specific example, in the case of a smartphone, the electronic device 110 may also include various structural elements typically included in smartphones, such as an accelerometer sensor, a gyroscope sensor, a camera module, various physical buttons, buttons utilizing a touchpad, input / output ports, a vibrator for vibration, etc.

[0045] The following describes specific embodiments of a method and system for storing timestamps in a manner associated with the identified persons and / or objects based on person and object recognition, and using them to provide additional information.

[0046] Figure 3 The figure is a diagram illustrating an example of the structural elements that a processor of an electronic device according to an embodiment of the present invention may include. Figure 4 A flowchart illustrating an example of a method for providing additional information executable by an electronic device according to an embodiment of the present invention.

[0047] In this embodiment, the electronic device 110 can form a computer-implemented additional information providing system. As an example, the additional information providing system can be in the form of a stand-alone program or an in-app application of a specific application. It can be configured to run on the aforementioned specific application and, depending on the situation, can link with the server 150 to provide additional information. For example, the additional information providing system can be in the form of an in-app application set within a video player on the electronic device 110, and can provide additional information within the video player using timestamp tags based on person or object recognition.

[0048] The additional information providing system formed on the electronic device 110 is executable based on the instructions provided by the application set on the electronic device 110. Figure 4 The method for providing additional information is shown. To perform the method for providing additional information, as follows... Figure 3 As shown, the processor 212 of the electronic device 110 may include structural elements such as a person recognition unit 310, a favorability judgment unit 320, an object recognition unit 330, and an additional information providing unit 340. According to an embodiment, the structural elements of the processor 212 may be selectively included in the processor 212 or may be excluded from the processor 212. Furthermore, according to an embodiment, in order to demonstrate the functions of the processor 212, the various structural elements of the processor 212 may be separated or combined.

[0049] Processor 212 can read necessary instructions from memory 211 loaded with instructions related to the control of electronic device 110. In this case, the read instructions may include instructions for controlling processor 212 to execute. Figure 4 The additional information shown provides instructions for the method.

[0050] Such processor 212 and its various structural elements are capable of controlling electronic device 110 by performing the various steps included in the additional information provision method. For example, processor 212 and its various structural elements can execute operating system code contained in memory 211 and instructions based on at least one program code.

[0051] The various structural elements of the processor 212 can be manifestations of multiple different functions of the processor 212 executed by the processor 212 according to instructions provided by the program code stored in the electronic device 110 (for example, instructions provided by the application program driven in the electronic device 110). For example, a person recognition unit 310 can be used as a functional expression of the processor 212 that controls the electronic device 110 according to the above instructions in a way that enables the electronic device 110 to recognize people within an image frame.

[0052] In step S410, the person recognition unit 310 can identify people within each image frame based on person recognition technology. (Refer to...) Figure 5 The person recognition unit 310 can identify a specific person from the candidate regions 501 segmented from the image frame 500 based on face recognition. The person recognition unit 310 tracks the positional changes of a specific person by utilizing object tracking technology and similarity between specific information such as the size, color, shape, and outline of the person identified from the image frame. The specific person recognition method can utilize at least one of known object recognition algorithms and object tracking algorithms.

[0053] In particular, the person recognition unit 310 can automatically generate a person-based timestamp tag by storing the timestamp of the image frame in which each person appears, based on the person recognition data. Furthermore, the person recognition unit 310 can analyze one or more image frames contained in the image to associate them with the people at each timestamp and store the data. (See reference...) Figure 6The person recognition unit 310 can record the timestamp 620 of the appearance of a person 610 identified from an image frame. As another example, it can also record the person 610 appearing in an image frame at a specific time point based on the timestamp 620. Therefore, the person recognition unit 310 does not specify the time of appearance of a particular person through a separate tool or manual operation, but can automatically generate a timestamp tag based on person recognition using person recognition technology.

[0054] Return to Figure 4 In step S420, the favorability determination unit 320 can determine the user's favorability towards the person identified in step S410. As an example, the favorability determination unit 320 can determine the user's favorability towards the identified person based on the user's resume information (e.g., gender, age, location, etc.). As another example, the favorability determination unit 320 can determine the user's favorability towards the identified person based on the resume information of the user of the electronic device 110 who has previously shown interest. In this case, the resume information can be extracted from the local information of the electronic device 110, for example, from information stored in a video player or similar device installed on the electronic device 110, or from user interest information on the network through linkage with the server 150. For example, the user's favorability towards a particular person can be determined based on the user's viewing history (the proportion of videos featuring a specific person in the videos viewed by the user in the past) or the resume of registered favorite figures.

[0055] In step S430, the object recognition unit 330 identifies objects related to a specific person within the image frame based on the user's favorability rating determined in step S420. The object recognition unit 330 may determine whether a person the user favors is identified in the image frame based on the user's favorability rating, and may determine whether an object is associated with a person based on factors such as distance within the image and whether the object is wearing (or overlapping) clothing. The object recognition unit 330 may identify at least one person among those identified in the image frame based on the user's favorability rating; for example, it may identify objects appearing alongside a specific person based on a person the user prefers. The object recognition unit 330 may also identify objects appearing alongside a person, other than the person themselves, within the image frame based on object recognition technology. (See reference...) Figure 7The object recognition unit 330 segments the image frame 500 into object candidate regions 702 and 703 based on object recognition technology, thereby recognizing objects within the segmented candidate regions 702 and 703. For a person identified in the image frame 500 through the person candidate region 501, the object recognition unit 330 can identify at least one object appearing with the corresponding person through the object candidate regions 702 and 703. Similarly, the object recognition unit 330 tracks the positional changes of a specific object based on target tracking technology and utilizing the similarity between specific information such as the size, color, shape, and outline of objects identified from the image frame. The specific object recognition method can utilize at least one of the well-known target recognition algorithms and target tracking algorithms. Similar to the person recognition unit 310, the object recognition unit 330 is based on target detection and target tracking. Although the person recognition unit 310 and the object recognition unit 330 are described as independent structural elements, they are not limited to this and can also be formed as a combined structural element.

[0056] In particular, the object recognition unit 330 can automatically generate object recognition-based timestamp tags by storing the timestamps of the image frames in which each object appears, according to the object recognition data. Furthermore, the object recognition unit 330 can analyze one or more image frames contained in an image to associate and store objects at each timestamp. (See reference...) Figure 8 The object recognition unit 330 can record the timestamp 840 of the appearance of each object 830 identified from an image frame. As another example, it can also record objects 830 that appear in image frames at specific timestamps 840. As another example, it can also generate timestamp tags by appending the timestamps of each person recorded in the person recognition unit 310 to the objects identified along with the person. (See reference...) Figure 9 The object recognition unit 330 can record objects that appear with a person in the same image frame as objects 830 related to the person 610, based on the timestamp 620 of person recognition. Therefore, the object recognition unit 330 does not specify the time of appearance of a specific object through a separate tool or manual operation, but can automatically generate a timestamp tag based on object recognition using object recognition technology.

[0057] Return to Figure 4In step S440, the additional information providing unit 340 can provide additional information related to the object identified in step S430. The additional information providing unit 340 can provide additional information related to a specific person or object by utilizing timestamp tags based on person and object recognition. In this case, the additional information providing unit 340 can provide additional information on the video player during image playback by overlaying a pop-up window, or by providing additional information in a separate area located at the bottom of the video player.

[0058] The additional information providing unit 340 can provide additional information based on user-preferred person or object information. For example, the additional information providing unit 340 can select objects related to user-preferred persons from the objects identified by the object recognition unit 330 and provide additional information related to the selected objects. Alternatively, the additional information providing unit 340 can distinguish objects related to user-preferred persons from the objects identified by the object recognition unit 330 and display them or provide additional information related to the corresponding objects in a manner that prioritizes them over other objects.

[0059] The supplementary information providing unit 340 associates supplementary information with the scene in which a specific person or object appears, based on timestamp tags that are based on person and object recognition. In this case, the supplementary information providing unit 340 can display supplementary information in a manner associated with the portion of the video frame in which a specific person or object is identified. As an example, the supplementary information providing unit 340 can display supplementary information at the time point of the first or last frame in which a specific person or object appears on the video player. As another example, if a specific person or object appears consecutively for a preset number of frames or more, the supplementary information providing unit 340 can maintain the display of supplementary information during the display of a frame interval containing the specific person or object. Furthermore, as another example, the supplementary information providing unit 340 can pre-display supplementary information at the time point of the appearance of video frames preceding a preset number of video frames in which the specific person or object begins to appear.

[0060] Figures 10 to 14 This is an example diagram of a service screen that provides additional information according to an embodiment of the present invention.

[0061] Reference Figure 10The additional information providing unit 340 can provide a directory of people 1010 and a directory of objects 1020 identified in the image frames being reproduced at the lower end of the video player 1000. As an example, the additional information providing unit 340 can provide a directory of objects 1020 that appears alongside each person included in the directory of people 1010. As another example, the additional information providing unit 340 can also provide the directory of people 1010 and the directory of objects 1020 as separate directories in a manner that does not associate people and objects. For example, the additional information providing unit 340 can provide the clothing of person A in the image, the vehicle driven by person A, the mobile phone used by person A, etc., as the directory of objects 1020. In this case, the directory of people 1010 or the directory of objects 1020 can be associated with at least one abbreviation. Figure 1 The thumbnail can be an image extracted from an image frame when a person or object is identified by the person recognition unit 310 or the object recognition unit 330. When the user inputs and selects through the person directory 1010 and the object directory 1020, the additional information providing unit 340 can provide additional information related to the specific person or object corresponding to the user's selection. The user can consume additional information related to a specific person or object appearing in the playing image not only while enjoying the image through the video player 1000, but also selectively consume additional information about a specific object appearing alongside a specific person. Therefore, in this embodiment, not only the image played back by the video player 1000 can be consumed, but also the people or objects appearing in the image can be consumed.

[0062] The additional information providing unit 340 can provide additional information related to a specific person based on user selection or preference among multiple people identified from the video frames being reproduced from the video player 1000.

[0063] Reference Figure 11 The additional information providing unit 340 can provide a video range directory of the corresponding character's appearance at the bottom of the video player 1000 as additional information 1130 for a specific character. The additional information providing unit 340 can also provide the video range directory 1130 on the progress bar of the video player 1000 in a form where an indicator shows the position of the specific character. The additional information providing unit 340 can provide a scene jump function that allows the user to jump to the selected range based on the range selection using the video range directory 1130.

[0064] Reference Figure 12The additional information providing unit 340 may provide at least one photo 1240 related to a specific person at the lower end of the video player 1000 as another example of additional information related to a specific person. In this case, the photo 1240 may include a scene in an image being reproduced by the video player 1000 where the specific person appears, or a photo of the specific person on the network obtained through a linkage with the server 150. The additional information providing unit 340 provides a function to search for photos of the corresponding person as additional information related to the specific person.

[0065] Reference Figure 13 The additional information providing unit 340 can provide a filmography of the corresponding person to the lower end of the video player 1000 as another example of additional information related to the specific person. The additional information providing unit 340 can provide a catalog of works performed by the specific person along with various statistical information based on user selection or preference. The additional information providing unit 340 can provide a function that allows searching for multiple works performed by the corresponding person as additional information related to the specific person.

[0066] Reference Figure 14 The additional information providing unit 340 can provide shopping information 1460 related to objects appearing alongside a specific person in the image being reproduced by the video player 1000. As an example, the additional information providing unit 340 can display shopping information 1460 on the video player 1000 via a pop-up window, thereby providing users with the ability to make direct purchases. The additional information providing unit 340 can display shopping information 1460 containing a page link for purchasing the corresponding product in a pop-up layer by matching the object identified by object recognition with pre-registered embedded advertising products. In this case, additional information 1130 related to a specific person can be provided at the bottom of the video player 1000, and shopping information 1460 matching the object appearing alongside the specific person can be provided on the video player 1000 via a pop-up window. Furthermore, as another example, the additional information providing unit 340 can display shopping information 1460 that includes a page link for searching objects identified by object recognition via the network 170 and purchasing the corresponding product in a pop-up window layer.

[0067] In addition to the method of displaying shopping information 1460 via a pop-up window, when a user directly selects a specific object in an image being played back by the video player 1000, in order to enable navigation to the purchase page of the product matching the corresponding object, the layer for receiving the user's object selection can be combined with the video player 1000. As another example, it is possible to... Figure 10The various items in the object catalog 1020 form a link structure that allows users to navigate to the purchase page of the product that matches the corresponding object.

[0068] As another example, the additional information providing unit 340 may provide a directory of all people and objects that can be identified throughout the entire image in a portion of the image being reproduced in the video player 1000 (e.g., the last frame of an interval or the last frame of the image). If the directory of people is provided first and at least one person is selected from the directory of people, then the directory of objects associated with the corresponding person can be provided along with each additional piece of information.

[0069] After identifying people and objects in the video being reproduced by the video player 1000 using target recognition and target tracking technologies, the processor 212 can provide additional information related to people, and further, additional information related to objects. In particular, after identifying at least one person from the people identified in the video based on user preference, the processor 212 can provide shopping information matching objects appearing with the specific person, as objects related to the specific person.

[0070] Additional information related to the object is described as shopping information, but is not limited to this. It may include comprehensive search results related to the object, related content such as videos or images, map information, etc.

[0071] According to various embodiments of the present invention, the consumer objects based on the video player are not limited to images, but are expanded to other elements such as people or objects in the images, thereby providing a more interesting viewing environment and making it easier to search for more information related to the people or objects appearing in the images during the viewing process.

[0072] The systems or devices described above can be embodied as hardware structural elements, software structural elements, and / or combinations of hardware and software structural elements. For example, the devices and structural elements described in the embodiments can be embodied using one or more common or special-purpose computers, such as processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field-programmable gate arrays (FPGAs), programmable logic units (PLUs), microprocessors, or other devices capable of executing and responding to instructions. The processing device can execute an operating system (OS) and one or more software applications running on that operating system. Furthermore, the processing device accesses, stores, manipulates, processes, and generates data in response to the execution of the software. For ease of understanding, the case of using only one processing device has been described. Those skilled in the art will recognize that a processing device can include multiple processing elements and / or various types of processing elements. For example, a processing device can include multiple processors or one processor and one controller. It can also be other processing configurations, such as parallel processors.

[0073] Software may comprise computer programs, code, instructions, or combinations thereof, configured to function as needed for a processing device or to issue instructions to the processing device independently or collectively. Software and / or data may embody any type of machinery, structural component, physical device, computer storage medium, or apparatus for interpretation by the processing device or for providing instructions or data to the processing device. Software is distributed across computer systems connected by a network, allowing for storage or execution via a distributed approach. Software and data may be stored on more than one computer-readable recording medium.

[0074] The method of this embodiment is embodied in recording program instructions executable by various computer units on a computer-readable medium. In this case, the medium can continue to store the program executable by a computer, or temporarily store it for execution or download. Furthermore, the medium can be a combination of multiple recording or storage units in the form of a single or multiple hardware components, and is not limited to a medium directly connected to a single computer system, but can exist distributed across a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic disks; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floppy disks; and read-only memory, random access memory, flash memory, etc., which can store program instructions. Furthermore, examples of other media include application stores distributing applications, web pages supplying or distributing various other software, and recording or storage media managed in servers, etc.

[0075] As described above, although multiple embodiments have been illustrated with reference to specific examples and accompanying drawings, those skilled in the art can make various modifications and variations based on the above description. For example, even if the described techniques are executed in a different order than the described methods and / or the structural elements of the described systems, structures, devices, circuits, etc., can be combined or integrated in a different manner than the described methods, or can be replaced or substituted by other structural elements or equivalent technical solutions, the corresponding results can still be achieved.

[0076] Therefore, other examples, other embodiments, and contents equivalent to the scope of protection of the invention also fall within the scope of protection of the invention.

Claims

1. A method for providing additional information, operating in a computer device, characterized in that, The aforementioned computer device includes at least one processor configured to execute a plurality of computer-readable instructions contained in memory. The methods for providing the above additional information include: The person recognition step involves identifying people from image frames during the video playback process of the video player using at least one of the aforementioned processors. The steps for determining user favorability involve using at least one of the aforementioned processors to determine user favorability related to the aforementioned individuals. The step of identification in the image frame involves, through at least one processor, identifying at least one person from the identified persons based on the user preference, and determining the presence of the specific person in the image frames. As the specific person is determined to exist in the image frames, objects related to the specific person are identified from the determined image frames. The steps involve providing additional information about objects related to the specific person, through at least one of the aforementioned processors. The user favorability rating associated with the aforementioned individuals is determined based on the user's resume information on the aforementioned computer device and / or the user's viewing history and / or the resume information of individuals registered as favorites. The aforementioned person recognition steps include: generating tags based on the timestamps of the individuals' appearances, according to the timestamps stored for each individual. The aforementioned steps for identification within image frames include: using the aforementioned timestamp based on person identification as a basis, adding objects that appear together with the aforementioned person within the same image frame as objects related to the aforementioned person to the timestamp based on the aforementioned person. The above-mentioned steps include: using the timestamp based on person recognition to provide a directory of people identified from the above-mentioned image frames, and providing a directory of objects that appear together with each person in the above-mentioned directory of people.

2. The method for providing additional information according to claim 1, characterized in that, The aforementioned person recognition steps include the following steps: using person recognition technology to identify the aforementioned person in the candidate regions segmented after segmenting the aforementioned image frames.

3. The method for providing additional information according to claim 1, characterized in that, The steps for recognition in the image frame include the following steps: using object recognition technology to identify the object in the candidate regions segmented from the image frame.

4. The method for providing additional information according to claim 1, characterized in that, The steps for recognition in the image frame include the following steps: generating a timestamp label based on object recognition by storing the timestamp of the object's appearance.

5. The method for providing additional information according to claim 1, characterized in that, The above-mentioned steps include the following steps: providing the above-mentioned additional information by associating it with a portion of the image frame in which the object is identified.

6. The method for providing additional information according to claim 1, characterized in that, The above-mentioned steps include providing the above-mentioned additional information at the time point in the first or last frame of the image frame in which the above-mentioned object appears.

7. The method for providing additional information according to claim 1, characterized in that, The above-mentioned steps include providing the above-mentioned additional information at the time point when the image frames appear before a preset number of image frames based on the image frames in which the above-mentioned object appears.

8. The method for providing additional information according to claim 1, characterized in that, The aforementioned steps include providing shopping information for goods that match the aforementioned object, as additional information for the object that appears with the aforementioned specific person.

9. A non-transitory computer-readable recording medium, characterized in that, The computer program is stored thereon and runs in a computer system the method for providing additional information according to any one of claims 1 to 8.

10. A computer device, characterized in that, Includes at least one processor configured to execute computer-readable instructions contained in memory. The aforementioned at least one processor includes: The person recognition unit is used to identify people from image frames during the video playback process of the video player. The favorability assessment department is used to determine the favorability of users related to the aforementioned individuals. An object recognition unit is configured to, based on the aforementioned user preference, identify at least one person from the identified persons, determine from the aforementioned image frames that contain the specific person, and, upon determining that the specific person exists in the aforementioned image frames, identify objects related to the specific person from the determined image frames; and The Additional Information Provision Department provides additional information about objects related to the aforementioned specific individuals. The user favorability rating associated with the aforementioned individuals is determined based on the user's resume information on the aforementioned computer device and / or the user's viewing history and / or the resume information of individuals registered as favorites. The aforementioned person recognition unit is used to generate tags based on the timestamps of the persons' appearances stored in the aforementioned person database. The object recognition unit is used to, based on the timestamps of the person recognition, add objects that appear together with the person in the same image frame as objects related to the person to the timestamps of the person. The aforementioned additional information providing unit is used to provide a directory of people identified from the aforementioned image frames using the aforementioned timestamp based on person recognition, and to provide a directory of objects that appear together with each person in the aforementioned directory of people.

11. The computer apparatus according to claim 10, characterized in that, The aforementioned person recognition unit uses person recognition technology to identify the aforementioned person in the candidate regions segmented from the aforementioned image frames.

12. The computer device according to claim 10, characterized in that, The object recognition unit uses object recognition technology to identify the object in the candidate regions segmented from the image frame. The object recognition unit generates a timestamp tag based on object recognition by storing the timestamp of the object's appearance.

13. The computer apparatus according to claim 10, characterized in that, The aforementioned additional information providing unit provides shopping information for goods matching the aforementioned object at the time point in the first or last frame of the image frame in which the aforementioned object appears, or at the time point in the image frame preceding the aforementioned object by a preset number of frames, as additional information for the aforementioned object that appears with the aforementioned specific person.

Images