Video distribution system, video distribution method

Abstract

To finely estimate interest objects of users who are watching video content.SOLUTION: An operation signal receiving unit 213 receives an operation signal for enlarging, reducing, or moving a screen on which video is displayed from a user terminal to which a video content is distributed. A cut out unit 214 cuts out a region in the video to be distributed to a user terminal in units of frame images on the basis of the received operation signal. A distribution unit 212 distributes video data of the cut out region to the user terminal. An attention region information holding unit 223 accumulates position information of a region of interest which the user paid attention to, on the basis of the operation signal received from the user terminal. An interest object identifying unit 216 identifies an object reflected in the region of interest in the frame image of the video content.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a video distribution system and a video distribution method for distributing video content to a user terminal. [Background technology]

[0002] A technology has been developed that extracts 2K video from 8K high definition video and streams it to user devices such as smartphones and tablets. In video distribution services that use this technology, users can pinch out / in on any area of ​​a video displayed on the touch panel display of their device to zoom in / out the video screen with the same feel as for still images. Users can also move the position of the video area displayed on the screen by dragging the video screen.

[0003] Meanwhile, there is known a targeted advertisement that estimates a user's taste based on the contents of the user's search and displays advertisements that match the user's taste on the user terminal (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2020-522820 Summary of the Invention [Problem to be solved by the invention]

[0005] Conventional targeted advertising has assumed the user's preferences based on the user's active actions (e.g., searching or browsing a specific web page). Objects that the user is only mildly interested in on the screen are hardly taken into consideration.

[0006] The present disclosure has been made in consideration of these circumstances, and one of its objectives is to provide a technology for precisely estimating the interests of a user who is viewing video content. [Means for solving the problem]

[0007] In order to solve the above problems, a video distribution system according to one aspect of the present disclosure includes an operation signal receiving unit that receives an operation signal for enlarging, reducing or moving a screen on which a video is displayed from a user terminal to which video content is distributed, an extraction unit that extracts an area within the video in frame image units based on the received operation signal and distributes it to the user terminal, a distribution unit that distributes video data of the extracted area to the user terminal, an attention area information storage unit that accumulates position information of the area that the user focuses on based on the operation signal received from the user terminal, and a target identification unit that identifies an object that appears in the attention area within the frame image of the video content.

[0008] Another aspect of the present disclosure is a video distribution method, which includes the steps of receiving an operation signal for enlarging, reducing, or moving a screen on which a video is displayed from a user terminal to which video content is distributed, cutting out an area in the video to be distributed to the user terminal on a frame image basis based on the received operation signal, distributing video data of the cut-out area to the user terminal, accumulating position information of an area focused on by a user based on the operation signal received from the user terminal, and identifying an object captured in the area of ​​interest in a frame image of the video content.

[0009] In addition, any combination of the above components, and any conversion of the expressions of the present disclosure between an apparatus, a system, a method, a computer program, a recording medium storing a computer program, etc. are also valid as aspects of the present disclosure. Effect of the Invention

[0010] According to the present disclosure, it is possible to precisely estimate the interests of a user who is viewing video content. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a block diagram illustrating components used in a video distribution service according to an embodiment. [Diagram 2] 2 is a block diagram showing a configuration example of a content holder terminal according to the embodiment; FIG. [Diagram 3] 11 is a diagram showing an example of registering an object shown in a frame image of video content. FIG. [Figure 4] 1 is a block diagram showing a configuration example of a video distribution system according to an embodiment. [Diagram 5] FIG. 2 is a block diagram showing a configuration example of a user terminal according to an embodiment. [Figure 6] FIG. 11 is a diagram showing an example of a video screen displayed on a touch panel display of a user terminal. [Figure 7] 11A and 11B are diagrams illustrating an example of a pinch out operation on a video screen displayed on a touch panel display of a user terminal. [Figure 8] FIG. 7 is a diagram showing the video screen after the area around the face of the female actor is enlarged from the video screen shown in FIG. 6. [Figure 9] FIG. 7 is a diagram showing a video screen after the area near the lower body of the male actor is enlarged from the video screen shown in FIG. 6. [Figure 10] 1 is a flowchart showing a flow of a process for uploading video content from a content holder terminal to a video distribution system according to the first embodiment. [Figure 11] 10 is a flowchart showing a process flow when video content is viewed by a user terminal according to the first embodiment. [Figure 12] 11 is a flowchart for explaining a first example of utilization of end-user interest information generated by the video distribution system according to the first embodiment. [Figure 13] 11 is a flowchart for explaining a second example of utilization of the end-user interest information generated by the video distribution system according to the first embodiment. [Figure 14] 13 is a flowchart for explaining a third example of use of the end-user interest information generated by the video distribution system according to the first embodiment. [Figure 15] 13 is a flowchart showing a flow of a process for uploading video content from a content holder terminal to a video distribution system according to a second embodiment. [Figure 16] 13 is a flowchart for explaining a first example of utilization of end-user interest information generated by the video distribution system according to the second embodiment. [Figure 17] 13 is a flowchart for explaining a second example of utilization of the end-user interest information generated by the video distribution system according to the second embodiment. [Figure 18] 13 is a flowchart for explaining a third example of use of the end-user interest information generated by the video distribution system according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] This embodiment relates to a video distribution service that utilizes a technology for extracting 2K (1920 x 1080) video from 8K (7680 x 4320) video. 8K video is a high-resolution image, and even if a part of the image is enlarged, the image can be displayed clearly. For example, it is said that a golf ball can be clearly confirmed even in a landscape image several tens of kilometers away. As of 2020, networks that can smoothly stream 8K video are not widespread, and a service that extracts 2K video from 8K video and streams the 2K video has been developed.

[0013] 1 is a block diagram showing components used in a video distribution service according to an embodiment. The video distribution service according to the embodiment mainly uses a content holder terminal 10, a video distribution system 20, and a user terminal 30. The video distribution system 20 is configured to be connectable to a plurality of content holder terminals 10 and a plurality of user terminals 30 via a network 50.

[0014] The content holder terminal 10 is a terminal device owned by a right holder of video content. For example, it may be a PC or a server. A camera 5 is connected to the content holder terminal 10, and video data captured by the camera 5 is read into the content holder terminal 10. Possible subjects that may be captured by the camera 5 include plays, concerts, and sports. Note that video content is not limited to these, and may also include movies, dramas, documentaries, variety shows, food reports (reports on food), and travel reports (reports on travel). The content holder terminal 10 can upload the video content loaded from the camera 5 to the video distribution system 20 via the network 50.

[0015] The video distribution system 20 is a system constructed and operated by a video distribution platformer. It is constructed by a plurality of servers and client PCs installed in the company's own data center or an external data center. The video distribution system 20 accumulates a plurality of video contents uploaded from a plurality of content holder terminals 10, and can stream video contents selected by end users to user terminals 30.

[0016] The user terminal 30 is a terminal device possessed by an end user that can access the network 50. For example, a smartphone or a tablet is an example of the user terminal 30. The end user can view any video content for a fee or free of charge by registering as a member of a video distribution service.

[0017] The network 50 is a general term for communication paths such as the Internet and dedicated lines, and the communication medium and protocol are not important. For example, an optical fiber network, an ADSL network, a CATV network, a mobile communication network, a wireless LAN, a wired LAN, etc. can be used as a communication medium. For example, a Transmission Control Protocol (TCP) / Internet Protocol (IP), a User Datagram Protocol (UDP) / IP, Ethernet (registered trademark), HTTP Live Streaming (HLS), Dynamic Adaptive Streaming over HTTP (MPEG-DASH), etc. can be used as a communication protocol.

[0018] 2 is a block diagram showing an example of the configuration of a content holder terminal 10 according to an embodiment. The content holder terminal 10 includes a processing unit 11, a storage unit 12, an operation unit 13, a display unit 14, and a communication unit 15. The processing unit 11 includes a video data reading unit 111, a video data editing unit 112, a video content transmitting unit 113, and a target placement information generating unit 114. The processing unit 11 can be realized by a combination of hardware resources and software resources, or by hardware resources alone. As the hardware resources, a CPU, a ROM, a RAM, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a chip set, and other LSIs can be used. As the software resources, a program such as an operating system, middleware, and an application can be used.

[0019] The storage unit 12 includes a video content storage unit 121 and a target placement information storage unit 122. The storage unit 12 includes a non-volatile recording medium such as a hard disk drive (HDD) or a solid state drive (SSD).

[0020] The operation unit 13 includes user interfaces such as a keyboard, a mouse, and a touch panel, and accepts operations from the person in charge of the content holder. The operation unit 13 outputs the accepted operation contents to the processing unit 11. The display unit 14 includes a display such as a liquid crystal display or an organic EL display, and displays an image generated by the processing unit 11. The communication unit 15 executes communication processing for communicating with the video distribution system 20 via the network 50.

[0021] The video data reading unit 111 reads video data captured by the camera 5 from the camera 5. In the present embodiment, the camera 5 is a camera capable of capturing 8K video, and the video data reading unit 111 reads the 8K video data from the camera 5.

[0022] The video data editing unit 112 edits the video data read from the camera 5 in response to an operation signal from the operation unit 13. An editor at the content holder can complete the video content by editing the video data, such as trimming, dividing, adding effects, transitions, etc. Note that the video data editing unit 112 is not necessary when the captured video data is uploaded as is without editing.

[0023] The video content transmitting unit 113 transmits the video content edited by the video data editing unit 112 or the video content as is, that is, the captured video data, to the video distribution system 20 via the network 50.

[0024] Video content storage unit 121 stores video content transmitted to video distribution system 20. Note that raw video data captured by camera 5 and video data in the middle of being edited may also be stored.

[0025] The object location information generating unit 114 generates object location information that specifies the position information of an object shown in each frame image of the video content. The object may be any object that may be of interest to the end user, such as a specific object (including an item), a specific person, a specific logo, a specific character, a specific animal, a specific vehicle, or a specific building.

[0026] The content holder's staff operates the operation unit 13 to surround an object to be registered among objects shown in each frame image of the video content with a frame, and inputs the name of the surrounded object (e.g., a product name, a person's name, etc.). The object placement information generation unit 114 associates the frame ID, the position information of the surrounded object (e.g., defined by the coordinate information of the vertices of a rectangular frame), and the input name, and registers them in the object placement information storage unit 122.

[0027] Fig. 3 is a diagram showing an example of registering an object shown in a frame image F1 of video content. The example shown in Fig. 3 shows an example of registering an object shown in a frame image F1 of video content filmed from a theatrical performance. The content holder's staff surrounds male actor A1's costume I1, left boot I2, right boot I3, female actor A2's costume I4, and earring I5 in frame image F1 with rectangular frames, and inputs the names of each item.

[0028] The object placement information generating unit 114 links the frame ID of the frame image F1, the position information of each of the enclosed items I1-I5, and the names of each of the input items I1-I5, and registers them in the object placement information holding unit 122.

[0029] The object location information generating unit 114 extracts features of each object in the enclosed area and generates a classifier for each object. The object location information generating unit 114 searches for each object in each subsequent frame image using the classifier for each object. As features for image recognition of each object, HOG (Histograms of Oriented Gradients) features, Haar-like features, LBP (Local Binary Patterns) features, etc. can be used.

[0030] When the object specified in the previous frame image is detected in each subsequent frame image, the object placement information generation unit 114 links the frame ID, the position information of the detected object, and the name of the object, and registers them in the object placement information storage unit 122.

[0031] The target location information generating unit 114 may track a target designated by a content holder in a certain frame image in a subsequent frame image. For example, a particle filter or a mean shift method may be used to track the target.

[0032] When the target placement information generation unit 114 generates a target placement information file describing the placement information of the target shown in each frame image of the video content, the video content transmission unit 113 transmits the target placement information file of the video content in addition to the video content to the video distribution system 20.

[0033] It is not essential to generate a target placement information file for video content. If a target placement information file is not generated, the target placement information generating unit 114 and the target placement information holding unit 122 are not necessary, and the video content transmitting unit 113 transmits only the video content to the video distribution system 20.

[0034] 4 is a block diagram showing a configuration example of a video distribution system 20 according to an embodiment. The video distribution system 20 includes a processing unit 21, a storage unit 22, an operation unit 23, a display unit 24, and a communication unit 25. The processing unit 21 includes a video content receiving unit 211, a distribution unit 212, an operation signal receiving unit 213, a cutout unit 214, an attention area identifying unit 215, an interest target identifying unit 216, and a notification unit 217. The processing unit 21 can be realized by a combination of hardware resources and software resources, or by hardware resources alone. As the hardware resources, a CPU, a ROM, a RAM, a GPU, a DSP, an ASIC, an FPGA, a chipset, and other LSIs can be used. As the software resources, a program such as an operating system, middleware, and an application can be used.

[0035] The storage unit 22 includes a member information storage unit 221, a video content storage unit 222, an attention area information storage unit 223, and an object placement information storage unit 224. The storage unit 22 includes a non-volatile recording medium such as an HDD or SSD.

[0036] The operation unit 23 includes user interfaces such as a keyboard, mouse, and touch panel, and accepts operations from a person in charge of the video distribution platform. The operation unit 23 outputs the accepted operation contents to the processing unit 21. The display unit 24 includes a display such as a liquid crystal display or an organic EL display, and displays an image generated by the processing unit 21. The communication unit 25 executes communication processing for communicating with the content holder terminal 10 or the user terminal 30 via the network 50.

[0037] The member information storage unit 221 stores member information of end users who have registered as members of the video distribution platform. For example, information such as user ID, password, name, date of birth, gender, nickname, email address, mobile email address, telephone number, mobile phone number, address, company name / organization, department name / job title, credit card information (brand name, card number, expiration date, account holder), bank account number (bank name, account number, account holder), etc. may be registered as member information even if optional information is left blank.

[0038] In addition, when a video distribution service is provided by a comprehensive platform provider that offers a variety of services over the Internet, such as shopping malls, mobile communications, finance, travel, e-books, and flea markets, you can log in to the video distribution service using SSO (Single Sign On) with your comprehensive platform provider's user ID and password.

[0039] The video content receiving unit 211 receives the video content transmitted from the content holder terminal 10, and stores it in the video content holding unit 222. When a target placement information file is added to the transmitted video content, the video content receiving unit 211 stores the target distribution information file in the target placement information holding unit 224.

[0040] 5 is a block diagram showing an example of the configuration of a user terminal 30 according to an embodiment. The user terminal 30 includes a processing unit 31, a storage unit 32, an operation unit 33, a display unit 34, and a communication unit 35. The processing unit 31 includes a video data receiving unit 311 and an operation signal transmitting unit 312. The processing unit 31 can be realized by a combination of hardware resources and software resources, or by hardware resources alone. As the hardware resources, a CPU, a ROM, a RAM, a GPU, a DSP, an ASIC, an FPGA, an SoC (System on a Chip), and other LSIs can be used. As the software resources, programs such as an operating system, middleware, and applications can be used. The storage unit 32 has a non-volatile recording medium (for example, a NAND type flash memory) and stores various programs and data.

[0041] The operation unit 33 and the display unit 34 are configured with a touch panel display TP. The touch panel display TP detects the coordinates of the touch position touched by the end user's finger, for example, by a capacitance method. The touch panel display TP displays an image generated by the processing unit 31, accepts operations by the end user, and outputs the contents of the accepted operations to the processing unit 31. Note that, in addition to the touch panel display TP, physical buttons (for example, a power button, a volume button) may be provided as the operation unit 33.

[0042] The communication unit 35 executes a communication process for communicating with the video distribution system 20 via the network 50. When the user terminal 30 is a smartphone or a tablet, the communication unit 35 executes a wireless communication process for wirelessly accessing a base station via a 4G network / 5G network. Alternatively, the communication unit 35 executes a wireless communication process for wirelessly accessing an access point via Wi-Fi (registered trademark). When the user terminal 30 is a PC or a TV, a LAN cable is connected to a router device, and the communication unit 35 executes a communication process conforming to Ethernet. When the user terminal 30 is a PC or a TV, the access point may be wirelessly accessed via Wi-Fi.

[0043] The end user operates the operation unit 33 of the user terminal 30 to access the site of the video distribution service and log in. The end user selects the video content that he or she wishes to view and presses the play button.

[0044] The delivery unit 212 of the video distribution system 20 starts distribution of the selected video content. The cutout unit 214 cuts out an area in the video to be distributed to the user terminal 30 in units of frame images. That is, the cutout unit 214 generates a low-resolution image from a high-resolution image in units of frame images of the video content. The cutout unit 214 can generate a low-resolution image from a high-resolution image by thinning or averaging pixels.

[0045] In this embodiment, the cropping unit 214 crops out a 2K (1920×1080) image from an 8K (7680×4320) image. At the start of distribution, the cropping unit 214 converts the size of the 8K (7680×4320) image into an image of 1 / 16 the size. The cropping unit 214 may generate a 2K image by thinning out 3 / 4 of the pixels vertically and 3 / 4 of the pixels horizontally, or may average adjacent 16 pixels and convert them into one pixel to generate a 2K image. Note that when the transmission bandwidth of the network 50 is wide, the cropping unit 214 may crop out a 4K (3840×2160) image from the 8K (7680×4320) image.

[0046] The video data receiving unit 311 of the user terminal 30 receives video data distributed from the video distribution system 20 and displays it on the touch panel display TP. The end user can touch the video screen displayed on the touch panel display TP to enlarge, reduce, or move the video screen with the same operation feeling as for a still image. Specifically, the end user can enlarge a specific area of ​​the video screen by pinching out with two fingers. The end user can reduce the enlarged video screen by pinching in with two fingers. In addition, when a part of the video screen is enlarged, the end user can move the position of the video screen displayed on the touch panel display TP by dragging the video screen with one finger.

[0047] The operation signal transmitting unit 312 transmits an operation signal including coordinate information of the touch position detected on the touch panel display TP to the video distribution system 20 via the network 50. The operation signal receiving unit 213 of the video distribution system 20 receives the operation signal transmitted from the user terminal 30.

[0048] The cropping unit 214 determines the cropping range of the image to be delivered to the user terminal 30 from each frame image based on the received operation signal. The cropping unit 214 determines the operation instruction content of the end user based on the time transition of the coordinates of the touch position. Specifically, the cropping unit 214 recognizes that the operation is an enlargement operation when the distance between the coordinate groups of two independent touch positions is increasing. At that time, the cropping unit 214 determines the enlargement ratio according to the distance. Also, the cropping unit 214 recognizes that the distance between the coordinate groups of two independent touch positions is decreasing as a reduction operation. At that time, the cropping unit 214 determines the reduction ratio according to the distance by which the two independent touch positions are decreasing. Also, the cropping unit 214 recognizes that the coordinate group of one touch position is moving as a screen movement operation. At that time, the cropping unit 214 determines the screen movement direction and the movement amount according to the direction and distance of the moved coordinate group.

[0049] The cropping unit 214 determines the cropping range from the 8K (7680×4320) image according to the operation instruction content of the end user. The resolution of the video data delivered to the user terminal 30 is always constant (2K resolution in this embodiment). For example, when the end user is watching video content by zooming in 4 times, the cropping range is 1 / 4 of the range of the 8K image. In this case, the cropping unit 214 converts the size of the cropping range (3840×2160) of the 8K (7680×4320) image to an image of 1 / 4 the size, and generates a 2K image for delivery. The cropping unit 214 adaptively changes the thinning rate or the number of pixels to be averaged according to the screen magnification rate by the end user.

[0050] In addition, if the end user's screen magnification is large, the size of the cropped range of the 8K image may be smaller than the size of the 2K image to be distributed. In this case, the cropping unit 214 interpolates pixels within the cropped range of the 8K image to generate a 2K image to be distributed.

[0051] The delivery unit 212 delivers the video data of the area cut out by the cutout unit 214 to the user terminal 30. The delivery unit 212 ends the delivery of the video data to the user terminal 30 when the playback of the video content ends.

[0052] Based on the operation signal received from the user terminal 30, the attention area identifying unit 215 identifies the position information of the area focused on by the end user on a frame image basis, and accumulates it as attention area information in the attention area information storage unit 223. The area focused on by the end user corresponds to the area displayed by enlarging the screen on the display unit 34 of the user terminal 30. Note that when the magnification is small, the video area displayed on the screen is large, and it can be estimated that the end user is unlikely to be focusing on something in the video.

[0053] Therefore, the attention area specifying unit 215 may accumulate only the position information of the cut-out range of the frame image viewed by the end user in a state where the magnification exceeds the first set value. The position information of the cut-out range can be specified by the coordinate information of the vertices of the cut-out range. Specifically, it may be specified by the coordinate information of four vertices, or may be specified by the coordinate information of two diagonal vertices.

[0054] The first setting value can be changed by the person in charge of the video distribution platform. The larger the first setting value is set, the fewer the number of samples in the end user's attention area, but the more reliable each sample is. Conversely, the smaller the first setting value is set, the more the number of samples in the end user's attention area, but the more reliable each sample is.

[0055] FIG. 6 is a diagram showing an example of a video screen displayed on the touch panel display TP of the user terminal 30. Here, the end user is Let's say you are interested in earrings E1.

[0056] 7 is a diagram showing an example of a pinch out operation on a video screen displayed on the touch panel display TP of the user terminal 30. The end user enlarges the area around the face of the female actor A2 by pinching out with two fingers Fin (thumb and index finger in FIG. 7), and moves the area around the face of the female actor A2 to the center of the screen by dragging.

[0057] Fig. 8 is a diagram showing a video screen after the area around the face of the female actor A2 is enlarged from the video screen shown in Fig. 6. After enlargement, the earring E1 is also enlarged, and the shape and color of the earring E1 can be confirmed in detail. The attention area identification unit 215 accumulates position information of the cut-out range (including the earring E1) of the frame image distributed from the distribution unit 212 in the attention area information storage unit 223 as position information of the end user's attention area.

[0058] Assume that an end user is interested in the boots B1 and B2 worn by a male actor A1. The end user pinches out with two fingers to enlarge the area of ​​the lower body of male actor A1, and drags the area near the lower body of male actor A1 to the center of the screen.

[0059] Fig. 9 is a diagram showing a video screen after the area around the lower body of male actor A1 is enlarged from the video screen shown in Fig. 6. After enlargement, boots B1 and B2 are also enlarged, allowing the shape and color of boots B1 and B2 to be confirmed in detail. The attention area identification unit 215 accumulates position information of the cut-out range (including boots B1 and B2) of the frame image distributed from the distribution unit 212 in the attention area information storage unit 223 as position information of the end user's attention area.

[0060] The interest target identification unit 216 of the video distribution system 20 identifies a target that is shown in a region of interest in a frame image of a video content. There are two main methods for identifying an interest target. The first identification method is an example of a case where a target placement information file is generated by a content holder, and the second identification method is an example of a case where a target placement information file is not generated by a content holder.

[0061] In the first identification method, the interest target identification unit 216 identifies an object shown in an attention area in a frame image of the video content based on the object location information of the video content and the attention area information of the video content. That is, the interest target identification unit 216 searches whether or not an object defined by the object location information of the video content is included in each attention area of ​​the frame image of the video content.

[0062] In the second identification method, the interest target identification unit 216 executes an image search for an attention target area that is highly popular among end users in each frame image of the video content, and identifies an object that appears in the attention target area. The interest target identification unit 216 reads attention target area information from the attention target area information storage unit 223 for each video content, and identifies an attention target area that is popular among a specific or unspecified number of end users. The interest target identification unit 216 overlaps attention targets of all frame images for each video content. The interest target identification unit 216 selects an area where the overlapping number exceeds a second set value as an attention target area that is highly popular among a specific or unspecified number of end users (hereinafter referred to as a selected attention target area).

[0063] Like the first setting value, the second setting value can also be changed by the person in charge of the video distribution platform. The larger the second setting value is set, the fewer the number of samples in the selected attention area, but the higher the reliability of each sample. Conversely, the smaller the second setting value is set, the more samples in the selected attention area will be, but the lower the reliability of each sample will be.

[0064] In addition, when selecting the selected attention region, the interest target identification unit 216 may take into consideration the size of the overlapping region of the attention regions. The interest target identification unit 216 may relax the criteria for selecting the selected attention region as the smaller the size of the overlapping region. In other words, the second setting value may be set to a lower value as the size of the overlapping region is smaller. The more the end user enlarges the screen, the smaller the size of the attention region identified by the attention region identification unit 215 becomes. When the end user enlarges the screen, it can be estimated that there is a high possibility that the end user is paying attention to something displayed in the enlarged screen.

[0065] The interest target identification unit 216 performs an image search using a predetermined image recognition algorithm based on image data in which the target to be detected appears within each selected region of interest. The interest target identification unit 216 extracts features of the target to be detected from the image data in which the target appears, and generates a classifier for the target. The interest target identification unit 216 searches for each target within each selected region of interest using the generated classifier for each target.

[0066] If the entity providing the video distribution service is a general platform operator that operates an online shopping mall, image data of products on the websites of each shop in the online shopping mall can be used as image data showing the target to be detected. In this case, only image data of products requested by shops wishing to promote their products may be searched for.

[0067] The notification unit 217 notifies information on a target within the video content that the end user has focused on to a party outside the video distribution system 20. Specific notification destinations and notification methods will be described later.

[0068] Example 1 The first embodiment is an example in which the above-mentioned first identification method is adopted as a method for identifying an end user's subject of interest.

[0069] 10 is a flowchart showing a flow of a process of uploading video content from the content holder terminal 10 to the video distribution system 20 according to the first embodiment. The video data reading unit 111 of the content holder terminal 10 reads video data captured by the camera 5 from the camera 5 and stores it in the video content holding unit 121 (S10). The video data editing unit 112 reads out the video data to be edited from the video content holding unit 121 and edits the video data based on an operation by an editor of the content holder (S11).

[0070] The target placement information generating unit 114 generates target placement information that specifies position information of targets shown in each frame image of the video content, and stores the generated information in the target placement information holding unit 122 (S12). The video content transmitting unit 113 reads out files of the video content to be uploaded and its target placement information from the video content holding unit 121 and the target placement information holding unit 122, respectively, and transmits the files to the video distribution system 20 via the network 50 (S13).

[0071] The video content receiving unit 211 of the video distribution system 20 receives the video content and its target placement information file transmitted from the content holder terminal 10 (S14), and stores them in the video content storage unit 222 and the target placement information storage unit 224, respectively (S15).

[0072] 11 is a flowchart showing a process flow when video content is viewed by the user terminal 30 according to the first embodiment. The operation signal transmitting unit 312 of the user terminal 30 accesses a site of a video distribution service in response to an operation by an end user, and transmits a user ID and a password to the video distribution system 20 (S20). The operation signal receiving unit 213 of the video distribution system 20 receives the user ID and password transmitted from the user terminal 30. An authentication unit (not shown) of the video distribution system 20 refers to the user ID and password held in the member information holding unit 221, and authenticates the transmitted user ID and password (S21). The authentication unit transmits the authentication result of the user ID and password to the user terminal 30.

[0073] If the user ID and password sent from the user terminal 30 are registered in the member information storage unit 221, the authentication unit permits login using that user ID, and if they are not registered, it denies login using that user ID.

[0074] When login using the user ID is permitted, the operation signal transmitting unit 312 of the user terminal 30 transmits a playback instruction signal for the video content selected by the end user to the video distribution system 20 (S22). The operation signal receiving unit 213 of the video distribution system 20 receives the playback instruction signal for the video content selected by the end user transmitted from the user terminal 30. The distribution unit 212 starts distribution of the selected video content (S23). The cut-out unit 214 cuts out an area within the video for each frame image to be distributed to the user terminal 30 (S24). The distribution unit 212 distributes the video data of the area cut out by the cut-out unit 214 to the user terminal 30 via the network 50 (S25).

[0075] The video data receiving unit 311 of the user terminal 30 receives the video data distributed from the video distribution system 20 and displays it on the touch panel display TP (S26). When a touch operation by the end user on the touch panel display TP is detected (Y of S27), the operation signal transmitting unit 312 transmits an operation signal corresponding to the touch operation by the end user to the video distribution system 20 (S28).

[0076] The operation signal receiving unit 213 of the video distribution system 20 receives an operation signal transmitted from the user terminal 30. The cropping unit 214 determines a cropping range of an image to be distributed to the user terminal 30 from each frame image based on the received operation signal (S29). When the size of the cropping range is equal to or larger than the size of the video screen enlarged by the first setting value, the attention area identifying unit 215 accumulates position information of the cropping range as attention area information in the attention area information storage unit 223 (S210).

[0077] On the user terminal 30 side, while the video content is being played back (N in S211), the process proceeds to step S26 and the processes of steps S26 to S28 are repeated. When the end user presses the playback end key on the touch panel display TP (Y in S211), the operation signal transmission unit 312 transmits a playback end instruction signal to the video distribution system 20.

[0078] On the video distribution system 20 side, while the video content is being distributed (N in S212), the process proceeds to step S24 and repeats the processes of steps S24, S25, S29, and S210. When a playback end instruction signal is received from the user terminal 30, or when the video content is played to the end (Y in S212), the distribution unit 212 ends the distribution of the video content to the user terminal 30.

[0079] 12 is a flowchart for explaining a first application example of the interest information of an end user generated by the video distribution system 20 according to the embodiment 1. The first application example is an example in which interest information of a specific end user is generated and targeted advertising is provided to the end user.

[0080] The interest target identifying unit 216 of the video distribution system 20 reads attention area information of one or more video contents viewed by a specific user ID from the attention area information storage unit 223 (S30). The interest target identifying unit 216 reads target placement information of each video content from the target placement information storage unit 224. The interest target identifying unit 216 refers to the target placement information of each read video content, and identifies targets included in each attention area of ​​the user ID (S31).

[0081] The interest target identification unit 216 calculates the degree of attention of the end user to the target according to at least one of the time the end user viewed the attention area including the target and the enlargement rate of the attention area including the target (S32). Specifically, the interest target identification unit 216 sets the degree of attention to a higher value as the time the end user viewed the attention area including the target is longer. Also, the interest target identification unit 216 sets the degree of attention to a higher value as the enlargement rate of the attention area including the target is larger.

[0082] The interest target identification unit 216 selects a target for which the calculated degree of interest of the end user is higher than the third set value as the interest target of the end user (S33). The third set value can be changed by the person in charge of the video distribution platform, similar to the first and second set values. The larger the third set value is set, the fewer the number of samples of the selected target will be, but the higher the reliability of each sample will be. Conversely, the smaller the third set value is set, the more the number of samples of the selected target will be, but the lower the reliability of each sample will be.

[0083] The notification unit 217 notifies an advertisement distribution system (not shown) of information on the end user's interests identified by the interest identification unit 216 (hereinafter referred to as interest information) (S34). When multiple interests are selected, the interest information may include attention level information for each interest.

[0084] If the entity providing the video distribution service is a comprehensive platform operator that operates an online shopping mall, and the comprehensive platform operator also operates and manages an advertisement distribution system, the interest information can be utilized as follows: The advertisement distribution system sends an email to the end user's email account, which includes a uniform resource locator (URL) that leads the end user to the website of a shop in the online shopping mall that sells a product that the end user is interested in.

[0085] In addition, when a site of a service provided by the integrated platform operator is viewed from the user terminal 30 used by the end user, the advertising distribution system uses the IP address of the user terminal 30 used by the end user to display a banner advertisement on the viewed site to guide the user to the site of the shop.

[0086] The advertisement delivery system may be a system operated and managed by an external Internet advertising agency. In this case, the notification unit 217 provides the end user's interest information to the advertisement delivery system within the scope of the personal information protection regulations. For example, a set of the IP address and the interest information may be sold to an external Internet advertising agency.

[0087] The notification unit 217 notifies the end user of the interest information to an account of a shop in an online shopping mall that sells the product that the end user is interested in (S35). The general platform provides the member stores that are open in the online shopping mall with tools and services necessary for operating an online shop. For example, the platform provides web page creation tools, order management and shipping tools, and customer attraction and sales promotion tools. The notification unit 217 may notify the account of the customer attraction and sales promotion tools of the end user's interest information. The notification unit 217 may also notify the mail account of the shop of the interest information of the end user by email. The shop can use the notified interest information to promote the sale of the product.

[0088] It should be noted that only one of the notification process in step S34 and the notification process in step S35 may be executed.

[0089] 13 is a flowchart for explaining a second application example of the end user interest information generated by the video distribution system 20 according to the first embodiment. The second application example is an example in which interest information of an unspecified number of end users for a specific video content is generated and provided to a distributor of the video content.

[0090] The interest target identifying unit 216 of the video distribution system 20 reads attention area information of specific video content viewed by all user IDs of the video distribution service from the attention area information storage unit 223 (S40). The interest target identifying unit 216 reads target placement information of the video content from the target placement information storage unit 224. The interest target identifying unit 216 refers to the read target placement information of the video content and identifies targets included in each attention area of ​​all user IDs (S41).

[0091] The interest target identification unit 216 calculates the degree of attention of each end user to the target according to at least one of the time each end user viewed the attention area including the target and the magnification rate of the attention area including the target. The interest target identification unit 216 calculates the degree of attention of all end users to the target by summing up the attention degrees of each end user (S42).

[0092] The interest target identification unit 216 selects a target with a calculated degree of interest of the entire end users higher than the fourth setting value as an interest target of the entire end users (S43). The fourth setting value can also be changed by the person in charge of the video distribution platform provider, similar to the first to third setting values.

[0093] The notification unit 217 notifies the content holder terminal 10, which is the distributor of the video content, of the interest information of all end users identified by the interest identification unit 216 (S44). The right holder of the video content can select goods based on the interest information provided by the video distribution system 20. In addition, when the interest is a person, the interest information can be used as a barometer to measure the popularity of each member of a theater company or idol group.

[0094] 14 is a flowchart for explaining a third application example of the interest information of end users generated by the video distribution system 20 according to the first embodiment. The third application example is an example in which interest information of an unspecified number of end users is generated and used for mass marketing advertisement.

[0095] The interest target identification unit 216 of the video distribution system 20 reads attention area information of one or more video contents viewed by all user IDs of the video distribution service from the attention area information storage unit 223 (S50). The interest target identification unit 216 reads target placement information of each video content from the target placement information storage unit 224. The interest target identification unit 216 refers to the target placement information of each video content that has been read, and identifies targets included in each attention area of ​​all user IDs (S51).

[0096] The interest target identification unit 216 calculates the degree of attention of each end user to the target according to at least one of the time each end user viewed the attention area including the target and the enlargement rate of the attention area including the target. The interest target identification unit 216 calculates the degree of attention of the entire end users to the target by summing up the attention levels of each end user (S52). The interest target identification unit 216 selects a target for which the calculated degree of attention of the entire end users is higher than a fourth set value as an interest target of the entire end users (S53). The notification unit 217 notifies an advertisement distribution system (not shown) of the interest target information of the entire end users identified by the interest target identification unit 216 (S54).

[0097] If the entity providing the video distribution service is a comprehensive platform operator that operates an online shopping mall, and said comprehensive platform operator also operates and manages an advertisement distribution system, the interest information can be utilized as follows: In principle, the advertisement distribution system will broadcast an e-mail containing a URL to direct end users to the website of a shop in the online shopping mall that sells products that are of interest to the end users as a whole to the e-mail accounts of all members of said comprehensive platform operator.

[0098] In addition, when a site for a service provided by the comprehensive platform operator is viewed from a user terminal 30 used by a member of the comprehensive platform operator, the advertising distribution system uses the IP address of the user terminal 30 used by the member to display a banner advertisement on the viewed site to guide the user to the site of the shop.

[0099] The advertisement delivery system may be a system operated and managed by an external Internet advertising agency. In this case, the notification unit 217 provides the interest information of all end users to the advertisement delivery system within the scope of the personal information protection regulations.

[0100] The notification unit 217 notifies the interest information of all end users to the account of the shop in the online shopping mall that sells the products of interest (S55). Based on the notified interest information, the shop can grasp the products that many end users are interested in.

[0101] It should be noted that only one of the notification process in step S54 and the notification process in step S55 may be executed.

[0102] Example 2 The second embodiment is an example in which the above-mentioned second identification method is adopted as a method for identifying an end user's interest.

[0103] Fig. 15 is a flowchart showing a flow of a process of uploading video content from the content holder terminal 10 to the video distribution system 20 according to the second embodiment. The uploading process of video content according to the second embodiment is different from the uploading process of video content according to the first embodiment shown in Fig. 10 in that the generation and transmission process of target placement information is omitted.

[0104] The video data reading unit 111 of the content holder terminal 10 reads video data captured by the camera 5 from the camera 5 and stores it in the video content holding unit 121 (S10). The video data editing unit 112 reads the video data to be edited from the video content holding unit 121 and edits the video data based on the operation of the editor of the content holder (S11). The video content sending unit 113 reads the video content file to be uploaded from the video content holding unit 121 and sends it to the video distribution system 20 via the network 50 (S13a). The video content receiving unit 211 of the video distribution system 20 receives the video content file sent from the content holder terminal 10 (S14a) and stores it in the video content holding unit 222 (S15a).

[0105] The process of viewing video content by the user terminal 30 is the same as that in the first embodiment, and therefore will not be described (see FIG. 11).

[0106] 16 is a flowchart for explaining a first application example of the interest information of an end user generated by the video distribution system 20 according to the embodiment 2. The first application example is an example of generating interest information of a specific end user and providing targeted advertising to the end user.

[0107] The interest target identifying unit 216 of the video distribution system 20 reads out attention area information of one or more video contents viewed by a specific user ID from the attention area information storage unit 223 (S60). The interest target identifying unit 216 identifies each attention area that the end user has paid attention to. The interest target identifying unit 216 overlaps the attention areas of all frame images for each video content. The interest target identifying unit 216 selects an area where the overlapping number exceeds a second set value as a selected attention area that has a high degree of attention of the end user (S61).

[0108] The interest target identification unit 216 performs an image search using a predetermined image recognition algorithm based on image data in which the target to be detected appears within each selected attention area, and detects the end user's interest target from within the selected attention area (S62). The processes in steps S63 and S64 are the same as those in steps S33 and S34 shown in the flowchart of FIG. 12, and therefore will not be described here.

[0109] 17 is a flowchart for explaining a second application example of the end user interest information generated by the video distribution system 20 according to the second embodiment. The second application example is an example in which interest information of an unspecified number of end users for a specific video content is generated and provided to a distributor of the video content.

[0110] The interest subject identification unit 216 of the video distribution system 20 reads out attention area information of specific video content viewed by all user IDs of the video distribution service from the attention area information storage unit 223 (S70). The interest subject identification unit 216 identifies each attention area that each end user paid attention to. The interest subject identification unit 216 overlaps the attention areas of all frame images of the video content. The interest subject identification unit 216 selects an area with an overlapping number exceeding a second set value as a selected attention area that is highly popular among all end users (S71).

[0111] The interest target identification unit 216 performs image search using a predetermined image recognition algorithm based on image data in which the target to be detected appears within each selected attention area, and detects the interest target of the entire end user from within the selected attention area (S72). The process of step S73 is the same as the process of step S43 shown in the flowchart of FIG. 13, so a description thereof will be omitted.

[0112] 18 is a flowchart for explaining a third application example of the interest information of end users generated by the video distribution system 20 according to the embodiment 2. The third application example is an example in which interest information of an unspecified number of end users is generated and used for mass marketing advertisement.

[0113] The interest subject identification unit 216 of the video distribution system 20 reads out attention area information of one or more video contents viewed by all user IDs of the video distribution service from the attention area information storage unit 223 (S80). The interest subject identification unit 216 identifies each attention area that each end user has paid attention to. The interest subject identification unit 216 overlaps the attention areas of all frame images for each video content. The interest subject identification unit 216 selects an area with an overlapping number exceeding a second set value as a selected attention area that is highly popular among all end users (S81).

[0114] The interest target identification unit 216 performs an image search using a predetermined image recognition algorithm based on image data in which the target to be detected appears within each selected attention area, and detects the interest target of the entire end user from within the selected attention area (S82). The processes in steps S83 and S84 are the same as those in steps S53 and S54 shown in the flowchart of FIG. 14, and therefore will not be described here.

[0115] As described above, according to this embodiment, the interests of an end user who is viewing video content can be estimated in detail. For example, in this embodiment, in a video distribution of a play, an end user can enlarge and view only a specific actor. If the actor is wearing an accessory, it can be estimated from the end user's operation of enlarging the screen that the end user may be interested in the accessory worn by the actor. Based on the interests estimated from such operation information of the end user, targeted advertising can be performed for the end user.

[0116] In targeted advertising using conventional image analysis technology, it is assumed that video analysis will be performed to match the user's tastes, and any object reflected in the video content that disrupts the worldview of the video content (for example, corn or milk reflected in a fashion video) is excluded from the advertising target. In other words, while targeted advertising using conventional image analysis technology can realize targeted advertising according to the user's direct tastes, it cannot pick up latent demand stimulated by items, etc. that are unexpectedly reflected in the video.

[0117] In contrast, according to the present embodiment, the potential interests of the end user can be unearthed from the end user's casual operations. This makes it possible to realize mild targeted advertising, which is intermediate between mass marketing advertising aimed at an unspecified number of people and targeted advertising. Mild targeted advertising has a lower hit rate per case compared to conventional targeted advertising based on the end user's clear interests, but it can increase the number of advertisements, which greatly contributes to increasing overall revenue. It is also expected that this could lead to a hit of an unexpected product that content holders and the like were not aware of. It can also be used not only to discover demand for goods, but also to investigate the actual popularity of actors, singers, idols, and the like.

[0118] In the first embodiment, the content holder can generate a target placement information file for the video content, thereby accurately registering the items and people that appear in the video. In addition, the content holder can investigate the popularity of the items that he / she wants to sell, and can utilize the information for merchandise sales.

[0119] In the second embodiment, the content holder does not need to generate a target placement information file, so the work of the content holder does not increase. In addition, it is possible to discover the existence of interest in unexpected items or people that the content holder did not intend. The content holder can also sell items to people other than the current fan base. In addition, by performing an image search of products in a shop in an online shopping mall operated by a comprehensive platform operator, it is possible to estimate the target of interest and use the results for targeted advertising, which can increase the profits of the shop and lead to the expansion of the economic sphere of the comprehensive platform operator.

[0120] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention.

[0121] In the above embodiment 1, an example was described in which a content holder generates a target placement information file of video content. In this regard, a video distribution platform may generate a target placement information file of video content uploaded from a content holder. For example, a person in charge of the video distribution platform may find a product that he or she wants to promote in the video content, and generate a target placement information file that specifies the position information of the product in a frame image.

[0122] In the above embodiment, the end user inputs an instruction to enlarge, reduce or move the video screen by touching the video screen displayed on the touch panel display TP of the user terminal 30. In this regard, the method of inputting an instruction to enlarge, reduce or move the video screen is not limited to a touch operation on the touch panel display TP. For example, if the user terminal 30 is a PC, the instruction to enlarge, reduce or move the video screen can be input by operating a mouse. Also, if the user terminal 30 is a TV, the instruction to enlarge, reduce or move the video screen can be input by operating a cross key or the like on a remote control.

[0123] Furthermore, if the user terminal 30 is equipped with a ToF (Time of Flight) sensor (e.g., LiDAR (Light Detection and Ranging)), the end user can input an instruction to enlarge, reduce or move the video screen by performing a gesture operation in the air. Furthermore, if the user terminal 30 is equipped with a camera that takes a picture of the end user (a so-called selfie camera) and an application program for gaze input, an instruction to enlarge, reduce or move the video screen can be input by the movement of the end user's gaze.

[0124] In the above embodiment, a two-dimensional video content is assumed. In this regard, the video content may be a 180-degree video or a 360-degree video in which the viewpoint can be freely switched. In this case, a VR headset is used as the user terminal 30. [Explanation of symbols]

[0125] 5 camera, 10 content holder terminal, 11 processing unit, 111 video data reading unit, 112 video data editing unit, 113 video content transmission unit, 114 object placement information generation unit, 12 memory unit, 121 video content holding unit, 122 object placement information holding unit, 13 operation unit, 14 display unit, 15 communication unit, 20 video distribution system, 21 processing unit, 211 video content receiving unit, 212 distribution unit, 213 operation signal receiving unit, 214 extraction unit, 215 attention area identification unit, 216 interest target identification unit, 217 notification unit, 22 memory unit, 221 member information holding unit, 222 video content holding unit, 223 attention area information holding unit, 224 object placement information holding unit, 23 operation unit, 24 display unit, 25 communication unit, 30 user terminal, 31 processing unit, 311 video data receiving unit, 312 operation signal transmitting unit, 32 memory unit, 33 operation unit, 34 display unit, TP touch panel display, 35 communication unit, 50 network.

Claims

1. An operation signal receiving unit that receives an operation signal for enlarging, reducing, or moving a screen on which video data is displayed from a user terminal to which the video data is delivered; a cutout unit that cuts out an area of ​​the video data to be delivered to the user terminal in units of frame images constituting the video data, based on the content of the screen enlargement, reduction, or movement indicated by the operation signal; a distribution unit that distributes the video data of the cut-out region to the user terminal; an attention area information storage unit that stores position information of the attention area, which is the area cut out based on the operation signal and is an area that a user has paid attention to; an object identifying unit that identifies an object appearing in the attention area within a frame image of the video data; The target identification unit further comprises: Calculating the degree of attention of the user to the target according to at least one of a time during which the user viewed the attention area including the target and a magnification rate of the attention area including the target; selecting an object having a calculated degree of attention higher than a predetermined set value as an object of interest to the user; Video distribution system.

2. Further comprising an object placement information storage unit that stores placement information of an object shown in each frame image of the video data, The object identification unit identifies an object appearing in a region of interest within a frame image of the video data based on object arrangement information and region of interest information of the video data. The video distribution system according to claim 1 .

3. The object identification unit performs an image search on a selected attention area in each frame image of the video data to identify an object appearing in the attention area. The video distribution system according to claim 1 .

4. The object reflected in the attention area includes a specific object or a specific person.

4. The video distribution system according to claim 2 or 3.

5. Further comprising a notification unit that notifies an external party of information regarding an object that a user has focused on within the video data. The video distribution system according to any one of claims 1 to 4.

6. The notification unit notifies an advertisement delivery system of information regarding a target that has attracted the attention of a specific or an unspecified number of users within the video data. The video distribution system according to claim 5 .

7. The notification unit notifies an account of a rights holder of the video data of information regarding a subject that has attracted the attention of an unspecified number of users within the video data.

7. The video distribution system according to claim 5 or 6.

8. The notification unit notifies information about an object that has attracted the attention of a specific or unspecified number of users in the video data to an account of a shop selling the object in an online shopping mall. The video distribution system according to any one of claims 5 to 7.

9. A step of receiving an operation signal for enlarging, reducing or moving a screen on which the video data is displayed from a user terminal to which the video data is delivered; cutting out an area of ​​the video data to be delivered to the user terminal in units of frame images constituting the video data, based on the content of the screen enlargement, reduction, or movement indicated by the operation signal; delivering the video data of the cropped region to the user terminal; accumulating position information of the region cut out based on the operation signal, which is a region that a user has focused on; and identifying an object appearing in the attention area within a frame image of the video data, The target step further comprises: Calculating the degree of attention of the user to the target according to at least one of a time during which the user viewed the attention area including the target and a magnification rate of the attention area including the target; selecting an object having a calculated degree of attention higher than a predetermined set value as an object of interest to the user; Video distribution methods.

Images