Receiving device and metadata generation system

Abstract

To efficiently process information on a broadcast program extracted for each frame using a limited system resource.SOLUTION: A receiving device according to an embodiment that receives a broadcast program and provides it for live viewing includes a data processing unit that generates conversion data capable of generating metadata representing the content of the broadcast program from a broadcast signal of the broadcast program while providing live viewing of the broadcast program, and a first transmitting / receiving unit that transmits the converted data to a server device that generates metadata.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] SUMMARY OF THE INVENTION An embodiment of the present invention relates to a receiving device and a metadata generation system. [Background technology]

[0002] To realize efficient viewing of various content, the usefulness of content-related metadata, including scene information, has been attracting attention. While manual creation of metadata for television broadcast programs has been the norm, in recent years attempts have begun to use artificial intelligence (AI) to automatically generate metadata. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-108984 [Patent Document 2] Japanese Patent Application Laid-Open No. 2006-109126 [Patent Document 3] Japanese Patent Application Laid-Open No. 2011-008676 Summary of the Invention [Problem to be solved by the invention]

[0004] However, automatic generation of metadata requires a huge amount of processing, so the challenge is how to process broadcast program information extracted for each frame, for example, with limited system resources.

[0005] The problem to be solved by the present invention is to provide a receiving device and a metadata generation system that can efficiently process broadcast program information extracted for each frame using limited system resources. [Means for solving the problem]

[0006] A receiving device according to an embodiment receives a broadcast program and provides it so that it can be viewed live, and includes: a data processing unit that generates, from a broadcast signal of the broadcast program, converted data that can generate metadata representing the content of the broadcast program while the broadcast program is being viewed live; and a first transmitting / receiving unit that transmits the converted data to a server device that generates the metadata. The data processing unit converts the broadcast program into a multidimensional array for each frame, and generates the converted data including an inference result obtained by inferring the content of the broadcast program from the multidimensional array. . [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a metadata generation system according to an embodiment. [Figure 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the server device according to the embodiment. [Figure 3] FIG. 3 is a block diagram illustrating an example of a functional configuration of the server device according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a hardware configuration of the television device according to the embodiment. [Figure 5] FIG. 5 is a block diagram illustrating an example of a functional configuration of the television device according to the embodiment. [Figure 6] FIG. 6 is a schematic diagram illustrating an example of how the metadata generation system according to the embodiment generates metadata. [Figure 7] FIG. 7 is a schematic diagram illustrating an example of how the metadata generation system according to the embodiment identifies an insertion position of an advertisement. [Figure 8] FIG. 8 is a schematic diagram illustrating an example of how tasks are assigned in the television device according to the embodiment. [Figure 9] FIG. 9 is a schematic diagram illustrating an example in which surplus resources are allocated to tasks of preprocessing of data conversion in the television device according to the embodiment. [Figure 10] FIG. 10 is a schematic diagram illustrating an example in which surplus resources are allocated to tasks of preprocessing of data conversion in the television device according to the embodiment. [Figure 11]FIG. 11 is a flowchart illustrating an example of a procedure of a metadata generation process performed by the metadata generation system according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] (Configuration of Metadata Generation System) Fig. 1 is a diagram showing an example of the configuration of a metadata generation system 1 according to an embodiment. As shown in Fig. 1, the metadata generation system 1 includes a server device 10 and a plurality of television devices 20 (20a, 20b, 20c, . . . 20n: where n is an arbitrary integer), and is configured to be able to generate metadata representing the contents of broadcast programs through cooperation between the server device 10 and the television devices 20.

[0009] The server device 10 and the plurality of television devices 20 are connected to each other wirelessly or via a wired connection via a network 30 such as the Internet. The network 30 may be, for example, a home network based on the Digital Living Network Alliance (DLNA) (registered trademark) or a home LAN (Local Area Network).

[0010] The television device 20 as a receiving device can receive various broadcast programs by receiving broadcast signals from broadcast stations, for example. The television device 20 is also configured to provide the received broadcast programs to users for live viewing, record them, and play back the recorded broadcast programs.

[0011] The television device 20 is also configured to be able to generate conversion data from the broadcast signal of a received broadcast program while providing the received broadcast program to a user, which can be used to generate metadata including scene information of the broadcast program, etc.

[0012] The server device 10 is configured as, for example, a cloud server placed on a cloud, etc. The server device 10 may be configured as one or more computers having physical components such as a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory).

[0013] The server device 10 receives the converted data converted by each television device 20 from the television devices 20 and generates metadata. The server device 10 provides the generated metadata to each television device 20.

[0014] (Server device configuration example) Next, a configuration example of the server device 10 according to the embodiment will be described with reference to FIGS.

[0015] 2 is a block diagram showing an example of a hardware configuration of the server device 10 according to the embodiment. As shown in FIG. 2, the server device 10 includes a CPU 101, a ROM 102, a RAM 103, a communication I / F (interface) 104, an input / output I / F 105, an input device 151, a display device 152, and a storage device 106.

[0016] The CPU 101 controls the entire server device 10. The ROM 102 functions as a storage area in the server device 10. Information stored in the ROM 102 is retained even when the power to the server device 10 is turned off. The RAM 103 functions as a primary storage device and serves as a working area for the CPU 101.

[0017] The CPU 101 loads, for example, a control program stored in the ROM 102 into the RAM 103 and executes it, thereby achieving the function of the server device 10 that generates metadata based on the converted data collected from a plurality of television devices 20.

[0018] The above control program can be provided by being recorded on various computer-readable recording media such as a flexible disk, CD-R, DVD (Digital Versatile Disk), Blu-ray Disc (registered trademark), semiconductor memory, etc.

[0019] The control program may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The control program may also be provided or distributed via a network such as the Internet.

[0020] The communication I / F 104 is configured to be connectable to a network 30 such as the Internet. The communication I / F 104 enables various types of information to be exchanged between the server device 10 and the plurality of television devices 20.

[0021] An input device 151 such as a keyboard or a mouse, and a display device 152 such as a monitor may be connected to the input / output I / F 105. This allows, for example, an administrator of the server device 10 to perform various operations on the server device 10.

[0022] The storage device 106 is a hard disk drive (HDD), a solid state drive (SSD), or the like, and functions as an auxiliary storage device for the CPU 101.

[0023] 3 is a block diagram showing an example of a functional configuration of the server device 10 according to the embodiment. As shown in FIG. 3, the server device 10 includes a transmitting / receiving unit 11, an integrating unit 12, an advertisement determination unit 13, a metadata generating unit 14, and a storage unit 15.

[0024] The above-described functional configuration of the server device 10 is realized by, for example, the CPU 101 executing the control program, or the hardware configuration of each part of the server device 10 operating under the control of the CPU 101, as shown in FIG.

[0025] The transmitting / receiving unit 11, which serves as a second transmitting / receiving unit, is configured to be able to exchange data between the plurality of television devices 20 and the server device 10. The transmitting / receiving unit 11 receives, for example, from the plurality of television devices 20, converted data that the television devices 20 have generated from broadcast signals of broadcast programs. The transmitting / receiving unit 11 also transmits metadata generated by the server device 10 to the plurality of television devices 20.

[0026] The integrating unit 12 integrates the converted data generated for each frame by the plurality of television devices 20 into time-series data arranged in chronological order. At this time, the integrating unit 12 selects and discards the converted data collected from the plurality of television devices 20 so as to obtain time-series data for all broadcast programs broadcast by the plurality of broadcast stations in a predetermined time slot.

[0027] For example, if a user of a certain television device 20 consistently watches broadcast programs from a single broadcast station, the integration unit 12 may generate time series data for a single broadcast program using only the converted data collected from that single television device 20.

[0028] Alternatively, in a case where a user of a certain television device 20 is watching multiple broadcast programs by repeatedly selecting channels, time series data for one broadcast program may be generated using conversion data collected from multiple television devices 20 under such circumstances.

[0029] The advertisement determination unit 13 determines the insertion position of an advertisement based on the time-series data. The converted data from the television device 20 includes information about the insertion position of the advertisement estimated by the television device 20. The advertisement determination unit 13 identifies the insertion position of the advertisement by referring to the information estimated by the television device 20.

[0030] The metadata generating unit 14 generates metadata representing the contents of the broadcast program, such as scene information, based on the converted data excluding the insertion positions of advertisements, that is, the converted data generated from the main part of the broadcast program.

[0031] The storage unit 15 stores various parameters, control programs, etc. required for the operation of the server device 10. The storage unit 15 may also store converted data collected from a plurality of television devices 20, time-series data generated from the converted data, information on identified insertion positions of advertisements, metadata generated based on the converted data, etc.

[0032] (Example of a television device configuration) Next, an example of the configuration of the television device 20 according to the embodiment will be described with reference to FIGS.

[0033] FIG. 4 is a diagram illustrating an example of a hardware configuration of the television device 20 according to the embodiment.

[0034] As shown in FIG. 4, the television device 20 includes an antenna 201, input terminals 202a to 202c, a tuner 203, a demodulator 204, a demultiplexer 205, an A / D (analog / digital) converter 206, a selector 207, a signal processing unit 208, a speaker 209, a display panel 210, an operation unit 211, a light receiving unit 212, an IP communication unit 213, a CPU 214, a memory 215, and a storage 216.

[0035] The antenna 201 receives a digital broadcast signal and supplies the received broadcast signal to a tuner 203 via an input terminal 202a.

[0036] The tuner 203 selects a broadcast signal of a desired channel from the broadcast signals supplied from the antenna 201 , and supplies the selected broadcast signal to the demodulator 204 .

[0037] The demodulator 204 demodulates the broadcast signal supplied from the tuner 203 and supplies the demodulated broadcast signal to a demultiplexer 205 .

[0038] The demultiplexer 205 separates the broadcast signal supplied from the demodulator 204 to generate a video signal and an audio signal, and supplies the generated video signal and audio signal to a selector 207 .

[0039] The selector 207 selects one of the signals supplied from the demultiplexer 205, the A / D converter 206, and the input terminal 202c, and supplies the selected signal to the signal processing unit 208.

[0040] The signal processing unit 208 performs predetermined signal processing on the video signal supplied from the selector 207, and supplies the processed video signal to the display panel 210. The signal processing unit 208 also performs predetermined signal processing on the audio signal supplied from the selector 207, and supplies the processed audio signal to the speaker 209.

[0041] The speaker 209 outputs voice or various sounds based on the audio signal supplied from the signal processing unit 208. Furthermore, the speaker 209 changes the volume of the voice or various sounds to be output based on the control of the CPU 214.

[0042] The display panel 210 displays still images, moving images, other images, text information, and the like, based on the video signal supplied from the signal processing unit 208 or under the control of the CPU 214 .

[0043] The input terminal 202b accepts analog signals such as video signals and audio signals input from the outside. The input terminal 202c accepts digital signals such as video signals and audio signals input from the outside. For example, the input terminal 202c can input digital signals from a recorder or the like equipped with a drive device that drives a recording and playback recording medium such as a BD (Blu-ray Disc) (registered trademark) for recording and playback.

[0044] The A / D converter 206 performs A / D conversion on the analog signal supplied from the input terminal 202 b to generate a digital signal, and supplies the digital signal to the selector 207 .

[0045] The operation unit 211 accepts operation inputs from the user.

[0046] The light receiving unit 212 receives infrared rays from the remote controller 219 .

[0047] The IP communication unit 213 is a communication interface for performing IP (Internet Protocol) communication via the network 30. However, the television device 20 may be connectable to a network different from the Internet, such as a LAN, and may be connected to the above-described server device 10 via such a network so as to be able to exchange various information.

[0048] The CPU 214 controls the entire television device 20 .

[0049] The memory 215 includes a ROM that stores various computer programs executed by the CPU 214, and a RAM that provides a working area for the CPU 214. For example, the ROM stores control programs and application programs for realizing various functions of the television device 20.

[0050] The storage 216 is a hard disk drive (HDD) or a solid state drive (SSD), etc. The storage 216 records, for example, the signal selected by the selector 207 as recording data.

[0051] 5 is a block diagram showing an example of a functional configuration of a television device 20 according to an embodiment. As shown in FIG. 5, the television device 20 includes a transmitter / receiver 21, a task distributor 22, a data processor 23, a broadcast receiver 24, an operation receiver 25, a live viewing processor 26, a recording processor 27, a playback processor 28, and a storage unit 29.

[0052] The above-described functional configuration of the television device 20 is realized by, for example, the CPU 214 executing a control program, or the hardware configuration of each unit of the television device 20 operating under the control of the CPU 214, as shown in FIG.

[0053] The broadcast receiving unit 24 receives broadcast signals of broadcast programs transmitted from broadcast stations. The broadcast signals include video information and audio information, and are multiplexed with service information (SI) indicating the contents of the broadcast programs, taking into consideration the convenience of program selection, etc. An example of service information is information related to an electronic program guide (EPG), which includes information equivalent to the TV listings in a newspaper.

[0054] The above-mentioned video information, audio information, and accompanying transmission control information, etc. are compressed in MPEG2 format and multiplexed to form a transport stream (TS).

[0055] The broadcast receiving unit 24 is capable of receiving video information, audio information, program arrangement information, and the like multiplexed onto a broadcast signal.

[0056] The operation receiving unit 25 receives various operations from the user, such as a live viewing operation, a recording operation, a recording reservation operation, and a playback operation.

[0057] The live viewing processing unit 26 processes live viewing of a broadcast program. Live viewing means, for example, displaying a broadcast program that is currently being broadcast in real time.

[0058] The recording processor 27 processes recording of broadcast programs in accordance with recording operations and recording reservation operations from the user. The recording processor 27 stores the recorded programs in the storage unit 29.

[0059] The playback processing unit 28 reads out the recorded program from the storage unit 29 and performs playback processing in accordance with a playback operation from the user.

[0060] The data processing unit 23 converts, for example, the broadcast signal of a broadcast program currently being provided for live viewing by the television device 20 into a format that enables the server device 10 to generate metadata. More specifically, the data processing unit 23 converts the data based on the broadcast signal of the broadcast program currently being provided for live viewing into a multidimensional array for each frame, and further generates converted data including an inference result obtained by inferring the content of the broadcast program from the multidimensional array.

[0061] The results of inferring the program content include, for example, the estimated start and end positions of advertisements for separating the main program from the advertisements, and information specifying the performers in the broadcast program.

[0062] When the data processing unit 23 performs data conversion processing, the task allocation unit 22 allocates tasks indicating the processing contents. When the data processing unit 23 performs data conversion processing, the tasks required for data conversion may differ depending on the content or frame of the broadcast program. The task allocation unit 22 identifies the necessary tasks as appropriate, for example, by referring to program sequence information included in the broadcast signal, and allocates them to the data processing unit 23.

[0063] The transmitting / receiving unit 21 as a first transmitting / receiving unit is configured to be able to exchange data between the television device 20 and the server device 10. The transmitting / receiving unit 21 transmits converted data generated by the television device 20 from a broadcast signal of a broadcast program to the server device 10. The transmitting / receiving unit 21 also receives metadata generated by the server device 10.

[0064] The storage unit 29 stores various parameters, control programs, etc. necessary for the operation of the television device 20. The storage unit 29 may also store recorded programs, converted data generated by the data processing unit 23, metadata received from the server device 10, etc.

[0065] It is assumed that the user of the television device 20 has given permission to use the resources of the television device 20 for the metadata generation process in the server device 10.

[0066] (Metadata generation example) Next, an example of metadata generation by the server device 10 and the television device 20 according to the embodiment will be described with reference to FIG.

[0067] 6 is a schematic diagram showing an example of how the metadata generation system 1 according to the embodiment generates metadata. In FIG. 6, time progresses from left to right on the page.

[0068] As shown in Fig. 6, for a broadcast program currently being viewed live on the television device 20, the data processing unit 23 captures the viewing screen, for example, frame by frame, and obtains a multidimensional array from the captured viewing screen IM. More specifically, the data processing unit 23 converts the transport stream of the broadcast program into a multidimensional array of floating-point values ​​or integer values. A multidimensional array is a multi-column array in which multiple variables are stored using the concept of a matrix.

[0069] By using a multidimensional array, it becomes possible to handle huge amounts of data. For example, a specific example will be given below, taking the viewing screen IM captured by the data processing unit 23 as an example. Assuming that the task allocation unit 22 selects the estimated start position and end position of an advertisement as tasks for the original viewing screen IM, in the case of terrestrial digital broadcasting, the number of pixels will be (3 x 1440 x 1080). By converting this into a multidimensional array, the output can be, for example, (1 x 576) pixels or (1 x 5) pixels. However, these specific numerical values ​​for input and output are merely examples. In any case, by using a multidimensional array, the number of array elements for the original viewing screen IM can be significantly reduced.

[0070] Therefore, by using a multidimensional array as input / output data to a deep neural network (DNN), etc., image recognition, voice recognition, etc. by the DNN become easier. In addition, as will be described later, by converting data into a multidimensional array, the amount of data handled by the server device 10 can be reduced, making it possible to collect and process information on more content.

[0071] The data processing unit 23 generates converted data including inference results obtained by using, for example, the above-mentioned DNN technology, to distinguish between the main content of a broadcast program and advertisements, performers in the broadcast program, and other scene information.

[0072] The transmitter / receiver 21 of the television device 20 transmits the generated converted data for each frame together with program sequence information to be multiplexed onto the received broadcast signal to the server device 10 via the network 30. The converted data is uploaded from the television device 20 to the server device 10 periodically, for example, every five minutes.

[0073] The integration unit 12 of the server device 10 selects and discards the converted data for each frame collected from the multiple television devices 20, and generates time-series data for, for example, all of multiple broadcast programs that are broadcast simultaneously. The time-series data generation process is also performed periodically, for example, every five minutes, in accordance with the timing of data uploads from the multiple television devices 20.

[0074] As described above, the advertisement determination unit 13 identifies the insertion position of the advertisement based on the time-series data. The metadata generation unit 14 generates metadata representing the program content of the main part of the broadcast program while referring to the program sequence information added to the converted data.

[0075] The metadata generated by the metadata generating unit 14 includes, for example, metadata indicating the distinction between the main content of a broadcast program and advertisements.

[0076] For example, the metadata generation unit 14 generates metadata indicating an advertisement portion by linking it to the time series data corresponding to the advertisement insertion position identified by the advertisement determination unit 13, or metadata indicating a main portion by linking it to the time series data corresponding to the portion excluding the advertisement insertion position, by linking it to the corresponding timing of the time series data.

[0077] Alternatively, if it is determined from the program arrangement information that the program is a broadcast program that does not include advertisements, such as NHK, the metadata generating unit 14 can omit the above process.

[0078] Furthermore, for example, if the broadcast program for which metadata is to be generated is a music program, the metadata generation unit 14 refers to the program arrangement information and identifies the singing timings of, for example, singer A, singer B, singer C, etc., and generates metadata including the names of these singers, etc., by linking them to the corresponding timings in the time-series data.

[0079] Furthermore, for example, if the broadcast program for which metadata is to be generated is a variety show, the metadata generation unit 14 refers to the program arrangement information and identifies the timing of appearances of, for example, talent A, talent B, talent C, etc., and generates metadata including the names of these talents, etc., by linking them to the corresponding timings in the time-series data.

[0080] For example, if the broadcast program for which metadata is to be generated is an anime or a drama, the metadata generation unit 14 refers to the program sequence information and identifies the timing at which, for example, an opening or ending theme song is played, and generates metadata indicating that it is the theme song part by linking it to the corresponding timing in the time-series data.

[0081] For example, if the broadcast program for which metadata is to be generated is a news program, the metadata generation unit 14 refers to the program sequence information and identifies, for example, the timing of broadcast from a studio or the timing of a live broadcast, and generates metadata indicating that it is a portion broadcast from a studio or a portion broadcast, by linking it to the corresponding timing in the time-series data.

[0082] The transmitter / receiver 11 of the server device 10 transmits the metadata generated as described above to the plurality of television devices 20. At this time, the metadata may be distributed to all television devices 20 connected to the server device 10, or may be transmitted to a television device 20 that has made a request among the plurality of television devices 20.

[0083] When a recorded program is played back, the television device 20 displays the corresponding metadata in accordance with the user's operation, allowing the user to efficiently watch the recorded program by referring to the metadata.

[0084] Note that playback of recorded programs includes not only playback of recorded programs at any time after the program has finished, but also playback of recorded content from the past while watching a live broadcast without waiting for the recording to finish. This type of playback method is also called time-shift playback.

[0085] (Example of identifying ad insertion position) Next, an example of specifying the insertion position of an advertisement in the metadata generation system 1 will be described with reference to FIG.

[0086] 7 is a schematic diagram showing an example of how the metadata generation system 1 according to the embodiment identifies the insertion position of an advertisement. Note that, in FIG. 7 as well, time progresses from left to right on the page.

[0087] 7, the data processing unit 23 of the television device 20 adds information on the estimated start position and estimated end position of an advertisement using, for example, DNN technology, based on a multidimensional array obtained by converting data based on the broadcast signal of a broadcast program. The information on the estimated start position and estimated end position may also include the accuracy of these estimates.

[0088] In the server device 10 that has received the converted data including this information, the advertisement determination unit 13 identifies the insertion position of the advertisement in the time-series data.

[0089] In the example of FIG. 7, for example, the beginning of the time-series data includes information on the estimated start position of the advertisement (accuracy: 80%) ((1) in FIG. 7). However, the time-series data does not subsequently include information on the estimated end position of the advertisement for a predetermined period of time. In this case, the advertisement determination unit 13 determines that the inference result of the estimated start position of the advertisement at the beginning of the time-series data is incorrect, and that the advertisement did not start at this timing.

[0090] Note that the broadcast time of an advertisement is usually standardized in 15-second increments up to one minute, etc. Therefore, the predetermined period used by advertisement determination unit 13 for determination can be set to, for example, one minute.

[0091] Furthermore, in the example of FIG. 7, the subsequent time-series data includes two pieces of information about the estimated start position of the advertisement (accuracy: 90%, accuracy: 85%) ((2) and (3) in FIG. 7). In contrast, there is only one piece of information about the estimated end position of the advertisement that is thought to correspond to this estimated start position (accuracy: 80%) ((4) in FIG. 7). In this case, advertisement determination unit 13 selects, from among the combinations of each of the two estimated start positions and one estimated end position, the combination in which the advertisement insertion time is closest to a multiple of the shortest broadcast period of the advertisement.

[0092] That is, as mentioned above, the advertisement broadcast time is standardized to increase in 15-second increments. In the example of Figure 7, when the earlier of the two estimated start positions ((2) in Figure 7) is combined with the subsequent estimated end position, the advertisement insertion time is 67 seconds. On the other hand, when the later of the two estimated start positions ((3) in Figure 7) is combined with the subsequent estimated end position, the advertisement insertion time is 60 seconds.

[0093] In this way, the combination of the later estimated start position and the subsequent estimated end position makes the advertisement broadcast time closer to a multiple of 15 seconds, so advertisement determination unit 13 adopts the later of the two estimated start positions as the start position of the advertisement.

[0094] In this example, the one with the lower accuracy is adopted out of the two estimated start positions. However, advertisement determination unit 13 may also use the accuracy as a determination criterion in addition to whether the start position is close to a multiple of the shortest broadcast period of the advertisement.

[0095] (Task allocation example) Next, an example of task allocation in the television device 20 of the embodiment will be described with reference to FIG.

[0096] 8 is a schematic diagram showing an example of how tasks are assigned in the television device 20 according to the embodiment. In FIG. 8, time passes from the top to the bottom of the page.

[0097] As described above, when the data processing unit 23 performs data conversion on a predetermined broadcast program, the tasks required for data conversion may differ depending on the content or frame of the broadcast program.

[0098] For example, in music programs or dramas, as described above, facial recognition of the singers and performers may be performed to identify the timing of each performer's appearance. On the other hand, in news programs, such processing is usually unnecessary.

[0099] For example, in commercial broadcasting programs, a process for identifying estimated start and end positions of advertisements may be performed, whereas in NHK broadcasting programs, such a process is not required.

[0100] Furthermore, for example, in the case of high-definition broadcasting, a process of resizing the volume of the captured viewing screen may be performed.

[0101] In the example of Figure 8, tasks 1 to 7 represent tasks that may vary depending on the content or frame of a broadcast program. Of these tasks 1 to 7, tasks 1 to 3 are tasks that can be executed with relatively few resources. On the other hand, of these tasks 1 to 7, tasks 4 to 7 are tasks that require relatively many resources.

[0102] For example, at the timing shown in the top row of Fig. 8, it is assumed that the data processing unit 23 is capturing a predetermined viewing screen IM ((1) in Fig. 8). The task allocation unit 22 refers to the program arrangement information at this timing and identifies a task required to generate conversion data based on the viewing screen IM captured by the data processing unit 23. More specifically, the task allocation unit 22 selects, for example, task 1 from a plurality of candidates for tasks 1 to 3, and allocates it to the data processing unit 23.

[0103] As described above, the processing of tasks 1 to 3, including task 1, can all be executed with relatively few resources. In this case, task allocator 22 determines that data processing unit 23 has surplus resources.

[0104] Also, for example, at the timing in the middle of Fig. 8, it is assumed that the data processing unit 23 is capturing another viewing screen IM ((2) in Fig. 8). The task allocation unit 22 refers to the program arrangement information at this timing and identifies tasks required to generate conversion data based on the viewing screen IM captured by the data processing unit 23. In the example of Fig. 8, the task allocation unit 22 selects, for example, task 1 and task 2 from a plurality of candidates for task 1 to task 3 and allocates them to the data processing unit 23.

[0105] In this case, although the processing of each of task 1, task 2, and task 3 can be executed with relatively few resources, the data processing unit 23 is busy processing two tasks, task 1 and task 2. In this case, the task allocation unit 22 determines that the data processing unit 23 does not have surplus resources and is unable to process any more tasks.

[0106] Also, for example, at the timing shown in the bottom row of Fig. 8, it is assumed that the data processing unit 23 is capturing yet another viewing screen IM ((3) in Fig. 8). The task allocation unit 22 refers to the program arrangement information at this timing, and identifies a task required to generate conversion data based on the viewing screen IM captured by the data processing unit 23.

[0107] 8, the task allocation unit 22 selects, for example, task 1 from among a plurality of candidates for tasks 1 to 3, and allocates it to the data processing unit 23. In addition, the task allocation unit 22 selects, for example, task 5 from among a plurality of candidates for tasks 4 to 7, and allocates it to the data processing unit 23.

[0108] At this time, it is assumed that task 5 of tasks 1 and 5 assigned to data processing unit 23 is a process of resizing viewing screen IM to create viewing screen im. Such a process requires a relatively large number of resources.

[0109] In this case as well, the task allocator 22 determines that the data processor 23 does not have any surplus resources and is therefore unable to process any more tasks.

[0110] The converted data for each frame including the inference results after these tasks is uploaded to the server device 10.

[0111] (Example of using surplus resources) Next, an example of how surplus resources are used in the television device 20 of the embodiment will be described with reference to FIGS.

[0112] 9 and 10 are schematic diagrams illustrating an example of a case where surplus resources are allocated to tasks of pre-processing of data conversion in the television device 20 according to the embodiment. In the examples of Fig. 9 and 10, the pre-processing of data conversion involves processing to extract features of performers in a predetermined drama in advance.

[0113] 6, when the data processing unit 23 of the television device 20 analyzes broadcast programs such as variety shows or dramas, it may perform processing to detect the appearance timing of predetermined performers in these broadcast programs. In this case, for example, face recognition using DNN technology is performed to identify individual performers and specify the timing when these performers appear on the viewing screen.

[0114] Therefore, for example, by extracting facial features of predetermined performers in advance, it becomes possible to quickly perform facial authentication of each performer based on the extracted features during data conversion processing.

[0115] 9(a), when surplus resources occur in the data processing unit 23 at a predetermined timing, the task allocation unit 22 of the television device 20 allocates, for example, a task of extracting facial features of performers in a predetermined drama to the data processing unit 23. Facial photographs of the individual performers for extracting facial features may be pooled in the server device 10, for example.

[0116] In this case, an administrator or the like of the server device 10 can store in advance, in the storage unit 15 of the server device 10, face photo data of the cast members of the drama that has been given permission to use. Alternatively, the television device 20 may autonomously acquire face photos of the cast members included in the program sequence information or the like and store them in the storage unit 29. In this case as well, comprehensive permission can be obtained in advance for the use of the program sequence information in the metadata generation system 1 including the television device 20.

[0117] As shown in Fig. 9(b), the data processing unit 23 of the television device 20 extracts facial features of each performer from the given facial photograph of the performer and stores the extracted features in, for example, the storage unit 29. Thereafter, when generating conversion data for a drama in which those performers appear, the individual performers can be identified by referring to the features stored in the storage unit 29. This process is shown in Fig. 10.

[0118] As shown in FIG. 10(a), for a drama from which facial features of cast members have already been extracted, the data processing unit 23 converts information contained in the viewing screen captured for each frame into a multidimensional array.

[0119] As shown in FIG. 10(b), the data processing unit 23 performs analysis based on the generated multidimensional array to extract facial features of the performers included in the viewing screen.

[0120] As shown in Figure 10(c), the data processing unit 23 reads out the facial feature data of each performer stored in the memory unit 29 and compares it with the facial feature data of the performers included in the viewing screen to identify the performers included in the viewing screen.

[0121] The television device 20 may transmit the generated facial features of the performers to the server device 10. The server device 10 may also distribute the features generated by a given television device 20 to other television devices 20, and when each television device 20 performs data conversion, the features generated by one television device 20 may be shared among these television devices 20.

[0122] (Metadata generation process example) Next, an example of metadata generation processing by the metadata generation system 1 according to the embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of the procedure of metadata generation processing by the metadata generation system 1 according to the embodiment.

[0123] As shown in FIG. 11, when viewing of a predetermined broadcast program begins, the task allocator 22 of the television device 20 allocates a task to the data processor 23 (step S101). That is, the task allocating unit 22 identifies tasks required for data conversion for each content or frame of a broadcast program while referring to program sequence information. The task allocating unit 22 also allocates the identified tasks to the data processing unit 23.

[0124] Furthermore, the task allocator 22 determines whether or not there is a possibility of excess resources occurring in the data processor 23 based on the identified tasks (step S102).

[0125] If it is determined that no surplus resources will be generated (step S102: No), the data processing unit 23 executes the processes of steps S103 to S105 based on the assigned tasks. If it is determined that surplus resources will be generated (step S102: Yes), the data processing unit 23 further assigns tasks according to the surplus resources to the data processing unit 23 (step S107).

[0126] Specifically, data processing unit 23 captures the viewing screen of the broadcast program that has started to be viewed, frame by frame (step S103). Data processing unit 23 also converts the information included in the viewing screen into a multidimensional array (step S104). Data processing unit 23 also performs various inferences based on the multidimensional array to generate inference results (step S105).

[0127] Furthermore, if a task corresponding to the surplus resources has been further assigned, the data processing unit 23 executes processing based on the assigned task in parallel with the processing of steps S103 to S105 (step S107).

[0128] The transmitting / receiving unit 21 transmits the converted data generated as described above to the server device 10 (step S106). The transmitting / receiving unit 11 of the server device 10 receives the converted data from the television device 20 (step S111).

[0129] The integrating unit 12 selects and discards the converted data collected from the plurality of television devices 20, integrates the plurality of converted data relating to the individual broadcast programs, and generates time-series data corresponding to each of these broadcast programs (step S112).

[0130] The advertisement determination unit 13 refers to, for example, program sequence information corresponding to the generated time-series data, and determines whether or not an advertisement is inserted in the time-series data (step S113).

[0131] In other words, the advertisement determination unit 13 determines from the program sequence information that if the broadcast program corresponding to the time-series data is an NHK broadcast program, no advertisements will be inserted, and if it is a commercial broadcast program, advertisements will be inserted. However, instead of or in addition to the program sequence information, the advertisement determination unit 13 may make the above determination based on whether or not the data processing unit 23 of the television device 20 has assigned estimated start and end positions of advertisements to the time-series data.

[0132] If it is determined that the time-series data contains an advertisement (step S113: Yes), the advertisement determination unit 13 identifies the advertisement insertion position based on the estimated start position and estimated end position of the advertisement added to the time-series data (step S114).If it is determined that the time-series data does not contain an advertisement (step S113: No), the advertisement determination unit 13 skips the process of step S114.

[0133] The metadata generation unit 14 generates metadata representing the content of the broadcast program for the main part of the time series data excluding the insertion position of the advertisement, based on the time series data and the inference result of the data processing unit 23 attached to the time series data (step S115).

[0134] The transmitting / receiving unit 11 transmits the metadata generated as described above to the television device 20 (step S116). The transmitting / receiving unit 21 of the television device 20 receives the metadata from the server device 10 (step S121) and stores it in the storage unit 29 so that it can be displayed, for example, when watching a recorded program (step S122).

[0135] This completes the metadata generation process by the metadata generation system 1 of the embodiment.

[0136] (Comparative Example) With the recent increase in content, there is a demand for content search functions and recommendation functions for individual content, and it is also important to provide efficient viewing methods. Metadata, such as scene information, is useful for realizing efficient content viewing.

[0137] Until now, metadata for television broadcast programs has mainly been created manually, which takes several hours after the broadcast ends, and the cost of creating the metadata is enormous. Therefore, attempts have begun to automatically generate metadata using artificial intelligence. However, automatic metadata generation has the following challenges, for example:

[0138] Television devices are required to perform real-time processing for viewing, recording, and playback, making it difficult to have television devices automatically generate metadata, which requires handling of time-series data.

[0139] Furthermore, television devices have limited system resources, making it difficult for them to perform complex processing using artificial intelligence and simultaneous parallel processing, which also makes it difficult for television devices to automatically generate metadata.

[0140] On the other hand, if the server device is to automatically generate metadata, the cost of the server device will be enormous if the server device processes a huge amount of content in a batch.

[0141] For example, in the technologies of Patent Documents 1 and 2, logo images and pattern files are prepared and distributed in advance to the cloud, allowing television devices to detect advertisement segments. However, logo images and other data must be purchased from advertisement providers, which incurs significant costs in order to expand the scope of target content. Furthermore, since advertisement segment detection is performed by the television device, there are concerns about processing delays.

[0142] In addition, the technology of Patent Document 3 above uses artificial intelligence to generate features in real time on a terminal device in the medical field, and then uses the features to execute processing using time-series data. However, it is difficult to apply such technology to television broadcast programs.

[0143] For television broadcasting, there is a demand for features such as time-shift playback, which allows playback of recorded content in the middle of live viewing, and the ability to record all broadcast programs on all channels at once. To accommodate these features, television sets must be equipped with multiple AV (Audio / Visual) decoders and AI computing machines for parallel processing of multiple contents in real time. Such a configuration poses significant obstacles in terms of both component costs and running costs, making it unrealistic to apply this technology to television sets, which are consumer devices.

[0144] According to the television device 20 of the embodiment, while a broadcast program is being provided for live viewing, converted data is generated from the broadcast signal of the broadcast program, from which metadata representing the content of the broadcast program can be generated.

[0145] In this way, the processing of a huge amount of content can be assigned to each television device 20, and unlike the case where the server device 10 processes the content collectively, for example, the load on the server device 10 can be reduced.

[0146] Furthermore, the above data conversion process can be performed using, for example, the existing hardware configuration of the television device 20. There is no need to add a new configuration to the television device 20 or to newly construct the television device 20 itself, which makes it possible to reduce component costs and running costs.

[0147] According to the television device 20 of the embodiment, the converted data is transmitted to the server device 10, which generates metadata. By increasing the frequency of uploads to the server device 10, the data handled by the server device 10 is frequently updated, which makes it possible to support, for example, time-shift playback, which requires processing in a short time.

[0148] According to the television device 20 of the embodiment, broadcast program information is converted into a multidimensional array for each frame to generate converted data. By converting data into a multidimensional array in this way, it becomes possible to handle huge amounts of data. Furthermore, for example, by using a multidimensional array for input / output data of a DNN, data analysis using DNN technology becomes easier. Furthermore, the amount of data handled by the server device 10 can be reduced, enabling the collection and processing of information on more content.

[0149] According to the television device 20 of the embodiment, tasks required for data conversion are identified based on program sequence information, and the identified tasks are assigned to the data processing unit 23. As a result, in a data conversion process that requires different tasks for each content or frame of a broadcast program, for example, it is possible to assign and execute appropriate tasks to the data processing unit 23. Furthermore, the limited resources of the data processing unit 23 can be used efficiently.

[0150] According to the television device 20 of the embodiment, when the data processing unit 23 has surplus resources, preprocessing tasks that can be used for data conversion are assigned to the data processing unit 23. This allows the resources of the data processing unit 23 to be used effectively.

[0151] According to the server device 10 of the embodiment, metadata is generated from the converted data that the television device 20 has converted.

[0152] This allows the server device 10 to take charge of generating metadata that requires handling of time-series data, thereby preventing any disruption to real-time processing of viewing, recording, and playback on the television device 20.

[0153] Furthermore, the server device 10 only needs to execute the process of generating metadata that deals exclusively with time-series data, and the load on the server device 10 can also be reduced.

[0154] According to the server device 10 of the embodiment, if information on the estimated end position is not included for a predetermined period of time from the estimated start position of the advertisement estimated by the data processing unit 23, it is determined that the advertisement was not inserted at the estimated start position. In this way, by making the above determination in the server device 10 capable of handling time-series data, the advertisement insertion position can be identified with high accuracy.

[0155] According to the server device 10 of the embodiment, when another estimated start position or another estimated end position is included between the estimated start position and estimated end position of an advertisement, the combination of these estimated start positions and estimated end positions whose advertisement insertion period is closest to a multiple of the shortest advertisement broadcast time is selected as the advertisement start position and end position. In this way, by performing the above determination in the server device 10 capable of handling time-series data, the advertisement insertion position can be identified with high accuracy.

[0156] According to the server device 10 of the embodiment, converted data collected from at least one television device 20 currently providing live viewing of a broadcast program among the plurality of television devices 20 is integrated into time-series data. In this way, by distributing the data conversion process among the plurality of television devices 20, the load on the television devices 20 can be further reduced.

[0157] In the above embodiment, the server device 10 generates time-series data by collecting and integrating converted data from, for example, multiple television devices 20 that are providing live viewing. However, a service provider that generates metadata may install multiple television devices 20 within its own premises and have the converted data uploaded from these television devices 20 to the server device 10.

[0158] In this case, it is possible to have each television device 20 consistently provide live viewing of broadcast programs from a specific broadcast station. This allows conversion data related to broadcast programs from a single broadcast station to be collected from a single television device 20, eliminating the need for processing to integrate conversion data from multiple television devices 20, for example.

[0159] In addition, in the above-described embodiment, the receiving device is the television device 20, but the configuration of the embodiment is not limited to this. For example, the receiving device may be other devices such as a personal computer, smartphone, tablet, or mobile phone that have a broadcast signal receiving function, a broadcasting function, and a voice recognition service function.

[0160] Although the embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. This novel embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the inventions and their equivalents as defined in the claims. [Explanation of symbols]

[0161] 1...metadata generation system, 10...server device, 11...transmitter / receiver unit, 12...integration unit, 13...advertisement determination unit, 14...metadata generation unit, 15...storage unit, 20...television device, 21...transmitter / receiver unit, 22...task allocation unit, 23...data processing unit, 24...broadcast receiving unit, 29...storage unit.

Claims

1. A receiving device that receives broadcast programs and provides them so that they can be viewed live, a data processing unit that generates, from a broadcast signal of the broadcast program while the broadcast program is being provided for live viewing, conversion data that enables the generation of metadata representing the content of the broadcast program; a first transmitting / receiving unit that transmits the converted data to a server device that generates the metadata; The data processing unit converting the broadcast program into a multidimensional array for each frame, and generating the converted data including an inference result obtained by inferring the content of the broadcast program from the multidimensional array; Receiving device.

2. A receiving device that receives broadcast programs and provides them for live viewing, a data processing unit that generates, from a broadcast signal of the broadcast program while the broadcast program is being provided for live viewing, conversion data that enables the generation of metadata representing the content of the broadcast program; a first transmitting / receiving unit that transmits the converted data to a server device that generates the metadata; a broadcast receiving unit that receives a broadcast signal of the broadcast program and program arrangement information multiplexed onto the broadcast signal; a task allocation unit that identifies a task required for generating the converted data based on the program sequence information and allocates the identified task to the data processing unit. Receiving device.

3. The task allocation unit if there are surplus resources in the data processing unit after the tasks are assigned to the data processing unit, assign a pre-processing task available for generating the converted data to the data processing unit.

3. The receiving device according to claim 2.

4. A receiving device according to any one of claims 1 to 3; a server device connected to the receiving device so as to be able to communicate with the receiving device; The server device a second transceiver that receives the converted data generated by the receiving device from the receiving device; a metadata generation unit that generates the metadata based on the converted data, The server device further comprising an advertisement determination unit that identifies an insertion position of an advertisement in the broadcast program; The metadata generation unit generating the metadata including information indicating an insertion position of the advertisement; Metadata generation system.

5. The data processing unit generating the conversion data for each frame, the conversion data including information indicating an estimated start position and an estimated end position of the advertisement; The advertisement determination unit determining an insertion position of the advertisement based on time series data obtained by arranging the converted data for each frame in time series; The metadata generation system of claim 4.

6. The advertisement determination unit If the information on the estimated end position is not included for a predetermined period from the estimated start position, it is determined that the advertisement has not been inserted at the estimated start position. The metadata generation system of claim 5 .

7. The advertisement determination unit If another estimated start position or another estimated end position is included between the estimated start position and the estimated end position, from among these combinations of estimated start positions and estimated end positions, a combination in which the advertisement insertion period is closest to a multiple of the shortest advertisement broadcast time is selected as the advertisement start position and end position. The metadata generation system of claim 5 .

8. The server device a receiving device connected to a plurality of receiving devices including the receiving device so as to be capable of communicating with the plurality of receiving devices; The system further includes an integration unit that integrates the converted data collected from at least one of the plurality of receiving devices that is currently providing live viewing of the broadcast program into time-series data. The metadata generation system of claim 4.

9. The server device transmitting the metadata generated from the converted data to the first transceiver unit of at least one of the plurality of receiving devices; The metadata generation system of claim 8.

Images