Question generation device, question generation system, and question generation method

Abstract

To easily generate a question for obtaining information related to an object displayed in a moving image.SOLUTION: A question generation device includes: an element output part which extracts a moving image element from the moving image on which an action is performed when the action is performed on the displayed moving image, and outputs it to an analyzer; a text acquisition part which acquires a text-converted element created according to the moving image element by the analyzer; and a question generation part which generates a question candidate related to the moving image element from the text-converted element.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a question generating device, a question generating system, and a question generating method. [Background technology]

[0002] In recent years, videos have been viewed on users' terminals (such as tablet terminals, smartphones, personal computers, etc.) For example, Patent Documents 1 to 3 describe viewing broadcast programs such as dramas on viewers' terminals. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-119833 A [Patent Document 2] JP 2007-306399 A [Patent Document 3] JP 2004-23425 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when a viewer of a video becomes interested in an object (hereinafter sometimes referred to as an "item") shown in the video, the viewer may try to collect information related to the item (hereinafter sometimes referred to as "item-related information"). For example, while watching a drama, a viewer may want to buy a bag carried by the main character in the drama, and may try to search online for the brand name and product name of the bag.

[0005] In addition, at that time, the viewer may try to collect item-related information, for example, by using an online chatbot. In recent years, chatbots often use software built based on large language models (LLMs). Viewers can obtain item-related information through text and voice dialogue with the chatbot.

[0006] However, even if viewers try to obtain item-related information through text or voice dialogue, it is difficult to find the desired item-related information if they do not know the search keywords for the item. In other words, since viewers do not know the search keywords for the item, they cannot easily come up with appropriate questions to ask the chatbot.

[0007] One object of the present invention is to easily create questions to obtain information related to an object shown in a video. Other objects of the present invention will become apparent from the description of this specification. [Means for solving the problem]

[0008] One aspect of the present invention is a question creation device that includes an element output unit that, when an action is performed on a displayed video, extracts video elements from the video on which the action was performed and outputs the video to an analysis device, a text acquisition unit that acquires text elements created by the analysis device based on the video elements, and a question creation unit that creates candidate questions related to the video elements from the text elements.

[0009] One aspect of the present invention is a question creation system that includes an element output unit that, when an action is performed on a displayed video, extracts video elements from the video on which the action was performed and outputs the video to an analysis device, a text acquisition unit that acquires text elements created by the analysis device based on the video elements, a question creation unit that creates candidate questions related to the video elements from the text elements, and a display unit that displays the candidate questions.

[0010] One aspect of the present invention is a question creation method comprising: when an action is performed on a displayed video, extracting a video element from the video on which the action was performed and outputting the video to an analysis device; obtaining a text element created by the analysis device based on the video element; and creating candidate questions related to the video element from the text element. Effect of the Invention

[0011] According to the above-described aspects of the present invention, questions for obtaining information related to a subject shown in a video can be easily created. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is an explanatory diagram showing an example of an action executed on a video according to the present embodiment. [Diagram 2] FIG. 2 is a diagram showing a question generation system 100 having a question generation device 10 of this embodiment. [Diagram 3] FIG. 3 is a flow diagram of a question creation process performed by the question creation device 10 of this embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a question template. [Diagram 5] FIG. 5 is a diagram showing a method of inputting text elements into a question template when there are a plurality of text elements. [Figure 6] FIG. 6 is a diagram showing an example of a screen in the first example of the question creation process. [Figure 7] FIG. 7 is a diagram showing an example of a screen when acquiring video elements and creating question candidates in the first example of the question creating process. [Figure 8] FIG. 8 is a diagram showing an example of a screen on which question candidates are displayed in the first example of the question generating process. [Figure 9] FIG. 9 is a diagram showing an example of a screen in the second example of the question creation process. [Figure 10]FIG. 10 is a diagram showing an example of a screen when acquiring a video element in the second example of the question creating process. [Figure 11] FIG. 11 is a diagram showing an example of a screen when acquiring text elements and creating question candidates in the second example of the question creation process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] At least the following points will become apparent from the following specification and drawings.

[0014] A question creation device is disclosed that includes an element output unit that, when an action is performed on a displayed video, extracts a video element from the video on which the action was performed and outputs the extracted video element to an analysis device, a text acquisition unit that acquires a text element created by the analysis device based on the video element, and a question creation unit that creates a question candidate related to the video element from the text element. With such a question creation device, it is possible to easily create a question for obtaining information related to an object shown in the video.

[0015] The video elements preferably include at least one of image information, audio information, and text information related to the video, which makes it easy to create questions to obtain information related to the subject shown in the video.

[0016] It is preferable that the video element has the image information including type information related to the video element, the text acquisition unit acquires the type information in association with the text element, and the question creation unit creates the question candidate using a question template corresponding to the type information. In this way, by utilizing the type information contained in the image information, it is possible to create highly accurate question candidates based on the type information.

[0017] It is preferable that the video element is at least one of image information, audio information, and character information not including type information related to the video element, the text acquisition unit acquires the text element linked to the type information, and the question creation unit creates the question candidate using a question template corresponding to the type information. In this way, even if a video element does not have type information, a question template corresponding to the type information can be used.

[0018] It is preferable that the text acquisition unit acquires a first text element associated with the first type information and a second text element associated with the second type information, the question template has a first input element related to the first type information and a second input element related to the second type information, and the question creation unit creates the question candidate by inputting the first text element into the first input element and the second text element into the second input element. This makes it possible to prevent the question creation unit from creating a question candidate that is expressed unnaturally as a question.

[0019] It is preferable to further include a question output unit that outputs the question candidates to a display device, whereby a viewer can visually check the question candidates via the display device.

[0020] It is preferable that the question creation unit creates a plurality of the question candidates, and the question output unit outputs the plurality of question candidates to the display device in a manner that allows a viewer of the video to select a question candidate, thereby allowing the viewer to select a question candidate that matches his / her intention from the plurality of question candidates.

[0021] It is preferable that the question output unit add priority information to the plurality of question candidates and output the candidate questions to the display device, thereby enabling a viewer to quickly select a candidate question that matches his / her intention from the plurality of candidate questions.

[0022] It is preferable that the question output unit outputs the question candidate determined by the viewer of the video to an answer creation device, whereby an answer to the question determined by the viewer can be automatically created.

[0023] The answer generation device is preferably a large-scale language model, which allows automatic generation of answers to questions established by the viewer.

[0024] The action is preferably performed by any one of clicks, touches, voice input, and keyboard input, or a combination of any one of these operations, which makes it easy to create questions to obtain information related to the object shown in the video.

[0025] A question creation system is disclosed that includes an element output unit that, when an action is performed on a displayed video, extracts a video element from the video on which the action was performed and outputs the extracted video element to an analysis device, a text acquisition unit that acquires a text element created by the analysis device based on the video element, a question creation unit that creates candidate questions related to the video element from the text element, and a display unit that displays the candidate questions. With such a question creation system, questions for obtaining information related to an object shown in a video can be easily created.

[0026] A question creation method is disclosed that includes, when an action is performed on a displayed video, extracting a video element from the video on which the action was performed and outputting the extracted video element to an analysis device, obtaining a text element created by the analysis device based on the video element, and creating a question candidate related to the video element from the text element. With such a question creation method, a question for obtaining information related to an object shown in the video can be easily created.

[0027] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. The same or equivalent components, members, etc. shown in each drawing are designated by the same reference numerals, and duplicated descriptions will be omitted as appropriate.

[0028] ===This embodiment=== <Video summary> Before describing the question creation device, the question creation system, and the question creation method of the present embodiment, first, with reference to FIG. 1, an example of a moving image used in the present embodiment and an action executed on the moving image will be described.

[0029] Fig. 1 is an explanatory diagram showing an example of an action executed on a video of this embodiment. Fig. 1A shows an example of a screen in the video of this embodiment, Fig. 1B shows an example of an operation (described later) for stocking item area information in the video, and Fig. 1C shows an example of a screen in a state in which the item area information has been stored by stocking.

[0030] As shown in FIG. 1A, a video is being played on the screen (here, a touch panel 6) of a user terminal 5 (for example, a viewer's smartphone or tablet device). A "video" is a moving image distributed as data from a distribution entity (here, a video distribution server 1, described below), and is, for example, a variety show, drama, news, sports, music, anime, or other program. The distribution format of the video is not limited to streaming format, as described below, but may also be download format, progressive download format, or live format.

[0031] 1A shows a person wearing sportswear (specifically, a jacket) on the upper half of his body. In this embodiment, an item area is set in advance in frames (still images) included in the video, and the video has information about the item for which the item area is set (hereinafter, may be referred to as "item area information").

[0032] Here, the "item area" refers to the area on the screen of an item shown in a video. In this embodiment, an item area is set for an item shown in a video, and the viewer is guided to the item area information, so that item area information in the video can be provided to the viewer in a simple manner. For example, when a viewer is interested in an item in a video, the viewer can display the item area information associated with the item area simply by selecting the item area. In this embodiment, an item area is set in advance in the jacket area on the screen. The item area is represented by a rectangular dashed frame 41 in FIG. 1B.

[0033] However, frame 41 representing the item area is not displayed on the actual touch panel 6. This can prevent frame 41 representing the item area from interfering with viewing of the video. That is, for the sake of convenience of explanation, frame 41 showing the item area is shown in FIG. 1B. However, frame 41 representing the item area may be displayed, rather than hidden, on the actual touch panel 6 during playback of the video.

[0034] When playing a video, frames (still images) are switched and displayed in sequence, so the area occupied by the jacket on the video screen (on the frame) changes from moment to moment. However, in this embodiment, the item area is set to change from moment to moment to follow the video, and the frame 41 representing the item area also changes from moment to moment.

[0035] As shown in FIG. 1B, when the user terminal 5 detects that an item area previously set in a video has been tapped, the user terminal 5 stores item area information associated with the item area as stock information. Here, the "tap operation" refers to an operation of touching the screen with a finger as if tapping and then releasing the finger, and is included in the "touch operation." Also, the "touch operation" is a general term for an operation performed by touching the screen with a finger. In addition to the above-mentioned "tap operation," the "touch operation" includes various operations such as "swipe," "double tap," "flick," "scroll," "drag," "long press," and "pinch in or pinch out." Note that, when the user terminal 5 detects a touch operation other than the tap operation, the user terminal 5 may store item area information associated with the item area as stock information.

[0036] As shown in FIG. 1C, when predetermined item area information is stored as stock information, an item image 44 (e.g., a thumbnail image) corresponding to the stock information (stocked item area information) is displayed on the stock information display section 43 on the touch panel 6 of the user terminal 5. That is, when a viewer is interested in a jacket in a video, the viewer can tap the jacket on the touch panel 6 to stock the item area information of that jacket. Furthermore, the viewer can confirm on the stock information display section 43 that the item area information of that jacket has been stored as stock information.

[0037] Although not shown in FIG. 1B and FIG. 1C, when a viewer stocks item area information, an item image (e.g., a thumbnail image) associated with the item area may be displayed. For example, when the user terminal 5 detects that an item area previously set in a video has been tapped, the user terminal 5 displays an item image associated with the item area. Even if the frame 41 (rectangular dashed line in FIG. 1B) indicating the item area is not displayed, the viewer can recognize that when the viewer is interested in a jacket in the video, the item image of the jacket is displayed by tapping the jacket on the touch panel 6, and related information about the jacket can be obtained. When the user terminal 5 detects a touch operation other than a tap operation, such as a double tap operation, the user terminal 5 may display an item image associated with the item area. The viewer may stock multiple items of item area information.

[0038] When the user terminal 5 detects that the area of ​​the item image 44 displayed on the stock information display unit 43 has been tapped, the user terminal 5 performs processing according to the event information associated with the item image 44 (item area information) (for example, displaying a purchase page (payment page) for a jacket). Note that when the user terminal 5 detects a touch operation other than a tap operation, the user terminal 5 may perform processing according to the event information associated with the item image 44 (item area information).

[0039] Here, the address of the purchase page (payment page) for the jacket is associated as the item area information for the jacket, and when the viewer taps on the item image 44 for the jacket, the purchase page (payment page) for that jacket is displayed on the touch panel 6. The display screen for the purchase page (payment page) for the jacket is the item area information for the jacket as well as the item image 44 for the jacket. The web page may be displayed as a multi-screen together with the video being played, or may be displayed alone.

[0040] In the following description, an operation of the terminal performed by the viewer due to the video being viewed may be referred to as an "action." Actions are not limited to the above-mentioned touch operations (tapping and swiping). An operation performed by the viewer due to the video being viewed may be, for example, voice input, or, if the user terminal 5 is a personal computer, mouse click or keyboard input. An action is executed by any of the operations of clicking, touching, voice input, and keyboard input, or a combination of any of the operations.

[0041] In the above example, the terminal (device) on which the action is executed is the same terminal (device) as the terminal (device) on which the video is displayed. That is, on the user terminal 5 on which the video is displayed, the storage of stock information by tapping operation and the display of the item purchase page (payment page) by tapping operation are executed.

[0042] However, the terminal (device) on which the action is executed may be a terminal (device) different from the terminal on which the video is displayed. The action may be executed on a terminal other than the user terminal 5 on which the video is displayed, for example, on another smartphone.

[0043] Furthermore, item regions do not have to be set in frames (still images) included in a video, i.e., the video does not have to have item region information.

[0044] <Question Creation System 100> Next, with reference to FIG. 2, a configuration of a question generation system 100 including the question generation device 10 of this embodiment will be described.

[0045] FIG. 2 is a diagram showing a question generation system 100 having a question generation device 10 of this embodiment.

[0046] The question creation system 100 is a system for creating questions to obtain item-related information. The item-related information includes information directly related to the item (e.g., the name, size, color, price, etc. of the item, and the item area information in FIG. 1 described above), as well as indirect information about the item (e.g., recommended information related to the item, etc.).

[0047] In this embodiment, when a viewer is interested in an object (item) shown in a video, the viewer can easily create a question for obtaining item-related information by simply performing an action on the displayed video. For example, if a viewer were to enter text on a user terminal when creating a question for a chatbot, this would interfere with viewing of the video. In contrast, according to this embodiment, a question for obtaining item-related information can be automatically created by simply performing an action on a displayed video, and this can prevent the viewer from being disturbed while viewing the video. In the question creation system 100, a question creation process is performed by multiple devices via a communication network 9, which will be described later.

[0048] 2, the question creation system 100 includes a video distribution server 1, a metadata distribution server 3, a user terminal 5, a question creation device 10, an analysis device 20, and an answer creation device 30. The video distribution server 1, the metadata distribution server 3, the user terminal 5, the question creation device 10, the analysis device 20, and the answer creation device 30 are communicatively connected to each other via a communication network 9. Here, the communication network 9 is, for example, the Internet, a telephone line network, a wireless communication network, a LAN, a VAN, etc., and the Internet is assumed here.

[0049] The video distribution server 1 is a server for distributing videos. In this embodiment, the video distribution server 1 distributes video data to the user terminal 5 in a streaming format. However, the video data may be distributed in a download format or a progressive download format. In the case of streaming format distribution, the video data is temporarily stored in the user terminal 5, and in the case of download format distribution, the downloaded video data is stored and held in the user terminal 5. Furthermore, the video distribution server 1 may distribute videos in a live format.

[0050] The metadata delivery server 3 is a server for delivering metadata including the above-mentioned item area information. In this embodiment, a part of the metadata is delivered in a preload format before video playback, and the other part of the metadata is delivered in a progressive download format. However, the method of delivering the metadata is not limited to this, and may be, for example, a download format or a streaming format.

[0051] In this embodiment, for convenience of explanation, the metadata and the video data are described separately, but the metadata may be stored in the video data (video file). When the video distribution server 1 distributes the video data with the metadata stored in the video data, the question creation system 100 does not need to include the metadata distribution server 3.

[0052] The metadata distributed by the metadata distribution server 3 may be created by the metadata distribution server 3 itself, or may be created by a metadata creation terminal (not shown).

[0053] The user terminal 5 is an information terminal (video playback device) capable of playing videos. Here, the user terminal 5 is a smartphone. However, the user terminal 5 is not limited to a smartphone, and may be, for example, a tablet-type mobile terminal or a personal computer. The user terminal 5 is equipped with hardware such as a CPU, memory, storage device, communication module, and touch panel 6 (corresponding to display unit 7A and input unit 7B), which are not shown. A video playback program is installed in the user terminal 5, and the above-mentioned various operations are realized by the user terminal 5 executing the video playback program. The video playback program can be downloaded to the user terminal 5 from a program distribution server, not shown.

[0054] The user terminal 5 includes a display unit 7A, an input unit 7B, a control unit 8A, and a communication unit 8B.

[0055] The display unit 7A is a function for displaying various screens. In this embodiment, the display unit 7A is realized by the display of the touch panel 6, a controller that controls the display of the display, and the like. The input unit 7B is a function for inputting and detecting instructions from a user. In this embodiment, the input unit 7B is realized by the touch sensor of the touch panel 6, and the like. In addition, when the user terminal 5 is a smartphone or a tablet-type mobile terminal, the display unit 7A and the input unit 7B are mainly realized by the touch panel 6, but the display unit 7A and the input unit 7B may be composed of separate parts. For example, when the user terminal 5 is a personal computer, the display unit 7A is composed of, for example, a liquid crystal display, and the input unit 7B is composed of a mouse, a keyboard, and the like.

[0056] The control unit 8A has a function of controlling the user terminal 5. The control unit 8A has a function for processing video data to play (display) videos, a function for processing metadata, and the like. The control unit 8A also has a browser function for acquiring web page information and displaying the web page. In this embodiment, the control unit 8A is realized by a CPU, a memory that stores a video playback program, a storage device, and the like (not shown).

[0057] The communication unit 8B is a function for connecting to the communication network 9. The communication unit 8B receives video data from the video distribution server 1, receives metadata from the metadata distribution server 3, and requests data from the video distribution server 1 and the metadata distribution server 3.

[0058] Although not shown in the figure, in addition to the above-mentioned configuration, the user terminal 5 may also have a video data storage unit having the function of storing video data, a metadata storage unit having the function of storing metadata, and a stock information storage unit that stores stocked item area information in correspondence with the video data.

[0059] The question creation device 10 is a device for creating questions for obtaining item-related information. The question creation device 10 executes a question creation process, which will be described later. In this embodiment, various processes including the question creation process can be executed in the question creation device 10 by cooperation between various hardware and software (not shown) of the question creation device 10.

[0060] The question creation device 10 is, for example, a computer such as a server, and is composed of an arithmetic unit (such as a CPU), a memory, a storage device, a communication device, etc. The storage device stores various programs including a question creation program and various data. The arithmetic unit reads out the question creation program stored in the storage device into memory and executes it, thereby realizing various processes including question creation processing, that is, the functions of each part described below (element output unit 11, text acquisition unit 12, question creation unit 13, and question output unit 14). The functions of the element output unit 11, text acquisition unit 12, question creation unit 13, and question output unit 14 will be described in detail later.

[0061] The question creation device 10 of this embodiment may have multiple computers. Various processes including the question creation process may be executed by the cooperation of the multiple computers via a network. In the question creation system 100 shown in Fig. 2, the question creation device 10 is connected to one user terminal 5 via a communication network 9, but the question creation device 10 may be connected to multiple user terminals 5 via the communication network 9.

[0062] The analysis device 20 is a device that analyzes video elements in a video and creates text elements.

[0063] Here, a video element is an element that can be extracted as data from a video, and the video element has, for example, at least one of image information, audio information, and text information related to the video. Note that image information is, for example, image data of people or objects in the video, and is not limited to image data that includes item area information (item area is set), but also includes image data that does not include item area information (item area is not set). Audio information is audio data such as sounds, lines, and background music in the video. Text information is not limited to text information within the video such as subtitles and tickers, but also includes text information outside the video such as viewer comments.

[0064] Moreover, the text element is character information that represents the above-mentioned video element. For example, if the video element is image data of a jacket shown in FIG. 1 above, the text information is "clothes," "sportswear," "jacket," etc., and if the color of the jacket is red, the text information is "red," etc. Note that video elements and text elements do not need to correspond one-to-one, and multiple "text elements" may correspond to one video element.

[0065] In this embodiment, the analysis device 20 is capable of executing various processes including the analysis process by cooperation of various hardware and software (not shown).

[0066] The analysis device 20 is, for example, a computer such as a server, and is composed of an arithmetic device (such as a CPU), a memory, a storage device, a communication device, and the like. The storage device stores various programs including an analysis program and various data. The arithmetic device reads out the analysis program stored in the storage device into the memory and executes it, thereby realizing various processes including the analysis process. The analysis device 20 of this embodiment may have multiple computers. The various processes including the analysis process may be executed by the multiple computers working together via a network.

[0067] Specifically, the analysis device 20 is a large-scale language model, a text search engine, an image search / image generation engine, a sound source search / sound source generation engine, a video search / video generation engine, a recommendation engine, etc. The analysis device 20 is not limited to a device using AI (Artificial Intelligence), and may be a device that does not use AI as long as it analyzes video elements in a video and creates text elements.

[0068] In the analysis device 20, for example, the text search engine and the image search engine may be configured as separate computers, or the computer configuring the text search engine may be configured as a plurality of computers.

[0069] In the present embodiment, the analysis device 20 has been described as being a device separate from the question creation device 10. That is, the analysis device 20 has been described as being a device external to the question creation device 10. However, the question creation device 10 may have the functions of the analysis device 20. Furthermore, the question creation system 100 does not necessarily have to include the analysis device 20.

[0070] The answer creating device 30 is a device that creates answers to question candidates related to video elements created from the above-mentioned text elements. In this embodiment, various processes including the answer creating process can be executed in the answer creating device 30 by cooperation of various hardware and software (not shown) of the answer creating device 30.

[0071] The answer creation device 30 is, for example, a computer such as a server, and is composed of an arithmetic device (such as a CPU), a memory, a storage device, a communication device, etc. The storage device stores various programs including an answer creation program and various data. The arithmetic device reads out the answer creation program stored in the storage device into the memory and executes it, thereby realizing various processes including the answer creation process. The answer creation device 30 of this embodiment may have multiple computers. The various processes including the answer creation process may be executed by the multiple computers working together via a network.

[0072] Specifically, the answer creation device 30 is a large-scale language model, a text search engine, an image search / image generation engine, a sound source search / sound source generation engine, a video search / video generation engine, a recommendation engine, or the like. In this embodiment, specifically, the answer creation device 30 is a generation AI constructed based on a large-scale language model such as CHATGPT (registered trademark). The answer creation device 30 is not limited to a device using AI, and may be a device that does not use AI as long as it is a device that creates answers to question candidates.

[0073] In this embodiment, the answer creation device 30 has been described as being a device separate from the question creation device 10. That is, the answer creation device 30 has been described as being a device external to the question creation device 10. However, the question creation device 10 may have the functions of the answer creation device 30. Also, the question creation system 100 does not necessarily have to include the answer creation device 30.

[0074] <Question Creation Device 10> As shown in FIG. 2, the question creation device 10 includes an element output unit 11, a text acquisition unit 12, a question creation unit 13, a question output unit 14, and a recording unit 16.

[0075] The element output unit 11 is a part that, when an action is executed on a displayed video, extracts a video element (e.g., image information of a jacket) from the video and outputs it to the analysis device 20. As described above, the analysis device 20 analyzes the video element (e.g., image information of a jacket) output from the element output unit 11 and creates a text element (e.g., character information of "jacket").

[0076] Furthermore, analysis device 20 may create multiple text elements (for example, character information such as "jacket", "clothes", and "sportswear") for one video element (for example, image information of a jacket). Analysis device 20 creates a text element (for example, character information such as "jacket") based on a video element (for example, image information of a jacket) output from element output unit 11, and outputs the text element to text acquisition unit 12.

[0077] The text acquisition unit 12 is a part that acquires at least one text element created by the analysis device 20 based on the video element.

[0078] The question creation unit 13 is a part that creates candidate questions (for example, text information such as "How much does a jacket cost?") related to video elements from text elements (for example, text information such as "jacket") acquired by the text acquisition unit 12. The question creation unit 13 may create multiple candidate questions (for example, text information such as "How much does a jacket cost?" and "Where can I buy a jacket?") from one text element (for example, text information such as "jacket").

[0079] The question output unit 14 is a part that outputs the question candidates created by the question creating unit 13 to a display device (for example, the display unit 7A of the user terminal 5). This allows the viewer to visually check the question candidates via the display device.

[0080] The recording unit 16 is a part where the data of the video elements, text elements, and question candidates described above is recorded. However, the recording unit 16 may record data other than the data of the video elements, text elements, and question candidates. Furthermore, the question creation device 10 may not have the recording unit 16, and the user terminal 5 may have the recording unit 16. Specifically, the recording unit 16 may be secured as a memory area in the user terminal 5. This allows specific video elements, text elements, and question candidates to be recorded in the user terminal 5 (i.e., a terminal owned by the viewer) while protecting the privacy of the viewer. Furthermore, both the question creation device 10 and the user terminal 5 may have the recording unit 16.

[0081] <Overview of question creation process> Next, a question creation process, which is an example of a question creation method according to the present embodiment, will be described with reference to FIG.

[0082] FIG. 3 is a flow diagram of a question creation process performed by the question creation device 10 of this embodiment.

[0083] First, when an action is performed on a displayed video, the element output unit 11 extracts a video element (e.g., image information of a jacket) from the video on which the action was performed, and outputs it to the analysis device 20 (S001). Here, the action is not limited to one, and multiple actions may be performed, and the element output unit 11 may extract at least one video element between the multiple actions. For example, the element output unit 11 may extract a video element during a period in which the viewer performs a touch operation multiple times within a predetermined time. This allows the element output unit 11 to extract not only the video element at the time the action is performed, but also video elements over a certain time span (a width within a predetermined time). Furthermore, the element output unit 11 may extract a video element during a period in which the viewer performs a long press operation within a predetermined time.

[0084] The analysis device 20 creates a text element (for example, character information of “jacket”) based on the video element (for example, image information of a jacket) output from the element output unit 11, and outputs the text element to the text acquisition unit 12.

[0085] Next, the text acquiring unit 12 acquires a text element (for example, the character information "jacket") from the analysis device 20 (S002). Next, the question creating unit 13 creates a question candidate (for example, the character information "How much does the jacket cost?") (S003).

[0086] In this embodiment, the question creation unit 13 creates question candidates using a question template corresponding to the category information. Here, the "category information" is character information related to the attributes of a video element, and is data that can be linked to the textual element of the video element. For example, if a video element is image data of a jacket, the textual element of the video element is the character information "jacket", and the category information is the character information "proper name / product name".

[0087] When a video element is image information including item area information (in which an item area is set) (image data of a video shown in FIG. 1), type information may be included in the item area information of the video element. In other words, type information may be included as item area information of the video. In this case, the text acquisition unit 12 acquires type information (e.g., character information such as "proper name / product name") by linking it to a text element (e.g., character information such as "jacket"). By using the type information contained in the image information, it is possible to create highly accurate question candidates based on the type information.

[0088] When a video element is at least one of image information, audio information, and character information that does not include item area information, analysis device 20 may analyze the video element and create a text element (e.g., character information such as "jacket") linked to type information (e.g., character information such as "proper name / product name"). Text acquisition unit 12 acquires the text element linked to type information. This allows a question template corresponding to type information to be used even if a video element does not have type information.

[0089] The question templates are pre-recorded in the recording unit 16, and the question creating unit 13 reads out the question template data from the recording unit 16 to create question candidates.

[0090] FIG. 4 is a diagram showing an example of a question template.

[0091] As shown in Fig. 4, question templates are provided for questions such as "I would like to know more about XXX", "How much does XXX cost?", "Where can I buy XXX?", "I would like to see different colors of XXX". Here, the "XXX" part of the question template is the part where a text element (for example, character information such as "jacket") is input (hereinafter, sometimes referred to as "input element"). In addition, the four question templates mentioned above correspond to the category information, which is "proper name / product name".

[0092] In the case of a text element (e.g., character information "jacket") linked to category information (e.g., character information "proper name / product name"), the question creating unit 13 uses a question template corresponding to the category information "proper name / product name" from among a plurality of question templates. By using a question template corresponding to the category information rather than the question creating unit 13 arbitrarily selecting and using a question template (irrespective of the category information), it is possible to create accurate question candidates that are more in line with the viewer's intentions.

[0093] FIG. 5 is a diagram showing a method of inputting text elements into a question template when there are a plurality of text elements.

[0094] A question template is not limited to having one input element, but may have multiple input elements (here, a first input element and a second input element) as shown in Fig. 5A. Furthermore, each input element may be linked to a predetermined type information. That is, the first input element (XXX) may be linked to the first type information (e.g., "person's name"), and the second input element (△△△) may be linked to the second type information (e.g., "proper name / product name").

[0095] If the text element input to the first input element and the text element input to the second input element are arbitrary (unrelated to the type information), a question candidate such as that shown in Fig. 5B may be created. In other words, a question candidate that is unnaturally expressed as a question may be created, such as "I would like to know more about Mr. ABC, the jacket worn by the person."

[0096] In this embodiment, since each input element is linked to a predetermined type information, the question creation unit 13 creates a question candidate in which a first text element (Mr. ABC) is input to a first input element (XXX) and a second text element (jacket) is input to a second input element (△△△) as shown in Fig. 5C. This makes it possible to create a question candidate that is naturally expressed as a question, such as "I would like to know more about the jacket that Mr. ABC was wearing." In other words, it is possible to prevent the creation of a question candidate that is unnaturally expressed as a question, such as that shown in Fig. 5B.

[0097] Next, the question output unit 14 outputs the question candidates to the display device (S003). This allows the viewer to visually check the question candidates via the display device. Furthermore, in this embodiment, the question candidates are displayed during playback of the video (in parallel with playback of the video), so that the viewer can check the question candidates without being disturbed while watching the video.

[0098] When the question creation unit 13 creates multiple question candidates, the question output unit 14 may output the multiple question candidates to the display device in a form selectable by the viewer of the video. This allows the viewer to select a question candidate that matches his / her intention from the multiple question candidates.

[0099] Furthermore, the question output unit 14 may add priority information to the multiple question candidates and output them to the display device. For example, the multiple question candidates may be displayed from top to bottom of the screen in order of priority. The multiple question candidates may also be displayed by screen transitions in order of priority. Furthermore, each of the multiple question candidates may be displayed with information on relevance (relevance x%) or priority order information (1st, 2nd, 3rd, etc.). This allows the viewer to quickly select a question candidate that matches his / her intention from the multiple question candidates.

[0100] Finally, the question output unit 14 outputs the question candidate selected by the viewer (hereinafter, may be simply referred to as a "question") to the answer creation device 30 (S004). The answer creation device 30 creates an answer to the question and outputs it to the question creation device 10. The question creation device 10 outputs the answer to the question to a display device (for example, the display unit 7A of the user terminal 5). This allows the viewer to visually check the answer to the question via the display device.

[0101] In the question creating device 10, the answers to the questions acquired from the answer creating device 30 may be recorded in the recording unit 16. This allows the answers to the questions to be further analyzed and utilized, for example by reflecting them in the videos to be distributed. In addition, it is also possible to extract viewer tendencies from the analysis results and create recommendations from the extracted tendencies.

[0102] <Example of question creation process> Next, a specific example of the question creation process will be described with reference to FIGS. First Example Fig. 6 is a diagram showing an example of a screen in the first example of the question creation process. Fig. 7 is a diagram showing an example of a screen when acquiring video elements and creating question candidates in the first example of the question creation process. Fig. 8 is a diagram showing an example of a screen on which question candidates are displayed in the first example of the question creation process.

[0103] The first example of the question creation process shown in Figures 6 to 8 is an example in which, when a viewer becomes interested in an item (i.e., a jacket) for which an item area is set, a question is created to obtain item-related information (information about the jacket) for that item.

[0104] As shown in Fig. 6, a question button 50 is located at the bottom right of the touch panel 6 of the user terminal 5. This allows the viewer to recognize the presence of a function that creates candidate questions. When the user terminal 5 detects that an item area that is set in advance in a video has been tapped as shown in Fig. 7, the user terminal 5 creates candidate questions using each of the components (functions) of the element output unit 11, text acquisition unit 12, and question creation unit 13 of the question creation device 10.

[0105] In this embodiment, a question for obtaining item-related information about a jacket can be automatically created by simply performing an action (here, a tap operation) on a displayed video, and thus it is possible to prevent the video from being disturbed while the user is watching the video. As described above, when the user terminal 5 detects that an item area set in advance in a video has been tapped, the user terminal 5 stores the item area information associated with the item area as stock information. Therefore, a single action (here, a tap operation) on a displayed video can simultaneously create a question for obtaining item-related information and store the information as stock information.

[0106] 7, although a candidate question has been created, the specific content of the candidate question is not displayed on the touch panel 6. However, the color of the question button 50 located at the bottom right of the touch panel 6 is changed, so that the viewer can recognize that a candidate question has been created.

[0107] As shown in FIG. 8, when the user terminal 5 detects that the question button 50 has been tapped, the question output unit 14 of the question generating device 10 displays the candidate question 52. The viewer can display the candidate question 52 at the timing he or she intends (the timing of tapping the question button 50), which prevents the unexpected display of the candidate question 52 from disrupting viewing of the video. In FIG. 8, the question output unit 14 outputs a plurality of candidate questions (three in this case) to the touch panel 6 in a manner that allows the viewer to select one. This allows the viewer to select a question candidate that matches his or her intention from among the plurality of candidate questions.

[0108] Second Example Fig. 9 is a diagram showing an example of a screen in the second example of the question creation process. Fig. 10 is a diagram showing an example of a screen when a video element is acquired in the second example of the question creation process. Fig. 11 is a diagram showing an example of a screen when a text element is acquired and a question candidate is created in the second example of the question creation process.

[0109] The second example of the question creation process shown in Figures 9 to 11 is an example in which, when a viewer becomes interested in an item (i.e., a person) other than the item for which the item area is set, a question is created to obtain item-related information (information about the jacket) for that item.

[0110] As shown in FIG. 9, when the user terminal 5 detects that an area other than an item area previously set in the video has been tapped, the element output unit 11 of the question creation device 10 outputs image information of an area of ​​a predetermined size including the tapped area to the analysis device 20. As shown in FIG. 10, when a viewer taps on the image information of the area of ​​the predetermined size, an item image within that area may be displayed. Even if the area is other than the item area previously set in the video, if the viewer is interested in a person in the video, the person's item image is displayed when the viewer taps on the person on the touch panel 6, so the viewer can recognize that item-related information about the person can be obtained.

[0111] The analysis device 20 analyzes the image information and creates a text element. As shown in Fig. 10, the text element ("ABC") is displayed on the touch panel 6, and it can be recognized that the text element has been created based on the image information of the tapped area. Furthermore, as shown in Fig. 11, the question creation unit 13 of the question creation device 10 creates a candidate question 52, and the question output unit 14 displays the candidate question 52.

[0112] Other examples Examples of the question creation process are not limited to the first and second examples described above. For example, in the first example described above, the question button 50 for displaying the question candidates 52 may not be arranged. In this case, when a viewer is interested in an item for which an item region is set (i.e., a jacket), for example, the viewer may perform a long press on the item to display the question candidates 52 as shown in FIG. 11 in the second example. Furthermore, in the second example described above, the question button 50 may be arranged, and the viewer may tap the question button 50 to display the question candidates 52.

[0113] ===Other=== The above-described embodiment is for the purpose of facilitating understanding of the present invention, and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and it goes without saying that the present invention includes equivalents thereof. [Explanation of symbols]

[0114] 1. Video distribution server 3. Metadata distribution server 5 User terminal 6 Touch Panel 7A Display section 7B Input section 8A Control section 8B Communications Department 9. Communication Networks 10 Question creation device 11 Element Output Section 12 Text Acquisition Section 13 Question Creation Department 14 Question output section 16 Recording Section 20 Analyzer 30 Answer Generation Device 41 slots 43 Stock information display section 44 Item Images 50 Question button 51 Text elements 52 Candidate questions 100 Question Creation System

Claims

1. an element output unit that, when an action is executed on a displayed moving image, extracts a moving image element from the moving image on which the action is executed and outputs the extracted moving image element to an analysis device; a text acquisition unit that acquires a text element created based on the video element by the analysis device; a question creation unit that creates question candidates related to the video element from the text element; A question creation device comprising:

2. The video element has at least one of image information, audio information, and text information related to the video. The question generating device according to claim 1 .

3. The video element has the image information including type information related to the video element, The text acquisition unit acquires the type information by linking it to the text element, the question creation unit creates the question candidates by using a question template corresponding to the category information. The question generating device according to claim 2.

4. The video element is at least one of image information, audio information, and text information that does not include type information related to the video element, The text acquisition unit acquires the text element associated with the type information, the question creation unit creates the question candidates by using a question template corresponding to the category information. The question generating device according to claim 2.

5. The text acquisition unit acquires a first text element associated with the first type information and a second text element associated with the second type information, the question template has a first input element associated with the first type information and a second input element associated with the second type information; the question creation unit creates the question candidate by inputting the first text element into the first input element and inputting the second text element into the second input element.

5. The question generating device according to claim 3 or 4.

6. A question output unit that outputs the question candidates to a display device. The question generating device according to claim 1 .

7. The question creation unit creates a plurality of the question candidates, the question output unit outputs the plurality of question candidates to the display device in a manner that allows a viewer of the video to select the question; The question generating device according to claim 6.

8. the question output unit adds priority information to the plurality of question candidates and outputs the candidates to the display device. The question generating device according to claim 7.

9. the question output unit outputs the question candidate, the viewer of the video being determined, to an answer creation device; The question generating device according to any one of claims 6 to 8.

10. The answer creation device is a large-scale language model. The question generating device according to claim 9.

11. The action is executed by any one of click, touch, voice input, and keyboard input, or a combination of any one of the operations. The question generating device according to claim 1 .

12. an element output unit that, when an action is executed on a displayed moving image, extracts a moving image element from the moving image on which the action is executed and outputs the extracted moving image element to an analysis device; a text acquisition unit that acquires a text element created based on the video element by the analysis device; a question creation unit that creates question candidates related to the video element from the text element; a display unit that displays the question candidates; A question creation system comprising:

13. When an action is executed on a displayed moving image, extracting a moving image element from the moving image on which the action was executed and outputting the extracted moving image element to an analysis device; obtaining a text element generated based on the video element by the analysis device; generating candidate questions related to the video element from the text element; A method for creating questions comprising the steps of:

Images