Server and computer program

Abstract

To improve retrieval of a moving image and its distributor.SOLUTION: A server includes: an image holding part for holding an image related to a moving image distributed by a moving image distribution platform; a request reception part for receiving a retrieval request from a user terminal of a user of the moving image distribution platform through a network; a generation image acquisition part for acquiring a generation image generated by a machine learning model in accordance with information inputted by the user included in the retrieval request by inputting the information to the machine learning model; a retrieval part for executing image retrieval to the image holding part on the basis of the generation image; and a provision part for providing the user terminal with a result of the image retrieval executed by the retrieval part through the network.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a server and a computer program. [Background technology]

[0002] With the development of IT technology, the way information is exchanged has also changed. In the Showa era, one-way information transmission such as newspapers and television was the norm. In the Heisei era, mobile phones and personal computers became widespread and Internet communication speeds improved significantly, leading to the rise of instant two-way communication services such as chat services, and on-demand video distribution services became more popular as storage costs decreased. Now, in the Reiwa era, with the increasing functionality of smartphones and further improvements in network speeds such as 5G, services that enable real-time communication through video, especially live streaming services, are rapidly gaining recognition. Live streaming services are seeing an increase in users, especially among young people, as they allow everyone to share the same fun time even when they are far apart.

[0003] In a live distribution platform, a broadcaster can start a live distribution whenever he / she likes, and a viewer can watch a live distribution of a broadcaster that he / she is interested in whenever he / she likes. Patent Document 3 discloses a technology that recommends live streaming content using machine learning. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2021-158612 A [Patent Document 2] International Publication No. 2018 / 180201 Brochure [Patent Document 3] Special Publication No. 2020-521207 [Non-patent literature]

[0005] [Non-Patent Document 1] "How to start virtual streaming", 17LIVE Inc., URL: https: / / jp.17.live / userguide / 23783 / [Non-Patent Document 2] "The image generation AI that changed the world has further evolved: Stable Diffusion XL (SDXL) is finally officially released", Kiyoshi Arata, URL: https: / / ascii.jp / elem / 000 / 004 / 145 / 4145553 / Summary of the Invention [Problem to be solved by the invention]

[0006] Information about live streaming and its broadcasters includes many images, such as profile pictures, avatar images, the videos themselves, etc. If such images could be searched, the accuracy of live streaming and broadcaster searches could be improved.

[0007] The present disclosure has been made in consideration of these problems, and its purpose is to provide technology that can improve searches for videos and their distributors. [Means for solving the problem]

[0008] One aspect of the present invention relates to a server, which includes an image storage unit that stores images related to videos distributed by a video distribution platform, a request receiving unit that receives a search request from a user terminal of a user of the video distribution platform via a network, a generated image acquisition unit that uses information input by the user included in the search request as an input to a machine learning model and acquires a generated image generated by the machine learning model corresponding to the information, a search unit that executes an image search on the image storage unit based on the generated image, and a provision unit that provides the result of the image search executed by the search unit to the user terminal via the network.

[0009] Another aspect of the present invention is a computer program that causes a terminal to realize a function of displaying, on a display, a search criteria acceptance screen that accepts input of search criteria, a function of displaying, on a display, a generated image display screen including a plurality of generated images generated by inputting the search criteria input via the search criteria acceptance screen into a machine learning model, a function of accepting, on the generated image display screen, a selection of at least one generated image from the plurality of generated images, and a function of displaying, on the display, an image search result display screen including a result of an image search corresponding to the result of the selection.

[0010] In addition, any combination of the above components, or mutual substitution of the components or expressions of the present invention between an apparatus, a method, a system, a computer program, a recording medium storing a computer program, etc., are also valid aspects of the present invention. Effect of the Invention

[0011] The present invention allows for improved discovery of videos and their distributors. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram illustrating a configuration of a live distribution system according to an embodiment of the present disclosure. [Diagram 2] 2 is a block diagram showing the functions and configuration of the user terminal of FIG. 1. [Diagram 3] 2 is a block diagram showing the functions and configuration of the server in FIG. 1. [Figure 4] 4 is a data structure diagram showing an example of a stream DB in FIG. 3. [Diagram 5] 4 is a data structure diagram showing an example of a user DB in FIG. 3. [Figure 6] 4 is a data structure diagram showing an example of a gift DB in FIG. 3. [Figure 7] 13 is a flowchart showing a series of processes related to a distributor search in a server. [Figure 8]FIG. 13 is a representative screen diagram of a search condition acceptance screen displayed on the display of a user terminal of an active user. [Figure 9] FIG. 13 is a representative screen diagram of a generated image display screen displayed on the display of a user terminal of an active user. [Figure 10] FIG. 13 is a representative screen diagram of an image search result display screen displayed on the display of a user terminal of an active user. [Figure 11] 1 is a block diagram showing an example of a hardware configuration of an information processing device according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the same or equivalent components, members, processes, and signals shown in each drawing will be denoted by the same reference numerals, and duplicated explanations will be omitted as appropriate. Also, in each drawing, some of the members that are not important for the explanation will be omitted.

[0014] In the live distribution system according to the embodiment, the user inputs search criteria that represent the image of the desired distributor. The server presents the user with distributors who have profile images or live distribution images that match the search criteria as search results. In this case, instead of performing a text-image search based on tags attached to the images, in this embodiment, an image corresponding to the search criteria is first generated using a machine learning model for image generation. The user checks whether the generated image matches or is close to what the user wants. When the user accepts the generated image, the server performs an image-image search on the profile images or live distribution images using the generated image as a search key. The server presents the user with distributors who have profile images with high matching scores as search results. This makes it possible to search for distributors based on images without tagging the images to be searched, thereby reducing the effort of tagging. In particular, since profile images are updated frequently on live distribution platforms, the burden on distributors and administrators can be reduced by eliminating the need for tagging each time an update is made. In addition, the user can check the images generated by the machine learning model and make corrections or selections, improving the accuracy of matching.

[0015] FIG. 1 is a schematic diagram showing a configuration of a live distribution system 1 according to an embodiment of the present disclosure. The live distribution system 1 provides an interactive live distribution service in which a distributor (also called a live streamer) LV and a viewer (also called an audience) AU (AU1, AU2, ...) can communicate in real time. As shown in FIG. 1, the live distribution system 1 includes a server 10, a user terminal 20 on the distributor side, and a user terminal 30 (30a, 30b, ...) on the viewer side. In addition to distributors who distribute live distribution and viewers who watch live distribution, there are also users who log in to the live distribution platform but do not distribute or watch. Such users are called active users. The distributors, viewers, and active users may be collectively called users. The server 10 may be configured by one or more information processing devices connected to a network NW. The user terminals 20 and 30 may be, for example, mobile terminals such as smartphones, tablet terminals, laptop PCs, recorders, portable game consoles, and wearable devices, or may be stationary devices such as desktop PCs. The server 10, the user terminal 20, and the user terminal 30 are connected to each other so as to be able to communicate with each other via various wired or wireless networks NW.

[0016] The live distribution system 1 involves a distributor LV, a viewer AU, and an administrator (not shown) who manages the server 10. The distributor LV is a person who transmits content in real time by recording and filming his / her own songs, talks, performances, fortune telling, game commentary, etc. on his / her own user terminal 20 and uploading the content to the server 10 as is. The administrator provides a platform for live distribution of content on the server 10, and also mediates or manages real-time exchanges between the distributor LV and the viewer AU. The viewer AU accesses the platform with the user terminal 30, selects desired content, and views it. During the live distribution of this content, the viewer AU performs operations to make comments, cheer, or request fortune telling through the user terminal 30, and the distributor LV providing the content responds to such comments, cheers, or requests, and the reactions are transmitted to the viewer AU by video and / or audio, thereby establishing two-way communication.

[0017] In this specification, "live distribution" may mean a data transmission mode that realizes a state in which content recorded on a user terminal 20 of a distributor LV is played back and made viewable on a user terminal 30 of a viewer AU in substantially real time, or may mean distribution itself realized by such a transmission mode. Live distribution may be realized using existing live distribution technologies such as HTTP Live Streaming, Common Media Application Format, Web Real-Time Communications, Real-Time Messaging Protocol, and MPEG DASH. Live distribution includes a transmission mode in which a viewer AU can view content with a predetermined delay while a distributor LV is recording the content. The amount of delay is at least as large as the amount of communication between the distributor LV and the viewer AU. However, live distribution is distinguished from so-called on-demand distribution in which the entire recorded data of the content is temporarily stored on a server, and the data is provided to a user from the server at any time thereafter in response to a request from the user.

[0018] In this specification, "moving image data" refers to data including image data (also called video data) generated by the imaging function of the user terminals 20 and 30, and audio data (also called audio data) generated by the audio input function of the user terminals 20 and 30. The moving image data is played back on the user terminals 20 and 30, allowing users to view the content. In this embodiment, it is assumed that processing such as compression, decompression, encoding, decoding, transcoding, and other processing is performed to change the format, size, and specifications of the data from when the moving image data is generated on the distributor's user terminal until it is played back on the viewer's user terminal. Since the contents (e.g., moving images and audio) represented by the moving image data before and after such processing are substantially unchanged, this embodiment will be described assuming that the moving image data after such processing is the same as the moving image data before such processing. In other words, when moving image data is generated on the distributor's user terminal and then played back on the viewer's user terminal via the server 10, the moving image data generated on the distributor's user terminal, the moving image data passing through the server 10, and the moving image data received and played on the viewer's user terminal are all the same moving image data.

[0019] In the example of Fig. 1, a broadcaster LV is live-streaming a talk. The user terminal 20 of the broadcaster LV records and records the image and audio of the broadcaster LV while he or she is talking, generating video data, and transmits the video data to the server 10 via the network NW. The user terminal 20 also displays the recorded video image VD of the broadcaster LV on the display of the user terminal 20, thereby enabling the broadcast content by the broadcaster LV to be confirmed.

[0020] The user terminals 30a, 30b of the viewers AU1, AU2 who have requested the platform to watch the live broadcast of the broadcaster LV each receive video data related to the live broadcast via the network NW, and play the received video data to display the videos VD1, VD2 on the display and output audio from the speaker. The videos VD1, VD2 displayed on each user terminal 30a, 30b are substantially identical to the video VD captured by the user terminal 20 of the broadcaster LV, and the audio output from each user terminal 30a, 30b is also substantially identical to the audio recorded by the user terminal 20 of the broadcaster LV.

[0021] The recording and recording in the user terminal 20 of the distributor LV and the playback of the video data in the user terminals 30a and 30b of the viewers AU1 and AU2 are performed substantially simultaneously. When one viewer AU1 inputs a comment on the content of the talk of the distributor LV to the user terminal 30a, the server 10 displays the comment in real time on the user terminal 20 of the distributor LV and also on the user terminals 30a and 30b of each viewer AU1 and AU2. When the distributor LV reads the comment and develops a talk that overlaps the content, the video and audio of the talk are output on the user terminals 30a and 30b of each viewer AU1 and AU2, and it is recognized that a conversation between the distributor LV and the viewer AU1 has been established. In this way, the live distribution system 1 realizes live distribution that enables two-way communication, not one-way.

[0022] Fig. 2 is a block diagram showing the functions and configuration of the user terminal 20 in Fig. 1. The user terminal 30 has the same functions and configuration as the user terminal 20. Each block shown in Fig. 2 and the following block diagrams can be realized in hardware terms by elements and mechanical devices such as a computer CPU, and in software terms by a computer program, etc., but here, functional blocks realized by the cooperation of these are depicted. Therefore, those skilled in the art who have read this specification will understand that these functional blocks can be realized in various ways by combining hardware and software.

[0023] The distributor LV and the viewer AU download and install a live distribution application program (hereinafter, referred to as a live distribution app) according to this embodiment from a download site to the user terminals 20 and 30 via the network NW. Alternatively, the live distribution app may be pre-installed in the user terminals 20 and 30. When the live distribution app is executed by the user terminals 20 and 30, the user terminals 20 and 30 communicate with the server 10 via the network NW to realize various functions. Hereinafter, functions realized by the user terminals 20 and 30 (processors such as CPUs) executing the live distribution app will be described as functions of the user terminals 20 and 30. These functions are actually functions that the live distribution app causes the user terminals 20 and 30 to realize. Note that in other embodiments, these functions may be realized by a computer program written in a programming language such as HTML (HyperText Markup Language), which is transmitted from the server 10 to a web browser of the user terminals 20 and 30 via the network NW and executed by the web browser.

[0024] The user terminal 20 includes a distribution unit 100 that generates video data recording the user's image and voice and provides it to the server 10, a viewing unit 200 that acquires the video data from the server 10 and plays it, and a non-distribution processing unit 400 that processes requests from active users. The user activates the distribution unit 100 when distributing, the viewing unit 200 when viewing, and the non-distribution processing unit 400 when searching for a live distribution or distributor to view, viewing a distributor's profile, or viewing an archive. The user terminal on which the distribution unit 100 is active is the user terminal on the distributor side, that is, the user terminal on the generating side of video data, the user terminal on which the viewing unit 200 is active is the user terminal on the viewer side, that is, the user terminal on the playing side of video data, and the user terminal on which the non-distribution processing unit 400 is active is the user terminal of an active user.

[0025] The delivery unit 100 includes an imaging control unit 102, an audio control unit 104, a video transmission unit 106, a delivery-side UI control unit 108, and a delivery-side communication unit 110. The imaging control unit 102 is connected to a camera (not shown in FIG. 2) and controls imaging by the camera. The imaging control unit 102 acquires image data from the camera. The audio control unit 104 is connected to a microphone (not shown in FIG. 2) and controls audio input by the microphone. The audio control unit 104 acquires audio data from the microphone. The video transmission unit 106 transmits video data including the image data acquired by the imaging control unit 102 and the audio data acquired by the audio control unit 104 to the server 10 via the network NW. The video transmission unit 106 transmits the video data in real time. That is, the generation of video data by the imaging control unit 102 and the audio control unit 104 and the transmission of the generated video data by the video transmission unit 106 are substantially performed simultaneously.

[0026] The distribution side UI control unit 108 controls the UI for the distributor. The distribution side UI control unit 108 is connected to a display not shown in FIG. 2, and displays a moving image on the display by playing back the moving image data to be transmitted by the moving image transmission unit 106. The distribution side UI control unit 108 is connected to input means such as a touch panel, a keyboard, and a display not shown in FIG. 2, and acquires input from the distributor via the input means. The distribution side UI control unit 108 superimposes a predetermined frame image on the moving image. The frame image includes various user interface objects (hereinafter simply referred to as objects) for receiving input from the distributor, comments input by the viewer, and information acquired from the server 10. The distribution side UI control unit 108 accepts, for example, a tap input on an object by the distributor.

[0027] The delivery-side communication unit 110 controls communication with the server 10 during live delivery. The delivery-side communication unit 110 transmits the contents of the input by the distributor acquired by the delivery-side UI control unit 108 to the server 10 via the network NW. The delivery-side communication unit 110 receives various types of information associated with the live delivery from the server 10 via the network NW.

[0028] The viewing unit 200 includes a viewing side UI control unit 202 and a viewing side communication unit 204. The viewing side communication unit 204 controls communication with the server 10 during live distribution. The viewing side communication unit 204 receives video data related to the live distribution in which the distributor and the viewers participate from the server 10 via the network NW.

[0029] The viewer-side UI control unit 202 controls the UI for the viewer. The viewer-side UI control unit 202 is connected to a display and a speaker (not shown in FIG. 2), and plays the received video data to display a video image on the display and output audio from the speaker. The output of an image on the display and audio from the speaker can be collectively referred to as "video data being played". The viewer-side UI control unit 202 is connected to input means (not shown in FIG. 2), such as a touch panel, a keyboard, or a display, and acquires input by the viewer through the input means. The viewer-side UI control unit 202 superimposes a predetermined frame image on the image of the video data acquired from the server 10. The frame image includes various objects for receiving input from the viewer, comments input by the viewer, and information acquired from the server 10. The viewer-side communication unit 204 transmits the contents of the input by the viewer acquired by the viewer-side UI control unit 202 to the server 10 via the network NW.

[0030] The non-broadcast processing unit 400 includes a non-broadcast UI control unit 402 and a non-broadcast communication unit 404. The non-broadcast UI control unit 402 controls a UI for active users. For example, the non-broadcast UI control unit 402 generates a live broadcast selection screen that displays a list of live broadcasts that can be currently participated in and accepts the selection of a live broadcast by an active user, and displays it on the display. The non-broadcast UI control unit 402 generates a profile screen of an arbitrary user and displays it on the display. The non-broadcast UI control unit 402 generates a search condition reception screen that accepts input of search conditions for searching live broadcasts and broadcasters, such as search keywords and search attributes, and displays it on the display. The non-broadcast UI control unit 402 generates a generated image display screen including at least one generated image generated by inputting the search conditions input via the search condition reception screen into a machine learning model, and displays it on the display. The non-broadcast UI control unit 402 accepts the selection of at least one generated image from the active user on the generated image display screen. The non-broadcast UI control unit 402 generates an image search result display screen including the result of the image search corresponding to the result of the active user's selection on the generated image display screen, and displays it on the display. The non-broadcast UI control unit 402 plays back archives generated by recording and filming past live broadcasts.

[0031] The non-broadcast communication unit 404 controls communication with the server 10 outside of live broadcasting. The non-broadcast communication unit 404 receives information for generating a live broadcast selection screen, generated images and their selection results, broadcaster / live broadcast search results, information for generating a profile screen, and archive data from the server 10 via the network NW. The non-broadcast communication unit 404 transmits the contents of input by active users to the server 10 via the network NW.

[0032] Fig. 3 is a block diagram showing the functions and configuration of the server 10 in Fig. 1. The server 10 includes a distribution information providing unit 302, a relay unit 304, a gift processing unit 308, a payment processing unit 310, a stream DB 314, a user DB 318, a gift DB 320, and a distributor search unit 322.

[0033] Fig. 4 is a data structure diagram showing an example of the stream DB 314 in Fig. 3. The stream DB 314 holds information on a live stream currently being performed and information on live streams (archives) performed in the past. In particular, the stream DB 314 holds images generated from videos of the live stream up to that point during the live stream. The stream DB 314 holds, in association with each other, a stream ID that identifies a live stream on a live streaming platform provided by the live streaming system 1, a broadcaster ID that is a user ID that identifies a broadcaster of the live stream, a viewer ID that is a user ID that identifies a viewer of the live stream, and an extracted image that is an image extracted from videos of the live stream up to the present.

[0034] In the live streaming platform provided by the live streaming system 1 according to the present embodiment, when a user performs live streaming, the user becomes a streamer, and when the same user watches a live stream by another user, the user becomes a viewer. Therefore, the distinction between streamer and viewer is not fixed, and a user ID registered as a streamer ID at one time may be registered as a viewer ID at another time.

[0035] The extracted image is an image that represents or well represents the live streaming currently being performed, and is generated or extracted from the video from the start of the live streaming to the present. For example, the server 10 may identify and extract an image in which the broadcaster appears from the video from the start of the live streaming to the present, and register the extracted image in the stream DB 314 in association with the live streaming. Alternatively, the server 10 may monitor the degree of excitement of the live streaming, and obtain an image when the degree exceeds a predetermined threshold. Extraction of an image from the video of the live streaming may be realized, for example, using the technology described in Patent Document 1.

[0036] FIG. 5 is a data structure diagram showing an example of the user DB318 in FIG. 3. The user DB318 holds information about a user. In particular, the user DB318 holds an image related to a distributor of a video distributed on a live distribution platform. The user DB318 holds a user ID that identifies a user, points owned by the user, a reward given to the user, attributes of the user, a profile image that is an image displayed on the profile screen of the user, and an avatar image of the user in association with each other. The attributes of the user include the age range of the user, the sex of the user, the hair color of the user, a region indicating the area where the user uses a live distribution app, and information indicating whether the user is a virtual distributor or a real distributor (hereinafter, referred to as a V real flag). A virtual distributor refers to a distributor who distributes not in the form of the distributor himself / herself but in the form of a character or object drawn two-dimensionally or three-dimensionally (for example, see Non-Patent Document 1). A profile image is registered by a user uploading a desired image to a live distribution platform. The avatar image may be generated by a live streaming app, or may be generated by a user using an image of their choice.

[0037] Points are electronic value circulated within the live streaming platform. Users purchase points by credit card or other payment methods. Rewards are electronic value defined within the live streaming platform and are an indicator for determining the amount of money that a broadcaster receives from the administrator of the live streaming platform. On the live streaming platform, when a viewer gives a gift to a broadcaster during or outside of a live streaming, the viewer's points are consumed and the broadcaster's reward increases accordingly.

[0038] Fig. 6 is a data structure diagram showing an example of the gift DB 320 in Fig. 3. The gift DB 320 holds information about gifts that can be used by viewers in live distribution. A gift is electronic data that has the following characteristics. -Can be purchased with points or money, or given free of charge. Something that viewers can give to the streamer. Giving a gift to a streamer is also called using a gift or throwing a gift. In some cases, the purchase and use of the gift occur simultaneously as a set, while in other cases the viewer can use the gift at any time after purchasing it. When a viewer gives a gift to a streamer, the streamer will receive a corresponding reward. When a gift is used, an effect associated with the gift may occur. For example, an effect corresponding to the gift may appear on the live streaming room screen.

[0039] The gift DB320 stores a gift ID that identifies a gift, a given reward that is given to a broadcaster when the gift is given to the broadcaster, and a value point that is a value to be paid when the gift is used, in association with each other. A viewer can give a gift to a broadcaster by paying the value points of a desired gift while watching a live broadcast. The payment of the value points may be made by an appropriate electronic payment means, for example, the viewer may pay the value points to the administrator. Alternatively, payment may be made by bank transfer or credit card. The relationship between the given reward and the value points can be arbitrarily set by the administrator. For example, the given reward may be set to the value points. Alternatively, the value points may be set to points obtained by multiplying the given reward by a predetermined coefficient such as 1.2, or the value points may be set to points obtained by adding a predetermined commission point to the given reward.

[0040] 3, when the distributor search unit 322 receives a search request for a distributor from the user terminal of an active user, the distributor search unit 322 executes a search for the distributor and returns the search result to the user terminal. The distributor search unit 322 includes a search request receiving unit 330, a generated image acquisition unit 332, an image generation model 338, an adjustment unit 340, an image search unit 334, and a provision unit 336.

[0041] The search request receiving unit 330 receives a search request from a user terminal of an active user of the live distribution platform via the network NW. The search request includes the user ID of the active user who made the request, a keyword to be searched for input by the active user (input in text format; hereinafter, referred to as a search keyword), and attributes of a distributor specified by the active user. The search keyword and attributes reflect the characteristics of a distributor that the active user who made the search request wants to find. The attributes specified here are the same as the attributes held in the user DB 318 described with reference to FIG. 5.

[0042] The image generation model 338 is a machine learning model for image generation that receives search keywords and attributes as input and outputs images corresponding to them. The image generation model 338 may be realized by a known image generation AI technology such as that described in Non-Patent Document 2. The image generation model 338 generates a plurality of different images (hereinafter, referred to as generated images) that correspond to the input search keywords and match the input attributes.

[0043] The generated image acquisition unit 332 inputs the search keywords and attributes included in the search request accepted by the search request acceptance unit 330 into the image generation model 338, and acquires multiple generated images generated by the image generation model 338 corresponding to the search keywords and attributes.

[0044] The adjustment unit 340 adjusts the multiple generated images acquired by the generated image acquisition unit 332 to specify or generate an image to be used as a search key in an image-image search executed by the image search unit 334 at a subsequent stage. The adjustment unit 340 presents multiple generated images to the active user who has made the search request, and allows the active user to select at least one generated image from among them. When one generated image is selected, that generated image becomes the search key. When multiple generated images are selected, a single search result may be obtained by synthesizing the results of an image-image search in which each of the selected generated images is used as a search key across the selected generated images, or a single new image may be generated by synthesizing the multiple selected generated images, and used as the search key.

[0045] The adjustment unit 340 may accept a change in the search conditions. In this case, a new generated image is generated under the changed search conditions and presented to the active user. The adjustment unit 340 may accept a refinement condition. In this case, among the multiple generated images acquired by the generated image acquisition unit 332, images that match the refinement condition are extracted and presented to the active user.

[0046] The adjustment unit 340 accepts the selection of at least one generated image from the multiple generated images by the active user who has made the search request. Specifically, the adjustment unit 340 generates a selection request including the multiple generated images acquired by the generated image acquisition unit 332, and transmits the selection request to the user terminal who has made the search request via the network NW. The user terminal generates a generated image display screen based on the received selection request, and displays the generated image on the display. The user terminal accepts the selection of at least one generated image from the multiple generated images displayed on the generated image display screen by the active user. The user terminal generates a selection response including information identifying the selected at least one generated image, and transmits the selection response to the adjustment unit 340 via the network NW. The adjustment unit 340 accepts the selection of at least one generated image from the multiple generated images by the active user based on the received selection response.

[0047] The image search unit 334 executes an image search on the stream DB 314 and the user DB 318 based on at least one selected generated image. When one generated image is selected, the image search unit 334 executes an image-image search on the extracted image in the stream DB 314 and the profile image and avatar image in the user DB 318 using the generated image as a search key. The result of the image-image search in the image search unit 334 includes a set of an extracted image / profile image / avatar image with a high matching score, a corresponding broadcaster ID, and a stream ID that specifies a live broadcast performed by the broadcaster specified by the broadcaster ID. The image-image search in the image search unit 334 may be realized using, for example, a search technique described in Patent Document 2.

[0048] When multiple generated images are selected, the image search unit 334 executes an image-image search using each generated image as a search key. The image search unit 334 generates a final search result by synthesizing the obtained search results across the multiple selected generated images. The synthesis of the search results may be realized by known techniques such as addition, averaging, and weighting.

[0049] The providing unit 336 provides the results of the image search executed by the image search unit 334 to the user terminal that originated the search request via the network NW.

[0050] The non-broadcast UI control unit 402 of the user terminal that originated the search request generates an image search result display screen based on the received image search results, and displays it on the display of the user terminal. When the non-broadcast UI control unit 402 accepts the selection of live streaming by the active user on the image search result display screen, it generates a distribution request including the stream ID of the selected live streaming, and transmits it to the server 10 via the network NW. The distribution information providing unit 302 starts providing the live streaming identified by the stream ID included in the received distribution request to the user terminal that originated the request. The distribution information providing unit 302 updates the stream DB 314 so that the viewer ID of the stream ID includes the user ID of the active user of the user terminal that originated the request. As a result, the active user becomes a viewer of the selected live streaming.

[0051] The relay unit 304 relays the transmission of video data from the user terminal 20 of the distributor to the user terminal 30 of the viewer in the live distribution started by the distribution information providing unit 302. The relay unit 304 receives a signal indicating a user input by the viewer during the live distribution, i.e., during the playback of the video data, from the viewing side communication unit 204. The signal indicating the user input may be an object designation signal indicating the designation of an object displayed on the display of the user terminal 30, and the object designation signal includes the viewer ID of the viewer, the distributor ID of the distributor performing the live distribution that the viewer is viewing, and an object ID that identifies the object. If the object is a gift icon, the object ID is a gift ID. In that case, the object designation signal is a gift use signal indicating the use of a gift by the viewer for the distributor. Similarly, the relay unit 304 receives a signal indicating a user input by the distributor during the playback of the video data, for example, an object designation signal, from the distribution side communication unit 110 of the distribution unit 100 of the user terminal 20.

[0052] The gift processing unit 308 updates the user DB 318 so as to increase the distributor's reward according to the reward for the gift specified by the gift ID included in the gift use signal. The gift processing unit 308 refers to the gift DB 320 and specifies the reward corresponding to the gift ID included in the received gift use signal. The gift processing unit 308 updates the user DB 318 so as to add the specified reward to the reward corresponding to the distributor ID included in the gift use signal.

[0053] In response to receiving the gift use signal, the payment processing unit 310 processes the payment of the value of the gift by the viewer. The payment processing unit 310 refers to the gift DB 320 and identifies the value points of the gift identified by the gift ID included in the gift use signal. The payment processing unit 310 updates the user DB 318 so as to subtract the identified value points from the points of the viewer identified by the viewer ID included in the gift use signal.

[0054] The operation of the live distribution system 1 configured as above will now be described. FIG. 7 is a flowchart showing a series of processes related to a distributor search in the server 10. The server 10 receives a search request from a user terminal via the network NW (S202). The server 10 inputs a search keyword and a desired attribute included in the received search request to the image generation model 338 (S204). The server 10 transmits a selection request including a generated image generated by the image generation model 338 to the requesting user terminal via the network NW (S206). The server 10 receives a selection response including information on the generated image selected by the user from the requesting user terminal (S208). The server 10 executes an image search for images held in the user DB 318 and images held in the stream DB 314 using the selected generated image as a search key (S210). The server 10 transmits information on the distributor obtained as a result of the search and information on the live distribution and archives performed by the distributor to the requesting user terminal (S212).

[0055] 8 is a representative screen diagram of a search condition reception screen 600 displayed on the display of the user terminal of an active user. The search condition reception screen 600 receives search conditions input by the active user. The search condition reception screen 600 has a keyword input area 602 that receives search keywords input by the active user in a free text input format, an attribute designation area 604 that receives attribute designation by the active user in a pull-down format, and a search button 606.

[0056] The active user inputs a desired search keyword in the keyword input area 602, selects a desired attribute in the attribute designation area 604, and presses the search button 606. When the non-broadcast communication unit 404 of the user terminal detects the pressing of the search button 606, it generates a search request including the search keyword input in the keyword input area 602, the attribute selected in the attribute designation area 604, and the active user's user ID, and transmits the search request to the server 10 via the network NW.

[0057] Fig. 9 is a representative screen diagram of a generated image display screen 608 displayed on the display of the user terminal of an active user. The generated image display screen 608 includes a plurality of generated images generated by inputting the search conditions (search keywords and attributes) input via the search condition reception screen 600 of Fig. 8 to the image generation model 338. The generated image display screen 608 has a generated image display area 610 that displays a plurality of generated images 612 included in a selection request received from the server 10 in response to a search request, a search condition addition button 614, and an image search execution button 616.

[0058] When an active user taps the search condition addition button 614, the non-broadcast UI control unit 402 provides an interface for adding or updating search conditions and redoing the generation of a generated image. The user terminal accepts an update request for the generated image via the interface. Alternatively, the non-broadcast UI control unit 402 provides an interface for adding search conditions and narrowing down search results.

[0059] The active user selects or specifies a generated image that is closest to the desired image of the distributor from among the multiple generated images displayed in the generated image display area 610, and presses the image search execution button 616. When the non-distribution UI control unit 402 detects the pressing of the image search execution button 616, it accepts the selection or specification of at least one generated image from the multiple generated images on the generated image display screen 610. The non-distribution communication unit 404 generates a selection response including information identifying the generated image selected or specified in the generated image display area 610, and transmits it to the server 10 via the network NW.

[0060] 10 is a representative screen diagram of an image search result display screen 618 displayed on the display of the user terminal of an active user. The image search result display screen 618 includes the result of an image search corresponding to the result of the selection of a generated image on the generated image display screen 608 of FIG. 9. The non-distribution communication unit 404 receives the result of an image search corresponding to the selection response transmitted via the generated image display screen 608 of FIG. 9 from the server 10. The non-distribution UI control unit 402 generates the image search result display screen 618 based on the received image search result and displays it on the display. The image search result display screen 618 includes a distributor ID 620 included in the result of the image search, a profile image 622 of the distributor identified by the distributor ID, a thumbnail 624 representing a live broadcast currently being performed by the distributor identified by the distributor ID, and a thumbnail 626 representing an archive of past live broadcasts by the distributor.

[0061] In the above-mentioned embodiment, examples of the storage unit are a hard disk and a semiconductor memory. Furthermore, based on the description in this specification, it will be understood by those skilled in the art who have read this specification that each unit can be realized by a CPU (not shown), an installed application program module, a system program module, a semiconductor memory that temporarily stores the contents of data read from a hard disk, or the like.

[0062] According to the live distribution system 1 of the present embodiment, by utilizing image generation from text, text-to-image search for distributors is possible without tagging images. This reduces the burden on users and administrators. In addition, search accuracy can be maintained or improved even when images are frequently updated, such as extracted images from live distribution.

[0063] Furthermore, in the live distribution system 1 according to the present embodiment, a plurality of different generated images generated from the search conditions are presented to the user as candidates. The user can select a generated image that is closest to what he or she is looking for from among the candidates. This can improve the accuracy of the distributor search. For example, it is possible to absorb differences in the meaning of text due to cultural differences. For example, the meaning of the text "cute" differs between Japan and other countries. By presenting the user with a generated image that reflects the meaning of "cute" in Japan and a generated image that reflects the meaning of "cute" in other countries, it is possible to suppress or prevent discrepancies in search results due to such discrepancies in meaning.

[0064] In conventional searches based on image tags, if there is no match between the search keyword and the tag, there is often no search result. In contrast, in the live distribution system 1 according to the present embodiment, a machine learning model generates a generated image based on the search criteria, and an image search is performed using the generated image as a search key. Therefore, it is possible to return some search results for any search criteria. This makes it possible to suppress a decrease in user satisfaction due to a lack of search results.

[0065] Furthermore, in the live distribution system 1 according to the present embodiment, the generated images and the selection results thereof can be acquired and utilized as data. For example, such data can be utilized when analyzing popular faces.

[0066] The hardware configuration of an information processing device according to the present embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing an example of the hardware configuration of an information processing device according to the present embodiment. The illustrated information processing device 900 can realize, for example, each of the server 10 and the user terminals 20 and 30 according to the present embodiment.

[0067] The information processing device 900 includes a CPU 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 925, and a communication device 929. The information processing device 900 further includes an imaging device (not shown) such as a camera. The information processing device 900 may also have a processing circuit such as a DSP (Digital Signal Processor) or an ASIC (Application Specific Integrated Circuit) instead of or in addition to the CPU 901.

[0068] The CPU 901 functions as an arithmetic processing device and a control device, and controls the overall operation or a part of the operation in the information processing device 900 according to various programs recorded in the ROM 902, the RAM 903, the storage device 919, or the removable recording medium 923. For example, the CPU 901 controls the overall operation of each functional unit included in the server 10 and the user terminals 20 and 30 in this embodiment. The ROM 902 stores programs and arithmetic parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change appropriately during the execution. The CPU 901, the ROM 902, and the RAM 903 are mutually connected by a host bus 907 constituted by an internal bus such as a CPU bus. Furthermore, the host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 909.

[0069] The input device 915 may be, for example, a device operated by a user, such as a mouse, a keyboard, a touch panel, a button, a switch, or a lever, or may be a device that converts a physical quantity into an electric signal, such as a sound sensor such as a microphone, an acceleration sensor, a tilt sensor, an infrared sensor, a depth sensor, a temperature sensor, or a humidity sensor. The input device 915 may be, for example, a remote control device that uses infrared or other radio waves, or an external connection device 927 such as a mobile phone that supports the operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user or a sensed physical quantity and outputs the input signal to the CPU 901. The user operates the input device 915 to input various data to the information processing device 900 or to instruct a processing operation.

[0070] The output device 917 is configured with a device capable of visually or audibly notifying the user of acquired information. The output device 917 can be, for example, a display such as an LCD, a PDP, or an OELD, an audio output device such as a speaker or a headphone, or a printer device. The output device 917 outputs the results obtained by the processing of the information processing device 900 as a video such as text or an image, or as a sound such as audio.

[0071] The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 900. The storage device 919 is configured, for example, with a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 919 stores programs and various data executed by the CPU 901, and various data acquired from the outside.

[0072] The drive 921 is a reader / writer for a removable recording medium 923 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and is built into or externally attached to the information processing device 900. The drive 921 reads out information recorded on the attached removable recording medium 923 and outputs the information to the RAM 903. The drive 921 also writes information to the attached removable recording medium 923.

[0073] The connection port 925 is a port for directly connecting a device to the information processing device 900. The connection port 925 can be, for example, a Universal Serial Bus (USB) port, an IEEE 1394 port, a Small Computer System Interface (SCSI) port, or the like. The connection port 925 can also be an RS-232C port, an optical audio terminal, a High-Definition Multimedia Interface (HDMI) (registered trademark) port, or the like. By connecting an external connection device 927 to the connection port 925, various types of data can be exchanged between the information processing device 900 and the external connection device 927.

[0074] The communication device 929 is, for example, a communication interface configured with a communication device for connecting to a network NW. The communication device 929 can be, for example, a communication card for a wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication device 929 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communications. The communication device 929 transmits and receives signals, for example, between the Internet and other communication devices using a predetermined protocol such as TCP / IP. The communication network NW connected to the communication device 929 is a network connected by wire or wirelessly, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication. The communication device 929 realizes a function as a communication unit.

[0075] An imaging device (not shown) such as a camera is a device that captures real space and generates a captured image using an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) and various members such as a lens for controlling the formation of a subject image on the imaging element. The imaging device may capture a still image or a moving image.

[0076] The above describes the configuration and operation of the live streaming system 1 according to the embodiment. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each process, and that such modifications are also within the scope of the present disclosure.

[0077] Although the embodiment has been described using a live streaming platform as an example, the present invention is not limited to this and the technical ideas of the embodiment can also be applied to other video streaming platforms such as VOD (Video On Demand).

[0078] In the embodiment, a case where a distributor is searched for has been described, but the present invention is not limited to this. For example, the technical idea according to the present embodiment can also be applied to a case where live distribution or archives are searched for.

[0079] In the embodiment, a case has been described in which a plurality of generated images are presented to the user to select one, but this is not limiting. For example, one generated image may be presented to the user and the user may be asked whether to adopt it or change the search conditions. Alternatively, after a generated image is generated, an image search may be performed using the generated image as a search key without asking the user.

[0080] In the embodiment, a case has been described in which the attributes designated by the user as search conditions are input to the image generation model 338, but this is not limiting. For example, instead of or in addition to being input to the image generation model 338, the attributes designated by the user as search conditions may be used to narrow down the streamers in advance in the user DB 318. For example, when a virtual streamer is designated as a search condition, the server may extract images of streamers whose V real flags are set to virtual from among images held in the user DB 318 in advance, and use these images as a population for image search.

[0081] In the embodiment, a case where a search request includes a search keyword and an attribute has been described, but the present invention is not limited to this, and a search request may include either a search keyword or an attribute. Alternatively, a search request may include an image, a voice, or profile information or a browsing history of a user who has made a request.

[0082] In the embodiment, a case has been described in which a user terminal of an active user generates a search request, but the present invention is not limited to this, and the technical idea according to the present embodiment can also be applied to, for example, a case in which a broadcaster is recommended to a user. In this case, the search request may include the user's profile information and browsing history. The browsing history may include a profile image of a broadcaster whose live broadcast the user has previously viewed. In this case, a generated image generated by an image generation model using such a profile image as an input is an image that the user is likely to like.

[0083] The conversion rates from gift points to given rewards in the embodiments are merely examples, and may be set appropriately by, for example, an administrator of the live distribution system.

[0084] The technical ideas according to the embodiments may be applied to live commerce. For example, an image of a product that the user wants may be generated from text entered by the user, and images of products handled in live commerce may be searched for using the generated image as a key, or images extracted from live streaming videos of live commerce may be searched for. For example, when a user enters "green clothes", the system generates multiple generated images of green clothes and presents them to the user. When the user selects a generated image that matches his / her taste from among them, the system executes an image search using the green clothes included in the selected generated image as a search key. As a result, product images of clothes similar in color and shape to the clothes in the search key are obtained as search results, or live commerce broadcasts of broadcasters wearing clothes similar to the clothes in the search key are obtained as search results.

[0085] In the processing procedures described in this specification, particularly in the processing procedures described using flow diagrams and flowcharts, it is possible to omit some of the steps that make up the processing procedures, to add steps that are not explicitly stated as steps that make up the processing procedures, and / or to rearrange the order of the steps, and processing procedures in which such omissions, additions, or changes in order have been made are included within the scope of this disclosure as long as they do not deviate from the spirit of this disclosure.

[0086] At least a part of the functions realized by the server 10 may be realized by a device other than the server 10, for example, the user terminals 20 and 30. At least a part of the functions realized by the user terminals 20 and 30 may be realized by a device other than the user terminals 20 and 30, for example, the server 10. For example, the superimposition of a predetermined frame image onto an image of video data performed at a viewer's user terminal may be performed at the server 10 or at a distributor's user terminal.

Claims

1. An image storage unit that stores images related to videos distributed on a video distribution platform; a request receiving unit that receives a search request from a user terminal of a user of the video distribution platform via a network; a generated image acquisition unit that acquires a generated image generated by the machine learning model in response to the information input by the user and included in the search request, by using the information as an input to the machine learning model; a search unit that executes an image search in the image storage unit based on the generated image; a providing unit that provides the result of the image search executed by the searching unit to the user terminal via the network.

2. the generated image acquisition unit acquires a plurality of generated images generated by the machine learning model in response to the information input by the user included in the search request; the server further includes a selection receiving unit configured to receive a selection of at least one generated image from the plurality of generated images by the user; The server according to claim 1 , wherein the search unit executes an image search in the image storage unit based on the selected at least one generated image.

3. the information entered by the user in the search request includes text and attributes; The server of claim 1 , wherein the machine learning model generates a generated image that corresponds to the text and matches the attributes.

4. The video distributed on the video distribution platform is distributed in real time by a distributor; The server according to claim 1 , wherein the image storage unit stores images generated from the video during distribution of the video.

5. A function for displaying a search condition reception screen for receiving input of search conditions on a display; a function of displaying on the display a generated image display screen including a plurality of generated images generated by inputting the search conditions input via the search condition receiving screen into a machine learning model; and a function of accepting, on the generated image display screen, a selection of at least one generated image from the plurality of generated images; A computer program for causing the terminal to realize the function of displaying, on the display, an image search result display screen including image search results corresponding to the result of the selection.

Images