Information processing device and information processing method

Abstract

We provide technology to generate videos summarizing the experiences of users who attended events in person. [Solution] An information processing device according to one embodiment includes an acquisition unit that acquires material image data, including images taken by participants who took part in an event held at a venue, and an output unit that outputs a video summarizing the participants' experiences at the venue, generated using the material image data. The acquisition unit acquires experience information indicating the participants' experiences at the venue, and the information processing device also includes a generation unit that automatically generates the video based on the experiences.

Description

Technical Field

[0001] The present invention relates to a technology for automatically generating a video that summarizes a user's experience.

Background Art

[0002] Techniques for generating videos related to events are known. For example, Patent Document 1 discloses an invention that automatically edits a video for personal viewing based on material video images including user tagging and screens according to the operation perspective of players for a game event in which players participate.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The invention described in Patent Document 1 was merely a video editing technique using captured video images such as operation screens provided by a game system as video materials.

[0005] On the other hand, the present invention provides a technology for generating a video that summarizes the experiences of users who participated in person at an event.

Means for Solving the Problems

[0006] An information processing apparatus according to an aspect of the present disclosure includes an acquisition unit that acquires material image data including images captured by participants who participated at a venue in an event held at the venue, and an output unit that outputs a video that summarizes the experiences of the participants at the venue, generated using the material image data.

[0007] An information processing method according to another aspect of the present disclosure includes the steps of: a computer acquiring material image data including images taken by a participant who attended an event held at a venue; and outputting a video summarizing the participant's experience at the venue, generated using the material image data. [Effects of the Invention]

[0008] According to the present invention, it is possible to generate a video summarizing the experiences of users who participated in an event in person. [Brief explanation of the drawing]

[0009] [Figure 1] A diagram illustrating the system configuration of an information processing system 1 according to one embodiment. [Figure 2] A diagram illustrating the functional configuration of information processing system 1. [Figure 3] A diagram illustrating the hardware configuration of the information processing device 10. [Figure 4] A sequence chart illustrating a method for generating experiential videos in Information Processing System 1. [Figure 5] A diagram illustrating application database 1000. [Figure 6] A diagram illustrating the Sub-Event Database 2000. [Figure 7] A diagram illustrating an example video of the experience. [Modes for carrying out the invention]

[0010] 1. Structure Figure 1 is a diagram illustrating the system configuration of an information processing system 1 according to one embodiment. In this example, the information processing system 1 (or simply the system) is a system for generating videos (hereinafter referred to as "experience videos") corresponding to the on-site experiences of participants who attended an event venue, based on video materials (examples of source image data) such as images taken by the participants. In this example, an event is an event held at a specific venue with participants such as spectators (hereinafter referred to as "users"), and includes, for example, sports matches, music concerts, and car races. An experience video is a video that summarizes the personal experiences of users who participated in the event, generated by combining various source image data. In this example, source image data includes, for example, photos or videos taken by the user with their own device and image data provided by a service provider operating a service related to the event (hereinafter referred to as the "official service provider").

[0011] Recently, it has become common for users attending events to take photos and videos using their smartphones while watching the event. Users can look back on the event later using the images they have taken, and may enjoy reminiscing about their event experience. However, when users look back at photos taken with their devices, they may only have fragmented images or a collection of similar photos, making it difficult to recall the unique excitement they felt at the event venue on the day. For example, regarding the performance of athletes or the outcome of a match, users are bound to have various impressions depending on the situation they witnessed (or they are bound to remember the impressions they felt at the time as an experience). However, with only fragmented photos or videos, it becomes difficult to recall the impressions of that moment, or the emotions may be diminished.

[0012] Conventional related technologies may involve users not only attending events in person but also utilizing services that distribute event-related video content (e.g., live broadcasts or digest videos). These services primarily distribute video content targeting a large audience based on official footage captured by cameras installed at the event venue. Therefore, even if event participants watch such video content, they may not necessarily be able to relive their own experiences and emotions. Furthermore, these services do not provide videos for personal viewing that focus on the experiences of event participants, nor do they utilize footage shot by the participants themselves. Therefore, this invention provides the following system to generate videos summarizing the experiences of users who attended events in person (i.e., original memory videos).

[0013] Information processing system 1 includes an information processing device 10, a user terminal 20, an application server 30, and an event terminal 40. In this example, each component of the system is connected to one another via a network 9. In this example, network 9 is a computer network such as the Internet or a mobile network.

[0014] The information processing device 10 is an information processing device or server device in the information processing system 1. In this example, the information processing device 10 generates an experience video in response to a request from a user who participated in the event in person (hereinafter referred to as the "target user"). In this example, the target user is a person who actually went to the venue where the event was being held and watched the event (for example, user U1), and who requests the generation of an experience video through the service provided by the application server 30. The information processing device 10 works in cooperation with the user terminal 20 and the application server 30 to acquire source image data. The information processing device 10 generates an experience video using the source image data and outputs it to the user terminal 20 via the application server 30.

[0015] The user terminal 20 is a terminal owned and used by a target user utilizing the information processing system 1. The user terminal 20 includes, for example, a smartphone, tablet, or personal computer. In this example, the user terminal 20 receives instructions from the target user to generate an experience video via an application installed on its terminal. The user terminal 20 uploads source image data recorded in the terminal's storage (so-called camera roll) to the information processing device 10. This allows the target user to incorporate photos or videos taken at the event site into the experience video. Furthermore, the user terminal 20 can receive processing requests from the target user regarding the experience video, such as video playback or editing operations. The user terminal 20 can also provide the information processing device 10 with various information used for generating the experience video.

[0016] The application server 30 is a server device that manages data related to applications in the information processing system 1. In this example, the application server 30 is a server that operates applications (hereinafter collectively referred to as "dedicated applications") for providing target users with services such as a service for distributing event video content or a service for generating experiential videos. The database of the application server 30 records data related to the generation of experiential videos, such as video data of the event (an example of source image data), attendance data accumulating records of attending the event, and user data about users who participated in the event. In this example, the source image data is categorized into free images and paid images among the images available to target users. In this example, free images include, for example, images distributed free of charge by video distribution companies, TV stations, sponsoring companies, or other participating companies that are recognized as official businesses related to the event. Paid images include images distributed for a fee by these official businesses. The application server 30 can output various data in response to requests from the information processing device 10.

[0017] The event terminal 40 is a terminal owned and used by an operator who operates events in the information processing system 1. The event terminal 40 includes, for example, a smartphone, a tablet, or a personal computer. In this example, the event terminal 40 records and holds, as video data, the state of an event photographed by a camera (not shown) installed at the event venue. Further, the event terminal 40 records and holds information related to various operations regarding the event, event promotion, ticket management, or operations such as merchandise sales. The event terminal 40 can provide material image data and various information related to event operations to the app server 30.

[0018] FIG. 2 is a diagram illustrating the functional configuration of the information processing system 1. In this embodiment, the information processing apparatus 10 has functional blocks (or components) called an acquisition unit 11, an output unit 12, a generation unit 13, a selection unit 14, a storage unit 191, and a control unit 192. In this example, the storage unit 191 stores various data, programs, and software including, for example, a database. In this example, the control unit 192 performs various controls.

[0019] The acquisition unit 11 acquires material image data including images photographed by users who participated in person in an event held at the venue. Further, the acquisition unit 11 acquires experience information indicating the experience of users at the venue. The acquisition unit 11 can cooperate with the app server 30 to acquire material image data and user experience information related to the event in which the user participated. Note that the experience information will be described later.

[0020] The output unit 12 outputs an experience video of the user at the venue generated using the material image data. In this example, the output unit 12 can output the experience video generated by the following generation unit 13 to the user terminal 20 via a dedicated app or the like. Thereby, the user can view or edit the experience video.

[0021] The generation unit 13 automatically generates an experience video based on the user's experience. The generation unit 13 refers to a database of events in which the user participated. This database defines multiple sub-events that describe the user's activity schedule during the event.

[0022] In this example, a sub-event refers to a predetermined division within the event's timeline, and includes various items that model the typical (or personal) schedules of users who participated in the event. Therefore, the generation unit 13 can generate an experience video using images selected from the source image data for each sub-event. Specific examples of sub-events will be described later.

[0023] The selection unit 14 works in cooperation with the generation unit 13 to select the corresponding material image data for each sub-event based on the experience information associated with the material image data. The following will explain specific examples of experience information and the functions of the corresponding components.

[0024] If the source image data includes corresponding location information, the generation unit 13 selects an image corresponding to a certain sub-event using the location information. In this example, the source image data includes various metadata such as geotags or timestamps.

[0025] The acquisition unit 11 can acquire the user's payment information. The generation unit 13 selects an image corresponding to a certain sub-event using the payment information. In this example, the payment information is information related to the payment of an event performed by the user, and includes, for example, information related to payments made for tickets, goods, food and beverages, or services.

[0026] The acquisition unit 11 can acquire motion information that indicates the physical movements of the user's body. The generation unit 13 selects an image corresponding to a certain sub-event using the motion information. In this example, the motion information includes, for example, information analyzed from source image data or information detected from sensors installed in the user terminal 20.

[0027] The acquisition unit 11 can acquire communication information (hereinafter referred to as "CC information") that indicates communication between users regarding an event from other applications. The generation unit 13 generates a video containing CC information for a certain sub-event. In this example, the CC information is information acquired from other applications such as SNS (social networking service) applications or messaging applications (hereinafter collectively referred to as "CC applications"), and includes, for example, text, messages, tags, images, or videos related to the event.

[0028] As for other functions, if the source image data includes official images provided by official event organizers (for example, companies operating video streaming services), the following implementation is envisioned. In this example, if the official images include paid images, the generation unit 13 generates a sample video that includes the paid images. Furthermore, depending on whether the user purchases this sample video, the output unit 12 can output the sample video including the paid images in an editable format. A specific example of a sample video will be described later.

[0029] Figure 3 illustrates the hardware configuration of the information processing device 10. Physically, the information processing device 10 is configured as a computer including a processor 101, memory 102, storage 103, communication device 104, input device (optional), display device (optional), and a bus connecting these. Each of these devices operates on power supplied from a battery (not shown). In the following description, the term "device" can be read as a circuit, device, unit, etc. The hardware configuration of the information processing device 10 may include one or more of the devices shown in Figure 3, or it may be configured without some of the devices. Alternatively, multiple devices with different enclosures may be connected via communication to constitute the information processing device 10.

[0030] Each function in the information processing device 10 is realized by loading predetermined software (programs) onto hardware such as the processor 101 and memory 102, which allows the processor 101 to perform calculations, control communication by the communication device 104, and control at least one of the reading and writing of data in the memory 102 and storage 103.

[0031] The processor 101 controls the entire computer, for example, by running the operating system. The processor 101 may consist of a central processing unit (CPU) that includes interfaces with peripheral devices, control units, arithmetic units, registers, etc. Alternatively, a baseband signal processing unit or a call processing unit may be implemented by the processor 101.

[0032] The processor 101 reads programs (program code), software modules, data, etc., from at least one of the storage 103 and the communication device 104 into the memory 102 and executes various processes accordingly. The program used is one that causes the computer to execute at least a part of the operations described later. The functional blocks of the information processing device 10 are stored in the memory 102 and may be realized by control programs that run on the processor 101. Various processes may be executed by one processor 101, or they may be executed simultaneously or sequentially by two or more processors 101. The processor 101 may be implemented by one or more chips. The program may also be transmitted to the information processing device 10 via a telecommunications line.

[0033] Memory 102 is a computer-readable recording medium and may consist of at least one of the following: ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. Memory 102 may also be called a register, cache, main memory, etc. Memory 102 can store executable programs (program code), software modules, etc., for carrying out the method according to this embodiment.

[0034] The storage 103 is a computer-readable recording medium and may consist of at least one of the following: an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disc, a digital multipurpose disc, a Blu-ray® disc), a smart card, flash memory (e.g., a card, a stick, a key drive), a floppy® disk, a magnetic strip, etc. The storage 103 may also be called an auxiliary storage device.

[0035] The communication device 104 is hardware (transceiver / receiver device) for communicating between computers via at least one of a wired network and a wireless network, and is also referred to as a network device, network controller, network card, communication module, etc.

[0036] Each device, such as the processor 101 and memory 102, is connected by a bus for communicating information. The bus may be configured using a single bus, or different buses may be used for each device.

[0037] The information processing device 10 may include hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array), and some or all of the functional blocks may be implemented by such hardware. For example, the processor 101 may be implemented using at least one of these hardware components.

[0038] In this example, the program stored in storage 103 includes a program (hereinafter referred to as the "server program") that causes the computer to function as a server in the information processing system 1. When the processor 101 is executing the server program, the processor 101, memory 102, storage 103, and communication device 104 are examples of functional blocks for operating the information processing device 10. The processor 101 is an example of a generation unit 13, a selection unit 14, and a control unit 192. At least one of the memory 102 and storage 103 is an example of a storage unit 191. The communication device 104 is an example of an acquisition unit 11 and an output unit 12.

[0039] Although a detailed explanation will be omitted, the application server 30 is a computer having a processor, memory, storage, and communication devices. In this example, the programs stored in the storage of the application server 30 include programs that cause the computer to function as an application server in the information processing system 1 (hereinafter referred to as "application programs").

[0040] The user terminal 20 and the event terminal 40 are computers having a processor, memory, storage, communication device, input device, and output device; specifically, they are, for example, smartphones, tablet terminals, or personal computers. In this example, the programs stored in the storage of the user terminal 20 and the event terminal 40 include programs that allow the computers to function as clients in the information processing system 1 (hereinafter referred to as "client programs"). The configuration of the information processing system 1 has now been described. Next, the operation of the information processing system 1 will be described.

[0041] 2.Operation Figure 4 is a sequence chart illustrating the method for generating an experiential video in Information Processing System 1. Here, we will explain the operation examples of each main device when a target user generates an experiential video using Information Processing System 1. Hereafter, an event in which a target user participates and which is designated as the target for generating an experiential video will be referred to as a "target event." In this example, the experiential video generation function is provided as a function of a dedicated application (an example of a client program) managed by the application server 30.

[0042] In step S101, the application server 30 acquires various data from the user terminal 20 (or event terminal 40). In this example, the application server 30 connects to various devices or terminals and acquires data used in the dedicated application as appropriate.

[0043] For example, if a dedicated app provides a function for generating experience videos as its main service, the app server 30 can acquire source image data and experience information data from the user terminal 20 and the event terminal 40, respectively. In this example, the user terminal 20 provides the app server 30 with data generated or detected while the target user is participating in the event. This data includes, for example, images taken with a camera, audio recorded with a microphone, location history from GPS (Global Positioning System), or information on vibration and acceleration acquired from various sensors. Alternatively, the user terminal 20 automatically uploads photos or videos stored in the terminal's camera roll, etc., as source image data to the app server 30 (or according to the target user's selection) based on the target user's request to generate an experience video. The event terminal 40 outputs source image data that can be provided by the official operator in the dedicated app to the app server 30. The app server 30 can record the acquired data in a predetermined database.

[0044] On the other hand, if the dedicated app provides functions related to video distribution services in addition to the experience video generation service, the app server 30 may acquire material image data from the event terminal 40 in advance. This could be, for example, the app server 30 acquiring various video data from the event terminal 40 in order to distribute live event footage, or the app server 30 itself recording the edited content. Alternatively, the app server 30 may acquire data about the target user from the user terminal 20, such as video content viewing data, personal data related to app membership registration, or data indicating the user's hobbies and preferences regarding the event.

[0045] In step S102, the application server 30 identifies the target event based on a database on the server. The following process is initiated when the user terminal 20 receives a request from the target user to generate an experience video via a dedicated application. In this example, the application server 30 receives instructions from the user terminal 20 regarding the upload of source image data and the generation of an experience video (for example, in step S101). At this point, the application server 30 can obtain basic information about the experience video from the target user (for example, instructions regarding the video length, image quality, or video structure).

[0046] Here, the application server 30 reads the database and retrieves data related to the target event. This process may also be performed when linking with other applications that manage participation in an event (for example, an application managed by the event organizing body) or other accounts (for example, an account for tickets to domestic sports events). Now, let's describe the database on the application server 30 that records data related to the dedicated application.

[0047] Figure 5 is an example of the app database 1000. The app database 1000 contains multiple records related to a user. Each record corresponds to a specific event in which a specific user (or account) participated. Each record includes a member ID, viewing information, uploaded materials, and experience information. The member ID is unique identification information for each user registered as a member of the dedicated app. Viewing information is information about the user's participation record of an event they actually went to and watched, and includes various items such as event ID, event details, date / time / location, and official materials. In this example, the event ID, event details, and date / time / location are, for example, identification information, information representing the content, or information indicating the schedule for the target event, such as domestic sports. Official materials (an example of material image data) are data related to videos or images that can be used to generate experience videos, corresponding to each event that the user actually participated in. In this example, official materials may include video content distributed in various forms, such as real-time broadcasts or recorded broadcasts, in the distribution service implemented by the app server 30, depending on the function of the dedicated app. Uploaded materials (an example of source image data) are photos or videos uploaded by the target user to the dedicated app to generate an experience video (for example, step S101).

[0048] Experience information is information corresponding to each image data set, defined to identify the experience of a target user at a target event. In this example, it is data in a format other than an image. Experience information includes at least one of several items, such as location information, payment information, motion information, and CC information. Location information is information showing the location history of the target user's user terminal 20 and is obtained from the user terminal 20. Payment information is information showing the payment made by the target user and is obtained from other service servers that cooperate with payment services, ticket reservation / sales services, or e-commerce services. Motion information is information showing the physical movement of the target user's body (e.g., body shaking or vibration) and is obtained from the user terminal 20 or the target user's wearable device. CC information is information showing the content of communication between the target user and other users and is obtained from other service servers that operate messaging apps or chat apps, for example. The information processing device 10 cooperates with these services, and the mapping of users between services is registered or defined in advance. The information processing device 10 acquires data from these other services during a predetermined time period based on the scheduled start time of the target event, and records it in the application database 1000 as experience information for the target event. This experience information is used by the information processing device 10 when selecting material image data corresponding to sub-events. Sub-events will be described later.

[0049] In this example, the uploaded material in a single record can include multiple photos and videos related to that event. For example, the uploaded material SU11 corresponding to user U1 and event E1 is data that encompasses multiple photos and videos. In practice, the uploaded material SU11 contains the individual photos and videos in a distinguishable form. The same applies to official materials. Similarly, each experience information can also include multiple units of data. For example, location information P11 is data that encompasses the history of location information of user U1 from leaving home to attend event E1 and returning home after the event. Payment information L11 is data that encompasses multiple payments related to event E1. Action information A11 is data that encompasses multiple units of actions of user U1 at event E1. CC information C11 is data that encompasses multiple communications of user U1 related to event E1.

[0050] Here, the information processing device 10 performs processing to identify and add various experience information to the source image data. This processing includes data processing related to metadata such as time information or location information (so-called metadata) contained in the source image data. For example, the information processing device 10 extracts location information and time information from the metadata of each image in the source image data. Alternatively, the information processing device 10 performs image analysis of the source image data using an image analysis AI or the like to identify experience information for each source image data. For example, the information processing device 10 can identify the location where the image was taken (an example of location information), the target user and the person's movements through person analysis (an example of motion information) from the image of the source image data, and record this information in the application database 1000.

[0051] Furthermore, the information processing device 10 may label each source image through image analysis. In this example, the labels correspond to the experience content shown in Figure 6 below, such as "Local Event Corner" or "Collaborating Artist Appearance." The labels (and timestamp set) may be recorded as experience information, or they may be recorded linked to each image in the uploaded image data. Other labels may relate to image quality, such as "Wrong Shot" or "High Noise." For example, the information processing device 10 labels an image if the proportion of subjects other than those related to the event (especially performers, merchandise, or user faces) in the overall image exceeds a threshold, or if the degree of noise related to a specific subject exceeds a threshold. By referring to these labels, the information processing device 10 can delete or correct elements when selecting candidates from the source image data.

[0052] Furthermore, the information processing device 10 may calculate various metrics from each image through image analysis. These metrics may be recorded as experience information. In this example, the information processing device 10 calculates metrics according to the type of experience information, such as the time spent at a specific location, the level of interest in purchased items, or the level of excitement that can be inferred from the user's actions, and records them in association with each other. These metrics are used when selecting materials. The information processing device 10 may also perform a process of acquiring experience information about the target user in cooperation with various services and later linking the acquired experience information to the material image data. In this case, any method may be applied to link the experience information and the material image data.

[0053] Returning to Figure 4, in step S103, the information processing device 10 receives a request from the application server 30 to generate an experience video related to the target event. In this example, the application server 30 receives a request from the user terminal 20 to generate an experience video or upload a video from the target user (for example, in step S101), and outputs the above-mentioned generation request to the information processing device 10. The information processing device 10 appropriately refers to the database of the application server 30 and obtains data used to generate the experience video. The application server 30 provides various data in response to the request of the information processing device 10.

[0054] In step S104, the information processing device 10 identifies sub-events included in the activity schedules of participants who took part in the target event. In this example, the information processing device 10 reads data related to the sub-events defined for each target event from the database. Here, we will describe the database managed by the information processing device 10.

[0055] Figure 6 illustrates the sub-event database 2000. The sub-event database 2000 contains multiple records related to events in which the target user participated. Each record contains information about sub-events defined for each event. Each record includes the event ID, sub-event, experience content, and selected materials. The event ID is unique identification information for each event, shared with the app database 1000. Here, we take the record for event E1, specified by the target user (e.g., user U1), as an example. Sub-events are items that indicate various categories of the action schedule that a typical participant (or the individual target user) is said to have performed during and around the timeline of the target event. For example, if we consider the actions of a user who participated in watching a domestic soccer match, the participant would have followed a schedule that included preparations such as purchasing tickets, traveling to the venue on the day of the event, watching the match, and then returning home. Therefore, by modeling such an action schedule, the user's experience can be organized chronologically. In this example, the experience content shows an example of events experienced by the user corresponding to each sub-event. Selected materials are, for example, image data selected according to the experience content of each sub-event. The sub-event database 2000 stores the selection results for materials uploaded by target users or provided by official service providers for each sub-event. Selected materials are managed with more detailed identification numbers, such as [SO111] or [SU111], for each data item within the official material group [SO] or the uploaded material group [SU]. The process for selecting image data is now explained.

[0056] Returning to Figure 4, in step S105, the information processing device 10 selects an image from the source image data based on the sub-event. That is, the information processing device 10 selects a corresponding image for each sub-event. Here, "selecting an image" means selecting a predetermined number of photos or a predetermined length of video from the source image data, i.e., multiple photos and multiple videos. More specifically, the information processing device 10 selects an image from the source image data of the app database 1000 that corresponds to experience information that is consistent with (or matches) the experience content defined for each sub-event. This allows the information processing device 10 to organize the experience of the target user in chronological order of events. The following describes a specific example of the process of selecting a source image corresponding to a sub-event based on the experience information contained in the source image data.

[0057] The procedure for associating source image data with sub-events basically takes into account the progress of the event itself and the schedules of the participating users, and is performed based on the history information of each source image data as follows: (1) The information processing device 10 sorts the source image data according to location information, classifying it into arbitrary categories related to location, such as home, travel, and venue. (2) Here, the information processing device 10 identifies the time periods for travel to the venue, the venue, and travel back from the history of time information (i.e., timestamps) contained in the source image data. For example, this determines the schedule of a target user, such as "13:00-13:40 travel", "13:40-16:30 venue", and "16:30-17:10 travel". Furthermore, by comparing this with the event schedule obtained from the event terminal 40, etc., the information processing device 10 confirms the time schedule for the day. This is a schedule that corresponds to sub-event items such as "10:00-13:00 Preparation", "13:00-13:40 Travel (outbound)", "13:40-14:00 Preparation for watching", "14:00-14:45 Event (first half)", "14:45-15:05 Break (halftime)", "15:05-15:55 Event (second half)", "15:55-16:30 After the event", and "16:30-17:10 Travel (return)". This allows the information processing device 10 to associate time (i.e., timestamp) with sub-events.

[0058] The information processing device 10 selects an image for each sub-event. In this example, the information processing device 10 identifies the correspondence between the sub-event and the source image data based on the experience information. The following section explains the selection method used by the information processing device 10, using specific examples of sub-events.

[0059] In the example in Figure 6, the sub-event "Preparation" corresponds to the user's experience during the preparation phase before the event. Experiences corresponding to the sub-event "Preparation" are defined in the sub-event database 2000, and one example is the experience of "Seat Selection". As a method for selecting images corresponding to this experience, it is defined that payment information is referenced. To obtain images corresponding to this experience, the information processing device 10 identifies the user's seat. For example, payment information related to the purchase of a seat ticket is used to identify the seat. The information processing device 10 obtains ticket information from the payment information in the application database 1000. The information processing device 10 selects an official photo of the stadium taken from an area near the seat purchased by the user as the image based on this experience. Alternatively, the information processing device 10 may select a photo taken by the user after they have taken their seat as the image based on this experience. In this case, the information processing device 10 determines whether the photo was taken near the user's seat based on the angle of view and direction of the photo, the position of objects and subjects in the photo, or more detailed coordinates within the venue attached to the photo data. In addition, the information processing device 10 may identify the location where a photograph or video was taken based on the degree of similarity when comparing an image taken from a camera installed at a specific location within the venue (an example of an official image) with an image taken by the user mentioned above.

[0060] In the example in Figure 6, the sub-event "Travel" corresponds to the target user's travel experience on the day of the event. Experiences corresponding to the sub-event "Travel" are defined in the sub-event database 2000, and one example is the experience of "Traveling by public transport". In this example, the information processing device 10 selects image data taken during the time the target user was traveling as an image based on this experience. Alternatively, the information processing device 10 may refer to experience information when selecting data. For example, the information processing device 10 identifies a post (including posts to SNS by the target user) corresponding to the experience "Traveling by public transport" from the CC information and selects an image corresponding to that post. An "image corresponding to a post" is an image taken during a time period based on the timestamp of that post (for example, 3 minutes before or after the post). In addition, when selecting an image, the information processing device 10 may add a string contained in the post to the selected image. For example, the information processing device 10 may generate a string of text from a social media post, such as "Successfully transferred on the XX line, but it took several times longer than usual due to a detour," and then generate an image by applying an effect (such as changing the color or making it decorative) to this string, and superimpose this image onto the source image.

[0061] Other experiences corresponding to sub-events (such as "Preparing to Watch," "Event Start," "Break Time," and "Event End" in the example in Figure 6) are also defined in the sub-event database 2000. The information processing device 10 identifies images corresponding to these experiences based on the target user's experience information. For example, for the sub-event "Preparing to Watch," the target user's experience includes "On-site Event Corner." The method for selecting images corresponding to this experience is defined as referring to labels assigned by image analysis. The information processing device 10 selects images corresponding to this experience by referring to labels obtained by image analysis. For the sub-event "Preparing to Watch," the information processing device 10 selects image data taken by the target user during this time period as images based on this experience. For the sub-event "Preparing to Watch," the information processing device 10 can select event promotional videos from official organizers or photos and videos taken by the target user during the experience as source image data, based on predefined experience information. The information processing device 10 performs similar processing and selects images for the sub-events "Event Start," "Break Time," and "Event End." The above describes various methods for the information processing device 10 to select source image data.

[0062] Return to Figure 4. In step S106, the information processing device 10 automatically generates an experience video using the selected material image data. In this example, the information processing device 10 obtains information from the database indicating the video structure defined as the format for the experience video (or based on the specification by the target user). Now, the method for generating the experience video will be explained.

[0063] Figure 7 is an example of an experiential video. Figure 7 is an example of a schematic diagram of a method for generating an experiential video. In this example, experiential video V1 is an example of a configuration model of an experiential video generated by the information processing device 10. The structure of the experiential video is broadly divided into, for example, an opening (OP in the figure), pre-event, during the event, going home, and an ending (ED in the figure), and sub-events (for example, sub-events SE1 to SE7) are assigned in advance on the time axis for each of these items. In addition, the progression of playback time (for example, about 8 to 10 minutes) is drawn in the direction shown in the experiential video V1. The video structure and settings such as playback time are predetermined according to the target event or based on the implementation of the information processing device 10. The information processing device 10 performs various editing and processing on the selected material image data according to the structure of experiential video V1. In this example, the information processing device 10 also performs processing related to the generation of text such as titles, captions, and subtitles, the use of effects, animations, and transitions, and the setting of audio, sound effects, and background music, using existing video editing technology. Using this method, the information processing device 10 generates a single experience video from multiple source image data.

[0064] Here, the information processing device 10 may generate a sample video (an example of an experience video) that includes paid images from the material image data provided by the official business operator as an option to present to the target user in the dedicated app. In this example, the information processing device 10 pre-manages the placement of paid images included in the sample video in a database or the like. The sample video is provided within the dedicated app according to the target user's purchase option or membership status (for example, whether they are a free member or a paid member). In addition, depending on whether the target user has purchased a paid sample video, the information processing device 10 may restrict at least some of the editing or viewing operations available to the target user in the dedicated app. If the target user purchases a paid sample video, the information processing device 10 unlocks the functions related to editing and viewing the sample video. Note that paid material includes images taken at close range to the performer, or footage taken in specific areas where entry by unauthorized persons is prohibited (for example, the performer's dressing room, various practice rooms, or inside a bus used for transportation). Whether the material image data is free or paid is pre-defined in a database or the like.

[0065] Return to Figure 4. In step S107, the information processing device 10 outputs the generated experience video data to the application server 30. In this example, the application server 30 stores the outputted experience video in a content folder assigned to the target user within the dedicated application. The application server 30 notifies the target user that the experience video has been generated.

[0066] In step S108, the application server 30 receives instructions from the user terminal 20 regarding the playback or editing of the experience video. This process includes predetermined operations executed within the dedicated application in response to the user's instructions. Referring again to Figure 7, display screen D1 is an example of an operation screen that presents editing options for performing the editing process of the experience video V1. In this example, the user can edit the experience video V1 by performing various operations on display screen D1 displayed on the user terminal 20. Field F1 is a field that displays the playback screen of the experience video V1. Field F1 can appropriately play a video generated by the information processing device 10 or a video that reflects the editing operations performed by the user. Field F2 is a field for the user to perform actual editing operations. In field F2, editing of videos, images (or photographs), and music can be performed, for example. In this example, in field F2, an image (material image data SO11 in the figure) corresponding to the playback position in field F1 is indicated by a playback bar, etc. The user can, for example, edit, delete, or replace this material image data SO11. Field F3 displays a list of images available as source material. In this example, the user can arbitrarily select from the images displayed in Field F3 to replace or add to the source material. Images in the source material list in Field F3 that are not available as free resources are grayed out with a key icon.

[0067] The user terminal 20 saves the edited experience video in a dedicated application according to the target user's actions. The target user can play, edit, or download the experience video stored in their folder as needed. The UI (User Interface) of the editing screen displayed on the user terminal 20 can be anything. Furthermore, the scope of the editing process is not limited to what is described above, and may include anything such as editing the title, or editing subtitles or audio. This allows the information processing device 10 to generate a more original experience video tailored to the target user's experience. Various functions may be implemented for the experience videos stored in the dedicated application, such as changing the sorting order, grouping (e.g., into folders), or searching.

[0068] As a result, the information processing device 10 can generate an experience video summarizing the experiences of target users who watched the event in person. Furthermore, target users can relive their memories (i.e., re-experience) by watching the generated experience video. This is expected to have the effect of supplementing memories even in cases where fragmented photos or videos alone are insufficient to recall the experience or the impression of the event, by watching the experience video.

[0069] 3. Variant The present invention is not limited to the embodiments described above, and various modifications are possible. Several modifications are described below. Two or more of the items described below may be combined and applied.

[0070] (1) Information processing system 1 The hardware and network configurations in the information processing system 1 are not limited to those exemplified in the embodiments. The information processing system 1 may have any hardware and network configuration as long as it can realize the required functions. For example, multiple physical devices may cooperate to function as the information processing system 1. For example, at least some of the functions of the information processing device 10 may be implemented in the user terminal 20 or application server 30. For example, the user terminal 20 or application server 30 may have at least some of the functions of the information processing device 10 related to the generation or editing of experience videos.

[0071] (2) Information processing device 10 Some of the functions of the information processing device 10 may be implemented on other servers. These servers may be, for example, physical servers or virtual servers (including so-called cloud servers). Furthermore, the correspondence between functional elements and hardware is not limited to those illustrated in the embodiments. For example, at least some of the functions described as being implemented on the information processing device 10 in the embodiments may be implemented on other devices or systems, or conversely, at least some of the functions described as being implemented on other devices or systems may be implemented on the information processing device 10. In this example, at least some of the functions of the information processing device 10 may be implemented on the application server 30, or conversely, the information processing device 10 may have at least some of the functions of the application server 30. The information processing device 10 may manage the application database 1000. The information processing device 10 may receive various data, such as data related to experience information, from existing systems, infrastructure, or services. The information processing device 10 may, for example, use a machine learning device (or AI) to perform processing related to the generation of experience videos. In this example, at least some of the functional elements implemented by the information processing device 10 may be implemented on, for example, AI, applications, or other software.

[0072] (3) User terminal 20, Event terminal 40 The user terminal 20 and event terminal 40 are not limited to those exemplified in the embodiment. The user terminal 20 and event terminal 40 may perform the above-described processing using any display screen, input device, or UI. The user terminal 20 may be configured to arbitrarily access various databases of the information processing device 10 or application server 30. The user terminal 20 may implement a function for identifying the experience of the target user. In this example, the user terminal 20 may perform automatic data acquisition and transmission to the information processing device 10 using an API or the like. The event terminal 40 may record various data related to events and may output various data in response to requests from the information processing device 10 or application server 30.

[0073] (4) Application Server 30 The application server 30 is not limited to those illustrated in the embodiment. The application server 30 may be a server for managing a database, or a server for performing analysis processing on the stored data. The application server 30 may output data from the database to the information processing device 10 as appropriate in response to a request from the information processing device 10. In this case, the databases held by the application server 30 and the information processing device 10 may function independently. In this embodiment, the experience video generation function was described as being provided as a function of a dedicated application, but the application program on which the experience video generation function is implemented may implement any functions. In this example, the application program may implement functions related to video distribution services, as well as functions related to SNS (Social Networking Service) services, functions related to event-related merchandise sales services, and functions related to event information distribution services.

[0074] (5) How to generate an experience video The sequence chart shown in Figure 4 is merely an example of operation, and the operation of the information processing system 1 is not limited to this. Some of the illustrated operations may be changed or omitted, the order may be changed, or new operations may be added. In step S101, the source image data is not limited to video captured by the target user's smartphone (an example of the user terminal 20), but may also be data captured by various shooting means such as smart glasses. The user terminal 20 can cooperate with these shooting means to acquire various data. In step S104, the sub-events may be defined in any way. For example, the information processing device 10 may define sub-events according to the location information or behavioral patterns of the target user and multiple users.

[0075] In step S105, the information processing device 10 may select the source image data by any method. Although the selection of source image data in step S105 has been described as an automatic selection process by the information processing device 10, it may also be combined with a process in which the target user themselves selects images from the camera roll or other locations on the user terminal 20. In this example, the information processing device 10 may present the target user with sub-events to be used in the experience video via the user terminal 20 and accept the selection of source image data accordingly.

[0076] In steps S106 to S108, the information processing device 10 may accept video uploads from the target user on the generation screen or editing screen related to the experience video. In this example, the information processing device 10 may perform processes such as replacing the user's personal video in the automatically generated experience video via these display screens.

[0077] (6) Database (data) The database (or the data itself) of the information processing system 1 shown in Figures 5, 6, and 7 is not limited to those illustrated in the embodiments. Any data may be registered in the database in this example. Any information may be attached to various experience information in the application database 1000. For example, tags, keywords, or labels for identifying experience information may be attached. The application database 1000 may record any data that identifies the experience of the target user. Experience information may include, for example, behavioral information, movement information, environmental information, contract information, or biometric information. Data related to attending an event (e.g., attendance information) may include information about management, sales, venue, seating, organizations, or performers. The sub-event database 2000 may define any sub-events depending on the event. The data recorded by the sub-event database 2000 may be any data relating to experience information or material image data corresponding to the sub-event. Experience videos may be of any kind and include various video data that are automatically generated based on predetermined playback time, video materials, templates, or effects.

[0078] (7) Others The various programs executed by processor 101 may be provided by downloading them over a network such as the Internet, or they may be provided recorded on a computer-readable non-temporary recording medium such as a DVD-ROM. Each processor may be, for example, a CPU, an MPU (Micro Processing Unit), or a GPU (Graphics Processing Unit).

[0079] The block diagrams used in the description of the above embodiments show functional units. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized using one device that is physically or logically coupled, or it may be realized using two or more physically or logically separated devices that are directly or indirectly connected (for example, using wired or wireless connections). A functional block may also be realized by combining the above one device or the above multiple devices with software.

[0080] Functions include, but are not limited to, judgment, decision, determination, calculation, calculation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, assumption, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating (mapping), and assigning. For example, a functional block (configuration part) that enables transmission is called a transmitting unit or transmitter. As mentioned above, the method of implementation is not particularly limited.

[0081] For example, the information processing device 10 in one embodiment of the present disclosure may function as a computer that performs the processing of the present disclosure.

[0082] Each aspect or embodiment described in this disclosure may be applied to at least one of the following systems: LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), FRA (Future Radio Access), NR (New Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), and other appropriate systems, as well as next-generation systems extended based thereon. Furthermore, multiple systems may be applied in combination (for example, a combination of at least one of LTE and LTE-A with 5G).

[0083] The processing procedures, sequences, flowcharts, etc., of each aspect or embodiment described in this disclosure may be reordered, provided they do not contradict each other. For example, the methods described in this disclosure present various step elements in an exemplary order and are not limited to the specific order presented.

[0084] Input and output information may be stored in a specific location (e.g., memory) or managed using a management table. Input and output information may be overwritten, updated, or appended to. Output information may be deleted. Input information may be sent to other devices.

[0085] The determination may be made by a value represented by 1 bit (0 or 1), by a boolean value (true or false), or by a numerical comparison (for example, a comparison with a predetermined value).

[0086] Although the present disclosure has been described in detail above, it will be clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the intent and scope of the present disclosure as defined by the claims. Therefore, the descriptions in the present disclosure are illustrative and not intended to be restrictive in any way.

[0087] Software should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., whether they are called software, firmware, middleware, microcode, hardware description languages, or by any other name. Furthermore, software, instructions, information, etc., may be transmitted and received via a transmission medium. For example, if software is transmitted from a website, server, or other remote source using at least one of wired technologies (such as coaxial cable, fiber optic cable, twisted pair, or digital subscriber line (DSL)) and wireless technologies (such as infrared or microwave), at least one of these wired and wireless technologies is included in the definition of a transmission medium.

[0088] The information, signals, etc., described herein may be represented using any of the following different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc., which may be referred to throughout the above description, may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof. Terms used herein and terms necessary for understanding this disclosure may be replaced with terms having the same or similar meaning.

[0089] Furthermore, the information, parameters, etc., described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0090] In this disclosure, the phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" means both "based solely on" and "based at least on."

[0091] Any reference to elements using designations such as “First,” “Second,” etc., as used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Accordingly, references to the First and Second elements do not imply that only two elements may be employed, or that the First element must precede the Second element in any way.

[0092] In the above-described configuration of each device, the term "part" may be replaced with "means," "circuit," "device," etc.

[0093] Where the terms “include,” “including,” and variations thereof are used in this disclosure, these terms are intended to be inclusive, as is the term “comprising.” Furthermore, the term “or” as used in this disclosure is not intended to mean exclusive OR.

[0094] In this disclosure, if articles are added by translation, such as a, an, and the in English, this disclosure may include the fact that the noun following these articles is plural.

[0095] In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combine" may be interpreted similarly to "different." [Explanation of Symbols]

[0096] 1...Information processing system, 10...Information processing device, 20...User terminal, 30...Application server, 40...Event terminal, 9...Network, 11...Acquisition unit, 12...Output unit, 13...Generation unit, 14...Selection unit, 191...Storage unit, 192...Control unit, 101...Processor, 102...Memory, 103...Storage, 104...Communication device, 1000...Application database, 2000...Sub-event database

Claims

1. An acquisition unit that acquires source image data, including images taken by participants who attended an event held at the venue, An output unit that outputs a video summarizing the participants' experiences at the venue, generated using the aforementioned material image data. An information processing device having

2. The acquisition unit acquires experience information indicating the participant's experience at the venue, Generation unit that automatically generates the video based on the aforementioned experience The information processing apparatus according to claim 1, having the following features.

3. Multiple sub-events are defined for the participant's activity schedule in the aforementioned event. The generation unit generates the video for each sub-event using an image selected from the source image data. The information processing apparatus according to claim 2.

4. The aforementioned material image data includes corresponding location information, The generation unit selects an image corresponding to a certain sub-event using the location information. The information processing apparatus according to claim 3.

5. The acquisition unit acquires the participant's payment information, The generation unit selects an image corresponding to a certain sub-event using the payment information. The information processing apparatus according to claim 3.

6. The acquisition unit acquires motion information indicating the physical movements of the participant's body. The generation unit selects an image corresponding to a certain sub-event using the operation information. The information processing apparatus according to claim 3.

7. The acquisition unit acquires communication information from other applications that indicates communication between the participant and other participants regarding the event. The generation unit generates the video containing the communication information for a certain sub-event. The information processing apparatus according to claim 3.

8. The aforementioned material image data includes official images provided by the event organizers. The information processing apparatus according to claim 1.

9. The aforementioned official images include paid images, It has a generation unit that generates a sample video including the aforementioned paid image, If the participant purchases the sample video, the output unit outputs the sample video, including the paid images, in an editable format. The information processing apparatus according to claim 8.

10. Computers The steps include: acquiring source image data, including images taken by participants who attended an event held at the venue; The steps include: outputting a video summarizing the participants' experiences at the venue, generated using the aforementioned material image data; and An information processing method having

Images