Event management server system and content image control method

Abstract

To provide an event management server system for enabling an event visitor to easily use an appropriate service through a portable terminal, and a content image control method to be achieved by the event management server system.SOLUTION: In an event management server system 1, when portable terminals 12a to c are held over a scene at an event site by users visiting the event, their picked-up images are acquired and displayed as a real time video. An event management server 10 acquires the picked-up images from the portable terminals 12a to c and acquires characteristics of the users by collating the picked-up images with registration data. Then, when a player to watch is determined to have left the users by more than a threshold value, an enlarged image of the player is superimposedly displayed on the portable terminals 12a to c.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an event management server system that provides services via mobile terminals at event venues such as sporting events, and a content image control method realized thereby. [Background technology]

[0002] Technologies have been put into practical use to improve convenience at event venues, tourist facilities, and the like by providing various information to visitors via their mobile devices. For example, a technology has been proposed that notifies visitors' smartphones of the availability of restrooms at stadiums and the locations of food and drink vendors (see, for example, Patent Document 1). Another technology has been proposed that uses a see-through display or the like to display supplementary information about a player of interest while watching a baseball game (see, for example, Patent Document 2). Meanwhile, a technology has been proposed that captures the state of a game held at a stadium from multiple viewpoints and delivers video from viewpoints that correspond to a client's request (see, for example, Patent Document 3). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-142080 [Patent Document 2] Special Publication No. 2019-531538 [Patent Document 3] Japanese Patent Application Publication No. 2019-68130 Summary of the Invention [Problem to be solved by the invention]

[0004] Most of the services mentioned above that event attendees can use on their mobile devices are relatively simple, presenting additional information one-way. As a result, there are few opportunities for attendees to look at or operate their devices, especially at events such as sporting events or music festivals where the audience's interest tends to be focused on the performance in the real world, and the situations in which the services can be used effectively are limited.

[0005] The present invention has been made in view of these problems, and its purpose is to provide a technology that enables event attendees to easily use appropriate services via their mobile terminals. [Means for solving the problem]

[0006] One aspect of the present invention relates to an event management server system, which includes a terminal information acquisition unit that acquires data of images captured by a terminal carried by a user who visits an event venue, a content image control unit that acquires user characteristics by comparing the captured image with registration data and selects a content image to be displayed on the terminal according to the characteristics, and an image data transmission unit that transmits data of the selected content image to the terminal for display.

[0007] Another aspect of the present invention relates to a content image control method, which includes the steps of acquiring data of an image captured by a terminal carried by a user who has visited an event venue, acquiring characteristics of the user by comparing the captured image with registered data and selecting a content image to be displayed on the terminal according to the characteristics, and transmitting data of the selected content image to the terminal for display.

[0008] Any combination of the above components, and any transformation of the present invention into a method, device, system, computer program, or recording medium on which a computer program is recorded, are also valid aspects of the present invention. [Effects of the Invention]

[0009] According to the present invention, event attendees can easily use appropriate services via their mobile terminals. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing the configuration of an event management system to which the present embodiment can be applied. [Figure 2] 10A and 10B are diagrams illustrating examples of screens displayed by a mobile terminal in the present embodiment. [Figure 3] 10A and 10B are diagrams illustrating examples of screens of a mobile terminal when promoting product sales using content images in the present embodiment. [Figure 4] 10A and 10B are diagrams illustrating examples of the screen of a mobile terminal when the entertainment value of an event is improved by a content image in the present embodiment. [Figure 5] FIG. 2 is a diagram illustrating an internal circuit configuration of an event management server according to the present embodiment. [Figure 6] FIG. 2 is a diagram showing an internal circuit configuration of a mobile terminal according to the present embodiment. [Figure 7] FIG. 2 is a diagram illustrating a functional block configuration of an event management server according to the present embodiment. [Figure 8] FIG. 2 is a diagram showing a functional block configuration of a mobile terminal according to the present embodiment. [Figure 9] 10A and 10B are diagrams for explaining an example of a matching process performed by a matching unit of a content image control unit in the event management server of the present embodiment. [Figure 10] 10A and 10B are diagrams for explaining examples of content images that a content image control unit of the event management server causes to be displayed on a mobile terminal in the present embodiment. [Figure 11] 10A and 10B are diagrams for explaining another example of a content image that the content image control unit of the event management server causes to be displayed on the mobile terminal in the present embodiment. [Figure 12]10A and 10B are diagrams illustrating an example of settings of the type of content image and the display trigger to be displayed on the mobile terminal by a content image control unit of the event management server in the present embodiment. [Figure 13] 10 is a flowchart showing a processing procedure in which the event management server selects and transmits a content image to be displayed on the mobile terminal in the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] This embodiment relates to a system that provides services to visitors at an event venue via mobile terminals. An event here refers to any event where multiple people visit a venue to watch or experience something, and the event content, scale, and type of venue are not particularly limited, and may include, for example, a sporting event, a music festival, a concert, or a show. Figure 1 shows the configuration of an event management system to which this embodiment can be applied. The event management system 1 includes an event management server 10, mobile terminals 12a to 12e, a virtual store management server 14, and a physical store management server 16.

[0012] Mobile terminals 12a to 12e are terminals carried by event attendees (hereinafter referred to as "users"), and display various information related to the event and accept user operations. In FIG. 1, mobile terminals 12a, 12b, and 12c are high-function mobile phones (smartphones), and mobile terminals 12d and 12e are wearable terminals such as AR (Augmented Reality) glasses. As such, the form of mobile terminals 12a to 12e is not limited as long as they are general terminals equipped with a display, an information processing device including communication functions, and an input device. Hereinafter, mobile terminals 12a to 12e will be simply referred to as "mobile terminal 12."

[0013] The event management server 10 establishes communication with the mobile terminal 12, receives captured images and user operation content, performs corresponding information processing, and returns necessary information. In recent years, with the introduction of 5G (fifth generation mobile communication system), it has become possible to transmit various data with low latency and large volumes even from mobile terminals. Therefore, such technology is appropriately used in communication between the mobile terminal 12 and the event management server 10.

[0014] The event management server 10 may actually be realized by multiple information processing devices and storage devices. For example, the event management server 10 may be distributed according to function using MEC (Multi-access Edge Computing), a 5G technology. In this case, functions requiring high responsiveness, such as the function of sending and receiving image data with the mobile terminal 12, may be realized by a server installed near the event venue, while other functions may be realized by a server installed in a separate data center, etc. Therefore, the "event management server" in this embodiment can be read as an "event management server system" regardless of the number of devices.

[0015] The virtual store management server 14 establishes communication with the event management server 10 and mediates sales and purchases between users and virtual stores on the network. The physical store management server 16 establishes communication with the event management server 10 and mediates sales and purchases between users and physical stores inside or outside the event venue. The physical store management server 16 may also provide the event management server 10 with information on the real-time status of the physical store, such as congestion, waiting times, and product inventory, as needed.

[0016] In this embodiment, the event management server 10 provides various services to users via the mobile terminal 12. The services provided here are broadly divided into mechanisms for making the event itself more enjoyable and sales promotion of related products. In this embodiment, various information related to the services can be viewed along with the state of the event via the mobile terminal 12. This allows users to use a variety of services without missing the real-time status of the event or losing interest.

[0017] Specifically, a user holds up mobile terminal 12 to capture images of the event using the in-camera, and the real-time video is displayed on the display. Meanwhile, event management server 10 collects and analyzes the images captured by mobile terminal 12 to estimate the characteristics and interests of each user using mobile terminal 12. Based on the estimation results, an optimal image to be superimposed on the real-time video is generated and displayed on each mobile terminal 12.

[0018] The superimposed image may be represented as a virtual object rendered as if it were actually present at the event venue. Augmented reality (AR) and mixed reality (MR) technologies can be used to blend virtual objects into images of real space. In this case, three-dimensional data representing the spatial structure of the event venue is used to define the placement of the virtual objects. In this embodiment, the three-dimensional data is also used to analyze captured images, making it possible to identify user characteristics and interests at low cost. Hereinafter, the image that the event management server 10 selects for each mobile terminal 12 and superimposes on the real-time video is referred to as a "content image."

[0019] FIG. 2 illustrates an example of a screen displayed by mobile terminal 12. In this example, and in FIGS. 3 and 4 described below, the event is a soccer match, and the user is watching the match taking place on the field from their seat. When the user holds up mobile terminal 12 in this state, mobile terminal 12 displays the state of the real space as seen from the user's viewpoint, as shown in (a), for example. This situation is realized, for example, by instantly displaying on the display a real-time image captured by a camera mounted on the back of mobile terminal 12. If the user changes the orientation of mobile terminal 12, the field of view of the image naturally changes accordingly.

[0020] (b) shows a state in which a content image is superimposed on the real-time video shown in (a). In this example, a cylindrical virtual signage image 200 is superimposed as the content image. The virtual signage is an object resembling an electronic scoreboard, with various information such as scores, player images, and advertisements displayed on its sides and interior walls, and is made to appear as if it were hanging from the sky. The virtual signage image 200 is drawn by mapping the necessary information as a texture onto a three-dimensional model that represents the shape of the signage. However, this is not intended to limit the type, shape, or drawing method of the object represented by the content image.

[0021] The virtual signage image 200 may be a moving image, and the presented information may rotate on the side of a cylinder. To realize such AR, the event management server 10 stores in advance three-dimensional data of the spatial structure of the soccer stadium, which is the event venue. The server then determines the placement of the virtual signage in the same three-dimensional space and notifies the mobile terminal 12 of the position information. The mobile terminal 12 virtually places the signage object at the corresponding position in real space and projects it onto the camera's screen coordinate system, thereby rendering the image 200 on the real-time video being displayed.

[0022] This allows the virtual signage to be fixed in a predetermined position in real space regardless of the camera's field of view, making it possible to display content images while avoiding important parts of the event, such as the soccer field.By converting the 3D data of the spatial structure into a highly accurate data format such as point cloud data, it is possible to display content images with high precision in a variety of locations, such as virtually displaying a logo on the underside of the roof of a soccer stadium.

[0023] If the user selects any of the information displayed on the virtual signage image 200, the event management server 10 acquires the user's operation and may further display, on the mobile terminal 12, a content image showing other information related to the selected information. For example, when a sponsor's logo displayed on the virtual signage is selected, the event management server 10 may additionally display an advertising video for the sponsor. When a player's face image is selected, the event management server 10 may additionally display an image showing the player's profile, record, related goods, etc.

[0024] A similar display may be triggered by a user pointing to an image of a real object displayed in real-time video. For example, if a user points to an image 201 on a signboard installed in the audience seats opposite, the event management server 10 displays a promotional video related to the image. The event management server 10 may also display a GUI (Graphical User Interface) that accepts purchases of related products in response to a user's pointing operation on the virtual signage image 200 or the image of the real object.

[0025] The content image may be constantly displayed while the user holds the mobile terminal 12 over the event venue, or may be switched between displayed and hidden in response to a specific operation by the user as described above. The specific operation may be a general instruction operation such as a tap on the screen, or a shooting operation such as a zoom magnification change operation or a still image shooting operation. Alternatively, the content image may be switched between displayed and hidden based on factors other than the user operation, such as the state of the contestants or the status of the event.

[0026] 3 shows an example of the screen of the mobile terminal 12 when promoting product sales using content images. In (a), sales information 206 for a pair of cleats, which is a product, and a GUI 208 for accepting purchases are superimposed on a real-time video. For example, when a user selects an image of a player displayed on the virtual signage image 200 shown in (b) of FIG. 2, sales information 206 and a GUI 208 are displayed for the same model of cleats worn by the player, along with information 202 about the player. These content images may be objects with three-dimensional models or flat images.

[0027] The spike image included in the sales information 206 may be displayed using mixed reality (MR) technology, allowing the user to change the orientation of the three-dimensional object or pull it closer to the user, allowing the user to check the details. In the illustrated example, the GUI 208 is represented by a horizontally elongated three-dimensional object displaying the words "Buy this." When a user operates the GUI 208, the event management server 10 accepts the operation and performs order processing and payment processing on the virtual store management server 14 or physical store management server 16 that manages the vendor.

[0028] Once the ordering and payment processes are complete, the event management server 10 superimposes an image 210 indicating the completion of the purchase on the mobile terminal 12, as shown in (b). In this example, the image 210 is represented by a horizontally elongated three-dimensional object displaying the words "Purchase Completed." Depending on the type and purchase location, purchased items may be available at the venue or delivered at a later date. In the former case, the items may be delivered to the attendee's seat, or the attendee may pick them up at a store.

[0029] 4 shows an example of the screen of the mobile terminal 12 when content images are used to enhance the entertainment value of an event. (a) is an example of a screen displayed at the time a goal is scored, and information 220 about the player who scored and an image 222 showing a virtual celebration of the goal using lights and text are superimposed as content images on the real-time video. (b) shows a real-time image 224 of a player on the field superimposed as a content image at any time during the game. By enlarging and displaying the player image 224 as shown in the figure, it is possible to check in detail how the player is performing, which would be difficult to see from one's seat.

[0030] In particular, in the case of content images such as those shown in Figure 4, whether a user finds the display results desirable or bothersome largely depends on the user's individual preferences, such as the team they support or the player they are interested in. Also, depending on the user's seat and the position of the player, there may be no need to enlarge the player's image. Furthermore, the information a user wants to see or the products they want to purchase can change from moment to moment due to factors that are difficult to predict in advance, such as the game situation, weather, temperature, the behavior of other spectators, and the user's individual physical condition.

[0031] Based on this, in this embodiment, the event management server 10 collects information such as images captured by the mobile device 12, details of user operations such as shooting operations and instruction operations, and user profiles, and estimates the characteristics and interests of each user based on this information. Furthermore, taking into account the status of participants and the venue, the server optimizes the content images to be displayed and the timing of their display. This minimizes the effort required for advance registration and menu operations, allowing users to enjoy the event more and easily purchase desired products via the mobile device 12.

[0032] 5 shows the internal circuit configuration of the event management server 10. The event management server 10 includes a CPU (Central Processing Unit) 23, a GPU (Graphics Processing Unit) 24, and a main memory 26. These components are interconnected via a bus 30. An input / output interface 28 is also connected to the bus 30. Connected to the input / output interface 28 are a communication unit 32 that establishes communication with the mobile terminal 12, the virtual store management server 14, and the physical store management server 16, a storage unit 34 such as a hard disk drive or nonvolatile memory, an output unit 36 ​​that outputs data to a display (not shown), an input unit 38 that inputs data from an input device (not shown), and a recording medium drive unit 40 that drives a removable recording medium such as a magnetic disk, optical disk, or semiconductor memory.

[0033] The CPU 23 controls the entire event management server 10 by executing an operating system stored in the storage unit 34. The CPU 23 also executes various programs read from removable recording media and loaded into the main memory 26. The GPU 24 performs drawing processing in accordance with drawing commands from the CPU 23 and outputs the results to the output unit 36. The main memory 26 is made up of RAM (Random Access Memory) and stores programs and data required for processing.

[0034] 6 shows the internal circuit configuration of the mobile terminal 12. The mobile terminal 12 includes a CPU 50, a main memory 52, a display unit 54, an input unit 56, an audio output unit 59, a communication unit 60, a motion sensor 62, a camera 64, and a position acquisition unit 66. These units are connected to each other via a bus 58. The CPU 50 processes information acquired from each unit of the mobile terminal 12 via the bus 58, and supplies output data to the display unit 54 and the audio output unit 59. The main memory 52 stores programs and data necessary for processing by the CPU 50.

[0035] The display unit 54 is composed of a display panel such as a liquid crystal panel or an organic EL panel and its drive mechanism, and displays images such as those exemplified in Figures 2 to 4. The input unit 56 is composed of a touchpad that covers the screen of the display unit 54 and hardware keys provided on the housing, and accepts instruction operations for the displayed image and shooting operations such as operations to change the zoom magnification and operations to shoot still images. The audio output unit 59 is composed of a speaker or earphones and their control mechanism, and allows the user to hear audio as needed.

[0036] The communication unit 60 establishes communication with the event management server 10 and transmits and receives necessary data. In this embodiment, mobile communication of the mobile terminal 12 is realized using, for example, 5G communication technology. The motion sensor 62 is composed of at least one of an acceleration sensor, a gyro sensor, a geomagnetic sensor, etc., and measures the posture and movement of the mobile terminal 12 in three-dimensional space. The camera 64 captures video of the real space in the opposite direction to the screen of the display unit 54. In this embodiment, in particular, the video is immediately displayed on the display unit 54, allowing the state of the real space to be viewed in real time. Data of the captured image is also transmitted to the event management server 10 via the communication unit 60 and used to select content images.

[0037] The position acquisition unit 66 constantly acquires the position information of the device itself using well-known technologies such as GPS (Global Positioning System), A-GPS (Assisted-GPS), etc. Measurement values ​​from the motion sensor 62 and the position acquisition unit 66 and images captured by the camera 64 can be used to determine a screen coordinate system for real space when rendering a virtual object, which is a content image.

[0038] Figure 7 shows the functional block configuration of the event management server 10 in this embodiment. Each functional block shown in Figure 7 and Figure 8, which will be described later, can be realized in hardware terms by electronic components such as the CPU and main memory shown in Figure 5 or Figure 6, and in software terms by programs that perform various functions such as data input, data storage, image processing, and communication, loaded into the main memory from a recording medium, etc. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms, using only hardware, only software, or a combination of these, and are not limited to any one of them.

[0039] The event management server 10 includes a terminal information acquisition unit 70 that acquires captured image data and user operation details from the mobile terminal 12, a content image control unit 72 that controls the content images to be displayed on the mobile terminal 12, a sales processing unit 74 that implements the product sales procedures, an information storage unit 76 that stores information necessary for controlling the content images and the product sales procedures, a participant status information acquisition unit 78 that acquires specified information related to the status of the event participants, an event status information acquisition unit 80 that acquires specified information related to the event status, and an image data transmission unit 82 that transmits content image data.

[0040] The terminal information acquisition unit 70 acquires data on images captured in real time by the camera 64 of the mobile terminal 12. The terminal information acquisition unit 70 also acquires the content of user operations on the mobile terminal 12 at any time. Here, user operations include, for example, shooting operations such as operations to change the zoom magnification of the camera 64 or operations to capture still images, operations to give instructions on images being displayed on the mobile terminal 12, and operations to purchase products. The terminal information acquisition unit 70 may continuously acquire data on the captured images at predetermined time intervals while the mobile terminal 12 is capturing images of the event venue, or may acquire data on the captured images only when the user performs a predetermined operation, in association with the content of the operation.

[0041] The terminal information acquisition unit 70 acquires information related to the captured image data and the details of user operations in association with the identification information of the mobile terminal 12, thereby enabling subsequent processing to be performed individually for each mobile terminal 12. The content image control unit 72 appropriately selects and generates content images to be displayed on each mobile terminal 12 for each individual user, based on the data and information acquired by the terminal information acquisition unit 70. In detail, the content image control unit 72 includes a matching unit 90, an image selection unit 92, and an image generation unit 94.

[0042] The matching unit 90 compares the data and information acquired by the terminal information acquisition unit 70 with three-dimensional data showing the spatial structure of the event venue, or with pre-registered template images of participants, logos, etc., to estimate the characteristics and interests of the user using the mobile terminal 12. For example, the matching unit 90 collates the data on the spatial structure of the event venue with the captured image to identify the location of the mobile terminal 12, and ultimately the user.

[0043] In this case, "matching" is realized, for example, by searching for three-dimensional data of the spatial structure based on the distribution of feature points in the captured image, and deriving the position and orientation of the mobile terminal 12 when the most matching angle of view is obtained. However, the means for identifying the user's position is not limited to this, and output values ​​of the motion sensor 62 or position acquisition unit 66 provided in the mobile terminal 12 may be used, or the output values ​​may be combined with the matching results of the captured image to improve accuracy.

[0044] The matching unit 90 may use the identified user's location to identify secondary information. For example, the matching unit 90 may derive the relationship between the distance or orientation between the subject, an event participant, and the user. Alternatively, the matching unit 90 may acquire the type of seat that includes the identified location, thereby determining the team the user is supporting or estimating a profile according to the seat rank.

[0045] The matching unit 90 may also infer the person or object the user is paying attention to by matching the image of the subject in the captured image with pre-prepared image or text information. For example, the matching unit 90 may extract a facial area from the captured image and match it with registered contestant facial images, or detect the jersey number or name shown on the uniform and match it with registered information to identify the contestant of interest. Alternatively, the matching unit 90 may detect signs or product logos appearing in the captured image and match them with registered company or product logo images to identify the company or product in which the user is currently interested. Technologies used in fields such as object detection and object recognition can be used to "match" images in this way.

[0046] The timing at which the matching unit 90 performs the above-described matching is not limited. For example, the matching unit 90 may perform the matching only once at the start of the event, or may perform the matching periodically while captured image data is being transmitted from the mobile terminal 12. Alternatively, the matching unit 90 may perform the matching when the event situation, the state of the participants, the content of the user operation, or the like meets predetermined conditions. For example, when the user performs a zoom-in operation, the matching unit 90 may use the image of the subject at the center of the zoom as the target for matching.

[0047] The image selection unit 92 selects a content image to be displayed on the mobile terminal 12 based on the result of the comparison by the comparison unit 90. For example, when the team the user is rooting for scores, the image selection unit 92 selects a content image representing the virtual performance shown in (a) of FIG. 4. Alternatively, when a contestant of interest is located far from the user, an enlarged image such as that shown in (b) of FIG. 4 is selected. Alternatively, when a sign in the stadium is zoomed in, an advertisement for a related company or product, or a GUI for purchasing related products such as that shown in FIG. 3, is selected as the content image.

[0048] The image generation unit 94 generates the content image selected by the image selection unit 92 in a state that can be displayed on the mobile terminal 12. When the content image is represented as a three-dimensional object that blends with real space, the image generation unit 94 associates the model data of the three-dimensional object with a position in a three-dimensional coordinate system that represents the event venue. Furthermore, the image generation unit 94 may generate texture data to be mapped to the three-dimensional object based on real-time information, or may cut out a real-time image of a specific contestant from live-action video. Alternatively, the image generation unit 94 may render the content image as a two-dimensional image.

[0049] The participant status information acquisition unit 78 continuously acquires status information including at least one of the position and orientation of the participant in the event. For example, the participant status information acquisition unit 78 acquires real-time video of the participant from multiple viewpoints from an external camera (not shown) and analyzes the video to derive status information such as the position, orientation, speed, and type of movement of the participant at the event venue. Such image analysis is widely used in fields such as sports broadcasting.

[0050] The contestant status information acquisition unit 78 sequentially provides the contestant's status information and video data to the content image control unit 72. The content image control unit 72 uses the information and data to control the content image. For example, the content image control unit 72 determines whether to display an enlarged image of the contestant depending on the distance between the user and the contestant. Alternatively, the content image control unit 72 changes at least one of the position and direction of the virtual viewpoint when generating the content image so as to show the contestant facing away from the user.

[0051] This allows for improved visibility regardless of seating, even when the contestants are difficult to see in a large venue, etc. Furthermore, the content image control unit 72 may control the display so that virtual effects that emphasize the quality of the contestant's performance or commentary about the performance are displayed depending on the quality of the contestant's performance.

[0052] The event situation information acquisition unit 80 acquires event situation information such as weather, time zone, season, temperature, humidity, the progress of the match, and the progress of the show. The event situation information may be input as needed by the administrator of the event management server 10 or the event organizer before the start of the event or during the event, or may be determined from output values ​​of various sensors (not shown), such as a thermometer, illuminometer, and hygrometer. Alternatively, the event situation information acquisition unit 80 may acquire situation information by analyzing captured images sent from the mobile terminal 12 or real-time video captured by an external camera (not shown).

[0053] The event status information acquisition unit 80 appropriately provides event status information to the content image control unit 72. The content image control unit 72 uses this information to control the content image. For example, the content image control unit 72 may display a content image that accepts the purchase of rain gear in response to rainfall during the event, or a content image that accepts the purchase of drinking water in response to a rise in temperature.

[0054] In response to a point being scored, the content image control unit 72 may also cause the mobile terminal 12 of the user who is rooting for the team that scored to display a content image showing a virtual effect or information about the scorer, as shown in (a) of Figure 4. Furthermore, the content image control unit 72 may cause the mobile terminal 12 of the user who is rooting for the team that scored to display a discount coupon that can be used to purchase related products. In this case, the discount rate may be changed depending on the situation, such as increasing the discount rate as the number of points increases.

[0055] The matching unit 90 of the content image control unit 72 may also switch the spatial structure data to be matched with the captured image based on the situation information. For example, in the case of a stadium with a retractable roof, the state of the roof changes depending on the weather. For this reason, spatial structure data is prepared in multiple patterns with different roof states, and the matching unit 90 selects the data representing the corresponding roof state based on the weather included in the situation information and uses it for matching. This makes it possible to stabilize the accuracy of the matching results. The event situation information acquisition unit 80 may also acquire the state of the spatial structure itself, such as whether the roof is open or closed, as situation information from the captured image, etc.

[0056] Furthermore, when a content image is expressed using AR, the spatial structure data may be changed based on the situation information when the image selection unit 92 selects a content image. This avoids problems such as displaying a virtual logo as a content image in an attic that does not actually exist. The above-described contestant status information and event situation information may be used by the content image control unit 72 to select content, and may also be used as a trigger for displaying a content image.

[0057] The buying and selling processing unit 74 carries out the buying and selling procedure in response to a purchase operation on a GUI for purchasing a product that is displayed as a content image on the mobile terminal 12. At this time, the buying and selling processing unit 74 acquires pre-registered personal information of the user of the mobile terminal 12, thereby eliminating the need for the user to re-enter the personal information on the spot. Here, personal information includes information necessary for payment, such as name, credit card number, and security code, as well as information such as a delivery address and telephone number if the product is to be mailed.

[0058] As mentioned above, the product to be purchased may be picked up by the user by visiting a store within the venue, or it may be delivered from the store to the user's seat. In the latter case, the buying and selling processing unit 74 may identify the delivery address by referring to the user's seat information obtained as a result of the matching by the matching unit 90. In either case, the buying and selling processing unit 74 establishes communication with the virtual store management server 14 or the physical store management server 16 in response to the user's purchase operation, and performs order processing and payment processing. Conventional technology can be applied to the mechanism of electronic commerce using payment methods such as credit cards and electronic money.

[0059] If a purchase operation is performed without specifying a store, or if the target product is sold at multiple chain stores, the buying and selling processing unit 74 may select, as the ordering destination, the store closest to the user's location or seat obtained as a result of the matching by the matching unit 90. After placing the order, the buying and selling processing unit 74 may also cause a content image indicating that the order has been completed to be displayed on the purchasing mobile terminal 12 via the content image control unit 72. Through these procedures, the user can purchase the desired product with simple operations and with minimal obscuration of the real-time event video, as shown in FIG. 3.

[0060] After placing an order with a store within the venue, the buying and selling processing unit 74 may obtain the store's location, order number, waiting time, etc. from the ordering store, and display a content image representing this information on the mobile terminal 12 from which the purchase was made via the content image control unit 72. The buying and selling processing unit 74 may also obtain store status information, such as the level of congestion, product inventory, and waiting time, of physical stores within the venue or nearby, as needed, and display a content image representing this information on the mobile terminal 12 via the content image control unit 72.

[0061] The information storage unit 76 stores information necessary for the content image control unit 72 and the trading processing unit 74 to perform the above-mentioned processes. In detail, the information storage unit 76 includes a user information storage unit 100, a related information storage unit 102, a spatial information storage unit 104, a matching rule storage unit 106, and an image data storage unit 108. The user information storage unit 100 stores identification information of the mobile terminal 12 and personal information of the user in association with each other. The personal information is used by the trading processing unit 74 in the product trading procedure as described above, and can also be used by the content image control unit 72 when selecting a content image.

[0062] For example, the user information storage unit 100 stores at least one of the user's age, gender, address, purchase history, event participation history, etc. as personal information. The content image control unit 72 can refer to this information to optimize the products to be displayed as content images, and can improve the accuracy of identifying the user's characteristics and target of interest by integrating this information with the matching results by the matching unit 90. For example, if multiple contestants or products appear in the captured image, the matching unit 90 may narrow down which contestant or product is being focused on based on the personal information described above.

[0063] The related information storage unit 102 stores template data used for matching by the matching unit 90 in association with information about the subject it represents. Examples of template data used for matching here include images of the faces, uniforms, and costumes of participants, team logos, text information indicating team names, images of equipment such as balls, images of signboards, logos and marks of products and companies, etc.

[0064] Each image and text information is associated with the profile of the contestant or team, the profile of the company, or sales information for the product it represents. The matching unit 90 estimates the contestant, company, or product that the user is interested in based on the results of matching the captured image with the template data. This allows the image selection unit 92 to appropriately select the contestant, advertisement, sales information, etc. to be displayed as the content image.

[0065] The related information storage unit 102 may store product sales information in association with display conditions for the associated content image. The display conditions can be set, for example, for the status information of the contestants or the status information of the event. This allows the image selection unit 92 to display product sales information at appropriate times without relying on user operation, such as displaying a GUI for purchasing rain gear in response to rainfall or a discount coupon in response to a score.

[0066] The related information storage unit 102 may further store, as sales information for the product, the URL (Uniform Resource Locator) of the sales store, the location of the physical store, etc. This allows the buying and selling processing unit 74 to place an order with an appropriate store via the virtual store management server 14 or the physical store management server 16.

[0067] The spatial information storage unit 104 stores data that represents the structure of the real space of the event venue as three-dimensional information. While there are no particular limitations on the data format, it is conceivable to use, for example, point cloud data, which represents the surface positions of objects existing in the real space of the venue as three-dimensional position coordinates of a group of points that are discretely distributed on the surface. By using point cloud data, the position and orientation of virtual objects can be precisely specified using the same three-dimensional coordinate system, allowing for a highly accurate representation of the fusion of real space and virtual objects.

[0068] As described above, in this embodiment, the spatial structure data is also used to identify the user's position by matching it with the captured image. The spatial information storage unit 104 further stores the divided regions of the spatial structure data in association with real-world categories such as seats, fields, stages, aisles, and stores. This makes it possible to identify the characteristics of a user according to their seat and narrow down the direction in which the camera of the mobile terminal 12 is facing.

[0069] Furthermore, as described above, if the structure of the event venue is variable, the spatial information storage unit 104 may store multiple patterns of spatial structure data, and the matching unit 90 may switch the data used for matching in accordance with a predetermined rule. For this reason, the spatial information storage unit 104 may store selection criteria set in the situation information, etc., in association with each piece of spatial structure data.

[0070] The matching rule storage unit 106 stores rules for matching processing by the matching unit 90. The matching rules define, for example, as shown in the following table, the target for matching a photographed image, the trigger for matching, and the target to be identified by matching.

[0071] [Table 1]

[0072] In this example, it is specified that the user's position and seat are identified by comparing the captured image with the spatial structure of the event venue when the application is launched on the mobile terminal 12. It is also specified that when a zoom-in operation is performed on the mobile terminal 12 or when a command is given to the displayed image, the target image is compared with a template image to identify the contestant, related company, or product of interest.

[0073] The table above also shows the types of content images that the image selection unit 92 selects based on the specified information. Specifically, by specifying the user's location, seat, and the contestant they are focusing on, the system limits the teams and contestants targeted for event content, i.e., for improving visibility and displaying virtual effects on their performances. In other words, content images for teams and contestants the user is not supporting will not be displayed, reducing the likelihood that the content will be perceived as excessive or unpleasant.

[0074] Furthermore, by identifying companies and products of interest, promotional content, i.e., sales information and advertising videos, can be displayed and purchases can be limited. This avoids the annoyance of displaying information about companies and products that are not of interest, and increases sales effectiveness with less information. However, the above table is an example and is not intended to limit the scope of this embodiment. For example, content images for purchasing merchandise related to a contestant of interest can be positioned as promotional content.

[0075] The image data storage unit 108 stores image data used by the image generation unit 94 to generate content images. The image data storage unit 108 stores, for example, 3D model data for rendering virtual objects, texture data, and layout information in the 3D space of the event venue. The image data storage unit 108 also stores content image formats showing information about contestants and products, image data for products and advertisements, and the like. These pieces of data are associated with information about contestants, companies, and products stored in the related information storage unit 102.

[0076] The image data transmission unit 82 transmits data of the content image selected and generated by the content image control unit 72 to the mobile terminal 12. The data transmitted by the image data transmission unit 82 here may be the content image itself, or partial data necessary for the mobile terminal 12 to interpret and draw the image, such as the placement of virtual objects in real space and text information to be expressed therein.

[0077] In the latter case, the mobile terminal 12 places the object in a coordinate system representing real space based on the 3D model stored internally and the transmitted position information, and then draws the image in a screen coordinate system formed by the display screen. Alternatively, the image data transmission unit 82 may generate an image by making the area other than the content image transparent, and then transmit the image to the mobile terminal 12. In this case, the mobile terminal 12 only needs to overlay the transmitted image on the real-time video. Those skilled in the art will understand that there are various possibilities for dividing the data stored in the event management server 10 and the mobile terminal 12 and the processing.

[0078] 8 shows a functional block configuration of mobile terminal 12 in this embodiment. All or part of the illustrated functions may be configured by downloading from a distribution server that distributes applications, based on a user's selection or autonomous request control of mobile terminal 12, or by installing the functions at the time of shipping mobile terminal 12. Mobile terminal 12 may also be configured by a terminal on which an application is installed, such as a high-performance mobile phone, and a wearable display, such as AR glasses, that displays images. In this case, the two may transmit and receive data via short-range wireless communication, such as Bluetooth (registered trademark), or may be connected by wire.

[0079] The mobile terminal 12 includes an operation receiving unit 110 that receives operations by the user, a photographing unit 111 that photographs images of the real space, a terminal information transmitting unit 112 that transmits data of the photographed images and the contents of user operations to the event management server 10, a status information acquiring unit 114 that acquires information related to the status of the mobile terminal 12 itself, an image data receiving unit 116 that receives content image data from the event management server 10, and a display unit 118 that displays real-time video and content images.

[0080] The mobile terminal 12 further includes a personal information storage unit 120 that stores personal information of the user, and an image data storage unit 122 that stores data used to display content images. The operation reception unit 110 receives user operations such as pointing operations and shooting operations for images displayed on the display unit 118. Means of user operations include tapping and pinching on the screen, as well as operations that virtually point to virtual objects in the real space within the field of view of the camera using MR technology, and hand gestures within the field of view.

[0081] The photographing unit 111 includes the camera 64 shown in Fig. 5 and photographs an image of the real space in which the event is taking place. In particular, in this embodiment, the basis is to photograph a video of the event while the user is holding up the mobile terminal 12. The terminal information transmission unit 112 transmits the images photographed by the photographing unit 111, i.e., the data of the video frames, and the content of the user operation accepted by the operation acceptance unit 110, together with the identification information of the mobile terminal 12, to the event management server 10.

[0082] For this purpose, the personal information storage unit 120 stores at least the identification information of the mobile terminal 12. The personal information storage unit 120 may also store personal information typically recorded on a terminal, such as the user's name, phone number, login ID, and password. The state information acquisition unit 114 includes the motion sensor 62 and position acquisition unit 66 shown in FIG. 5, and acquires the position, posture, and movement of the mobile terminal 12 as state information as needed. This state information may be transmitted to the event management server 10 via the terminal information transmission unit 112.

[0083] In this case, the event management server 10 may more accurately identify the user's position and orientation by integrating the collation result using the captured image with the status information of the mobile terminal 12. The image data receiving unit 116 receives content image data from the event management server 10. The content image data may be received intermittently, such as once or multiple times as needed, or may be received continuously at predetermined time intervals.

[0084] The display unit 118 displays the real-time video captured by the imaging unit 111, and also superimposes and displays a content image based on the data received by the image data receiving unit 116. For example, the display unit 118 draws the content image by projecting a three-dimensional model of an object to be displayed as the content image onto a screen coordinate system corresponding to the position and attitude of the mobile terminal 12 acquired by the state information acquiring unit 114. The display unit 118 may also move or deform the object displayed as the content image so that the object interacts with the user's finger or the like.

[0085] The image data storage unit 122 stores some of the data necessary for displaying content images. As described above, the data stored in the image data storage unit 122 may vary depending on the division of processing between the mobile terminal 12 and the event management server 10. Note that the content images displayed by the display unit 118 are not limited to being fixed to a coordinate system in real space. For example, depending on the type of image and the timing of display, images such as corporate advertising videos, images for purchasing products, and discount coupons may be displayed at a predetermined position in the screen coordinate system.

[0086] 9 is a diagram illustrating an example of the matching process performed by the matching unit 90 of the content image control unit 72 in the event management server 10. The upper part of (a) shows a part of the spatial structure 230 of a soccer stadium, which is the event venue. This data may actually be represented in the form of point cloud data in a three-dimensional coordinate system with axes set at predetermined positions and directions, and is stored in advance in the spatial information storage unit 104.

[0087] When a captured image 232 is sent from the mobile terminal 12, the matching unit 90 extracts feature points from the image and identifies the range 234 of the spatial structure that provides the distribution with the highest match rate, as shown by the arrow. (b) shows a schematic bird's-eye view of a soccer stadium, with spectator seats 238 arranged around a field 236. Based on the matching results shown in (a), the matching unit 90 identifies the position of the mobile terminal 12 that provides a field of view of range 234, and ultimately the user's position 240.

[0088] The matching unit 90 may further identify the category of the seat corresponding to the position 240 based on information associated with the spatial structure data. In this example, the user's position 240 corresponds to a seat behind the goal post 242, typically reserved for supporters of one of the teams. Which team's supporter seat it is is uniquely determined by the match settings.

[0089] As a result, the matching unit 90 can identify the user's location and the team they are supporting by matching the captured image with the spatial structure data. Furthermore, if seats are ranked, it is possible to estimate the range of expenditures the user is expecting for the event. Furthermore, if the user is sitting in a special seat, it can be assumed that the user is a valued customer. These user characteristics can be used to determine the team to be displayed in event content, the price range of products to be displayed in sales promotion content, and so on.

[0090] The matching unit 90 may obtain not only the user's position 240 but also the attitude of the mobile terminal 12 and, ultimately, the camera's line of sight direction 244 by matching the captured image with the spatial structure data. A technique such as Visual SLAM (Simultaneous Localization and Mapping) may be used to obtain the camera's position and attitude from the captured image. By obtaining the line of sight direction 244, the content image control unit 72 may estimate that a real object, such as a contestant, present in the line of sight is the user's target of attention and use this information to select a content image. This allows, for example, the displayed content images to be changed one after another by moving the camera's field of view.

[0091] 10 is a diagram illustrating an example of a content image that the content image control unit 72 in the event management server 10 causes to be displayed on the mobile terminal 12. In this example, the event venue is also assumed to be a soccer stadium, and the user is in a seat behind one of the goal posts, as shown in (b) of FIG. 9. (a) and (b) are real-time images displayed on the mobile terminal 12 from that seat.

[0092] In soccer, the goal posts that each team attacks are switched between the first and second halves. This limits the period of time that users behind the goal posts can watch their favorite team attack the goal up close. For example, in the first half, as shown in (a), they can see the players of their favorite team shooting from a relatively close distance, whereas in the second half, as shown in (b), the shots are shot at the opposite goal post, making it difficult to see. This can lead to a lack of excitement in the second half compared to the first half.

[0093] Therefore, when the distance between the user and the player is greater than a predetermined value, the image selection unit 92 determines to display an enlarged image 250a of the player's image, as shown in (c). The distance to the player may be acquired by the player status information acquisition unit 78 as described above, or may be determined by the image selection unit 92 from the size of the player's image in the captured image. Furthermore, the target to be enlarged may be limited to a player on the team the user is supporting, which can be determined from the user's seat, or a player the user is paying attention to, which can be estimated by a zoom-in operation, etc.

[0094] Two or more thresholds may be set for the distance between the user and the player, and the magnification of the enlarged image 250a may be increased stepwise each time the distance exceeds a threshold. Alternatively, no threshold may be set, and the magnification of the enlarged image 250a may be increased continuously as the distance increases. When the distance decreases, the magnification of the enlarged image 250a may be reduced or the enlarged image 250a may not be displayed by the reverse control.

[0095] The enlarged image 250a shown in (c) is an enlarged image of the player as viewed from the same or substantially the same direction as the user's viewpoint. Meanwhile, as another embodiment, an enlarged image 250b viewed from a direction that is not visible from the user's position may be displayed. In this example, by displaying an image of the player viewed from the front, it becomes possible to check facial expressions that would not normally be visible. The user may also be allowed to freely control the orientation of the player displayed in the enlarged image. Technology that uses multi-viewpoint video to enable players to be viewed from any viewpoint during play has been put to practical use, and this technology may be applied.

[0096] Note that the image viewed from a virtual viewpoint different from the user's viewpoint may be displayed at the same magnification. Furthermore, the trigger for display does not have to depend on the distance between the user and the contestant. In any case, at least one of the magnification of the image and the orientation of the contestant on display is changed depending on the relative positions and orientations of the user and the contestant. This allows any performer on stage, for example at a concert, to be viewed from any direction, reducing the difficulty of viewing due to the angle of the seats.

[0097] For clarity, only one player is shown in the diagram, but naturally, there are multiple players on the field. Therefore, multiple players may be enlarged. In this case, the magnification rate may be adjusted to allow the relative positions of the players, i.e., the formation, to be confirmed. Alternatively, the player's image may be replaced with a graphic or the like, and a bird's-eye view may be shown to more clearly show the formation. Furthermore, once the player in question has been identified as described above, information such as the player's performance and play details may be automatically displayed, regardless of whether the player is enlarged.

[0098] 11 is a diagram illustrating another example of a content image that the content image control unit 72 in the event management server 10 causes to be displayed on the mobile terminal 12. (a) shows a real-time video image of a user pointing the camera at the spectator seats in a soccer stadium. Here, suppose the user focuses on a product 260 held by a customer and zooms in on the vicinity of the product 260 as shown in (b).

[0099] When the event management server 10 acquires information about such a shooting operation and data on the captured image at that time, the matching unit 90 of the content image control unit 72 matches the zoomed-in captured image with images of companies and products registered in the related information storage unit 102. In this way, the matching unit 90 identifies information about the company that manufactures the product 260 and the product itself. In the illustrated example, the matching unit 90 identifies the product 260 as French fries called "AA Potatoes" based on the character string "AA," the color of the rectangle surrounding it, the shape and size of the package, etc.

[0100] In response to this, the image selection unit 92 selects a content image associated with the relevant product. In the illustrated example, an image 262 representing a free coupon for "AA fries" is superimposed. For example, when the user taps the coupon image 262 on the screen of the mobile terminal 12, the mobile terminal 12 stores the image in memory in a state in which it can be displayed again, thereby making it appear as if the coupon has been acquired. In this case, when the user goes to a store and presents the coupon, the "AA fries" are presented to the user.

[0101] Note that content images displayed by photographing real objects are not limited to coupons; they can also be information about related companies, information about other products sold by those companies, or GUIs for purchasing. Similar content images can also be realized by photographing signs displayed within the venue, as shown in Figure 2. On the other hand, as shown in Figure 11, by using products held by other visitors as recognition targets and as the basis for displaying content images, diversity can be expanded, leading to unexpected encounters with companies and products. Furthermore, being able to see actual products eaten or worn by people evokes images and further promotes sales.

[0102] In the illustrated example, the matching and content image display are triggered by a zoom-in operation by the user, but other shooting operations such as zooming out or still image capture may also be triggered. For example, during a zoom-out operation, if a real object becomes visible in its entirety as a result of the operation, the matching unit 90 may use the image of the real object as the target for matching. Furthermore, when a still image is captured, the matching unit 90 may match the still image with registered data, as in the case of zooming in.

[0103] 12 shows an example of the type of content image and the display trigger that the content image control unit 72 of the event management server 10 sets to display on the mobile terminal 12. This setting information is stored in advance, for example, in the internal memory of the image selection unit 92. As described above, content images are broadly divided into event content and sales promotion content.

[0104] The former are images that improve the user's visibility and evoke feelings of support, and are preferably limited to images related to the team or player the user is rooting for. The latter are images that promote sales of various products, and are preferably limited to images that the user is interested in or that suit the event situation and the user's characteristics. However, depending on the content presented, there may be content images that are common to all users, such as the virtual signage image 200 shown in Figure 2.

[0105] In the example shown in the figure, the event content is set to display a virtual celebration of a goal scored in a soccer game, etc., as well as a discount coupon for merchandise related to the scoring team. Also, while the distance between the user and a contestant is equal to or greater than a threshold, an enlarged image of the contestant is displayed, and when the user zooms in on or gives instructions to zoom in on the contestant, an image of the contestant from a direction different from the user's line of sight (another-viewpoint image) is displayed.

[0106] Additionally, as sales promotion content, when a user zooms in on a sign or product within the venue, or when a change in the weather, temperature, or other environmental factors occurs that exceeds a predetermined value, a GUI is displayed that allows users to purchase the associated product. Also, when a user zooms in on a sign or product within the venue, a discount coupon for the associated product is displayed. However, these settings are merely examples, and should be optimized according to the content of the event, the size and characteristics of the venue, etc.

[0107] In addition to the trigger for displaying a content image, the setting items are not limited to those shown in the figure, and may include the setting for hiding a displayed content image. In any case, the setting is made so that appropriate content images can be displayed flexibly depending on the situation of the event and the position and orientation relationship between the user and the contestants, not just when the user intentionally instructs it. This allows the user to naturally get excited and easily purchase desired products without losing focus on the event.

[0108] Next, an operation that can be realized by the above configuration will be described. Fig. 13 is a flowchart showing the processing procedure in which the event management server 10 selects and transmits a content image to be displayed on the mobile terminal 12. First, when a user at the event venue starts an application that enables the display of the content image and performs authentication procedures such as logging in, the event management server 10 acquires the information entered by the user and performs authentication processing (S10). This allows the event management server 10 to refer to the user's personal information stored in the user information storage unit 100 and smoothly carry out procedures such as the purchase and sale of products.

[0109] If the authentication is successful, the event management server 10 notifies the mobile terminal 12 of this, which activates the camera on the mobile terminal 12 and displays real-time video on the display. The terminal information acquisition unit 70 of the event management server 10 then begins acquiring terminal information such as data on images captured by the camera and information related to the content of user operations (S12). In response, the content image control unit 72 begins matching the captured image with registered data (S14) and begins acquiring the user's characteristics and interests based on the matching results (S16).

[0110] Here, the registered data includes three-dimensional data of the spatial structure of the event venue, and template images of real objects including contestants, signs, and merchandise. The user's characteristics and interests include the user's location, seat, team they are supporting, and signs and merchandise they are paying attention to. The user's characteristics and interests may be narrowed down by adding the user's personal information stored in the user information storage unit 100 to the result of matching the captured images. As described above, matching of the captured images may be performed at any time, such as at a predetermined timing or when a predetermined user operation is performed.

[0111] The content image control unit 72 then starts selecting a content image suited to the user's characteristics and interests (S18). The selection of a content image may also be performed whenever the user's characteristics and interests are acquired. However, instead of determining a specific content image each time, the content images may be narrowed down in stages according to the acquired characteristics and interests.

[0112] For example, after identifying the team the user is supporting based on the seat of the user, the content image control unit 72 first extracts a set of event content for that team as selection candidates. Then, when the time comes for the user to perform a zoom-in operation, a player to score, or the like, and a specific content image can be selected, the content image control unit 72 selects a specific content image, such as an enlarged image of the player, from the selection candidates.

[0113] The content image control unit 72 waits until the timing arrives to select and display a specific content image (N in S20), and when that timing arrives (Y in S20), it generates the content image (S22) and transmits the data to the mobile terminal 12 (S24). Note that in the example in the figure, the flow is such that image generation and transmission are each performed only once at the timing to display the content image, but this is not intended to limit the present embodiment.

[0114] For example, generation and data transmission may be continuously repeated for one content image. In this case, the process may be terminated by further determining the timing to hide the content image. Although not shown, when a purchase operation is performed on a GUI for purchasing a product displayed on the mobile terminal 12 as a content image, the event management server 10 may execute the buying and selling process as described above.

[0115] Thereafter, the content image control unit 72 repeats the process of selecting a content image at an appropriate timing, generating data, and transmitting the data (N in S26, S20 to S24). When it becomes necessary to stop the process related to the content image, for example, when the user closes the application, the event management server 10 ends the process (Y in S26).

[0116] According to the present embodiment described above, the event management server controls content images to be superimposed on real-time video of the real space displayed on devices carried by users attending the event. The event management server collects images captured by the mobile devices and estimates the characteristics and interests of the users based on the images, thereby optimizing the content images to be displayed. This allows users to enjoy the event and various presentations and easily use the presented information without being distracted by unnecessary content images.

[0117] In addition, by comparing the 3D data of the spatial structure of the event venue with the photographed image, the user's position and seat can be identified. By using the 3D data that is originally required to express content images in AR, compatibility with AR technology is high, making it possible to realize a low barrier to entry.

[0118] Furthermore, even if the user does not explicitly specify a menu or image, an appropriate content image is selected and displayed at an appropriate timing when a shooting operation such as zooming in is performed, or when there is a change in the position or orientation of the user and the contestants, the event situation, environment, etc. This allows the user to access desired information and images without missing an event, increasing the opportunities for using the service via a mobile device.

[0119] The present invention has been described above based on the embodiments. The above embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the respective components and treatment processes, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0120] 1 Event management system, 10 Event management server, 12 Mobile terminal, 14 Virtual store management server, 16 Physical store management server, 23 CPU, 26 Main memory, 32 Communication unit, 50 CPU, 52 Main memory, 54 Display unit, 60 Communication unit, 62 Motion sensor, 64 Camera, 66 Position acquisition unit, 70 Terminal information acquisition unit, 72 Content image control unit, 74 Buying and selling processing unit, 76 Information storage unit, 78 Participant status information acquisition unit, 80 Event status information acquisition unit, 82 Image data transmission unit, 90 Matching unit, 92 Image selection unit, 94 Image generation unit, 110 Operation reception unit, 111 Photography unit, 112 Terminal information transmission unit, 114 Status information acquisition unit, 116 Image data reception unit, 118 Display unit, 120 Personal information storage section, 122 image data storage section.

Claims

1. a terminal information acquisition unit that acquires data of images captured by a terminal held by a user who has visited the event venue; a content image control unit that acquires the user's seat category by comparing the registered data with the captured image, and selects an image related to the user's seat category as a content image to be displayed on the terminal; an image data transmission unit that transmits data of the selected content image to the terminal and displays it; a spatial information storage unit that stores three-dimensional data of the spatial structure of the event venue as the registration data, the content image control unit identifies a user's position by comparing the captured image with the spatial structure, and further acquires a seat category corresponding to the identified user's position based on information associated with the three-dimensional data of the spatial structure. Event management server system.

2. The event management server system according to claim 1 , wherein the content image control unit identifies the team that the user is supporting from among the teams playing in the event based on the seat category.

3. the content image control unit generates the content image as a virtual object arranged in the spatial structure; The event management server system of claim 1 or 2, characterized in that the image data transmission unit transmits positional information of the virtual object relative to the three-dimensional data of the spatial structure in the content image data, thereby superimposing an image of the virtual object at a corresponding position in the captured image being displayed on the terminal.

4. the spatial information storage unit stores data of the spatial structure in a plurality of patterns; 4. The event management server system according to claim 1, wherein the content image control unit switches the spatial structure data used for the matching in accordance with a predetermined rule.

5. the terminal information acquisition unit further acquires information on a user operation for changing a zoom magnification of the captured image; 5. The event management server system according to claim 1, wherein the content image control unit selects the content image when a user operation to change the zoom magnification is performed.

6. the terminal information acquisition unit further acquires information about a user operation to designate any one of the captured images being displayed on the terminal; 6. The event management server system according to claim 1, wherein the content image control unit selects the content image corresponding to a specified image.

7. 7. The event management server system according to claim 1, wherein the content image control unit identifies a subject that the user is paying attention to through the matching and selects the content image corresponding to the subject.

8. a participant status information acquisition unit that acquires status information including the location of the participant at the event venue; 8. The event management server system according to claim 1, wherein the content image control unit changes the content image in accordance with a positional relationship between the user and the contestant.

9. The event management server system described in claim 8, characterized in that the content image control unit selects an enlarged image showing the appearance of the contestant as the content image, and changes the magnification ratio of the enlarged image depending on the positional relationship.

10. The event management server system described in claim 8 or 9, characterized in that the content image control unit selects as the content image an image showing the appearance of the contestant from a virtual viewpoint different from the user's viewpoint, determined based on the positional relationship.

11. further comprising an event status information acquisition unit that acquires a status of the event; 11. The event management server system according to claim 1, wherein the content image control unit changes the content of the content image depending on the status of the event.

12. The event management server system described in claim 11, characterized in that the content image control unit selects an image of a discount coupon for a related product as the content image and changes the discount rate indicated by the discount coupon depending on the situation of the event.

13. 13. The event management server system according to claim 11, wherein the content image control unit selects, as the content image, an image that represents a virtual presentation according to a situation of the event.

14. 14. The event management server system according to claim 1, wherein the content image control unit selects, as the content image, an image that represents information about a product related to the subject appearing in the photographed image.

15. 15. The event management server system according to claim 14, wherein the content image control unit determines the related product by detecting a signboard or product corresponding to the registration data from the photographed image.

16. the content image control unit selects a content image that accepts purchase of the related product; the terminal information acquisition unit further acquires information on a purchase operation of the related product performed on the terminal; The event management server system includes: a user information storage unit for storing correspondence information between the terminal and the user's personal information; a buying and selling processing unit that, when the purchase operation is performed on the terminal, acquires the personal information associated with the terminal to carry out a buying and selling procedure; 16. The event management server system according to claim 14, further comprising:

17. An event management server system acquires data of images taken by a terminal held by a user who visits the event venue; The event management server system acquires the user's seat category by comparing the registered data with the captured image, and selects an image related to the user's seat category as a content image to be displayed on the terminal; a step in which the event management server system transmits data of the selected content image to the terminal and displays it; the event management server system stores three-dimensional data of the spatial structure of the event venue as the registration data; In the selecting step, the event management server system identifies the user's position by comparing the photographed image with the spatial structure, and further acquires a seat category corresponding to the identified user's position based on information associated with the three-dimensional data of the spatial structure. Content image control method.

18. A function to acquire data of images taken by devices held by users who visit the event venue; a function of acquiring a seat category of the user by comparing the registered data with the captured image, and selecting an image related to the seat category of the user as a content image to be displayed on the terminal; a function of transmitting data of the selected content image to the terminal and displaying it; A function of storing three-dimensional data of the spatial structure of the event venue as the registered data; A computer program that causes a computer to implement the following: the selecting function identifies the user's position by comparing the captured image with the spatial structure, and further acquires a seat category corresponding to the identified user's position based on information associated with the three-dimensional data of the spatial structure. Computer program.

Images