Information processing device, mobile device, and information processing system

Abstract

The system generates a map that determines a safe travel route that avoids collisions with performers and objects on the stage, and then drives the image-capturing robot along the route determined based on the map. [Solution] A potential map is generated that defines the permissible travel area for an image-capturing robot that moves around the stage to take pictures. The data processing unit acquires data on the performers' planned actions on the stage, the planned placement of objects, and the planned lighting control on the stage. Based on the acquired data, it generates a potential map that defines the permissible travel area as an area where the robot will not collide with performers or objects and will not be conspicuous due to lighting. Furthermore, the robot's travel route is determined based on the generated map, and the robot is made to move.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, a mobile device, and an information processing system. More specifically, in a configuration where a performance such as a singer's song of a performer on a stage is photographed by a camera mounted on a mobile device (autonomous driving robot) that moves on the stage, a map is generated that sets a route that does not collide with objects such as performers and speakers on the stage and does not obstruct the line of sight of the audience, and an information processing apparatus, a mobile device, and an information processing system that perform movement control according to the map are related.

Background Art

[0002] When performing various performances on a stage such as a music live, a process of photographing an image of the performance being performed on the stage with a moving camera that travels on the same stage may be performed.

[0003] When performing such image photographing, the camera is mounted on a mobile device (cart) such as an autonomous driving robot and moves to various positions on the stage to photograph images from various angles. In this case, the camera needs to select a travel route so as not to collide with performers moving around on the stage and equipment such as microphones and speakers installed on the stage and move. Also, it is required to move so as not to obstruct the line of sight of the audience in front of and around the stage.

[0004] As prior arts that disclose movement control of mobile devices such as robots, for example, there are Patent Document 1 (Japanese Patent Application Laid-Open No. 2020-087061) and Patent Document 2 (Japanese Patent No. 5160322).

[0005] Patent Document 1 discloses an unmanned mobile body that monitors a person so as not to obstruct the person to be monitored. Specifically, it is a configuration in which an area where the mobile body is difficult to be sensed by the sensory organs of the person to be monitored is determined, and monitoring is performed from the determined area.

[0006] Furthermore, Patent Document 2 discloses a robotic device that tracks an object, and discloses a configuration that allows the tracking process to continue even if an obstacle enters between the object being tracked and the robot, causing the robot to almost lose sight of the object.

[0007] Patent Document 1 discloses a configuration for selecting an area that is difficult for the person being monitored to perceive and monitoring from that selected position, and Patent Document 2 discloses a configuration for continuing tracking when an obstacle appears between the robot and the object being tracked. Each of these only discloses a configuration for achieving a specific purpose.

[0008] In contrast, when filming performers moving on stage with a camera that also moves on the same stage, it is necessary to determine the camera's position based on various factors, such as the performers' movement, the location of equipment on stage, and the audience's line of sight. The above-mentioned Patent Documents 1 and 2 do not disclose a control configuration for determining the optimal path for moving a camera, taking into account such various circumstances. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] Japanese Patent Publication No. 2020-087061 [Patent Document 2] Patent No. 5160322 [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] This disclosure was made, for example, in view of the above-mentioned problems, and provides an information processing device, a mobile device, and an information processing system that determine the optimal position and route of a camera according to various situations such as the movement position of performers on stage, the position of equipment on stage, and the line of sight of the audience, and control the movement of a mobile device (camera). [Means for solving the problem]

[0011] The first aspect of this disclosure is, It has a data processing unit that generates the travel route of a moving device that moves on the stage, The aforementioned data processing unit The system acquires data on the planned locations of obstacles, including people and objects on the stage, that may obstruct the movement of the mobile device, or data on planned environmental control of the stage environment that changes the visibility level of the mobile device from outside the stage, and The information processing device generates a travel route that reduces the possibility of physical interference between the mobile device and the obstacle, or at least one of the level of visibility of the mobile device from outside the stage, based on acquired data.

[0012] Furthermore, the second aspect of this disclosure is, A storage unit that stores travel route information that reduces the possibility of physical interference between the mobile device and the obstacles, or the visibility level of the mobile device from outside the stage, based on data of the planned positions of obstacles on the stage, including people and objects, that may obstruct the mobile device's movement on the stage, or planned environmental control data of the stage environment that changes the visibility level of the mobile device from outside the stage, or It has at least one communication unit that acquires the aforementioned driving route information from an external device, The mobile device travels on the stage according to either the travel route information obtained from the storage unit or the travel route information obtained via the communication unit.

[0013] Furthermore, a third aspect of this disclosure is: It is an information processing system that includes a mobile device that moves around on the stage and a server. The aforementioned server, The system includes a data processing unit that generates the travel route of the aforementioned mobile device, The aforementioned data processing unit The system acquires data on the planned locations of obstacles on the stage, including people and objects, that may obstruct the movement of the mobile device on the stage, or data on planned environmental control of the stage environment that changes the visibility level of the mobile device from outside the stage. Based on the acquired data, a travel route is generated that reduces the possibility of physical interference between the mobile device and the obstacle, or at least one of the visibility level of the mobile device from outside the stage. The aforementioned mobile device is The system in question is an information processing system that travels according to the travel route generated by the aforementioned server.

[0014] Further purposes, features, and advantages of this disclosure will become apparent from the more detailed descriptions based on the embodiments and accompanying drawings described below. In this specification, a system is a logical combination of multiple devices, and the devices in each configuration are not necessarily located within the same enclosure.

[0015] According to one embodiment of the present disclosure, it is possible to generate a map that determines a safe travel route that does not collide with performers or objects on the stage, and to have the image-capturing robot travel according to the route determined based on the map. Specifically, for example, a potential map is generated that defines the permissible movement area for an image-capturing robot that moves around the stage to take pictures. The data processing unit acquires data on the performers' planned actions on the stage, the planned placement of objects, and the planned lighting control on the stage. Based on the acquired data, it generates a potential map that defines the permissible movement area as an area where the robot will not collide with performers or objects and will not be conspicuous due to the lighting. Furthermore, the robot's travel route is determined based on the generated map, and the robot is made to move. This configuration allows for the generation of a map that determines a safe travel route that avoids collisions with performers or objects on the stage, and enables the image-capturing robot to travel along the route determined based on the map. Note that the effects described in this specification are merely illustrative and not limiting, and there may be additional effects.

Brief Description of Drawings

[0016] [Figure 1] It is a diagram for explaining the outline of the live stage and the outline of the processing of the present disclosure. [Figure 2] It is a diagram for explaining the outline of the live stage and the outline of the processing of the present disclosure. [Figure 3] It is a diagram for explaining the outline of the live stage and the outline of the processing of the present disclosure. [Figure 4] It is a diagram for explaining the configuration and processing of the information processing apparatus of the present disclosure. [Figure 5] It is a diagram for explaining the configuration and processing of the information processing system of the present disclosure. [Figure 6] It is a diagram for explaining an example of the performer-based potential map generated by the individual potential map generation unit. [Figure 7] It is a diagram for explaining an example of the lighting-based potential map generated by the individual potential map generation unit. [Figure 8] It is a diagram for explaining an example of the object-based potential map generated by the individual potential map generation unit. [Figure 9] It is a diagram for explaining an example of the pre-generated potential map generated by the potential map synthesis unit. [Figure 10] It is a diagram for explaining the configuration and processing of the information processing apparatus of the present disclosure. [Figure 11] It is a diagram for explaining an example of the image capturing robot travel route determined based on the pre-generated potential map. [Figure 12] It is a diagram showing a flowchart for explaining the generation processing sequence of the performer-based potential map executed by the information processing apparatus of the present disclosure. [Figure 13]This figure shows a flowchart illustrating the process sequence for generating an illumination-based potential map performed by the information processing device of the present disclosure. [Figure 14] This figure shows a flowchart illustrating the process sequence for generating an illumination-based potential map performed by the information processing device of the present disclosure. [Figure 15] This figure shows a flowchart illustrating the process sequence for generating an object-based potential map executed by the information processing device of the present disclosure. [Figure 16] This figure shows a flowchart illustrating the generation process sequence for pre-generated potential maps performed by the information processing device of this disclosure. [Figure 17] This figure illustrates an example of a simulation image generated by the information processing device of the present disclosure. [Figure 18] This figure illustrates the process in Example 2 of the present disclosure. [Figure 19] This figure illustrates the process in Example 2 of the present disclosure. [Figure 20] This figure illustrates an example of the configuration and processing of an information processing device in Embodiment 2 of the present disclosure. [Figure 21] This figure illustrates an example of a priority audience & TV camera-based potential map generated by the information processing device of Embodiment 2 of this disclosure. [Figure 22] This figure shows a flowchart illustrating the processing sequence for generating a priority audience & TV camera-based potential map performed by the information processing device of this disclosure. [Figure 23] This figure shows a flowchart illustrating the pre-generated potential map generation process sequence performed by the information processing device of Example 2 of this disclosure. [Figure 24] This figure illustrates the configuration and processing of an information processing apparatus in Example 3 of the present disclosure. [Figure 25] This figure illustrates the configuration and processing of the information processing system in Embodiment 3 of this disclosure. [Figure 26]This figure illustrates an example of a potential map generated in the information processing device of Embodiment 3 of this disclosure. [Figure 27] This figure illustrates an example of a potential map generated in the information processing device of Embodiment 3 of this disclosure. [Figure 28] This figure shows a flowchart illustrating the real-time data reflection potential map generation process sequence performed by the information processing device of Embodiment 3 of this disclosure. [Figure 29] This figure shows a flowchart illustrating the real-time data reflection potential map generation process sequence performed by the information processing device of Embodiment 3 of this disclosure. [Figure 30] This figure illustrates an example of the hardware configuration of the information processing device disclosed herein. [Modes for carrying out the invention]

[0017] The details of the information processing device, mobile device, and information processing system disclosed herein will be described below with reference to the drawings. The explanation will be conducted according to the following items. 1. Overview of the processing of this disclosure 2. (Example 1) Details of the configuration and processing of the information processing device of Example 1 of this disclosure 3. The sequence of processes executed by the information processing device of Embodiment 1 of this disclosure 3-(1) Regarding the generation sequence of map a "performer-based potential map" 3-(2) Generation sequence of map b "Illumination-based potential map" 3-(3) Regarding the generation sequence of map c "Object-based potential map" 3-(4) Generation sequence of pre-generated potential maps synthesized from individual potential maps 4. (Example 2) Details of the configuration and processing of an information processing device that creates a map considering the gaze of the audience and the gaze of television cameras on the audience side. 5. (Example 3) An example of generating a real-time data reflection potential map using information from live execution. 6. Examples of hardware configurations for each device 7. Summary of the structure of this disclosure

[0018] [1. Overview of the processing of this disclosure] First, an overview of the processing described herein will be explained with reference to Figure 1 and subsequent figures.

[0019] Figure 1 shows an example of a performance such as a live music show that takes place on a stage. The example shown in Figure 1 is a live music show performed by a duo of idol singers, performers 20, on stage 10. In front of Stage 10, there are 30 spectators watching 20 performances.

[0020] In situations like music concerts where various performances take place on stage, images of the performances are sometimes captured using mobile cameras that move around the same stage.

[0021] The image-capture robot 50 shown in Figure 1 is a mobile device equipped with a camera, i.e., a mobile robot, which moves around the stage to capture the performer's performance from various angles.

[0022] The image-capturing robot 50 is an automated mobile robot (mobile device) such as a cart equipped with a camera, and for example, it travels on the stage according to a predetermined route and takes images from various angles.

[0023] The image-capturing robot 50 needs to choose a safe route to move around the stage, avoiding collisions with performers moving around on the stage or with equipment such as microphones and speakers installed on the stage. Furthermore, performers are required to move in a way that does not obstruct the view of the audience in front of or around the stage.

[0024] As shown in Figure 2, the stage 10 where the live performance actually takes place is equipped with speakers 12, monitors 13, and various decorative objects 14, and lighting 11 is also shone on the performers 20 and others.

[0025] The performers 20 move around the stage in accordance with the progress of the live performance, and the position, brightness, and color of the lighting 11 change accordingly. The decorative objects 14 are also changed one after another in accordance with the progress of the live performance.

[0026] For example, the stage state shown in Figure 2 represents a scene at a certain time t1 during a live performance. At a later time t2, the stage state will be different, for example, as shown in Figure 3, with changes to the performer 20's position, as well as the position, brightness, and color of the lighting 11. Furthermore, the decorative objects 14 will also be replaced.

[0027] In this way, the positions of the performers 20 on stage, the positions, brightness, and color of the lighting 11, and the positions of decorative objects are changed sequentially depending on the time of day when live performances or other events are taking place. During the performance of such a live event, the image-capturing robot 50 needs to be driven in a way that avoids collisions with the performers 20, decorative objects 14 on the stage, speakers 12, and other objects placed on the stage.

[0028] Furthermore, it is important to select a route that does not obstruct the view of the 30 spectators. One effective method for achieving this is to control the vehicle to travel through dark areas outside of brightly lit zones.

[0029] This disclosure describes a method for controlling the movement of a mobile device (camera) by determining the optimal position and travel route of the camera in accordance with various conditions such as the movement position of performers on stage, the position of equipment on stage, lighting, and the audience's line of sight. The details of the structure and processing of this disclosure are described below.

[0030] [2. (Example 1) Details of the configuration and processing of the information processing device of Example 1 of this disclosure] The configuration and processing details of the information processing device of Embodiment 1 of this disclosure will be described below.

[0031] Figure 4 shows an example configuration of the information processing device 100 of Embodiment 1 of this disclosure. The information processing device 100 may be configured inside a mobile device that travels on the stage shown in Figures 1 to 3, i.e., an image-capturing robot 50 equipped with a camera, or it may be a device independent of the image-capturing robot 50, such as a device that can communicate with the image-capturing robot 50.

[0032] Figure 5 shows an example configuration of an information processing system when the information processing device, which has the configuration shown in Figure 4, is a separate device from the image acquisition robot 50. For example, as shown in Figure 5, an information processing system 180 is constructed by connecting an information processing device (server) 100, a live venue information acquisition device 60, and an image-capturing robot 50 within the live venue via a communication network. The information processing device (server) 100 has the configuration shown in Figure 4.

[0033] The live venue information acquisition device 60 consists of a camera that takes images of the live venue, a microphone that acquires audio information of the live venue, an illuminance meter that detects the lighting conditions, a color analyzer, etc., and transmits the acquired images and other information to the information processing device (server) 100 via a communication network.

[0034] The information processing device (server) 100 performs processes such as generating a map (potential map) to determine the travel route of the image-capturing robot 50 within the live venue, determining the travel route using the map, and generating travel control information for the image-capturing robot 50 according to the determined travel route.

[0035] The information processing device (server) 100 further transmits the generated driving control information to the image-capturing robot 50 via the communication network. The image-capturing robot 50 moves around the stage according to the driving control information it receives from the information processing device (server) 100. For example, it is possible to perform processing using such an information processing system.

[0036] The configuration and processing of the information processing device 100 shown in Figure 4 will be described below. As shown in Figure 4, the information processing device 100 includes a storage unit 110, an individual potential map generation unit 120, a potential map synthesis unit 130, and a travel route generation unit 160. As mentioned above, the information processing device 100 may be configured inside a mobile device that travels on the stage shown in Figures 1 to 3, i.e., an image-capturing robot 50 equipped with a camera, or it may be an independent device from the image-capturing robot 50, such as the information processing device (server) 100 shown in Figure 5, which is capable of communicating with the image-capturing robot 50.

[0037] The memory unit 110 stores the following three pieces of data: A. Performer's activity schedule data 111, B. Stage lighting control planned data 112, C. Stage object placement plan data 113,

[0038] These three schedule data are data that are prepared in advance before the start of the live performance and stored in the memory unit 110. In other words, they are schedule data that are prepared in advance according to a program such as a live performance schedule that is prepared before the start of the live performance.

[0039] The information processing device 100 uses this scheduled data to generate a map (potential map) for setting a safe travel route that prevents the image-capturing robot 50 from colliding with performers, speakers, or other objects. During the actual live performance, the image-capturing robot 50 is driven according to the safe travel route selected using the generated potential map.

[0040] Details of the data stored in the memory unit 110 will be explained below. A. Performer movement schedule data 111 is time-series position data of performers moving around on stage during the live performance. In other words, it is time-series position data of performers from the start to the end of the live performance.

[0041] B. Stage lighting control schedule data 112 is time-series data of lighting control information, including lighting setting information such as lighting position, brightness, and color from the start to the end of the live performance.

[0042] C. Stage object placement plan data 113 is time-series data of stage object placement information, including the placement positions of objects on the stage from the start to the end of the live performance. Stage objects include speakers, monitors, decorative objects, and other items placed on the stage.

[0043] These three types of data all represent time-series planned data from the start to the end of a live performance. In other words, for example, for a one-hour live performance, A. Performer's activity schedule data 111, B. Stage lighting control planned data 112, C. Stage object placement plan data 113, These three types of hourly time-series schedule data are recorded in the storage unit 110.

[0044] The data stored in this memory unit 110 is used in the individual potential map generation unit 120. The individual potential map generation unit 120 uses the three types of time-series data A, B, and C individually to generate the following three types of individual potential maps. Map a = Performer-based potential map, Map b = Illumination-based potential map, Map c = Object-based potential map,

[0045] The potential map, as explained with reference to Figures 1-3, defines the no-travel zone, the caution zone, and the permitted zone for the image-capturing robot 50. The three maps a-c described above are all time-series maps.

[0046] For example, map a = performer-based potential map is a time-series map in which each region (no-travel zone, caution zone, permitted zone) is dynamically changed according to the performer's position from the start to the end of the live performance. Map b, the lighting-based potential map, is a time-series map in which each area (no-driving area, caution area, permitted driving area) is dynamically changed according to the lighting conditions (lighting position, brightness, color, etc.) from the start to the end of a live performance. Map c, the object-based potential map, is a time-series map in which each region (no-go zone, caution zone, and permitted zone) changes dynamically according to the object placement positions from the start to the end of the live (performance).

[0047] As shown in Figure 4, the individual potential map generation unit 120 executes each of the processes shown in steps S11 to S13 in the figure. In other words, In step S11, map a = performer-based potential map is generated. In step S12, map b = illumination-based potential map is generated. In step S13, a map c = object-based potential map is generated. Furthermore, these steps S11 to S13 can be executed as parallel processes. The following explains these processes in order.

[0048] (Step S11) In step S11, the individual potential map generation unit 120 generates map a = "performer-based potential map". The "Performer-Based Potential Map," as described above, is a map that dynamically changes each region (no-travel zone, caution zone, permitted zone) according to the performer's position from the start to the end of a live performance, and is a time-series map with the following three regions (a1 to a3). a1. No-driving zone (red) = close proximity to the performer's position a2. Driving caution area (yellow) = medium distance from the performer's position a3. Permitted driving area (blue) = Distance from the performer's position

[0049] Specifically, for example, a1. No-Drive Zone (Red) = Area where the probability of collision with the performer is above the predetermined first threshold Tha1. a2. Driving caution area (yellow) = Area where the possibility of collision with the performer is within the range of the first threshold Tha1 to the second threshold Tha2, which are predetermined. a3. Permissible driving area (blue) = Area where the possibility of collision with the performer is less than or equal to the predefined second threshold Tha2. For example, this is a map with these area divisions.

[0050] Referring to Figure 6, a specific example of map a = performer-based potential map will be explained. The "performer-based potential map" is a time-series map in which each region (no-travel zone, caution zone, permitted zone) is dynamically changed according to the performer's position from the start to the end of the live performance. Figure 6 shows examples of performer-based potential maps at four different timings (t1 to t4).

[0051] The map in the upper left of Figure 6 (t1) is an example of a "performer-based potential map" at time t1. As explained earlier with reference to Figures 1 to 3, there are two performers on the stage. As mentioned above, the potential map is a map generated before the actual live performance begins, and the performers' positions are estimated based on the live program, such as the live schedule. The planned action data, which shows the performers' positions from the start to the end of the live performance, is pre-generated and recorded in the memory unit 110 as A. Performer Action Plan Data 111.

[0052] The "performer-based potential map" is a map in which different colors are assigned to each region according to the area (no-driving area, caution area, permitted driving area) determined by the distance from the performer's position, as follows: Areas in close proximity to the performers are marked in red as no-go zones. The performer's position at a medium distance is set to yellow as a driving caution area. The performer's position at a distance is set to blue as the permissible driving area.

[0053] The map in the upper right of Figure 6 (t2) is an example of a "performer-based potential map" at time t2, a certain period of time after time t1. At time t2, the two performers have moved to different locations than at time t1. Along with this movement of the performers, the settings for the three zones—the no-travel zone (red), the caution zone (yellow), and the permitted zone (blue)—also change.

[0054] The map in the lower left of Figure 6 (t3) is an example of a "performer-based potential map" at time t3, a certain amount of time has elapsed since time t2. At time t3, the two performers have moved to different locations than at times t1 and t2. Along with this movement of the performers, the settings for the three zones—the no-travel zone (red), the caution zone (yellow), and the permitted zone (blue)—also change.

[0055] The map in the lower right of Figure 6 (t4) is an example of a "performer-based potential map" at time t4, a certain period of time has elapsed since time t3. At time t4, the two performers have moved to different locations than those at times t1-t3. Along with this movement of the performers, the settings for the three zones—the no-travel zone (red), the caution zone (yellow), and the permitted zone (blue)—also change.

[0056] Thus, the "performer-based potential map" is a time-series map in which each region (no-travel zone, caution zone, permitted zone) is dynamically changed according to the performer's position from the start time to the end time of the live performance. In step S11, the individual potential map generation unit 120 generates such a map a = "performer-based potential map".

[0057] Furthermore, the action schedule data indicating the performers' positions from the start to the end of the live performance is pre-generated and recorded in the memory unit 110 as A. Performer action schedule data 111. In step S11, the individual potential map generation unit 120 acquires A. performer action schedule data 111 from the memory unit 110, and, referring to the acquired data, generates a time-series map a = "performer-based potential map" in which each region (no-travel region, caution region, permitted region) changes dynamically from the start to the end of the live performance.

[0058] (Step S12) Furthermore, in step S12, the individual potential map generation unit 120 generates map b = "Illumination base potential map". The "lighting base potential map," as described above, is a time-series map in which each region (no-driving area, caution area, permitted driving area) is dynamically changed according to the lighting conditions (lighting position, brightness, color, etc.) from the start to the end of a live performance. It is a time-series map with the following three regions (b1 to b3). b1. No-travel zone (red) = Areas that stand out due to lighting conditions (bright areas or areas with lighting of a different color from the image-capturing robot) b3. Permissible Driving Area (Blue) = Areas that are not noticeable due to lighting conditions (dark areas or areas with lighting of the same color as the image-capturing robot) b2. Driving caution area (yellow) = intermediate area between b1 and b3

[0059] Specifically, for example, b1. No-travel zone (red) = Areas where the lighting condition value calculated based on the lighting conditions (lighting position, brightness, color, etc.) is equal to or greater than the predetermined first threshold Thb1 (= conspicuous areas (bright areas or areas with lighting of a different color from the image-capturing robot)). b3. Permissible Driving Area (Blue) = Area where the illumination state value calculated based on the illumination state (illumination position, brightness, color, etc.) is less than the predetermined second threshold Thb2 (= inconspicuous area (dark area or illumination area of ​​the same color as the image-capturing robot)). b2. Driving caution area (yellow) = intermediate area between b1 and b3 For example, this is a map with these area divisions.

[0060] Refer to Figure 7 to explain a specific example of map b = illumination base potential map. The "lighting base potential map" is a time-series map in which each region (no-driving area, caution area, permitted driving area) is dynamically changed according to the lighting conditions (lighting position, brightness, color, etc.) from the start to the end of a live performance. Figure 7 shows examples of lighting base potential maps for four different timings (t1 to t4).

[0061] The map in the upper left of Figure 7 (t1) is an example of the "lighting base potential map" at time t1. This map (t1) is an example of a map where the lighting is generally dim, for example, at the start of a live performance. In this case, since the entire stage is set to be dark, the entire stage is set as a drivable area (blue) = an area that is inconspicuous due to the lighting conditions (dark areas or areas with lighting of the same color as the image-capturing robot).

[0062] The map in the upper right of Figure 7 (t2) is an example of the "lighting base potential map" at time t2, a certain period of time after time t1. At time t2, the lighting conditions (lighting position, brightness, color, etc.) are set to a different state than at time t1. Along with this change in lighting conditions, the settings for the no-driving zone (red), the caution zone (yellow), and the permitted driving zone (blue) also change.

[0063] The "lighting base potential map" is a map in which different colors are assigned to each region according to the lighting conditions (lighting position, brightness, color, etc.), which are determined according to the region (no driving zone, driving caution zone, driving permitted zone). In other words, areas that stand out due to lighting conditions (bright areas or areas with lighting of a different color from the image-capturing robot) are set to red as no-go zones. Areas that are inconspicuous due to lighting conditions (dark areas or areas with lighting of the same color as the image-capturing robot) are set to blue as areas where movement is permitted. The area between the prohibited driving zone and the permitted driving zone is set to yellow as a driving caution zone.

[0064] Thus, at time t2, the settings of the three areas—the no-driving area (red), the caution area (yellow), and the permitted driving area (blue)—change in accordance with the changes in the lighting conditions (lighting position, brightness, color, etc.).

[0065] The map in the lower left of Figure 7 (t3) is an example of the "lighting base potential map" at time t3, which is a certain period of time after time t2. At time t3, the lighting conditions (lighting position, brightness, color, etc.) will be different from those at times t1 and t2. Along with this change in lighting conditions, the settings for the no-driving zone (red), the caution zone (yellow), and the permitted driving zone (blue) will also change.

[0066] The map in the lower right of Figure 7 (t4) is an example of the "lighting base potential map" at time t4, which is a certain period of time after time t3. Even at time t4, the lighting conditions (lighting position, brightness, color, etc.) are set to a different state than those at times t1 to t3. Along with this change in lighting conditions (lighting position, brightness, color, etc.), the settings for the no-driving zone (red), the caution zone (yellow), and the permitted driving zone (blue) also change.

[0067] Thus, the "lighting-based potential map" is a time-series map in which each area (no-driving area, caution area, permitted driving area) is dynamically changed according to the changes in lighting conditions (lighting position, brightness, color, etc.) from the start time to the end time of a live performance.

[0068] The individual potential map generation unit 120 generates such a map b = "Illumination base potential map" in step S12.

[0069] Furthermore, the planned transition data for lighting conditions (lighting position, brightness, color, etc.) from the start to the end of the live performance is pre-generated and recorded as B. Stage lighting control planned data 112 in the storage unit 110. In step S12, the individual potential map generation unit 120 acquires the B. Stage lighting control scheduled data 112 from the storage unit 110, and, referring to the acquired data, generates a time-series map b = "lighting base potential map" in which each region (no driving area, driving caution area, driving permitted area) changes dynamically from the start to the end of the live (performance).

[0070] (Step S13) In step S13, the individual potential map generation unit 120 generates a map c = "object-based potential map". As described above, the "object-based potential map" is a map that dynamically changes each region (no-go zone, caution zone, and permitted zone) according to the placement of objects on the stage from the start to the end of the live performance, and is a time-series map with the following three regions (c1 to c3). c1. No-travel zone (red) = proximity to the object placement location c2. Driving caution area (yellow) = Mid-range position of object placement c3. Permissible travel area (blue) = Distance from the object placement location

[0071] Specifically, for example, c1. No-travel zone (red) = Area where the probability of collision with an object is defined as a first threshold Thc1 or higher. c2. Driving caution area (yellow) = Area where the possibility of collision with an object is within the range of the first threshold Thc1 to the second threshold Thc2, which are predetermined. c3. Permissible driving area (blue) = Area where the probability of collision with an object is less than or equal to the predefined second threshold Thc2. For example, this is a map with these area divisions.

[0072] An object, in this context, refers to an object placed on the stage, such as speakers, monitors, or decorative objects.

[0073] Refer to Figure 8 to explain a specific example of map c = object-based potential map. The "object-based potential map" is a time-series map in which each region (no-go zone, caution zone, and permitted zone) is dynamically changed according to the object placement position from the start to the end of the live performance. Figure 8 shows examples of object-based potential maps for four timings (t1 to t4).

[0074] The map in the upper left of Figure 8 (t1) is an example of an "object-based potential map" at time t1. Speakers, monitors, decorative objects, and other items are placed on the stage, some of which are moved or replaced as the live performance progresses.

[0075] An "object-based potential map" is a map in which different colors are assigned to each region according to the area (no-driving area, driving caution area, driving permitted area) determined by the distance from the object's placement location, as shown below. The areas in close proximity to the object placement location are set to red as no-go zones. The mid-range positions of object placement are set to yellow as areas requiring caution while driving. Object placement locations at long distances are set to blue as the permissible travel area.

[0076] The map in the upper right of Figure 8 (t2) is an example of an "object-based potential map" at time t2, a certain period of time after time t1. At time t2, the objects on the stage have moved to or been replaced in a different position than at time t1. Along with this movement or replacement of objects, the settings of the three areas—no entry zone (red), caution zone (yellow), and permitted entry zone (blue)—also change.

[0077] The map in the lower left of Figure 8 (t3) is an example of an "object-based potential map" at time t3, a certain amount of time has elapsed since time t2. At time t3, the objects on the stage have moved to or been replaced in positions different from those at times t1 and t2. Along with this movement or replacement of objects, the settings for the three areas—no entry zone (red), caution zone (yellow), and permitted entry zone (blue)—also change.

[0078] The map in the lower right of Figure 8 (t4) is an example of an "object-based potential map" at time t4, a certain period of time has elapsed since time t3. At time t4, the objects on the stage have moved to or been replaced with objects in different positions than those at times t1-t3. Along with this movement or replacement of objects, the settings for the three areas—no entry zone (red), caution zone (yellow), and permitted entry zone (blue)—also change.

[0079] Thus, the "object-based potential map" is a time-series map in which each area (no-go zone, caution zone, and permitted zone) is dynamically changed according to the placement of objects on the stage from the start time to the end time of the live performance. In step S13, the individual potential map generation unit 120 generates such a map c = "object-based potential map".

[0080] Furthermore, the planned stage object placement data, which indicates the placement positions of objects from the start to the end of the live performance, is pre-generated and recorded in the storage unit 110 as C. Planned stage object placement data 113. In step S13, the individual potential map generation unit 120 acquires the C. Stage object placement planned data 113 from the memory unit 110, and, referring to the acquired data, generates a time-series map c = "object-based potential map" in which each region (no-travel area, caution area, permitted area) changes dynamically from the start to the end of the live (performance).

[0081] As described above, the individual potential map generation unit 120 shown in Figure 4 generates the following three maps in steps S11 to S13. That is, In step S11, map a = performer-based potential map is generated. In step S12, map b = illumination-based potential map is generated. In step S13, a map c = object-based potential map is generated.

[0082] These three individual potential maps generated by the individual potential map generation unit 120, namely, Map a = Performer-based potential map, Map b = Illumination-based potential map, Map c = Object-based potential map, These three maps are input to the potential map synthesis unit 130.

[0083] The potential map synthesis unit 130 performs a synthesis process of the three individual potential maps generated by the individual potential map generation unit 120 to generate a pre-generated potential map 150. Furthermore, the pre-generated potential map 150 is a time-series map in which each region (no-driving region, cautionary driving region, permitted driving region) is dynamically changed from the start to the end of the live (performance).

[0084] The pre-generated potential map 150 generated by the potential map synthesis unit 130 is composed of three individual potential maps generated by the individual potential map generation unit 120, i.e., Map a = Performer-based potential map, Map b = Illumination-based potential map, Map c = Object-based potential map This is generated as composite data that reflects all three of these time-series data.

[0085] A specific example of a pre-generated potential map 150 generated by the potential map synthesis unit 130 is shown in Figure 9.

[0086] Figure 9 shows an example of 150 pre-generated potential maps for four timings (t1-t4), similar to the individual potential maps described with reference to Figures 6-8.

[0087] The pre-generated potential maps for the four timings (t1-t4) shown in Figure 9 are maps generated by combining the four individual potential maps for the same timings (t1-t4) shown in Figures 6-8 for each timing unit.

[0088] The map in the upper left of Figure 9 (t1) is the "pre-generated potential map" at time t1. This "pre-generated potential map" at time t1 is, The performer-based potential map at time t1 shown in Figure 6(t1), The illumination base potential map at time t1 shown in Figure 7(t1), The object-based potential map at time t1 is shown in Figure 8(t1). This map is a composite of these three individual potential maps at the same timing (t1).

[0089] On the stage, there are two performers in the same positions as in Figure 6(t1), and objects such as speakers, monitors, and decorative objects are positioned in the same locations as in Figure 8(t1). The lighting is set to be dark throughout the stage, as in Figure 7(t1).

[0090] The specific synthesis sequence will be described later, but for example, the synthesis process is performed using the following steps. The regions of each individual potential map (driving prohibited area, driving caution area, driving permitted area) are quantified. for example, No-driving zone = 10, Driving caution zone = 5, Driving tolerance range = 0 This quantification process is performed, the numerical values ​​for each region of each individual potential map are added together, and a pre-generated potential map is generated as a composite map based on the summation result.

[0091] For example, a composite map, i.e., a pre-generated potential map, is created with settings such that the region with an added value of 10 or more is a no-driving region, the region with an added value between 5 and 10 is a driving caution region, and the region with an added value less than 5 is a driving permitted region.

[0092] The "pre-generated potential map" shown in Figure 9(t1) is, The performer-based potential map at time t1 shown in Figure 6(t1), The illumination base potential map at time t1 shown in Figure 7(t1), The object-based potential map at time t1 is shown in Figure 8(t1). This map was generated by quantifying and adding the values ​​of each region (driving prohibited region, driving caution region, driving permitted region) of these three individual potential maps at the same timing (t1), and then dividing the regions based on the above added values.

[0093] The "pre-generated potential map" is a map in which the no-go zone, caution zone, and permissible zone are determined by considering the performer's position, lighting conditions (lighting position, brightness, color), and object placement, and the following different colors are assigned to each determined zone. No-go zones are marked in red. The area requiring caution while driving is set to yellow. The permissible driving area is set to blue.

[0094] The map in the upper right of Figure 9 (t2) is an example of a composite map at time t2, which is a certain time after time t1 has elapsed, i.e., a "pre-generated potential map". At time t2, the performer's position, lighting conditions (lighting position, brightness, color), and object placement will be set differently from those at time t1.

[0095] This "pre-generated potential map" at time t2 is, The performer-based potential map at time t2 shown in Figure 6(t2), The illumination base potential map at time t2 is shown in Figure 7(t2), The object-based potential map at time t2 is shown in Figure 8(t2). This map was generated by quantifying and adding the values ​​of each region (driving prohibited region, driving caution region, driving permitted region) of these three individual potential maps at the same timing (t2), and then dividing the regions based on the added values.

[0096] The map in the lower left of Figure 9 (t3) is an example of a composite map of time t3, which is a certain period of time after time t2, i.e., a "pre-generated potential map". At time t3, the performer's position, lighting conditions (lighting position, brightness, color), and object placement will be set differently from those at times t1 and t2.

[0097] This "pre-generated potential map" at time t3 is, The performer-based potential map at time t3 shown in Figure 6(t3), The illumination base potential map at time t3 is shown in Figure 7(t3), The object-based potential map at time t3 is shown in Figure 8(t3). This map was generated by quantifying and adding the values ​​of each region (driving prohibited region, driving caution region, driving permitted region) of these three individual potential maps at the same timing (t3), and then dividing the regions based on the added values.

[0098] The map in the lower right of Figure 9 (t4) is an example of a composite map of time t4, which is a certain period of time after time t3, i.e., a "pre-generated potential map". At time t4, the performer's position, lighting conditions (lighting position, brightness, color), and object placement will be set differently from those at times t1-t3.

[0099] This "pre-generated potential map" at time t4 is, The performer-based potential map at time t4 shown in Figure 6(t4), The illumination base potential map at time t4 is shown in Figure 7(t4), The object-based potential map at time t4 is shown in Figure 8(t4). This map was generated by quantifying and adding the values ​​of each region (driving prohibited region, driving caution region, driving permitted region) of these three individual potential maps at the same timing (t4), and then dividing the regions based on the added values.

[0100] Thus, the potential map synthesis unit 130 combines the three individual potential maps generated by the individual potential map generation unit 120, that is, Map a = Performer-based potential map, Map b = Illumination-based potential map, Map c = Object-based potential map These individual potential maps are combined to generate a pre-generated potential map 150.

[0101] The potential map synthesis unit 130 quantifies each region (driving prohibited region, driving caution region, driving permitted region) of multiple individual potential maps at the same timing, adds them together, performs region division based on the added values, and generates a pre-generated potential map 150.

[0102] The pre-generated potential map 150 generated by the potential map synthesis unit 130 is provided to the travel route generation unit 160, which then determines the travel route of the image acquisition robot 50 from the start to the end of the live performance based on the pre-generated potential map 150.

[0103] In other words, as shown in Figure 10, the route generation unit 160 receives the pre-generated potential map 150 generated by the potential map synthesis unit 130 and generates route information 165 that sets a route for the image-capturing robot 50 to travel in an area that does not collide with performers or objects on the stage from the start to the end of the live performance, and that is not conspicuous due to lighting.

[0104] The route generation unit 160 generates a route that travels only within the permissible travel area in the pre-generated potential map 150, for example.

[0105] The generated travel route is provided to the travel control unit 170, which controls the movement of the image-capturing robot 50, and the travel control unit 170 makes the image-capturing robot 50 move according to the generated travel route information 165. The driving control unit 170 may be configured as an information processing device within the image-capturing robot 50, or as an information processing device capable of communicating with robots outside the image-capturing robot 50.

[0106] In this way, by having the image-capturing robot 50 travel according to the travel route information 165 generated using the pre-generated potential map 150, it becomes possible to have the image-capturing robot 50 travel from the start to the end of the live performance without colliding with performers or objects on the stage, and also select areas that are not conspicuous due to lighting.

[0107] Referring to Figure 11, an example of driving route information 165 generated using a pre-generated potential map 150 will be explained. Figure 11 shows an example of the travel route of the image acquisition robot 50 from time (t3) to (t4).

[0108] The travel route of the image acquisition robot 50 at time (t3) to (t4) shown in Figure 11 is a travel route set to travel only within the permissible travel area within the pre-generated potential map 150 generated by the potential map synthesis unit 130.

[0109] By having the image-capturing robot 50 travel along a route set to travel only within the permissible travel area in the pre-generated potential map 150, it becomes possible to travel through the entire live performance (from start to finish) without colliding with performers or objects on the stage, and to select areas that are not conspicuous due to lighting.

[0110] [3. Sequence of processes executed by the information processing device of Embodiment 1 of this disclosure] Next, the sequence of processes performed by the information processing device of Embodiment 1 of this disclosure will be described.

[0111] The flowcharts shown in Figure 12 and subsequent figures illustrate the sequence of processes performed by the information processing device 100 of this disclosure, which was previously explained with reference to Figure 4. The flowcharts shown in Figures 12 to 15 represent the sequence of processes executed by the individual potential map generation unit 120 of the information processing device 100, and correspond to the following three types of individual potential map generation sequences. (1) Figure 12 = Generation sequence of map a "Performer-based potential map" (2) Figures 13-14 = Generation sequence of map b "Illumination-based potential map" (3) Figure 15 = Generation sequence of map c "Object-based potential map"

[0112] Furthermore, the flowchart shown in Figure 16 is the generation process sequence for the "pre-generated potential map," which is a composite map executed by the potential map synthesis unit 130 of the information processing device 100 shown in Figure 4.

[0113] The processing described below according to the flow can be executed, for example, according to a program stored in the memory of the information processing device, and is performed under the control of a control unit having program execution capabilities, such as a CPU. The details of the processing in the flow shown in Figure 14 will be described in order below.

[0114] [3-(1) Regarding the generation sequence of map a "performer-based potential map"] First, referring to the flowchart shown in Figure 12, we will explain the generation sequence of map a, "performer-based potential map," executed by the individual potential map generation unit 120 of the information processing device 100.

[0115] The process shown in the flowchart in Figure 12 corresponds to the detailed sequence of the process for generating map a = performer-based potential map, which is the process of step S11 executed by the individual potential map generation unit 120, as explained earlier with reference to Figure 4.

[0116] In other words, this is a detailed sequence of the generation process for map a = performer-based potential map, which is a time-series map in which each region (no-travel zone, caution zone, permitted zone) is dynamically changed according to the performer's position from the start to the end of the live performance.

[0117] The following describes the processing of each step in the flow shown in Figure 12. (Step S101) First, in step S101, the data processing unit (individual potential map generation unit 120) of the information processing device 100 obtains the number of performers = na during the live performance period of the live performance for which the performer-based potential map is to be generated. This is executed, for example, as a process to retrieve performer action schedule data generated based on a pre-set live program, that is, "A. Performer action schedule data 111" stored in the storage unit 110 of the information processing device 100 shown in Figure 4.

[0118] (Step S102) Next, in step S102, the data processing unit of the information processing device 100 selects one presenter P to be analyzed.

[0119] (Step S103) Next, in step S103, the data processing unit of the information processing device 100 acquires time-series behavioral data of the performer P being analyzed, from the start to the end of the live performance.

[0120] This process is also executed, for example, as a process of acquiring performer action schedule data generated based on a pre-set live program, that is, "A. Performer action schedule data 111" stored in the storage unit 110 of the information processing device 100 shown in Figure 4.

[0121] (Step S104) Next, in step S104, the data processing unit of the information processing device 100 generates a potential map based on the time-series behavioral data of the performer P being analyzed, from the start to the end of the live performance.

[0122] This process is executed according to the procedure previously explained with reference to Figures 4 and 6. It determines three regions (no driving region, caution region, and driving permitted region) based on the distance from the position of the performer P being analyzed at each time point from the start to the end of the live performance, and then assigns different colors to each region according to the determined region.

[0123] Specifically, the following area and color settings will be applied. The area in close proximity to performer P's position will be marked in red as a no-go zone. The mid-range position of performer P is set to yellow as a driving caution area. The distant position of performer P is set to blue as the permissible driving area.

[0124] This process generates a performer-based potential map corresponding to one performer P being analyzed.

[0125] (Step S105) Next, in step S105, the data processing unit of the information processing device 100 determines whether or not there are any presenters whose data has not yet been processed for analysis. In other words, it is determined whether the generation of all performer-based potential maps for the number of performers na obtained in step S101 has been completed.

[0126] If there are any performers who have not yet been processed, the determination in step S105 will be Yes. In this case, the processing from step S102 onwards will be performed for the unprocessed performers. On the other hand, if there are no unprocessed performers, that is, if it is determined that the generation of all performer-based potential maps for the number of performers na obtained in step S101 has been completed, the determination in step S105 will be No. In this case, proceed to step S106.

[0127] (Step S106) Once the generation of all performer-based potential maps is complete, the data processing unit of the information processing device 100 executes the following processes:

[0128] First, in step S106, the setting range for each of the na individual potential maps corresponding to all presenters 1 to na is quantified. For example, No-driving zone = 10, Driving caution zone = 5, Driving tolerance range = 0 This involves quantifying the data at the domain level.

[0129] (Step S107) Next, in step S107, the data processing unit of the information processing device 100 calculates the sum of region-corresponding values ​​for each region by adding up the region-corresponding values ​​of the potential maps for all performers 1 to na.

[0130] For example, suppose there are 3 performers, and individual performer-based potential maps (m1~m3) corresponding to the 3 performers have been generated, and the domain setting for a certain stage position (x1, y1) at a certain time tx is as follows. Map m1 value = 5 (Caution required when driving) Map m2 value = 5 (Caution advised area for driving) Map m3 value = 0 (driving tolerance zone) In this case, the sum is 5 + 5 + 0 = 10. This addition process is performed for all maps. Note that the maps are time-series data, meaning the process is performed for all stage positions on the map at all points in time.

[0131] (Step S108) Next, in step S108, the data processing unit of the information processing device 100 resets the region divisions based on the sum of the region correspondences of the potential maps of all performers 1 to na.

[0132] For example, the area will be reconfigured according to the following rules. If the added value is 10 points or more, it means driving is prohibited. A score of 5 or more indicates a driving caution zone. A score of less than 5 points = acceptable driving range. For example, the area can be reconfigured according to the rules above.

[0133] (Step S109) Next, in step S109, the data processing unit of the information processing device 100 outputs the potential map, whose region divisions were reset in step S108, to the potential map synthesis unit 130 as a "performer-based potential map".

[0134] The above is a detailed sequence of the process for generating map a = performer-based potential map executed by the individual potential map generation unit 120. This process generates map a = performer-based potential map, which is a time-series map in which each region (no-travel zone, caution zone, permitted zone) changes dynamically according to the performer's position from the start to the end of the live performance.

[0135] [3-(2) Regarding the generation sequence of map b "Illumination-based potential map"] Next, with reference to the flowcharts shown in Figures 13 to 14, the generation sequence of map b, "Illumination-Based Potential Map," executed by the individual potential map generation unit 120 of the information processing device 100 will be explained.

[0136] The process shown in the flowcharts in Figures 13 and 14 corresponds to the detailed sequence of the map b = illumination-based potential map generation process, which is the process of step S12 executed by the individual potential map generation unit 120, as explained earlier with reference to Figure 4.

[0137] In other words, this is a detailed sequence of the generation process for map b = lighting-based potential map, which is a time-series map in which each area (no-driving area, caution area, permitted driving area) is dynamically changed according to the lighting conditions (lighting position, brightness, color, etc.) from the start to the end of the live performance.

[0138] The following describes the processing of each step in the flow shown in Figures 13 and 14. (Step S121) First, in step S121, the data processing unit (individual potential map generation unit 120) of the information processing device 100 obtains the number of illumination position division regions = nb during the live execution period of the live performance for which the illumination base potential map is to be generated.

[0139] This process is executed, for example, by retrieving lighting control schedule data generated based on a pre-set live program, that is, "B. Stage lighting control schedule data 112" stored in the storage unit 110 of the information processing device 100 shown in Figure 4.

[0140] (Step S122) Next, in step S122, the data processing unit of the information processing device 100 selects one illumination position division region Q to be analyzed.

[0141] (Step S123) Next, in step S123, the data processing unit of the information processing device 100 acquires time-series illumination brightness information for one illumination position division area Q selected as the target of analysis, from the start to the end of the live broadcast.

[0142] This, for example, is executed as a process to acquire lighting control schedule data generated based on a pre-set live program, that is, "B. Stage lighting control schedule data 112" stored in the storage unit 110 of the information processing device 100 shown in Figure 4.

[0143] (Step S124) Next, in step S124, the data processing unit of the information processing device 100 generates a potential map based on the time-series illumination brightness data from the start to the end of the live session for one illumination position division region Q selected as the target of analysis.

[0144] This process is executed according to the process previously explained with reference to Figures 4 and 7. It determines three areas (no driving area, driving caution area, and driving permitted area) based on the brightness of the lighting position area Q at each time from the start to the end of the live event, and then assigns different colors to each area according to the determined area.

[0145] Specifically, the following area and color settings will be applied. If the brightness of the lighting position area Q is equal to or greater than the specified threshold Thd1, it will be set to red as a no-travel area. If the brightness of the lighting position area Q is within the specified threshold range of Thd1 to Thd2, it will be set to yellow as a driving caution area. If the brightness of the illumination position classification area Q is below the specified threshold Thd2, the area is set to blue as a drivable area.

[0146] This process generates an illumination brightness-based potential map corresponding to one illumination position region Q.

[0147] (Step S125) Next, in step S125, the data processing unit of the information processing device 100 determines whether or not there are any illumination position division areas that have not been analyzed. In other words, it is determined whether the generation of all illumination brightness-based potential maps for the number of illumination position division regions nb obtained in step S121 has been completed.

[0148] If there are unprocessed lighting position division areas, the determination in step S125 is Yes. In this case, the processing from step S122 onwards is performed on the unprocessed lighting position division areas. On the other hand, if there are no unprocessed illumination position division regions, that is, if it is determined that the generation of all illumination brightness-based potential maps for all illumination position division regions nb acquired in step S121 has been completed, the determination in step S125 will be No. In this case, proceed to step S126.

[0149] (Step S126) Once the generation of all illumination brightness-based potential maps is complete, the data processing unit of the information processing device 100 executes the following processes:

[0150] First, in step S126, one illumination position division region Q to be analyzed is selected.

[0151] (Step S127) Next, in step S127, the data processing unit of the information processing device 100 acquires time-series color information of the illumination from the start to the end of the live illumination of one illumination position division area Q selected as the target of analysis.

[0152] This, for example, is executed as a process to acquire lighting control schedule data generated based on a pre-set live program, that is, "B. Stage lighting control schedule data 112" stored in the storage unit 110 of the information processing device 100 shown in Figure 4.

[0153] (Step S128) Next, in step S128, the data processing unit of the information processing device 100 generates a potential map based on the time-series illumination color data from the start to the end of the live broadcast for one illumination position division region Q selected as the target of analysis.

[0154] This process is executed according to the process previously explained with reference to Figures 4 and 7. It determines three areas (no driving area, driving caution area, and driving permitted area) based on the illumination light color of the lighting position division area Q at each time from the start to the end of the live event, and then assigns different colors to each area according to the determined area.

[0155] Specifically, the following area and color settings will be applied. If the illumination color of the illumination position area Q is different from that of the image-capturing robot, it will be set to red as a no-travel area. Even if the illumination color of the illumination position classification area Q is different from that of the image-capturing robot, if it is not a similar color, it will be set to yellow as a driving caution area. If the illumination color of the illumination position classification area Q is similar to the color of the image-capturing robot, the area is set to blue as the permissible travel area.

[0156] This process generates a color-based potential map corresponding to one illumination position region Q.

[0157] (Step S129) Next, in step S129, the data processing unit of the information processing device 100 determines whether or not there are any illumination position classification areas that have not been analyzed. In other words, it is determined whether the generation of all illumination color-based potential maps for the number of illumination position division regions nb obtained in step S121 has been completed.

[0158] If there are unprocessed lighting position division areas, the determination in step S129 is Yes. In this case, the processing from step S126 onwards is performed on the unprocessed lighting position division areas. On the other hand, if there are no unprocessed illumination position division regions, that is, if it is determined that the generation of all illumination color base potential maps for the illumination position division region nb acquired in step S121 has been completed, the determination in step S129 will be No. In this case, proceed to step S131.

[0159] (Step S131) Once the generation of all illumination color-based potential maps is complete, the data processing unit of the information processing device 100 executes the following processes:

[0160] In step S131, the data processing unit of the information processing device 100 quantifies the setting ranges of the illumination brightness-based potential map and the illumination color-based potential map. For example, No-driving zone = 10, Driving caution zone = 5, Driving tolerance range = 0 This involves quantifying the data at the domain level.

[0161] (Step S132) Next, in step S132, the data processing unit of the information processing device 100 adds the values ​​corresponding to the setting areas of the illumination brightness-based potential map and the illumination color-based potential map for each area to calculate the sum of the area-corresponding values.

[0162] For example, suppose the region setting for a certain stage position (x1, y1) at a certain time tx is as follows: The value of the lighting brightness-based potential map m1 is 5 (drive caution area). The value of the lighting color base potential map m2 = 0 (driving tolerance range) In this case, the sum is 5 + 0 = 5. This addition process is performed for all maps. Note that the maps are time-series data, meaning the process is performed for all stage positions on the map at all points in time.

[0163] (Step S133) Next, in step S133, the data processing unit of the information processing device 100 resets the region divisions based on the sum of the region-corresponding values ​​of the illumination brightness-based potential map and the illumination color-based potential map.

[0164] For example, the area will be reconfigured according to the following rules. If the added value is 10 points or more, it means driving is prohibited. A score of 5 or more indicates a driving caution zone. A score of less than 5 points = acceptable driving range. For example, the area can be reconfigured according to the rules above.

[0165] (Step S134) Next, in step S134, the data processing unit of the information processing device 100 outputs the potential map, whose region divisions were reset in step S133, as an "illumination base potential map" to the potential map synthesis unit 130.

[0166] The above is a detailed sequence of the map b = illumination-based potential map generation process performed by the individual potential map generation unit 120. This process generates map b = lighting-based potential map, which is a time-series map in which each area (no-driving area, caution area, permitted driving area) is dynamically changed according to the lighting conditions (lighting position, brightness, color, etc.) from the start to the end of the live performance.

[0167] [3-(3) Regarding the generation sequence of map c "Object-based potential map"] First, referring to the flowchart shown in Figure 15, we will explain the generation sequence of the map c "object-based potential map" executed by the individual potential map generation unit 120 of the information processing device 100.

[0168] The process shown in the flowchart in Figure 15 corresponds to the detailed sequence of the map c = object-based potential map generation process, which is the process of step S13 executed by the individual potential map generation unit 120, as explained earlier with reference to Figure 4.

[0169] In other words, this is a detailed sequence of the generation process for map c = object-based potential map, which is a time-series map in which each region (no-travel zone, caution zone, and permitted zone) is dynamically changed according to the placement of objects on the stage from the start to the end of the live performance.

[0170] An object, in this context, refers to an object placed on the stage, such as speakers, monitors, or decorative objects.

[0171] The following describes the processing of each step in the flow shown in Figure 15. (Step S151) First, in step S151, the data processing unit (individual potential map generation unit 120) of the information processing device 100 obtains the number of objects = nc during the live execution period of the live performance that will be the target of generating the object-based potential map. This is executed, for example, as a process to retrieve data from the planned stage object placement data, which is generated based on a pre-configured live program, i.e., "C. Planned stage object placement data 113" stored in the storage unit 110 of the information processing device 100 shown in Figure 4.

[0172] (Step S152) Next, in step S152, the data processing unit of the information processing device 100 selects one object Ob to be analyzed.

[0173] (Step S153) Next, in step S153, the data processing unit of the information processing device 100 acquires time-series placement data of the object Ob to be analyzed from the start to the end of its live performance.

[0174] This process is also executed, for example, as a process of obtaining data from the planned stage object placement data, which is generated based on a pre-configured live program, that is, "C. Planned stage object placement data 113" stored in the storage unit 110 of the information processing device 100 shown in Figure 4.

[0175] (Step S154) Next, in step S154, the data processing unit of the information processing device 100 generates a potential map based on the time-series placement data of the object Ob to be analyzed from the start to the end of its live performance.

[0176] This process is executed according to the procedure previously explained with reference to Figures 4 and 8. It determines three regions (no-travel zone, caution zone, and permitted zone) based on the distance from the position of the object Ob to be analyzed at each time point from the start to the end of the live session, and then assigns different colors to each region according to the determined region.

[0177] Specifically, the following area and color settings will be applied. The areas in close proximity to the placement of object Ob will be set to red as no-go zones. The mid-range positions of object Ob are set to yellow as areas requiring caution while driving. The distant positions of object Ob are set to blue as the permissible travel area.

[0178] This process generates an object-based potential map corresponding to a single object Ob being analyzed.

[0179] (Step S155) Next, in step S155, the data processing unit of the information processing device 100 determines whether or not there are any unprocessed objects. In other words, it is determined whether the generation of all object-based potential maps for the number of objects nc obtained in step S151 has been completed.

[0180] If there are unprocessed objects, the determination in step S155 is Yes. In this case, the processing from step S152 onwards is performed on the unprocessed objects. On the other hand, if there are no unprocessed objects, that is, if it is determined that the generation of all object-based potential maps for the number of objects nc obtained in step S151 has been completed, the determination in step S155 will be No. In this case, proceed to step S156.

[0181] (Step S156) When the generation of all object-based potential maps is completed, the data processing unit of the information processing apparatus 100 executes the processes below step S156.

[0182] First, in step S156, the setting areas of each of the nc potential maps corresponding to the individual object correspondences for all objects 1 to nc are digitized. For example, Travel prohibited area = 10, Travel caution area = 5, Travel allowed area = 0 Such digitization is performed in units of areas.

[0183] (Step S157) Next, in step S157, the data processing unit of the information processing apparatus 100 adds up the numerical values corresponding to the areas of the potential maps of all objects 1 to nc for each area to calculate the added value corresponding to the area.

[0184] For example, if the number of objects = 3 and the individual object-based potential maps (m1 to m3) corresponding to the three objects have been generated, and the area setting at a certain stage position (x1, y1) at a certain time tx is as follows. The numerical value of map m1 = 5 (travel caution area) The numerical value of map m2 = 5 (travel caution area) The numerical value of map m3 = 0 (travel allowed area) In this case, the added value is 5 + 5 + 0 = 10. Such addition processing is executed for all maps. Note that the maps are time-series data, that is, it is executed for all stage positions of all maps at all times.

[0185] (Step S158) Next, in step S158, the data processing unit of the information processing apparatus 100 re-sets the area classification based on the added values corresponding to the areas of the potential maps of all objects 1 to nc.

[0186] For example, the area will be reconfigured according to the following rules. If the added value is 10 points or more, it means driving is prohibited. A score of 5 or more indicates a driving caution zone. A score of less than 5 points = acceptable driving range. For example, the area can be reconfigured according to the rules above.

[0187] (Step S159) Next, in step S159, the data processing unit of the information processing device 100 outputs the potential map, whose region divisions were reset in step S158, as an "object-based potential map" to the potential map synthesis unit 130.

[0188] The above is a detailed sequence of the map c = object-based potential map generation process executed by the individual potential map generation unit 120. This process generates a time-series map c = object-based potential map, which dynamically changes each region (no-travel zone, caution zone, and permitted zone) according to the object placement positions from the start to the end of the live (performance).

[0189] [3-(4) Regarding the generation sequence of pre-generated potential maps synthesized from individual potential maps] Next, with reference to the flowchart shown in Figure 16, the generation sequence of pre-generated potential maps executed by the potential map synthesis unit 130 of the information processing device 100 will be described.

[0190] As explained earlier with reference to Figure 4, the potential map synthesis unit 130 combines the three individual potential maps generated by the individual potential map generation unit 120, that is, Map a = Performer-based potential map, Map b = Illumination-based potential map, Map c = Object-based potential map These three potential maps are combined to generate a pre-generated potential map, which is a composite data set that reflects all of the time-series data of these three potential maps.

[0191] The flowchart shown in Figure 16 represents the generation sequence of pre-generated potential maps performed by the potential map synthesis unit 130. The processing of each step in the flowchart shown in Figure 16 will be explained sequentially below.

[0192] (Step S171) First, in step S171, the data processing unit (potential map synthesis unit 130) of the information processing device 100 shown in Figure 4 processes the three individual potential maps generated by the individual potential map generation unit 120, that is, Map a = Performer-based potential map, Map b = Illumination-based potential map, Map c = Object-based potential map The setting range for each of these three potential maps is quantified. For example, No-driving zone = 10, Driving caution zone = 5, Driving tolerance range = 0 This involves quantifying the data at the domain level.

[0193] (Step S172) Next, in step S172, the data processing unit of the information processing device 100 generates three individual potential maps, namely, Map a = Performer-based potential map, Map b = Illumination-based potential map, Map c = Object-based potential map The numerical values ​​corresponding to the respective setting regions of these three potential maps are added together for each region to calculate the sum of the region-specific values.

[0194] For example, suppose the region setting for a certain stage position (x1, y1) at a certain time tx is as follows: Map a = Numerical value of the actor-based potential map = 5 (Driving caution area) Map b = Numerical value of the lighting-based potential map = 0 (Driving permitted area) Map c = Numerical value of the object-based potential map = 5 (Driving caution area) In this case, the added value is 5 + 0 + 5 = 10. Such addition processing is performed for all maps. Note that the maps are time-series data, that is, it is performed for all stage positions of the maps at all times.

[0195] (Step S173) Next, the data processing unit of the information processing apparatus 100, in step S173, for the three individual potential maps, namely Map a = Actor-based potential map, Map b = Lighting-based potential map, Map c = Object-based potential map The data processing unit of the information processing apparatus 100 reconfigures the area classification based on the added values of the area correspondence of these three potential maps.

[0196] For example, the area reconfiguration is performed according to the following rules. Added value of 10 or more = Driving prohibited area, Added value of 5 or more = Driving caution area, Added value less than 5 = Driving permitted area, For example, the area reconfiguration is performed according to the above rules.

[0197] (Step S174) Next, the data processing unit of the information processing apparatus 100, in step S174, generates the potential map with the area classification reconfigured in step S173 as a composite map, that is, a "pre-generated potential map".

[0198] The above is the detailed sequence of the generation process of the pre-generated potential map 150 executed by the potential map synthesis unit 130. Thus, the potential map synthesis unit 130 combines the three individual potential maps generated by the individual potential map generation unit 120, that is, Map a = Performer-based potential map, Map b = Illumination-based potential map, Map c = Object-based potential map These individual potential maps are combined to generate a pre-generated potential map 150.

[0199] The pre-generated potential map 150 generated by the potential map synthesis unit 130 is provided to the travel route generation unit 160, which then determines the travel route of the image acquisition robot 50 from the start to the end of the live performance based on the pre-generated potential map 150.

[0200] In other words, as explained earlier with reference to Figure 10, the travel route generation unit 160 receives the pre-generated potential map 150 generated by the potential map synthesis unit 130 and generates travel route information 165 that sets a route for the image-capturing robot 50 to travel in an area that does not collide with performers or objects on the stage from the start to the end of the live performance, and that is not conspicuous due to lighting. For example, a driving route is generated that travels only within the permissible driving area in the pre-generated potential map 150.

[0201] The generated travel route is provided to the travel control unit 170, which controls the movement of the image-capturing robot 50, and the travel control unit 170 makes the image-capturing robot 50 move according to the generated travel route information 165. In this way, by having the image-capturing robot 50 travel according to the travel route information 165 generated using the pre-generated potential map 150, it becomes possible to have the image-capturing robot 50 travel from the start to the end of the live performance without colliding with performers or objects on the stage, and also select areas that are not conspicuous due to lighting.

[0202] Furthermore, the travel route of the image-capturing robot 50, which is generated using the pre-generated potential map 150, can be displayed as a simulated image on the display unit of the information processing device 100, for example.

[0203] The data processing unit of the information processing device 100 generates simulation data to display the planned route of the image acquisition robot 50 from the start to the end of the live event, based on the travel route information 165 generated using the pre-generated potential map 150. A specific example of this simulation data is shown in Figure 17.

[0204] As shown in Figure 17, users such as robot control operators can check the travel position of the image-capturing robot 50 at each time point from the start (ts) to the end (te) of the live session by moving the slider left or right. Figure 17 shows an example of display data for simulation data, specifically the travel position of the image-capturing robot 50 at time t3 and time t4.

[0205] By referring to this simulation data, users such as robot control operators can verify the travel route generated using the pre-generated potential map 150.

[0206] [4. (Example 2) Details of the configuration and processing of the information processing device that creates a map considering the gaze of the audience and the gaze of the television cameras on the audience side] Next, as Embodiment 2 of this disclosure, the configuration and processing details of an information processing device that creates a map taking into account the gaze of the audience and the gaze of television cameras on the audience side will be described.

[0207] In the above-described embodiment 1, the configuration involved generating and utilizing the following three individual maps when generating a potential map to determine the travel route of the image-capturing robot 50. Map a = Performer-based potential map, Map b = Illumination-based potential map, Map c = Object-based potential map,

[0208] In the above-described embodiment 1, the three individual potential maps were generated individually, then combined to generate a pre-generated potential map, and the travel route of the image acquisition robot 50 was determined using the generated pre-generated potential map. For example, the configuration involved selecting a travelable area within a pre-generated potential map to determine the travel route of the image-capturing robot 50.

[0209] The second embodiment described below adds to the processing of the first embodiment described above by creating a fourth individual potential map that takes into account the gaze of the audience and the gaze of the television cameras on the audience side, and then generates a pre-generated potential map that also takes this fourth individual potential map into account.

[0210] Referring to Figure 18, we will explain the line of sight of the audience and the line of sight of the television camera on the audience side that are considered in this embodiment 2.

[0211] The image-capture robot 50 shown in Figure 18, like the image-capture robot 50 shown in Figure 1 described earlier, is a mobile device equipped with a camera, i.e., a mobile robot, that moves around the stage and captures the performer's performance 20 from various angles.

[0212] However, as shown in Figure 18, when the image-capturing robot 50 is, for example, photographing the performer 20 from the front, it may get in the way of the audience 30 on the audience side or the TV camera 31 and the performer 20.

[0213] In this situation, even if the audience 30 turns their gaze towards the performer 20, the image-capturing robot 50 enters their field of view, making it difficult to see the performer 20. The same applies to the TV camera 31 on the audience side; even if the TV camera 31 tries to film the performer 20 and directs its camera direction (line of sight) towards the performer, the image capture robot 50 enters the captured image, interfering with the filming of the performer 20. The following Example 2 is an example that solves this problem.

[0214] Figure 19 shows an example of a live music venue where the processing of this second embodiment takes place. As shown in Figure 19, the stage 10 where the live performance actually takes place is equipped with speakers 12, monitors 13, and various decorative objects 14, and the performers 20 are also illuminated with lighting 11.

[0215] The performers 20 move around the stage in accordance with the progress of the live performance, and the position, brightness, and color of the lighting 11 change according to their movements. The decorative objects 14 are also changed in various ways.

[0216] Furthermore, as shown in Figure 19, there are many spectators 30 on the audience side, and TV cameras 31 are also positioned to film the live performance of the performers 20. In this embodiment 2, a potential map is generated that takes into account the viewpoint positions of some of the audience members 30, namely the priority audience members 35 shown in the figure, and the viewpoint position of the TV camera 31.

[0217] If a potential map is generated that takes into account the viewpoint positions of all spectators, the entire stage will be set as a no-go zone, resulting in a potential map where the image-capturing robot 50 can barely move. Therefore, processing is performed that takes into account the viewpoint positions of some of the priority spectators 35.

[0218] The priority seating area 35 shown in the diagram is, for example, a seat reserved for staff or a designated seat reserved for an important guest. In this embodiment 2, in addition to the processing described in embodiment 1, a potential map is generated that takes into account the viewpoint position of the priority audience member 35 and the viewpoint position of the TV camera 31. The details of this embodiment 2 will be explained with reference to Figure 20 and subsequent figures.

[0219] Figure 20 shows an example configuration of the information processing device 100b of Embodiment 2 of the present disclosure. Furthermore, this information processing device 100b may be configured inside a mobile device that travels on the stage shown in Figures 1 to 3, that is, an image-capturing robot 50 equipped with a camera, or it may be a device that can communicate with the image-capturing robot 50, or it may be a device independent of the image-capturing robot 50.

[0220] The configuration of the information processing device 100b shown in Figure 20 will be explained below. As shown in Figure 20, the information processing device 100b includes a storage unit 110, an individual potential map generation unit 120, a potential map synthesis unit 130, and a travel route generation unit 160. These basic configurations are the same as those of the information processing device 100 described earlier with reference to Figure 4.

[0221] Furthermore, the information processing device 100b shown in Figure 20 may be configured inside the mobile device that travels on the stage shown in Figures 1 to 3, that is, the image-capturing robot 50 equipped with a camera, or it may be a device independent of the image-capturing robot 50, such as a device (server) that can communicate with the image-capturing robot 50, as explained earlier with reference to Figure 5.

[0222] The memory unit 110 stores the following four pieces of data: A. Performer's activity schedule data 111, B. Stage lighting control planned data 112, C. Stage object placement plan data 113, D. Priority audience, TV camera viewpoint position data 114,

[0223] These four scheduled data points are data that are prepared in advance before the start of the live performance and stored in the memory unit 110. In other words, they are scheduled data that are prepared in advance according to a program such as a live performance schedule that is prepared before the start of the live performance.

[0224] Data A to C are the same data as those described in the previous Example 1 with reference to Figure 4. That is, A. Performer movement schedule data 111 is time-series position data of performers moving around on stage during the live performance. In other words, it is time-series position data of performers from the start to the end of the live performance.

[0225] B. Stage lighting control schedule data 112 is time-series data of lighting control information, including lighting setting information such as lighting position, brightness, and color from the start to the end of the live performance. C. Stage object placement plan data 113 is time-series data of stage object placement information, including the placement positions of objects on the stage from the start to the end of the live performance. Stage objects include speakers, monitors, decorative objects, and other items placed on the stage.

[0226] In this embodiment 2, in addition to data A to C, D. Priority audience, TV camera viewpoint position data 114, This data D is stored in the storage unit 110.

[0227] D. Priority audience and TV camera viewpoint position data 114 is data on the viewpoint positions of priority audience members in the audience seating area and the viewpoint positions of TV cameras from the start to the end of the live performance.

[0228] Furthermore, if the viewpoints of the priority audience and the TV cameras change to various positions from the start to the end of the live performance, then D. Priority audience and TV camera viewpoint position data 114 will be time-series data that changes dynamically over time. However, if the viewpoint of the priority audience and the viewpoint of the TV camera remain in the same position from the start to the end of the live performance, then D. Priority audience and TV camera viewpoint position data 114 can be treated as a single fixed data.

[0229] The data A to D stored in this memory unit 110 are used in the individual potential map generation unit 120. The individual potential map generation unit 120 uses the four types of data A, B, C, and D individually to generate the following four types of individual potential maps. Map a = Performer-based potential map, Map b = Illumination-based potential map, Map c = Object-based potential map, Map d = Priority audience, TV camera viewpoint position-based potential map.

[0230] Maps a to c are the individual potential maps described earlier in Example 1. Map a, the performer-based potential map, is a time-series map in which each region (no-travel zone, caution zone, permitted zone) is dynamically changed according to the performer's position from the start to the end of the live performance. Map b, the lighting-based potential map, is a time-series map in which each area (no-driving area, caution area, permitted driving area) is dynamically changed according to the lighting conditions (lighting position, brightness, color, etc.) from the start to the end of a live performance. Map c, the object-based potential map, is a time-series map in which each region (no-go zone, caution zone, and permitted zone) changes dynamically according to the object placement positions from the start to the end of the live (performance).

[0231] Map d = Priority Spectator, TV Camera Viewpoint-Based Potential Map is a map that defines various areas (no-driving area, caution area, permitted driving area) according to the priority spectator and TV camera viewpoint positions on the spectator side. Furthermore, this map is time-series data in which the designated areas (no-driving areas, driving caution areas, driving permitted areas) dynamically change over time, as the viewpoints of priority spectators and TV cameras change to various positions from the start to the end of the live performance.

[0232] However, if the viewpoint of the priority audience and the viewpoint of the TV camera remain in the same position from the start to the end of the live performance, the set areas (no-driving area, caution area, permitted driving area) will not change dynamically, resulting in a single map.

[0233] In this embodiment 2, the individual potential map generation unit 120 generates the following four maps in steps S11 to S14 shown in Figure 20. That is, In step S11, map a = performer-based potential map is generated. In step S12, map b = illumination-based potential map is generated. In step S13, a map c = object-based potential map is generated. In step S14, map d = priority audience, TV camera viewpoint position-based potential map is generated.

[0234] As shown in Figure 20, in step S14, the individual potential map generation unit 120 divides the area from the near-field to the far-field of the straight line connecting the priority audience viewpoint position and the TV camera viewpoint position to the center of the stage into a no-travel area (red), a travel caution area (yellow), and a travel permitted area (blue). Map d = Priority audience, TV camera viewpoint position-based potential map Generates.

[0235] Figure 21 shows a specific example of "map d = priority audience, TV camera viewpoint position-based potential map" generated by the individual potential map generation unit 120.

[0236] "Map d = Priority Spectator, TV Camera Viewpoint-Based Potential Map" is a map that defines various areas (no-driving areas, cautionary driving areas, permitted driving areas) according to the priority spectator and TV camera viewpoint locations on the spectator side. As shown in Figure 21, a map is generated that divides the area from the priority audience viewpoint and the straight line connecting the TV camera viewpoint to the center of the stage, from near to far, into no-driving zones (red), driving caution zones (yellow), and driving permitted zones (blue).

[0237] As mentioned above, the setting areas (no-driving areas, caution areas, and permitted driving areas) of this "map d = priority spectator, TV camera viewpoint position-based potential map" can be either a dynamically changing time-series map or a single fixed map where the setting areas do not change.

[0238] In this way, the individual potential map generation unit 120 generates the following four individual potential maps. Map a = Performer-based potential map, Map b = Illumination-based potential map, Map c = Object-based potential map, Map d = Priority audience, TV camera viewpoint position-based potential map.

[0239] The four individual potential maps generated by the individual potential map generation unit 120 are input to the potential map synthesis unit 130.

[0240] The potential map synthesis unit 130 performs a synthesis process of the four individual potential maps generated by the individual potential map generation unit 120 to generate pre-generated potential maps b and 150b. Furthermore, the pre-generated potential maps b and 150b are time-series maps in which each region (no-driving region, cautionary driving region, and permitted driving region) is dynamically changed from the start to the end of the live (performance).

[0241] In this embodiment 2, the "pre-generated potential map b" is a map in which the no-travel zone, travel caution zone, and travel permitted zone are determined by considering the performer's position, lighting conditions (lighting position, brightness, color), object placement, and both the preferred audience viewpoint and the TV camera viewpoint, and the following different colors are assigned to each determined zone. No-go zones are marked in red. The area requiring caution while driving is set to yellow. The permissible driving area is set to blue.

[0242] The pre-generated potential maps b and 150b generated by the potential map synthesis unit 130 are provided to the travel route generation unit 160, which then determines the travel route of the image acquisition robot 50 from the start to the end of the live performance based on the pre-generated potential maps b and 150b.

[0243] The route generation unit 160 receives pre-generated potential maps b and 150b generated by the potential map synthesis unit 130 and generates route information 165 that sets a route for the image-capture robot 50 to travel in, selecting areas that will not collide with performers or objects on the stage from the start to the end of the live performance, areas that will not be conspicuous due to lighting, and areas that will not interfere with the line of sight of priority audience members 35 or the filming of the TV camera 31.

[0244] The route generation unit 160 generates a route that travels only within the permissible travel areas in the pre-generated potential maps b and 150b, for example.

[0245] The generated travel route is provided to the travel control unit 170, which controls the movement of the image-capturing robot 50, and the travel control unit 170 makes the image-capturing robot 50 move according to the generated travel route information 165.

[0246] Thus, by having the image-capturing robot 50 travel according to the travel route information 165 generated using the pre-generated potential maps b, 150b generated in this embodiment 2, it becomes possible to have the image-capturing robot 50 travel from the start to the end of the live performance without colliding with performers or objects on the stage, and also select areas that are not conspicuous due to lighting, and areas that do not interfere with the line of sight of the priority audience 35 or the filming of the TV camera 31.

[0247] Next, referring to the flow shown in Figure 22, we will explain the generation sequence of map d, "Priority Audience, TV Camera Viewpoint Position-Based Potential Map," executed by the individual potential map generation unit 120 of the information processing device 100.

[0248] The process shown in the flowchart in Figure 22 corresponds to the detailed sequence of the process in step S14, which is performed by the individual potential map generation unit 120 as explained earlier with reference to Figure 20.

[0249] In other words, it is a detailed sequence of processes for generating a map in which each area (no-driving area, caution area, permitted driving area) is defined according to the viewpoints of priority audience members and TV cameras from the start to the end of the live performance. The following describes the processing of each step in the flow shown in Figure 22.

[0250] (Step S201) First, in step S201, the data processing unit (individual potential map generation unit 120) of the information processing device 100 obtains the number of priority audience members and TV camera viewpoints = nd during the live performance period of the live performance for which the priority audience and TV camera viewpoint-based potential map will be generated.

[0251] This is executed, for example, as a process to acquire data from "D. Priority Audience, TV Camera Viewpoint Position Data 114" stored in the storage unit 110 of the information processing device 100 shown in Figure 20, which is generated based on a pre-set live program.

[0252] (Step S202) Next, in step S202, the data processing unit of the information processing device 100 selects one preferred audience viewpoint or TV camera viewpoint S to be analyzed.

[0253] (Step S203) Next, in step S203, the data processing unit of the information processing device 100 acquires time-series position data of the priority audience viewpoint or TV camera viewpoint S to be analyzed, from the start to the end of the live broadcast.

[0254] This process is also executed, for example, by acquiring data from the priority audience viewpoint and TV camera viewpoint position data generated based on a pre-configured live program, i.e., "D. Priority audience, TV camera viewpoint viewpoint position data 114" stored in the storage unit 110 of the information processing device 100 shown in Figure 4.

[0255] As mentioned above, the time-series position data from the priority audience viewpoint or TV camera viewpoint S from the start to the end of the live performance may be data that changes dynamically over time, or it may be a single, fixed data point that does not change.

[0256] (Step S204) Next, in step S204, the data processing unit of the information processing device 100 generates a potential map based on the positional data of the preferred audience viewpoint or TV camera viewpoint S selected as the target of analysis, from the start to the end of the live broadcast.

[0257] For example, a map like the one previously explained with reference to Figure 21 is generated, which sets out different areas (no-driving areas, areas requiring caution when driving, and areas where driving is permitted) according to the priority spectators on the spectator side and the TV camera viewpoint. As mentioned above, the setting areas (no-driving areas, caution areas, and permitted driving areas) of this "map d = priority spectator, TV camera viewpoint position-based potential map" can be either a dynamically changing time-series map or a single fixed map where the setting areas do not change.

[0258] This process generates a priority audience or TV camera viewpoint-based potential map corresponding to the single priority audience viewpoint or TV camera viewpoint S that was analyzed.

[0259] (Step S205) Next, in step S205, the data processing unit of the information processing device 100 determines whether there are any priority audience viewpoints or TV camera viewpoints that have not yet been analyzed and processed. In other words, it is determined whether the generation of potential maps based on all priority audiences and TV camera viewpoints (nd) obtained in step S201 has been completed.

[0260] If there are any unprocessed priority audience viewpoints or TV camera viewpoints, the determination in step S205 is Yes. In this case, the processing from step S202 onwards is performed for the unprocessed priority audience viewpoints or TV camera viewpoints. On the other hand, if there are no unprocessed priority audience viewpoints or TV camera viewpoints, that is, if it is determined that the generation of all priority audience and TV camera viewpoint-based potential maps for the number of priority audience and TV camera viewpoints nd obtained in step S201 has been completed, then the determination in step S205 is No. In this case, proceed to step S206.

[0261] (Step S206) Once the generation of all priority audience and TV camera viewpoint-based potential maps is complete, the data processing unit of the information processing device 100 executes the following processes:

[0262] First, in step S206, the setting area of ​​each of the nd potential maps corresponding to individual priority audience viewpoints or TV camera viewpoints, corresponding to all priority audience viewpoints or TV camera viewpoints 1 to nd, is quantified. For example, No-driving zone = 10, Driving caution zone = 5, Driving tolerance range = 0 This involves quantifying the data at the domain level.

[0263] (Step S207) Next, in step S207, the data processing unit of the information processing device 100 calculates the sum of region-corresponding values ​​for each region by adding the region-corresponding values ​​of the potential maps for all priority audience viewpoints or TV camera viewpoints 1 to nd.

[0264] For example, suppose the total number of priority audience and TV camera viewpoints is 3, and three individual priority audience and TV camera viewpoint-based potential maps (m1~m3) corresponding to the three priority audience viewpoints or TV camera viewpoints have been generated, and the region setting for a certain stage position (x1, y1) at a certain time tx is set as follows. Map m1 value = 5 (Caution required when driving) Map m2 value = 5 (Caution advised area for driving) Map m3 value = 0 (driving tolerance zone) In this case, the sum is 5 + 5 + 0 = 10. This addition process is performed for all maps. If the maps are time-series data, this process is performed for all stage positions on the map at all time points.

[0265] (Step S208) Next, in step S208, the data processing unit of the information processing device 100 resets the region divisions based on the sum of the region correspondences of the potential maps for all priority audience viewpoints or TV camera viewpoints 1 to nd.

[0266] For example, the area will be reconfigured according to the following rules. If the added value is 10 points or more, it means driving is prohibited. A score of 5 or more indicates a driving caution zone. A score of less than 5 points = acceptable driving range. For example, the area can be reconfigured according to the rules above.

[0267] (Step S209) Next, in step S209, the data processing unit of the information processing device 100 outputs the potential map, whose region divisions were reset in step S208, to the potential map synthesis unit 130 as a "priority audience, TV camera viewpoint-based potential map".

[0268] The above is a detailed sequence of the generation process for map a = preferred audience, TV camera viewpoint-based potential map executed by the individual potential map generation unit 120. This process generates a time-series map where each area (no-driving area, caution area, permitted driving area) dynamically changes according to the priority audience viewpoint or TV camera viewpoint position from the start to the end of the live performance, or a map a = priority audience, TV camera viewpoint-based potential map where there is no dynamic change.

[0269] Next, with reference to the flowchart shown in Figure 23, the generation sequence of pre-generated potential maps executed by the potential map synthesis unit 130 of the information processing device 100 will be described.

[0270] As explained earlier with reference to Figure 20, the potential map synthesis unit 130 combines the four individual potential maps generated by the individual potential map generation unit 120, that is, Map a = Performer-based potential map, Map b = Illumination-based potential map, Map c = Object-based potential map Map d = Priority audience, TV camera viewpoint position-based potential map. These four potential maps are combined to generate a pre-generated potential map b, which is a composite data that reflects all four potential maps.

[0271] The flowchart shown in Figure 23 represents the generation sequence of the pre-generated potential map b performed by the potential map synthesis unit 130. The processing of each step in the flowchart shown in Figure 23 will be explained sequentially below.

[0272] (Step S221) First, in step S221, the data processing unit (potential map synthesis unit 130) of the information processing device 100 shown in Figure 20 processes the four individual potential maps generated by the individual potential map generation unit 120, that is, Map a = Performer-based potential map, Map b = Illumination-based potential map, Map c = Object-based potential map Map d = Priority audience, TV camera viewpoint position-based potential map. The setting range for each of these four potential maps is quantified. For example, No-driving zone = 10, Driving caution zone = 5, Driving tolerance range = 0 This involves quantifying the data at the domain level.

[0273] (Step S222) Next, in step S222, the data processing unit of the information processing device 100 processes four individual potential maps, namely, Map a = Performer-based potential map, Map b = Illumination-based potential map, Map c = Object-based potential map Map d = Priority audience, TV camera viewpoint position-based potential map. The numerical values ​​corresponding to the setting regions of each of these four potential maps are added together for each region to calculate the sum of the region-corresponding values.

[0274] For example, suppose the region setting for a certain stage position (x1, y1) at a certain time tx is as follows: Map a = Performer-based potential map value = 5 (Cautionary driving area) Map b = Lighting base potential map value = 0 (Driving permissible area) Map c = Object-based potential map value = 5 (Cautionary driving area) Map d = priority spectators, TV camera viewpoint position base potential map = 5 (driving caution area), In this case, the sum is 5 + 0 + 5 + 5 = 15. This addition process is performed for all maps. Note that the maps are time-series data, meaning the process is performed for all stage positions on the map at all points in time.

[0275] (Step S223) Next, in step S223, the data processing unit of the information processing device 100 processes four individual potential maps, namely, Map a = Performer-based potential map, Map b = Illumination-based potential map, Map c = Object-based potential map Map d = Priority audience, TV camera viewpoint position-based potential map The region divisions are redefined based on the sum of the region correspondences of these four potential maps.

[0276] For example, the area will be reconfigured according to the following rules. If the added value is 10 points or more, it means driving is prohibited. A score of 5 or more indicates a driving caution zone. A score of less than 5 points = acceptable driving range. For example, the area can be reconfigured according to the rules above.

[0277] (Step S224) Next, in step S224, the data processing unit of the information processing device 100 generates a composite map, or "pre-generated potential map b", from the potential map whose region divisions were reset in step S223.

[0278] The above is a detailed sequence of the generation process of pre-generated potential maps b and 150b performed by the potential map synthesis unit 130. Thus, the potential map synthesis unit 130 combines the four individual potential maps generated by the individual potential map generation unit 120, that is, Map a = Performer-based potential map, Map b = Illumination-based potential map, Map c = Object-based potential map Map d = Priority audience, TV camera viewpoint position-based potential map These individual potential maps are combined to generate pre-generated potential maps b and 150b.

[0279] The pre-generated potential maps b and 150b generated by the potential map synthesis unit 130 are provided to the travel route generation unit 160, which determines the travel route of the image acquisition robot 50 from the start to the end of the live performance based on the pre-generated potential map 150.

[0280] In other words, as explained earlier with reference to Figure 10, the route generation unit 160 receives pre-generated potential maps b and 150b generated by the potential map synthesis unit 130 and generates route information 165 that sets a route for the image-capturing robot 50 to travel in, selecting areas that will not collide with performers or objects on the stage from the start to the end of the live performance, areas that will not be conspicuous due to lighting, and areas that will not obstruct the view of priority audience members or TV cameras. For example, a driving route is generated that travels only within the permissible driving area in the pre-generated potential map 150.

[0281] The generated travel route is provided to the travel control unit 170, which controls the movement of the image-capturing robot 50, and the travel control unit 170 makes the image-capturing robot 50 move according to the generated travel route information 165. In this way, by having the image-capturing robot 50 travel according to the travel route information 165 generated using the pre-generated potential map 150, it becomes possible to have the image-capturing robot 50 travel from the start to the end of the live performance without colliding with performers or objects on the stage, and in areas that are not conspicuous due to lighting and do not obstruct the view of priority audience members or TV cameras.

[0282] [5. (Example 3) An example of generating a real-time data reflection potential map using information from live execution] Next, as Example 3, we will describe an example in which a real-time data reflection potential map is generated using information from live execution.

[0283] The above-described Examples 1 and 2 generated potential maps using data that could be obtained before the live performance on stage began, that is, the following data stored in the storage unit 110 of the information processing device 100 shown in Figures 4 and 20. A. Performer's activity schedule data 111, B. Stage lighting control planned data 112, C. Stage object placement plan data 113, D. Priority audience, TV camera viewpoint position data 114,

[0284] The data A to D used in Examples 1 and 2 were generated on the premise that, for example, performers would act according to a live program, lighting would be controlled, and objects would be placed accordingly. However, once the live performance actually begins, the performers' movements, lighting control, and other aspects may differ from the original program.

[0285] In such cases, if the image-capturing robot 50 is driven using the pre-generated potential map generated based on the above-described examples 1 and 2, there is a possibility that it may come into contact with the performer. The third example described below aims to prevent such a situation and is an example in which real-time data is acquired during the actual live performance, i.e., the time when the performers are performing, and a potential map is generated.

[0286] Figure 24 shows an example configuration of the information processing device 200 in this embodiment 3. As shown in Figure 24, the information processing device 200 includes a real-time stage information acquisition unit 201, a real-time audience seating information acquisition unit 202, a real-time network information acquisition unit 203, a real-time stage information analysis unit 204, a storage unit 205, a pre-generated potential map modification unit 206, a real-time area of ​​interest analysis unit 207, and a real-time data reflection potential map generation unit 208.

[0287] The information processing device 200 shown in Figure 24 may be configured inside, for example, a mobile device that travels on the stage shown in Figures 1 to 3, i.e., an image-capturing robot 50 equipped with a camera, or it may be a device independent of the image-capturing robot 50, such as a device that can communicate with the image-capturing robot 50.

[0288] Figure 25 shows an example configuration of an information processing system when the information processing device having the configuration shown in Figure 24 is a separate device from the image acquisition robot 50. For example, as shown in Figure 25, an information processing system 280 is constructed by connecting an information processing device (server) 200, a live venue information acquisition device 60, an image capture robot 50 within the live venue, and an SNS server 290 via a communication network. The information processing device (server) 100 has the configuration shown in Figure 4.

[0289] The live venue information acquisition device 60 consists of a camera that takes images of the live venue, a microphone that acquires audio information of the live venue, an illuminance meter that detects the lighting conditions, a color analyzer, etc., and transmits the acquired images and other information to the information processing device (server) 200 via a communication network.

[0290] The information processing device (server) 200 analyzes the information received from the live venue information acquisition device 60 and the information acquired from the SNS server 290, and performs processes such as generating a map (potential map) to determine the travel route of the image-capturing robot 50 within the live venue, determining the travel route using the map, and generating travel control information for the image-capturing robot 50 according to the determined travel route.

[0291] The SNS server 290 is a server that collects comment information, such as tweets, from users who are viewing the live stream online. The information processing device (server) 200 analyzes the information obtained from the SNS server 290 to analyze areas of interest of users who are watching the live stream via the internet.

[0292] The information processing device (server) 200 further transmits the generated driving control information to the image-capturing robot 50 via the communication network. The image-capturing robot 50 moves around the stage according to the driving control information it receives from the information processing device (server) 100. For example, it is possible to perform processing using such an information processing system.

[0293] The configuration and processing of the information processing device 100 shown in Figure 24 will be described below. The real-time stage information acquisition unit 201 is specifically composed of, for example, cameras and light meters, and acquires real-time information about the actual live performance, i.e., the stage where the performers are performing.

[0294] The real-time stage information acquisition unit 201 acquires the following information as real-time data, as shown in the figure. Performer position 211, Lighting condition 212, Object position 213 on the stage, Image capture robot position 214, The lighting status includes information on the position of the stage illuminated by the lights, the brightness of the lights, and the color of the lights.

[0295] The real-time stage information acquisition unit 201 continuously acquires performer positions 211, lighting status 212, stage object positions 213, and image capture robot positions 214 as real-time data during the live performance period, and outputs the acquired data to the real-time stage information analysis unit 204.

[0296] The real-time stage information analysis unit 204 compares the performer positions 211, lighting conditions 212, stage object positions 213, and image capture robot positions 214, which are input from the real-time stage information acquisition unit 201, with the scheduled data 231 based on the live program stored in the storage unit 205 beforehand, and the pre-generated potential map 150 generated according to the previous embodiment 1 or embodiment 2.

[0297] The scheduled data 231 based on the live program stored in the memory unit 205 is the same data as the following information previously explained with reference to Figures 4 and 20. A. Performer's activity schedule data 111, B. Stage lighting control planned data 112, C. Stage object placement plan data 113, Each of these data points is pre-generated based on the live performance schedule before the live event begins.

[0298] The real-time stage information analysis unit 204 compares the performer positions 211, lighting status 212, stage object positions 213, and image capture robot positions 214, which are input from the real-time stage information acquisition unit 201, with the scheduled data 231 based on the live program that is stored in the storage unit 205 in advance, to check whether there are any differences.

[0299] If there are no differences, the image-capturing robot 50 can be driven along a route set using the pre-generated potential map 150 generated according to Example 1 or Example 2, without colliding with performers or objects, and without being conspicuous due to lighting. However, if differences exist, when the image-capturing robot 50 is driven along a route set using the pre-generated potential map 150 generated according to Example 1 or Example 2, there is a possibility of contact with performers or objects, or the robot may drive in a way that is conspicuous due to lighting.

[0300] The real-time stage information analysis unit 204 detects differences between the performer positions 211, lighting conditions 212, stage object positions 213, and image-capturing robot positions 214 input from the real-time stage information acquisition unit 201, and the scheduled data 231 based on the live program stored in the storage unit 205 beforehand. Furthermore, it determines whether there are any problems with driving along the driving route according to the pre-generated potential map 150 generated according to the previous embodiment 1 or embodiment 2.

[0301] If a problem is detected, that is, if it is determined that there is a possibility of contact with performers or objects, or that the robot may move in a way that is conspicuous due to lighting, when the image-capturing robot 50 is driven along the route set using the pre-generated potential map 150, a request to modify the pre-generated potential map 150 is output to the pre-generated potential map modification unit 206.

[0302] When the pre-generated potential map modification unit 206 receives a request from the real-time stage information analysis unit 204 to modify the pre-generated potential map 150, it modifies the pre-generated potential map 150 to reduce the possibility of contact with performers or objects, and the possibility of conspicuous movement due to lighting. The pre-potential map modification unit 206 modifies the pre-generated potential map 150 to generate a real-time stage information reflecting potential map.

[0303] The real-time stage information reflection potential map is a map that reflects the real-time performer positions 211, lighting conditions 212, stage object positions 213, and image-capturing robot positions 214 acquired by the real-time stage information acquisition unit 201, and sets each area (no-movement area, caution area, permitted area).

[0304] A specific example of the generation process of the "real-time stage information reflected potential map" executed by the pre-potential map modification unit 206 will be explained with reference to Figure 26. Figure 26 shows the following two potential maps. (1) Pre-generated potential map (2) Real-time stage information reflection potential map

[0305] (1) The pre-generated potential map is the pre-generated potential map 150 generated according to the previous example 1 or example 2, and is the potential map stored in the storage unit 205 of the information processing device 200 shown in Figure 24.

[0306] (2) The real-time stage information reflection potential map is a map generated by the pre-potential map modification unit 206 by modifying the pre-generated potential map 150. That is, it is a map in which each region (no-movement region, cautionary movement region, permitted movement region) is reset to reflect the real-time performer position 211, lighting status 212, stage object position 213, and image capture robot position 214 acquired by the real-time stage information acquisition unit 201.

[0307] Comparing (1) and (2) in Figure 26, for example, the presenter's position in (1) and the presenter's position in (2) are slightly different, with the presenter's position in (2) being slightly shifted to the right. Thus, unexpected situations can occur during actual live performances. The pre-potential map modification unit 206 modifies the pre-generated potential map using real-time stage information to generate a real-time stage information-reflected potential map.

[0308] The real-time stage information reflection potential map generated by the pre-potential map modification unit 206 is a map that reflects the real-time performer positions, lighting conditions, and object positions, and for example, the map will have the following area settings. The permissible movement area (blue) = the area furthest from the performer's and object's real-time positions, and determined to be the least conspicuous based on real-time lighting conditions. No-travel zone (red) = Areas close to the real-time performer's position and object's position, and which are judged to be conspicuous based on the real-time lighting conditions. The area requiring caution while driving (yellow) is the intermediate area between the permitted driving area (blue) and the prohibited driving area (red).

[0309] The real-time stage information-reflected potential map generated by the pre-potential map modification unit 206 is output to the real-time data-reflected potential map generation unit 208.

[0310] The real-time audience seating information acquisition unit 202 is specifically composed of, for example, cameras, and acquires real-time information about the audience seating in front of the stage where the actual live performance, i.e., the performers' performance, is taking place.

[0311] The real-time audience seating information acquisition unit 202 acquires the following information as real-time data, as shown in the figure. Audience line of sight direction 215, TV camera shooting direction 216 on the spectator side,

[0312] The audience viewpoint position 215 is the direction of the audience's line of sight in front of the stage where the live performance is taking place. The audience-side TV camera shooting direction 216 is the direction in which the TV camera in front of the stage where the live performance is taking place is shooting. The real-time information acquired by the real-time audience seating information acquisition unit 202 is input to the real-time area of ​​interest analysis unit 207.

[0313] Furthermore, the real-time network information acquisition unit 203 has a communication unit connected to a communication network such as the internet, and acquires so-called SNS information 217, such as tweets and comments from many live viewers on the internet, and inputs the acquired SNS information 217 to the real-time attention area analysis unit 207.

[0314] The real-time attention area analysis unit 207 analyzes the audience gaze direction 215 input from the real-time audience seat information acquisition unit 202, the audience-side TV camera shooting direction 216, and the SNS information 217 input from the real-time internet information acquisition unit 203 to determine where the audience is looking on the stage, where the TV cameras are shooting on the stage, and where online live viewers are focusing their attention on the stage. Based on these analysis results, the real-time focus region analysis unit 207 estimates the current focus region on the stage and outputs the estimated region as real-time focus region information 221 to the real-time data reflection potential map generation unit 208.

[0315] The real-time data reflection potential map generation unit 208 receives the following data: (1) The "real-time stage information reflected potential map" generated by the pre-potential map modification unit 206 (2) Real-time focus region information 221 generated by the real-time focus region analysis unit 207 The real-time data reflection potential map generation unit 208 takes each of these data as input and generates a real-time data reflection potential map 230 based on these input data.

[0316] The real-time data reflection potential map generation unit 208 sequentially executes the following processes 1 and 2. (Process 1) Based on the real-time focus region information 221 generated by the real-time focus region analysis unit 207, a real-time focus region reflection potential map is generated that reflects the real-time focus region. (Process 2) The pre-potential map modification unit 206 generates a real-time data reflection potential map 230 by combining the "real-time stage information reflection potential map," which sets each region (no-travel area, caution area, permitted area) based on the performer's position 211, lighting status 212, stage object position 213, and image capture robot position 214 in real time, with the real-time focus area reflection potential map generated in Process 1.

[0317] Specific examples of these (Process 1) and (Process 2) will be explained with reference to Figure 27. Figure 27 shows the following three potential maps. (2) Real-time stage information reflection potential map (3) Real-time potential map reflecting areas of interest (4) Real-time data reflection potential map 230

[0318] (2) The real-time stage information reflection potential map is the map described earlier with reference to Figure 26, and is the "real-time stage information reflection potential map" generated by the pre-potential map modification unit 206 using real-time stage information.

[0319] (3) The real-time focus region reflected potential map is a map generated by the (process 1) above, which is performed by the real-time data reflected potential map generation unit 208, and is a potential map generated that reflects the real-time focus region based on the real-time focus region information 221 generated by the real-time focus region analysis unit 207.

[0320] The real-time attention area reflection potential map is a map generated by reflecting real-time audience, TV camera, and internet information, and the map will have the following area settings. Driving Permitted Area (Blue) = The least attention-grabbing, non-attention area based on real-time spectators, TV cameras, and internet information. No-Driving Area (Red) = The most noteworthy area based on real-time spectators, TV cameras, and online information. The area requiring caution while driving (yellow) is the intermediate area between the permitted driving area (blue) and the prohibited driving area (red).

[0321] The (4) Real-time data reflection potential map 230 shown in Figure 27 is shown in Figure 27. (2) Real-time stage information reflection potential map (3) Real-time potential map reflecting areas of interest This map is generated by combining two potential maps.

[0322] This map synthesis process generates the map by quantifying the settings of the "(2) Real-time Stage Information Reflection Potential Map" and the "(3) Real-time Area of ​​Interest Reflection Potential Map," adding the values ​​for each area, and then resetting the areas (driving prohibited area, driving caution area, driving permitted area) based on the summation result. A detailed example of the generation sequence of "(4) Real-time data reflection potential map 230" based on this map synthesis process will be explained later with reference to the flowchart.

[0323] The real-time data reflection potential map 230 generated by the real-time data reflection potential map generation unit 208 is a map generated by reflecting real-time information on the stage, real-time audience information, and real-time network information.

[0324] The real-time data reflection potential map 230 generated by the real-time data reflection potential map generation unit 208 is a map that reflects all real-time performer positions, lighting conditions, object positions, audience, TV camera, and network information, and the map has the following area settings. Permitted area for movement (blue) = The least conspicuous, non-focused area based on real-time performer positions, lighting conditions, object positions, audience, TV cameras, and internet information. No-go zone (red) = The most prominent area of ​​interest based on real-time performer positions, lighting conditions, object positions, audience, TV cameras, and internet information. The area requiring caution while driving (yellow) is the intermediate area between the permitted driving area (blue) and the prohibited driving area (red).

[0325] The real-time data reflection potential map 230 generated by the real-time data reflection potential map generation unit 208 is provided to the travel route generation unit 240, which then determines the travel route of the image acquisition robot 50 based on the real-time data reflection potential map 230.

[0326] The route generation unit 240 receives the real-time data reflection potential map 230 as input and generates route information that sets a route for the image-capturing robot 50 to travel in, selecting areas that do not collide with performers or objects on the stage, areas that are not conspicuous due to lighting, and areas that are not the focus of attention of the audience, TV cameras, or online viewers.

[0327] The route generation unit 240 can set such a route by, for example, generating a route that travels only within the permitted travel area in the real-time data reflection potential map 230.

[0328] The generated travel route is provided to the travel control unit, which controls the movement of the image-capturing robot 50, and the travel control unit makes the image-capturing robot 50 move according to the generated travel route information.

[0329] Thus, by having the image-capturing robot 50 travel according to the travel route information generated using the real-time data reflection potential map 230 generated in this embodiment 3, it becomes possible to have the image-capturing robot 50 travel without colliding with the real-time positions of performers or objects on the stage where a live performance is taking place, and also select areas that are not conspicuous due to lighting, as well as areas that are not the focus of attention of the audience, TV cameras, or online viewers.

[0330] Next, with reference to the flow shown in Figure 28, the real-time data reflection potential map generation process sequence executed by the information processing device 200 of this embodiment 3 will be described. The following describes the processing of each step in the flow shown in Figure 28.

[0331] (Step S301) First, in step S301, the information processing device 200 acquires real-time stage information (performer positions, lighting status, stage object positions, and image capture robot positions).

[0332] This process is executed by the real-time stage information acquisition unit 201 of the information processing device 200 shown in Figure 24. The real-time stage information acquisition unit 201 is specifically composed of, for example, cameras and light meters, and acquires information about the actual live performance, i.e., the stage where the performers are performing, i.e., real-time stage information, including the performers' positions, lighting conditions, the positions of objects on the stage, and the positions of the image-capturing robot.

[0333] (Step S302) Next, in step S302, the information processing device 200 modifies the previously generated pre-generated potential map based on the real-time stage information acquired in step S301 to generate a real-time stage information reflected potential map.

[0334] The process in step S302 is performed by the pre-potential map modification unit 206 of the information processing device 200 shown in Figure 24.

[0335] Although omitted in the flow shown in Figure 28, after the processing in step S301, the real-time stage information analysis unit 204 of the information processing device 200 shown in Figure 24 analyzes the difference between the real-time stage information (performer position, lighting status, stage object position, image capture robot position) input from the real-time stage information acquisition unit 201 and the planned data 231 based on the live program stored in the storage unit 205 in advance, and analyzes any problems with driving the driving route according to the pre-generated potential map 150.

[0336] If a problem is detected, that is, if it is determined that there is a possibility of contact with performers or objects, or that the robot may move in a way that is conspicuous due to lighting, when the image-capturing robot 50 is driven along the route set using the pre-generated potential map 150, a request to modify the pre-generated potential map 150 is output to the pre-generated potential map modification unit 206.

[0337] The process in step S302 is a subsequent process in which the pre-generated potential map modification unit 206, upon receiving a request from the real-time stage information analysis unit 204 to modify the pre-generated potential map 150 to reduce the possibility of contact with performers or objects, and the possibility of conspicuous movement due to lighting, and then generates a real-time stage information reflected potential map.

[0338] The real-time stage information reflection potential map generated by the pre-potential map modification unit 206 is a map that reflects the real-time performer positions, lighting conditions, and object positions, and for example, the map will have the following area settings. The permissible movement area (blue) = the area furthest from the performer's and object's real-time positions, and determined to be the least conspicuous based on real-time lighting conditions. No-travel zone (red) = Areas close to the real-time performer's position and object's position, and which are judged to be conspicuous based on the real-time lighting conditions. The area requiring caution while driving (yellow) is the intermediate area between the permitted driving area (blue) and the prohibited driving area (red).

[0339] (Step S303) Next, in step S303, the information processing device 200 acquires real-time audience seating information and real-time network information.

[0340] These processes are executed by the real-time audience seating information acquisition unit 202 and the real-time network information acquisition unit 203 of the information processing device 200 shown in Figure 24.

[0341] As explained earlier with reference to Figure 24, the real-time audience information acquisition unit 202 acquires the following information as real-time data. Audience line of sight direction 215, TV camera shooting direction 216 on the spectator side, Furthermore, the real-time network information acquisition unit 203 acquires so-called SNS information 217, such as tweets and comments from many live viewers on the internet, and inputs the acquired SNS information 217 into the real-time attention area analysis unit 207.

[0342] (Step S304) Next, in step S304, the information processing device 200 analyzes the real-time area of ​​interest based on the real-time audience seating information and real-time network information acquired in step S303.

[0343] This process is performed by the real-time focus area analysis unit 207 of the information processing device 200 shown in Figure 24.

[0344] The real-time attention area analysis unit 207 analyzes the audience gaze direction 215 input from the real-time audience seat information acquisition unit 202, the audience-side TV camera shooting direction 216, and the SNS information 217 input from the real-time internet information acquisition unit 203 to determine where the audience is looking on the stage, where the TV cameras are shooting on the stage, and where online live viewers are focusing their attention on the stage.

[0345] The real-time focus region analysis unit 207 estimates the current focus region on the stage based on these analysis results and outputs the estimated region as real-time focus region information 221 shown in Figure 24 to the real-time data reflection potential map generation unit 208.

[0346] (Step S305) Next, in step S305, the information processing device 200 generates a "real-time attention area reflection potential map" that reflects the real-time attention area analyzed based on real-time audience seating information and real-time network information.

[0347] This process is performed by the real-time data reflection potential map generation unit 208 of the information processing device 200 shown in Figure 24.

[0348] The real-time data reflection potential map generation unit 208 generates a "real-time attention area reflection potential map" that reflects real-time attention areas analyzed based on real-time audience seating information and real-time network information.

[0349] The real-time attention area reflection potential map is a map generated by reflecting real-time audience, TV camera, and internet information, and the map will have the following area settings. Driving Permitted Area (Blue) = The least attention-grabbing, non-attention area based on real-time spectators, TV cameras, and internet information. No-Driving Area (Red) = The most noteworthy area based on real-time spectators, TV cameras, and online information. The area requiring caution while driving (yellow) is the intermediate area between the permitted driving area (blue) and the prohibited driving area (red).

[0350] (Step S306) Next, in step S306, the information processing device 200 quantifies the "real-time stage information reflection potential map" and the "real-time area of ​​interest reflection potential map," as well as the setting areas of each map.

[0351] This process is also executed by the real-time data reflection potential map generation unit 208 of the information processing device 200 shown in Figure 24. for example, No-driving zone = 10, Driving caution zone = 5, Driving tolerance range = 0 This involves quantifying the data at the domain level.

[0352] (Step S307) Next, in step S307, the information processing device 200 calculates the sum of region-corresponding values ​​for each region by adding the region-corresponding values ​​of the "real-time stage information reflection potential map" and the "real-time focus region reflection potential map" for each region.

[0353] This process is also executed by the real-time data reflection potential map generation unit 208 of the information processing device 200 shown in Figure 24.

[0354] For example, suppose the region settings for the same stage position (x1, y1) in the "Real-time Stage Information Reflection Potential Map" and the "Real-time Area of ​​Interest Reflection Potential Map" are as follows. Real-time stage information reflection potential map value = 5 (driving caution area) Real-time attention area reflection potential map value = 5 (driving caution area) In this case, the sum is 5 + 5 = 10. This addition process is performed for all stage areas within the map.

[0355] (Step S308) Next, in step S308, the data processing unit of the information processing device 200 resets the area divisions based on the area-corresponding sum calculated in step S307.

[0356] This process is also executed by the real-time data reflection potential map generation unit 208 of the information processing device 200 shown in Figure 24.

[0357] The real-time data reflection potential map generation unit 208 of the information processing device 200 performs area resetting according to rules such as the following: If the added value is 10 points or more, it means driving is prohibited. A score of 5 or more indicates a driving caution zone. A score of less than 5 points = acceptable driving range. For example, the area can be reconfigured according to the rules above.

[0358] (Step S309) Next, in step S309, the data processing unit of the information processing device 200 outputs the potential map, whose region divisions were reset in step S308, as a "real-time data-reflected potential map".

[0359] This process is also executed by the real-time data reflection potential map generation unit 208 of the information processing device 200 shown in Figure 24.

[0360] The above is a detailed sequence of the real-time data reflection potential map generation process performed by the information processing device 200 of Embodiment 3 shown in Figure 24.

[0361] By using the real-time data reflection potential map 230 generated by the information processing device 200 of Embodiment 3 shown in Figure 24 to generate the travel route information of the image-capturing robot 50, it becomes possible to move the image-capturing robot 50 without colliding with the real-time positions of performers or objects on the stage where a live performance is taking place, and to select areas that are not conspicuous due to lighting, as well as areas that are not the focus of attention of the audience, TV cameras, or online viewers.

[0362] In addition, the following processes are performed in steps S301 to S302 of the flowchart shown in Figure 28. As shown in Figure 24, the information processing device 200 performs the acquisition of real-time stage information (performer position, lighting status, stage object position, image capture robot position) by the real-time stage information acquisition unit 201, and the generation of a real-time stage information-reflecting potential map by modifying the pre-generated potential map by the pre-potential map modification unit 206.

[0363] After the processing in steps S301 to S302, the processing in steps S303 to S309 is executed, and finally in step S309, a real-time data reflection potential map 230 is generated. Based on the generated real-time data reflection potential map 230, the travel route of the image acquisition robot 50 is determined and travel control is performed.

[0364] However, the route of the image-capturing robot 50 cannot be changed at any time during the live run. There are periods when changes are permitted (= period during which route changes are allowed) and periods when changes are not permitted (= period during which route changes are not allowed). Therefore, it is preferable that the real-time stage information reflection potential map generation process in steps S301 to S302 is executed only during the period in which the travel route of the image-capturing robot 50 can be changed.

[0365] The processing sequence in which the real-time stage information reflection potential map generation process in steps S301 to S302 of the flow shown in Figure 28 is set to be executed only during the period in which the travel route change of the image acquisition robot 50 is permitted will be explained with reference to the flowchart shown in Figure 29.

[0366] The flowchart shown in Figure 29 is a flowchart that allows execution by replacing the processes in steps S301 to S302 of the flowchart shown in Figure 28. The following describes the processing of each step in the flow shown in Figure 29.

[0367] (Step S321) First, the information processing device 200 shown in Figure 24 determines in step S321 whether or not it is time to acquire real-time stage information.

[0368] This process is performed by the real-time stage information acquisition unit 201 of the information processing device 200 shown in Figure 24, or by the control unit that controls the real-time stage information acquisition unit 201. The timing for acquiring real-time stage information is predetermined, for example, every 10 seconds, to be at regular intervals.

[0369] In step S321, it is determined whether or not it is time to acquire real-time stage information according to this provision. If it is determined that it is time to acquire real-time stage information, the process proceeds to step S322.

[0370] (Step S322) If it is determined in step S321 that it is time to acquire real-time stage information, the information processing device 200 acquires real-time stage information (performer positions, lighting status, stage object positions, image capture robot positions) in step S322.

[0371] This process is performed by the real-time stage information acquisition unit 201 of the information processing device 200 shown in Figure 24. The real-time stage information acquisition unit 201 is specifically composed of, for example, cameras and light meters, and acquires information about the actual live performance, i.e., the stage where the performers are performing, i.e., real-time stage information, including the performers' positions, lighting conditions, the positions of objects on the stage, and the positions of the image-capturing robot.

[0372] (Step S323) Next, in step S323, the information processing device 200 determines whether or not it is the end of the travel time for the image-capturing robot.

[0373] Information regarding the completion time of the image-capturing robot's journey is pre-stored in the memory unit 205 of the information processing device 200. For example, it is recorded in association with the planned route of the image-capturing robot. The information processing device 200 refers to this recorded information and determines whether the current time is the end of the image-capturing robot's journey.

[0374] If the current time is the end of the image-capture robot's travel time, the process will be terminated. On the other hand, if the current time is not the end of the image-capturing robot's journey, the process proceeds to step S324.

[0375] (Step S324) If it is determined in step S323 that the time for the image-capturing robot to finish its journey has not yet ended, the information processing device 200 determines in step S324 whether or not it is within the permitted time for changing the travel route of the image-capturing robot.

[0376] Information regarding whether or not it is within the permissible time for changing the travel route of the image-capturing robot is pre-stored in the storage unit 205 of the information processing device 200. For example, it is recorded in association with the planned travel route of the image-capturing robot. The information processing device 200 refers to this recorded information and determines whether the current time is within the permitted period for changing the travel route of the image-capturing robot.

[0377] If the current time is not within the allowable time for changing the image-capturing robot's travel route, the process does not proceed to step S325, but returns to step S321, and steps S321 to S324 are repeated. Step S325 is initiated only if it is determined that the current time is within the permissible time for changing the image-capturing robot's travel route.

[0378] (Step S325) If, in step S324, it is determined that the current time is within the allowable time for changing the travel route of the image acquisition robot, the process in step S325 is executed.

[0379] In this case, in step S325, the information processing device 200 performs a comparison process between the real-time stage information (performer position, lighting status, stage object position, image capture robot position) acquired in step S321 and the pre-generated potential map that has been generated earlier.

[0380] The process in step S325 is performed by the real-time stage information analysis unit 204 of the information processing device 200 shown in Figure 24.

[0381] (Step S326) Next, in step S326, the information processing device 200 determines whether the travel route of the image-capturing robot based on the pre-generated potential map is set to pass through a dangerous area estimated from the real-time stage information (performer position, lighting status, stage object position, image-capturing robot position) acquired in step S321.

[0382] The process in step S326 is also executed by the real-time stage information analysis unit 204 of the information processing device 200 shown in Figure 24.

[0383] The real-time stage information analysis unit 204 analyzes the difference between the real-time stage information (performer positions, lighting status, stage object positions, image capture robot positions) input from the real-time stage information acquisition unit 201 and the planned data 231 based on the live program stored in the storage unit 205 beforehand, and analyzes the risks of the travel route according to the pre-generated potential map 150.

[0384] If it is determined that there is no danger, that is, if the image-capturing robot 50 is driven according to the driving route set using the pre-generated potential map 150, it is determined that there is no possibility of contact with performers or objects, or of the robot driving in a way that is conspicuous due to lighting, then the process returns to step S321 without proceeding to step S327, and the processing from step S321 onward is repeated.

[0385] On the other hand, if it is determined that there is a risk, that is, if it is determined that there is a possibility of contact with performers or objects, or that the robot may move in a way that is conspicuous due to lighting, when the image-capturing robot 50 is driven along the route set using the pre-generated potential map 150, the process proceeds to step S327.

[0386] (Step S327) In step S326, if the information processing device determines that the driving route set using the pre-generated potential map 150 poses a risk, such as potentially coming into contact with the performer, it executes the process in step S327.

[0387] In this case, in step S327, the information processing device 200 modifies the pre-generated potential map 150 stored in the memory unit 205 and generates a real-time stage information reflected potential map.

[0388] The process in step S327 is performed by the pre-generated potential map modification unit 206 of the information processing device 200 shown in Figure 24.

[0389] The pre-potential map modification unit 206 modifies the pre-generated potential map 150 to reduce the possibility of contact with performers or objects, and the possibility of conspicuous movement due to lighting, thereby generating a real-time stage information-reflecting potential map.

[0390] The real-time stage information reflection potential map generated by the pre-potential map modification unit 206 is a map that reflects the real-time performer positions, lighting conditions, and object positions, and for example, the map will have the following area settings. The permissible movement area (blue) = the area furthest from the performer's and object's real-time positions, and determined to be the least conspicuous based on real-time lighting conditions. No-travel zone (red) = Areas close to the real-time performer's position and object's position, and which are judged to be conspicuous based on the real-time lighting conditions. The area requiring caution while driving (yellow) is the intermediate area between the permitted driving area (blue) and the prohibited driving area (red).

[0391] After the processing of steps S321 to S327 shown in Figure 29, the processing from step S303 onwards in the flow shown in Figure 28 is executed. In other words, the processes in steps S303 to S309 are executed, and finally, in step S309, a real-time data reflection potential map 230 is generated. Based on the generated real-time data reflection potential map 230, the travel route of the image acquisition robot 50 is determined and travel control is performed.

[0392] By performing steps S321 to S327 shown in the flowchart of Figure 29 instead of steps S301 to S302 shown in Figure 28, the real-time stage information reflection potential map generation process can be performed only during the period in which the travel route of the image-capturing robot 50 can be changed.

[0393] [6. Examples of Hardware Configurations for Information Processing Devices] Next, an example of the hardware configuration of an information processing device that performs the processing according to the above-described embodiment will be explained with reference to Figure 30. The hardware shown in Figure 30 is an example of the hardware configuration of the information processing device 100, which was explained earlier with reference to Figure 4, the information processing device 100b, which was explained with reference to Figure 20, and the information processing device 200, which was explained with reference to Figure 24. The hardware configuration shown in Figure 30 will be explained below.

[0394] The CPU (Central Processing Unit) 301 functions as a data processing unit that executes various processes according to the program stored in the ROM (Read Only Memory) 302 or the storage unit 308. For example, it executes processes according to the sequence described in the above embodiment. The RAM (Random Access Memory) 303 stores the program and data that the CPU 301 executes. These CPU 301, ROM 302, and RAM 303 are interconnected by a bus 304.

[0395] The CPU 301 is connected to the input / output interface 305 via the bus 304. The input / output interface 305 is connected to an input section 306 consisting of various sensors, cameras, switches, keyboards, mice, microphones, etc., and an output section 307 consisting of displays, speakers, etc.

[0396] The storage unit 308, connected to the input / output interface 305, consists of, for example, a hard disk and stores programs executed by the CPU 301 and various data. The communication unit 309 functions as a data transmission and reception unit for data communication via a network such as the Internet or a local area network, and communicates with external devices.

[0397] The drive 310 connected to the input / output interface 305 drives removable media 311 such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memory such as memory cards, and performs data recording or reading.

[0398] [7. Summary of the structure of this disclosure] The embodiments of this disclosure have been described in detail above with reference to specific examples. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of this disclosure. In other words, the present invention has been disclosed in the form of examples and should not be interpreted restrictively. To determine the gist of this disclosure, one should refer to the claims section.

[0399] Furthermore, the technology disclosed in this specification can have the following configuration. (1) It has a data processing unit that generates the travel route of a mobile device that moves on the stage, The aforementioned data processing unit The system acquires data on the planned locations of obstacles, including people and objects on the stage, that may obstruct the movement of the mobile device, or data on planned environmental control of the stage environment that changes the visibility level of the mobile device from outside the stage, and An information processing device that generates a travel route based on acquired data that reduces the possibility of physical interference between the mobile device and the obstacle, or at least one of the level of visibility of the mobile device from outside the stage.

[0400] (2) The data processing unit is: The information processing device according to (1) that generates the travel route using a map that defines at least one of the following area divisions: area divisions corresponding to the possibility of physical interference between the mobile device and the obstacle, or area divisions corresponding to the visibility level of the mobile device from outside the stage.

[0401] (3) The map is, (2) The information processing device described in (2), which is a map that defines the area where the mobile device is permitted to travel and the area where it is prohibited to travel.

[0402] (4) The map is, (3) The information processing device described in (3), which is a map that defines a driving caution area between the driving permissible area and the driving prohibited area.

[0403] (5) The data processing unit is: An information processing device according to any of (2) to (4) that generates the aforementioned driving route using the time-series data of the aforementioned map.

[0404] (6) The data for the planned environmental control is The planned control data for the lighting on the aforementioned stage is as follows: The aforementioned data processing unit An information processing device according to any one of (1) to (5), which generates a region as the travel route in which the visibility level of the mobile device from outside the stage decreases due to the control of the lighting on the stage.

[0405] (7) The planned control data for the lighting is An information processing device according to any one of (1) to (6), which generates the aforementioned travel route using a map that defines area divisions according to the brightness of the lights illuminating the stage, which is the planned data for controlling the brightness of the lights.

[0406] (8) The planned control data for the lighting is: An information processing device according to any one of (1) to (7), which generates the aforementioned travel route using a map that defines area divisions according to the color of the lighting to be projected onto the stage, which is the planned data for controlling the color of the lighting.

[0407] (9) The data processing unit is: (a) A performer base map that defines areas on the stage where the likelihood of collision with performers is low as a permissible driving area, based on the performer's planned actions on the stage. (b) An object-based map that defines areas on the stage where the likelihood of collision with objects is low as a driveable area, based on the planned placement data of objects on the stage. (c) Based on the planned lighting control data on the stage, a lighting base map is defined as a driving permissible area in which the area where the visibility level from outside the stage to the mobile device is low, calculated based on the lighting irradiated onto the stage, An information processing device according to any of (1) to (8) that generates the aforementioned driving route using a pre-generated map created by combining the three types of individual maps described in (a) to (c) above.

[0408] (10) The pre-generated map is The information processing device according to (9), which is a map that defines as a permissible travel area an area in which the likelihood of collision with performers and objects on the stage is low, and in which the visibility level of the mobile device from outside the stage, calculated based on lighting, is low.

[0409] (11) The pre-generated map is The information processing device described in (9) or (10) is a map generated by performing area-level numerical processing on each of the three types of individual maps (a) to (c) defined therein, calculating an added value by adding the numerical values ​​of each individual map after numerical processing, and resetting the areas of the area

[0410] (12) The data processing unit, An information processing device according to any one of (1) to (11), which acquires viewpoint position data of at least one of the following: viewpoint position data of spectators on the audience side of the stage and viewpoint position data of a camera, and generates the driving route using a map that defines a driving permissible area as a position far from the line connecting the viewpoint position of the spectator or the camera to the center position of the stage, based on the acquired data.

[0411] (13) The data processing unit, (a) A performer base map that defines areas on the stage where the likelihood of collision with performers is low as a permissible driving area, based on the performer's planned actions on the stage. (b) An object-based map that defines areas on the stage where the likelihood of collision with objects is low as a driveable area, based on the planned placement data of objects on the stage. (c) Based on the planned lighting control data on the stage, a lighting base map is defined as a driving permissible area in which the area where the visibility level from outside the stage to the mobile device is low, calculated based on the lighting irradiated onto the stage, (d) A spectator & camera base map defined as a permissible driving area, based on at least one of the planned viewpoint data of spectators on the audience seating side viewing the stage and the planned viewpoint data of cameras, where the area far from the line connecting the planned viewpoint of the spectator or camera to the center of the stage is defined. An information processing device according to any of (1) to (12) that generates the aforementioned driving route using a pre-generated map created by combining the four types of individual maps described in (a) to (d) above.

[0412] (14) The data processing unit shall An information processing device according to any one of (1) to (13), which acquires real-time data of either the performer's actions or the placement of objects on the stage as real-time data during the performance by the performer on the stage, and generates the driving route using a real-time data reflection map that defines areas on the stage where the likelihood of collision with the performer or objects is low as a driving-permitted area based on the acquired real-time data.

[0413] (15) The data processing unit shall As real-time data during the performance by the performer on the aforementioned stage, The information processing device according to (14), which acquires data on the viewing direction of spectators on the audience side of the stage and the shooting direction of the camera, analyzes the area of ​​interest based on the acquired data, and generates the driving route using a real-time data reflection map in which the area other than the area of ​​interest is defined as a driving-permitted area.

[0414] (16) The data processing unit further: The information processing device according to (14) or (15), which obtains user comments from users viewing the performance by the performer on the stage via the internet, analyzes the areas of interest of the internet viewing users based on the obtained comments, and generates the driving route using a real-time data reflection map that defines areas other than the areas of interest as areas where driving is permitted.

[0415] (17) A storage unit that stores travel route information that reduces the possibility of physical interference between the mobile device and the obstacles, or the visibility level of the mobile device from outside the stage, based on data of the planned positions of obstacles on the stage, including people and objects on the stage, which may interfere with the mobile device's movement on the stage, or data of planned environmental control of the stage environment that changes the visibility level of the mobile device from outside the stage, or It has at least one communication unit that acquires the aforementioned driving route information from an external device, A mobile device that travels on the stage according to either the travel route information obtained from the storage unit or the travel route information obtained via the communication unit.

[0416] (18) The aforementioned route information is The mobile device according to (17), which is a mobile device that generates travel route information using a map that defines at least one of the following area divisions: area divisions corresponding to the possibility of physical interference between the mobile device and the obstacle, or area divisions corresponding to the visibility level of the mobile device from outside the stage.

[0417] (19) An information processing system having a mobile device that moves on the stage and a server, The aforementioned server, The system includes a data processing unit that generates the travel route of the aforementioned mobile device, The aforementioned data processing unit The system acquires data on the planned locations of obstacles on the stage, including people and objects, that may obstruct the movement of the mobile device on the stage, or data on planned environmental control of the stage environment that changes the visibility level of the mobile device from outside the stage. Based on the acquired data, a travel route is generated that reduces the possibility of physical interference between the mobile device and the obstacle, or at least one of the visibility level of the mobile device from outside the stage. The aforementioned mobile device is An information processing system that travels according to a travel route generated by the aforementioned server.

[0418] (20) The data processing unit of the server, The information processing system according to (19) that generates the travel route using a map that defines at least one of the following area divisions: area divisions corresponding to the possibility of physical interference between the mobile device and the obstacle, or area divisions corresponding to the visibility level of the mobile device from outside the stage.

[0419] Furthermore, the series of processes described in the specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in the memory of a computer embedded in dedicated hardware, or the program can be installed and executed on a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing from the recording medium to a computer, the program can also be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as a built-in hard disk.

[0420] Furthermore, the various processes described in this specification may not only be executed sequentially as described, but may also be executed in parallel or individually as needed, depending on the processing capacity of the device performing the process. In addition, in this specification, a system is a logical combination of multiple devices, and the devices in each configuration are not limited to being located in the same enclosure. [Industrial applicability]

[0421] As described above, according to the configuration of one embodiment of the present disclosure, it is possible to generate a map that determines a safe travel route that does not collide with performers or objects on the stage, and to make the image-capturing robot travel according to the route determined based on the map. Specifically, for example, a potential map is generated that defines the permissible movement area for an image-capturing robot that moves around the stage to take pictures. The data processing unit acquires data on the performers' planned actions on the stage, the planned placement of objects, and the planned lighting control on the stage. Based on the acquired data, it generates a potential map that defines the permissible movement area as an area where the robot will not collide with performers or objects and will not be conspicuous due to the lighting. Furthermore, the robot's travel route is determined based on the generated map, and the robot is made to move. This configuration allows for the generation of a map that determines a safe travel route that avoids collisions with performers or objects on the stage, and enables the image-capturing robot to travel along the route determined based on the map. [Explanation of Symbols]

[0422] 10 stages 11 Lighting 12 speakers 13 monitors 14 Decorative Objects 20 performers 30 spectators 31 TV cameras 35 Priority Spectators 50 Image-capturing robots 100 Information Processing Devices 110 Storage section 111 Performer's Schedule Data 112 Stage lighting control planned data 113 Stage Object Placement Plan Data 114 Priority audience, TV camera viewpoint position data 120 Individual Potential Map Generation Unit 130 Potential Map Synthesis Section 150 Pre-generated potential maps 160 Driving route determination section 165 Driving Route Information 170 Driving Control Unit 180 Information Processing Systems 200 Information Processing Devices 201 Real-time stage information acquisition unit 202 Real-time Audience Seating Information Acquisition Unit 203 Real-time Internet Information Acquisition Department 204 Real-time Stage Information Analysis Department 205 Storage section 206 Pre-generated potential map modification section 207 Real-time Area of ​​Interest Analysis Unit 208 Real-time data reflection potential map generation unit 240 Driving route generation unit 280 Information Processing Systems 290 SNS servers 301 CPU 302 ROM 303 RAM 304 Bus 305 Input / Output Interface 306 Input section 307 Output section 308 Storage section 309 Communications Department 310 Drive 311 Removable Media

Claims

1. It has a data processing unit that generates the travel route of a moving device that moves on the stage, The aforementioned data processing unit The system acquires data on the planned locations of obstacles, including people and objects on the stage, that may obstruct the movement of the mobile device, or data on planned environmental control of the stage environment that changes the visibility level of the mobile device from outside the stage, and An information processing device that generates a travel route based on acquired data that reduces the possibility of physical interference between the mobile device and the obstacle, or at least one of the level of visibility of the mobile device from outside the stage.

2. The aforementioned data processing unit The information processing device according to claim 1, which generates the travel route using a map that defines at least one of the following area divisions: area divisions corresponding to the possibility of physical interference between the mobile device and the obstacle, or area divisions corresponding to the visibility level of the mobile device from outside the stage.

3. The aforementioned map is The information processing device according to claim 2, which is a map defining the area where the mobile device is permitted to travel and the area where it is prohibited to travel.

4. The aforementioned map is The information processing device according to claim 3, which is a map that defines a driving caution area between the driving permit area and the driving prohibition area.

5. The aforementioned data processing unit The information processing device according to claim 2, which generates the driving route using the time-series data of the map.

6. The data for the aforementioned environmental control plan is: The planned control data for the lighting on the aforementioned stage, The aforementioned data processing unit The information processing device according to claim 1, which generates a region as the travel route in which the visibility level of the mobile device from outside the stage decreases due to the control of the lighting on the stage.

7. The planned control data for the aforementioned lighting is: The information processing device according to claim 1, which generates the travel route using a map that defines area divisions according to the brightness of the lighting projected onto the stage, which is data for controlling the brightness of the lighting.

8. The planned control data for the aforementioned lighting is: The information processing device according to claim 1, which generates the travel route using a map that defines area divisions according to the color of the lighting to be projected onto the stage, which is the planned data for controlling the color of the lighting.

9. The aforementioned data processing unit (a) A performer base map that defines areas on the stage where the likelihood of collision with performers is low as permissible driving areas, based on the performer's planned actions on the stage. (b) An object-based map that defines areas on the stage where the likelihood of collision with objects is low as a drivable area, based on the planned placement data of objects on the stage. (c) A lighting base map defined as a driving permissible area, in which the area where the visibility level of the mobile device from outside the stage is low, calculated based on the lighting irradiated onto the stage, is determined based on the planned lighting control data on the stage. The information processing device according to claim 1, which generates the driving route using a pre-generated map obtained by synthesizing the three types of individual maps described in (a) to (c) above.

10. The aforementioned pre-generated map is The information processing device according to claim 9, which is a map that defines as a permissible travel area an area in which the likelihood of collision with performers and objects on the stage is low, and in which the visibility level of the mobile device from outside the stage, calculated based on lighting, is low.

11. The aforementioned pre-generated map is The information processing device according to claim 9, wherein each of the three types of individual maps (a) to (c) is subjected to area-level numerical processing for each of the permitted driving area, driving caution area, and driving prohibited area, a numerical value is calculated by adding the numerical values ​​of each individual map after numerical processing, and a map is generated by resetting the areas of the permitted driving area, driving caution area, and driving prohibited area based on the calculated numerical value.

12. The aforementioned data processing unit The information processing device according to claim 1, which acquires viewpoint position data of at least one of the viewpoint position data of spectators on the audience seating side viewing the stage and viewpoint position data of a camera, and generates the driving route using a map that defines a driving permissible area as a position far from the line connecting the viewpoint position of the spectator or the camera to the center position of the stage, based on the acquired data.

13. The aforementioned data processing unit (a) A performer base map that defines areas on the stage where the likelihood of collision with performers is low as permissible driving areas, based on the performer's planned actions on the stage. (b) An object-based map that defines areas on the stage where the likelihood of collision with objects is low as a drivable area, based on the planned placement data of objects on the stage. (c) A lighting base map defined as a driving permissible area, in which the area where the visibility level of the mobile device from outside the stage is low, calculated based on the lighting irradiated onto the stage, is determined based on the planned lighting control data on the stage. (d) A spectator & camera base map that defines a permissible driving area as a position far from the line connecting the spectator or camera's planned viewpoint position to the center of the stage, based on planned viewpoint position data of at least one of the spectator's planned viewpoint position data on the spectator's side of the spectator seating area and the camera's planned viewpoint position data. The information processing device according to claim 1, which generates the driving route using a pre-generated map obtained by synthesizing the four types of individual maps described in (a) to (d) above.

14. The aforementioned data processing unit The information processing device according to claim 1, which acquires real-time data of at least one of the performer's behavior data or the placement data of objects on the stage as real-time data during the performance by the performer on the stage, and generates the driving route using a real-time data reflection map that defines areas on the stage where the likelihood of collision with the performer or objects is low as a driving-permitted area based on the acquired real-time data.

15. The aforementioned data processing unit As real-time data during the performance by the performer on the aforementioned stage, The information processing device according to claim 14, which acquires data on at least one of the viewing direction of spectators on the audience side of the stage and the shooting direction of the camera, analyzes the area of ​​interest based on the acquired data, and generates the driving route using a real-time data reflection map which defines the area other than the area of ​​interest as a driving-permitted area.

16. The aforementioned data processing unit further, The information processing device according to claim 14, which obtains user comments from users viewing the performance by the performers on the stage via the internet, analyzes the areas of interest of the internet users based on the obtained comments, and generates the driving route using a real-time data reflection map that defines areas other than the areas of interest as areas where driving is permitted.

17. A storage unit that stores travel route information that reduces the possibility of physical interference between the mobile device and the obstacles, or the visibility level of the mobile device from outside the stage, based on data of the planned positions of obstacles on the stage, including people and objects, that may obstruct the mobile device's movement on the stage, or planned environmental control data of the stage environment that changes the visibility level of the mobile device from outside the stage, or It has at least one communication unit that acquires the aforementioned driving route information from an external device, A mobile device that travels on the stage according to either the travel route information obtained from the storage unit or the travel route information obtained via the communication unit.

18. The aforementioned driving route information is, The mobile device according to claim 17, wherein the travel route information is generated using a map that defines at least one of the following area divisions: area divisions corresponding to the possibility of physical interference between the mobile device and the obstacle, or area divisions corresponding to the visibility level of the mobile device from outside the stage.

19. It is an information processing system that includes a mobile device that moves around on the stage and a server. The aforementioned server, The system includes a data processing unit that generates the travel route of the aforementioned mobile device, The aforementioned data processing unit The system acquires data on the planned locations of obstacles on the stage, including people and objects, that may obstruct the movement of the mobile device on the stage, or data on planned environmental control of the stage environment that changes the visibility level of the mobile device from outside the stage. Based on the acquired data, a travel route is generated that reduces the possibility of physical interference between the mobile device and the obstacle, or at least one of the visibility level of the mobile device from outside the stage. The aforementioned mobile device is An information processing system that travels according to a travel route generated by the aforementioned server.

20. The data processing unit of the server is The information processing system according to claim 19, which generates the travel route using a map that defines at least one of the following area divisions: area divisions corresponding to the possibility of physical interference between the mobile device and the obstacle, or area divisions corresponding to the visibility level of the mobile device from outside the stage.

Images