Information processing device, mobile device, and information processing system

Abstract

The present invention generates a map for determining a safe travel route that avoids collision with a performer and an object on a stage, and causes an image capturing robot to travel in accordance with a route determined on the basis of the map. The present invention generates a potential map defining a travel allowable area of the image capturing robot that moves on the stage and captures an image. A data processing unit acquires planned action data of the performer on the stage, planned placement data of an object on the stage, and planned data of lighting control on the stage, and uses the acquired data to generate a potential map defining, as the travel allowable area, an area that avoids collision with the performer and the object and is not noticeable due to lighting. Further, the present invention determines the travel route of the robot on the basis of the generated map and causes the robot to travel.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a mobile device, and an information processing system. More specifically, the present disclosure relates to an information processing device, a mobile device, and an information processing system that, in a configuration in which a performance such as a singer by a performer on stage is filmed by a camera attached to a mobile device (autonomous traveling robot) that moves on the stage, generates a map that sets a route that does not collide with the performer or objects such as speakers on the stage and does not obstruct the view of the audience, and controls movement according to the map. [Background technology]

[0002] When various performances are held on stage, such as live music concerts, a process of capturing images of the performance on stage with a mobile camera traveling on the same stage may be performed.

[0003] When taking such images, the camera is attached to a mobile device (cart) such as an automatic traveling robot and moves to various positions on the stage to take images from various angles. In this case, the camera must select a travel route that will avoid colliding with performers moving around on the stage or with equipment such as microphones and speakers installed on the stage. They are also required to move in a way that does not obstruct the view of the audience in front of or around the stage.

[0004] Incidentally, examples of prior art that disclose movement control of moving devices such as robots include Patent Document 1 (JP 2020-087061 A) and Patent Document 2 (JP Patent No. 5160322 A).

[0005] Patent Document 1 discloses an unmanned mobile object that monitors people without getting in the way of the people being monitored. Specifically, the device determines an area where the mobile object is unlikely to be detected by the sensory organs of the people being monitored, and monitors from that determined area.

[0006] Furthermore, Patent Document 2 discloses a robot device that follows a certain object, and discloses a configuration that enables the robot to continue the following process even when an obstacle comes between the object and the robot and the robot is about to lose sight of the object.

[0007] The above-mentioned Patent Document 1 discloses a configuration for selecting an area where it is difficult for the person being monitored to sense and monitoring from the selected position, while the above-mentioned Patent Document 2 discloses a configuration for continuing tracking when an obstacle appears between the robot and the target being followed; each of these merely discloses a configuration for achieving a specific purpose.

[0008] On the other hand, when filming performers moving around on a stage with a camera moving around the same stage, it is necessary to determine the position of the camera depending on various circumstances, such as the performers' positions, the position of the equipment on the stage, and the audience's line of sight. The above-mentioned Patent Documents 1 and 2 do not disclose a control configuration for determining the optimum path for moving the camera in consideration of such various situations. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Japanese Patent Publication No. 2020-087061 [Patent Document 2] Patent No. 5160322 Summary of the Invention [Problem to be solved by the invention]

[0010] The present disclosure has been made in consideration of the above-mentioned problems, for example, and provides an information processing device, a mobile device, and an information processing system that determine the optimal position and travel route of a camera in accordance with various circumstances, such as the movement position of a performer performing on stage, the position of equipment on stage, and the line of sight of the audience, and thereby controls the movement of the mobile device (camera). [Means for solving the problem]

[0011] A first aspect of the present disclosure provides: a data processing unit that generates a potential map that defines an allowable travel area for an image capturing robot that moves on the stage and captures images; The data processing unit The information processing device acquires at least one of data on the planned behavior of performers on the stage or data on the planned placement of objects on the stage, and based on the acquired data, generates a potential map that defines an allowable driving area where the probability of collision with performers or objects on the stage is below a specified threshold.

[0012] Furthermore, a second aspect of the present disclosure is a memory unit storing travel route information generated based on a potential map, which is a map generated based on at least one of planned behavior data for performers on stage and planned placement data for objects on the stage, and which defines an allowable travel area where the probability of collision with performers or objects on the stage is below a specified threshold; or a communication unit that acquires the travel route information from an external device, The mobile device executes a travel process in accordance with either the travel route information acquired from the storage unit or the travel route information acquired via the communication unit.

[0013] Furthermore, a third aspect of the present disclosure is An information processing system having an image capturing robot and a server, the image capturing robot is an image capturing robot that moves on a stage and captures images, The server a data processing unit that generates a potential map that defines an allowable travel area for the image capturing robot; The data processing unit acquiring at least one of planned behavior data of performers on the stage and planned placement data of objects on the stage, and generating a potential map based on the acquired data that defines an allowable travel area where the probability of collision with performers or objects on the stage is below a specified threshold; The image capturing robot The information processing system travels along a travel route determined based on the potential map generated by the server.

[0014] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description of the embodiments of the present disclosure and the accompanying drawings. Note that in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are located within the same housing.

[0015] According to the configuration of one embodiment of the present disclosure, a map can be generated to determine a safe driving route that will avoid colliding with performers or objects on stage, and an image-capturing robot can be made to drive along the route determined based on the map. Specifically, for example, a potential map is generated that defines the allowable travel area for an image capturing robot that moves around a stage and captures images. The data processing unit acquires planned data on the actions of performers on the stage, planned data on the placement of objects, and planned data on lighting control on the stage, and generates a potential map that defines, as the allowable travel area, an area that will not collide with performers or objects and will not be noticeable due to lighting, based on the acquired data. Furthermore, the robot's travel route is determined based on the generated map, and the robot is made to travel. This configuration makes it possible to generate a map that determines a safe driving route that will avoid colliding with performers or objects on the stage, and to make the image-taking robot drive along the route determined based on the map. The effects described in this specification are merely examples and are not limiting, and additional effects may also be present. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram illustrating an overview of a live stage and an overview of the processing of the present disclosure. [Figure 2] 1 is a diagram illustrating an overview of a live stage and an overview of the processing of the present disclosure. [Figure 3] 1 is a diagram illustrating an overview of a live stage and an overview of the processing of the present disclosure. [Figure 4] 1 is a diagram illustrating a configuration and processing of an information processing device according to the present disclosure. [Figure 5] FIG. 1 is a diagram illustrating the configuration and processing of an information processing system according to the present disclosure. [Figure 6] 10A and 10B are diagrams illustrating an example of a performer-based potential map generated by an individual potential map generating unit. [Figure 7] 10A and 10B are diagrams illustrating an example of an illumination-based potential map generated by an individual potential map generating unit. [Figure 8] 10A and 10B are diagrams illustrating an example of an object-based potential map generated by an individual potential map generating unit. [Figure 9] 10A and 10B are diagrams illustrating an example of a pre-generated potential map generated by a potential map synthesis unit; [Figure 10] 1 is a diagram illustrating a configuration and processing of an information processing device according to the present disclosure. [Figure 11] FIG. 10 is a diagram illustrating an example of an image capturing robot travel route determined based on a pre-generated potential map. [Figure 12] FIG. 10 is a diagram showing a flowchart illustrating a process sequence for generating a performer-based potential map executed by an information processing device of the present disclosure. [Figure 13] FIG. 10 is a diagram showing a flowchart illustrating a generation processing sequence of an illumination-based potential map executed by the information processing device of the present disclosure. [Figure 14] FIG. 10 is a diagram showing a flowchart illustrating a generation processing sequence of an illumination-based potential map executed by the information processing device of the present disclosure. [Figure 15] FIG. 10 is a diagram showing a flowchart illustrating a generation processing sequence of an object-based potential map executed by an information processing device of the present disclosure. [Figure 16] FIG. 10 is a diagram showing a flowchart illustrating a generation processing sequence of a pre-generated potential map executed by an information processing device of the present disclosure. [Figure 17] 10A and 10B are diagrams illustrating examples of simulation images generated by the information processing device of the present disclosure. [Figure 18] FIG. 10 is a diagram illustrating a process according to a second embodiment of the present disclosure. [Figure 19] FIG. 10 is a diagram illustrating a process according to a second embodiment of the present disclosure. [Figure 20] FIG. 10 is a diagram illustrating a configuration example and processing of an information processing device according to a second embodiment of the present disclosure. [Figure 21] FIG. 11 is a diagram illustrating an example of a priority spectator and TV camera-based potential map generated by an information processing device according to a second embodiment of the present disclosure. [Figure 22] FIG. 10 is a diagram showing a flowchart illustrating a processing sequence for generating a priority spectator and TV camera-based potential map executed by the information processing device of the present disclosure. [Figure 23] FIG. 10 is a diagram showing a flowchart illustrating a generation processing sequence of a pre-generated potential map executed by an information processing device according to a second embodiment of the present disclosure. [Figure 24] FIG. 11 is a diagram illustrating a configuration example and processing of an information processing device according to a third embodiment of the present disclosure. [Figure 25] FIG. 10 is a diagram illustrating the configuration and processing of an information processing system according to a third embodiment of the present disclosure. [Figure 26] FIG. 11 is a diagram illustrating an example of a potential map generated in an information processing device according to a third embodiment of the present disclosure. [Figure 27] FIG. 11 is a diagram illustrating an example of a potential map generated in an information processing device according to a third embodiment of the present disclosure. [Figure 28]FIG. 11 is a diagram showing a flowchart illustrating a generation processing sequence of a real-time data reflecting potential map executed by an information processing device according to a third embodiment of the present disclosure. [Figure 29] FIG. 11 is a diagram showing a flowchart illustrating a generation processing sequence of a real-time data reflecting potential map executed by an information processing device according to a third embodiment of the present disclosure. [Figure 30] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0017] The information processing device, mobile device, and information processing system of the present disclosure will be described in detail below with reference to the drawings. The description will be made according to the following items. 1. Overview of the Disclosure Process 2. (First embodiment) Details of the configuration and processing of an information processing device according to the first embodiment of the present disclosure 3. Sequence of processing executed by the information processing device according to the first embodiment of the present disclosure 3-(1) Map a: Generation sequence of the "performer-based potential map" 3-(2) Map b "Illumination-based Potential Map" Generation Sequence 3-(3) Map c: Generation sequence of "object-based potential map" 3-(4) Generation sequence of pre-generated potential map by combining individual potential maps 4. (Example 2) Configuration and processing details of an information processing device that creates a map taking into account the line of sight of spectators and television cameras on the spectator seats 5. (Example 3) Example of generating a potential map reflecting real-time data using information during live execution 6. Hardware configuration examples for each device 7. Summary of the Disclosure

[0018] [1. Overview of the Disclosure Process] First, an overview of the processing of the present disclosure will be described with reference to FIG. 1 and subsequent figures.

[0019] 1 is a diagram showing an example of a performance such as a live music concert held on a stage. In the example shown in FIG. 1, performers 20, a duo of idol singers, are performing a live music concert on a stage 10. In front of the stage 10, there is a large audience 30 watching the performance of the performer 20.

[0020] When various performances are performed on stage, such as at a live music concert, a process of capturing images of the performance may be carried out using a mobile camera traveling on the same stage.

[0021] The image capturing robot 50 shown in FIG. 1 is a mobile device equipped with a camera, that is, a traveling robot, which moves around on the stage and captures the performance of the performer 20 from various angles.

[0022] The image capturing robot 50 is an automatically traveling robot (mobile device) such as a cart equipped with a camera, and captures images from various angles while traveling on the stage, for example, following a predetermined travel route.

[0023] The image capturing robot 50 must move along a safe route to avoid colliding with performers moving around on the stage or with equipment such as microphones and speakers installed on the stage. They are also required to move in a way that does not obstruct the view of the audience in front of or around the stage.

[0024] As shown in FIG. 2, on a stage 10 where a live performance actually takes place, speakers 12, monitors 13, and various decorative objects 14 are placed, and lighting 11 is also used to illuminate performers 20 and the like.

[0025] As the live performance progresses, the performers 20 move around the stage, and their movements change the position, brightness, and color of the lights 11. The decorative objects 14 are also replaced one after another as the live performance progresses.

[0026] For example, the stage state shown in Figure 2 is a scene from a live performance at time t1. In the stage state at a later time t2, the position of the performer 20 will be different, and the position, brightness, and color of the lighting 11 will also be changed, as shown in Figure 3. Furthermore, the decorative objects 14 will also be replaced.

[0027] In this way, the position of the performer 20 on stage, the position, brightness and color of the lighting 11, the position of the decorative objects, etc. are changed sequentially depending on the time of day when a live performance or the like is being held. During the performance of such a live performance, the image capturing robot 50 must travel in a manner that does not collide with the performers 20 or with objects placed on the stage, such as the decorative objects 14 and the speakers 12 .

[0028] It is also important to select a route that does not obstruct the line of sight of spectators 30. One effective method for this is to control the vehicle to run in a dark area other than the brightly lit area.

[0029] This disclosure controls the movement of a mobile device (camera) by determining the optimal position and travel route of the camera depending on various conditions, such as the movement positions of performers performing on stage, the position of equipment on the stage, lighting, and the audience's line of sight. The configuration and processing of the present disclosure will be described in detail below.

[0030] [2. (First embodiment) Details of configuration and processing of information processing device according to the first embodiment of the present disclosure] The configuration and processing of the information processing device according to the first embodiment of the present disclosure will be described in detail below.

[0031] FIG. 4 is a diagram illustrating a configuration example of the information processing apparatus 100 according to the first embodiment of the present disclosure. In addition, this information processing device 100 may be configured inside a mobile device that runs on the stage shown in Figures 1 to 3, i.e., an image-capturing robot 50 equipped with a camera, or it may be a device independent of the image-capturing robot 50, such as a device that can communicate with the image-capturing robot 50.

[0032] FIG. 5 shows an example of the configuration of an information processing system in which the information processing device having the configuration shown in FIG. 4 is a device independent of the image capturing robot 50. For example, as shown in FIG. 5, an information processing system 180 is constructed in which an information processing device (server) 100, a live venue information acquisition device 60, an image capturing robot 50 in the live venue, etc. are connected via a communication network. The information processing device (server) 100 has the configuration shown in FIG.

[0033] The live venue information acquisition device 60 is composed of a camera that captures images of the live venue, a microphone that acquires audio information from the live venue, a light meter that detects lighting conditions, a color analysis device, etc., and transmits information such as acquired images to the information processing device (server) 100 via a communication network.

[0034] The information processing device (server) 100 performs processes such as generating a map (potential map) for determining the travel route of the image capturing robot 50 within the live venue, determining the travel route using the map, and generating travel control information for the image capturing robot 50 according to the determined travel route.

[0035] The information processing device (server) 100 further transmits the generated driving control information to the image capturing robot 50 via a communication network. The image capturing robot 50 moves on the stage in accordance with the movement control information received from the information processing device (server) 100. For example, it is possible to perform processing using such an information processing system.

[0036] The configuration and processing of the information processing device 100 shown in FIG. 4 will be described below. 4, the information processing device 100 has a storage unit 110, an individual potential map generating unit 120, a potential map combining unit 130, and a travel route generating unit 160. As described above, the information processing device 100 may be configured inside the mobile device that travels on the stage shown in FIGS. 1 to 3, i.e., the image capturing robot 50 equipped with a camera, or may be a device independent of the image capturing robot 50, such as an information processing device (server) 100 shown in FIG. 5, which is a device that can communicate with the image capturing robot 50.

[0037] The storage unit 110 stores the following three pieces of data: A. Performer action schedule data 111, B. Stage lighting control schedule data 112, C. Planned object placement data on stage 113,

[0038] These three pieces of scheduled data are data that are prepared in advance before the start of the live performance and stored in the storage unit 110. In other words, they are scheduled data that are prepared in advance according to a program such as a live performance progress chart that is prepared before the start of the live performance.

[0039] The information processing device 100 uses this scheduled data to generate a map (potential map) for setting a safe travel route for the image capturing robot 50 that will prevent it from colliding with objects such as performers and speakers. During the actual live performance, the image capturing robot 50 is caused to travel along a safe travel route selected using the generated potential map.

[0040] The data stored in the storage unit 110 will now be described in detail. A. The performer behavior schedule data 111 is time-series position data of the performers who move around on stage during the live performance, that is, time-series position data of the performers from the start to the end of the live performance.

[0041] B. Stage lighting control schedule data 112 is time-series data of lighting control information including lighting setting information such as lighting position, brightness, color, etc. of lighting from the start to the end of a live performance (live show).

[0042] C. On-stage object placement plan data 113 is time-series data of on-stage object placement position information including the placement positions of on-stage objects from the start to the end of a live performance. The stage objects include speakers, monitors, decorative objects, and the like that are placed on the stage.

[0043] All three types of data are time-series schedule data from the start to the end of a live performance. For example, if the live performance lasts for one hour, A. Performer action schedule data 111, B. Stage lighting control schedule data 112, C. Planned object placement data on stage 113, These three types of one-hour time-series schedule data are recorded in the storage unit 110.

[0044] The data stored in the storage unit 110 is used in the individual potential map generating unit 120 . The individual potential map generating section 120 generates the following three types of individual potential maps by individually using the above three types of time series data A, B, and C. Map a = performer-based potential map, Map b = illumination-based potential map, map c = object-based potential map,

[0045] The potential map is a map that defines travel-prohibited areas, travel-caution areas, and travel-allowed areas for the image capturing robot 50 described with reference to FIGS. The above three types of maps a to c are all time-series maps.

[0046] For example, map a = performer-based potential map is a time-series map in which each area (no-driving area, driving caution area, driving permitted area) changes dynamically depending on the performer's position from the start to the end of the live performance. Map b = lighting-based potential map is a time series map in which each area (no driving area, driving caution area, driving permitted area) changes dynamically depending on the lighting conditions (lighting position, brightness, color, etc.) from the start to the end of the live performance. Map c = Object-based potential map is a time-series map in which each area (no-driving area, driving caution area, driving permitted area) changes dynamically depending on the object placement position from the start to the end of the live performance.

[0047] As shown in FIG. 4, the individual potential map generating section 120 executes the processes of steps S11 to S13 shown in the figure. That is, In step S11, a map a = a performer-based potential map is generated. In step S12, a map b=illumination-based potential map is generated. In step S13, a map c=an object-based potential map is generated. The processes in steps S11 to S13 can be executed as parallel processes. These processes will be explained below in order.

[0048] (Step S11) In step S11, the individual potential map generating unit 120 generates a map a = "performer-based potential map." As mentioned above, the "performer-based potential map" is a map in which each area (no driving area, driving caution area, driving permitted area) changes dynamically depending on the performer's position from the start to the end of the live performance, and is a time-series map with three areas (a1 to a3) set as follows. a1. No driving area (red) = close to the performer's position a2. Driving caution area (yellow) = mid-distance from the performer's position a3. Allowable travel area (blue) = far distance from the performer's position

[0049] Specifically, for example, a1. No-driving area (red) = Area where the possibility of collision with the performer is greater than or equal to the first threshold Tha1. a2. Driving caution area (yellow) = Area where the possibility of collision with the performer is within the range of the first threshold Tha1 to the second threshold Tha2 that has been predefined a3. Allowable driving area (blue) = Area where the possibility of collision with the performer is below the predefined second threshold Tha2 For example, a map with these area divisions.

[0050] A specific example of map a = performer-based potential map will be described with reference to FIG. The "performer-based potential map" is a time series map in which each area (no driving area, driving caution area, and driving permitted area) changes dynamically depending on the performer's position from the start to the end of the live performance. Figure 6 shows examples of performer-based potential maps for four timings (t1 to t4).

[0051] The map (t1) in the upper left of Figure 6 is an example of a "performer-based potential map" at time t1. As explained above with reference to FIGS. 1 to 3, there are two performers on the stage. As mentioned above, the potential map is a map generated before the actual start of the live performance, and the positions of the performers are the planned positions of the performers estimated according to the live program such as the live performance schedule. Planned behavior data indicating the positions of the performers from the start to the end of the live performance (performance) is generated and recorded in advance as A. Performer Planned Behavior Data 111 in the storage unit 110.

[0052] The "performer-based potential map" is a map in which different colors are assigned to each area (no-driving area, caution-driving area, permitted driving area) determined according to the distance from the performer's position as follows: Areas close to the performer's position are set in red as no-driving areas. The mid-distance position of the performer is set to yellow as a driving caution area. The far distance position of the performer is set in blue as a travel-allowed area.

[0053] The map (t2) in the upper right of Figure 6 is an example of a "performer-based potential map" at time t2, a certain time after time t1. At time t2, the two actors move to different positions than at time t1. As the actors move, the settings of the three areas—no driving zone (red), caution zone (yellow), and permitted zone (blue)—also change.

[0054] The map (t3) at the bottom left of Figure 6 is an example of a "performer-based potential map" at time t3, a certain time after time t2. At time t3, the two actors move to different positions than at times t1 and t2. As the actors move, the settings of the three areas—no driving zone (red), caution zone (yellow), and permitted zone (blue)—also change.

[0055] The map (t4) at the bottom right of Figure 6 is an example of the "performer-based potential map" at time t4, a certain time after time t3. At time t4, the two performers have moved to different positions than those at times t1 to t3. As the performers move, the settings of the three areas—no driving zone (red), caution zone (yellow), and permitted zone (blue)—also change.

[0056] In this way, the "performer-based potential map" is a time-series map in which each area (no-driving area, driving caution area, allowed driving area) changes dynamically depending on the performer's position from the start time to the end time of the live performance. In step S11, the individual potential map generating unit 120 generates such a map a = "performer-based potential map."

[0057] The behavior schedule data indicating the positions of the performers from the start to the end of the live performance is generated and recorded in advance as A. Performer behavior schedule data 111 in the storage unit 110. In step S11, the individual potential map generation unit 120 acquires A. performer behavior schedule data 111 from the memory unit 110, and by referring to the acquired data, generates a time series map a = "performer-based potential map" in which each area (no-driving area, driving caution area, driving permitted area) is dynamically changed from the start to the end of the live performance (live show).

[0058] (Step S12) Furthermore, in step S12, the individual potential map generating unit 120 generates map b="illumination-based potential map". As described above, the "lighting-based potential map" is a time series map in which each area (no driving area, driving caution area, driving permitted area) changes dynamically depending on the lighting conditions (lighting position, brightness, color, etc.) from the start to the end of the live performance, and is a time series map in which three areas (b1 to b3) are set as follows: b1. No-drive area (red) = Areas that stand out due to lighting conditions (bright areas or areas with lighting of a different color than the image-taking robot) b3. Allowable travel area (blue) = Area that is not noticeable due to lighting conditions (dark areas or areas with lighting of the same color as the image capture robot) b2. Driving caution area (yellow) = Area between b1 and b3

[0059] Specifically, for example, b1. No-travel area (red) = Area where the lighting condition value calculated based on the lighting condition (lighting position, brightness, color, etc.) is equal to or greater than the first threshold value Thb1 (= conspicuous area (bright area or lighting area of ​​a different color from the image-taking robot)) b3. Allowable travel area (blue) = Area where the lighting condition value calculated based on the lighting condition (lighting position, brightness, color, etc.) is less than the predetermined second threshold value Thb2 (= inconspicuous area (dark area or lighting area of ​​similar color to the image capturing robot)) b2. Driving caution area (yellow) = Area between b1 and b3 For example, a map with these area divisions.

[0060] A specific example of the map b (illumination-based potential map) will be described with reference to FIG. The "lighting-based potential map" is a time-series map in which each area (no-driving area, driving caution area, driving permitted area) changes dynamically according to the lighting conditions (lighting position, brightness, color, etc.) from the start to the end of the live performance. Figure 7 shows examples of lighting-based potential maps for four timings (t1 to t4).

[0061] The map (t1) in the upper left of FIG. 7 is an example of an "illumination-based potential map" at time t1. This map (t1) is an example of a map when the lighting is generally dark, for example, at the start of a live performance. In this case, since the entire stage is set to be dark, the entire stage is set as the permitted travel area (blue) = an area that is not noticeable due to the lighting conditions (a dark area or an area illuminated in a similar color to the image-taking robot).

[0062] The map (t2) in the upper right of FIG. 7 is an example of an "illumination-based potential map" at time t2, a certain time after time t1. At time t2, the lighting conditions (lighting position, brightness, color, etc.) are set to a different state than at time t1. As the lighting conditions (lighting position, brightness, color, etc.) change, the settings of these three areas - no-driving area (red), caution-driving area (yellow), and permitted-driving area (blue) - also change.

[0063] The "lighting-based potential map" is a map in which different colors are assigned to each area (no-driving area, caution-warning area, permitted driving area) that are determined according to the lighting conditions (lighting position, brightness, color, etc.) as follows: That is, areas that stand out due to lighting conditions (bright areas or areas illuminated in a different color from the image capturing robot) are set in red as no-travel areas. Areas that are not noticeable due to lighting conditions (dark areas or areas illuminated in a similar color to the image capturing robot) are set to blue as permitted travel areas. The intermediate area between the no-travel area and the permitted travel area is set in yellow as a caution area.

[0064] In this way, at time t2, the settings of these three areas - no-driving area (red), caution-driving area (yellow), and permitted driving area (blue) - change as the lighting conditions (lighting position, brightness, color, etc.) change.

[0065] The map (t3) at the bottom left of FIG. 7 is an example of an "illumination-based potential map" at time t3, a certain time after time t2. At time t3, the lighting conditions (lighting position, brightness, color, etc.) are different from those at times t1 and t2. As the lighting conditions (lighting position, brightness, color, etc.) change, the settings of the three areas - no-driving area (red), caution-driving area (yellow), and permitted-driving area (blue) - also change.

[0066] The map (t4) at the bottom right of FIG. 7 is an example of an "illumination-based potential map" at time t4, a certain time after time t3. At time t4, the lighting conditions (lighting position, brightness, color, etc.) are set to different states from those at times t1 to t3. Along with this change in lighting conditions (lighting position, brightness, color, etc.), the settings of these three areas - no-driving area (red), caution-driving area (yellow), and permitted-driving area (blue) - also change.

[0067] In this way, the "lighting-based potential map" is a time-series map in which each area (no-driving area, driving caution area, allowed driving area) changes dynamically in response to changes in the lighting conditions (lighting position, brightness, color, etc.) from the start time to the end time of the live performance.

[0068] The individual potential map generating unit 120 generates such a map b = "illumination-based potential map" in step S12.

[0069] The transition schedule data for the lighting state (lighting position, brightness, color, etc.) from the start to the end of the live performance is generated and recorded in advance as B. Stage lighting control schedule data 112 in the storage unit 110. In step S12, the individual potential map generation unit 120 acquires B. Stage lighting control schedule data 112 from the memory unit 110, and by referring to the acquired data, generates a time series map b = "lighting-based potential map" in which each area (no-driving area, driving caution area, driving permitted area) is dynamically changed from the start to the end of the live performance (performance).

[0070] (Step S13) In step S13, the individual potential map generating unit 120 generates map c="object-based potential map". As mentioned above, the "object-based potential map" is a map in which each area (no-driving area, driving caution area, driving permitted area) changes dynamically depending on the position of objects on the stage from the start to the end of the live performance, and is a time-series map with three areas (c1 to c3) set as follows. c1. No driving area (red) = Close to the object placement position c2. Driving caution area (yellow) = Mid-distance position of object placement position c3. Allowable travel area (blue) = far distance from object placement position

[0071] Specifically, for example, c1. No-driving area (red) = Area where the possibility of collision with an object is greater than or equal to the first threshold value Thc1. c2. Caution area (yellow) = Area where the possibility of collision with an object is within the range of the first threshold value Thc1 to the second threshold value Thc2. c3. Allowable travel area (blue) = Area where the possibility of collision with an object is below the second threshold value Thc2 specified in advance For example, a map with these area divisions.

[0072] Note that the objects are objects that are placed on the stage, such as speakers, monitors, and decorative objects.

[0073] A specific example of the map c (object-based potential map) will be described with reference to FIG. The "object-based potential map" is a time-series map in which each area (no-driving area, caution-driving area, permitted driving area) changes dynamically depending on the object placement position from the start to the end of the live performance. Figure 8 shows examples of object-based potential maps at four timings (t1 to t4).

[0074] The map (t1) in the upper left of FIG. 8 is an example of an "object-based potential map" at time t1. Speakers, monitors, decorative objects, etc. are placed on the stage, some of which are moved or replaced as the live performance progresses.

[0075] The "object-based potential map" is a map in which different colors are assigned to each area (no-driving area, caution area, permitted area) determined according to the distance from the object placement position as follows: The area close to the object placement position is set in red as a no-travel area. The mid-distance position of the object placement position is set to yellow as a driving caution area. The far-distance position of the object placement position is set in blue as a travel-allowed area.

[0076] The map (t2) in the upper right of FIG. 8 is an example of an "object-based potential map" at time t2, a certain time after time t1. At time t2, the objects on the stage have been moved or replaced to different positions than at time t1. As the objects are moved or replaced, the settings of the three areas - no-driving area (red), caution area (yellow), and allowed area (blue) - also change.

[0077] The map (t3) at the bottom left of FIG. 8 is an example of an "object-based potential map" at time t3, a certain time after time t2. At time t3, the object on the stage has moved or been replaced to a different position than at times t1 and t2. As the object moves or is replaced, the settings of the three areas - no-driving area (red), caution-driving area (yellow), and allowed-driving area (blue) - also change.

[0078] The map (t4) at the bottom right of FIG. 8 is an example of an "object-based potential map" at time t4, a certain time after time t3. At time t4, the objects on the stage have been moved to different positions or replaced from those at times t1 to t3. As the objects are moved or replaced, the settings of the three areas—no driving area (red), caution area (yellow), and allowed area (blue)—also change.

[0079] In this way, the "object-based potential map" is a time-series map in which each area (no-driving area, driving caution area, driving permitted area) changes dynamically depending on the position of objects on the stage from the start time to the end time of the live performance. In step S13, the individual potential map generating unit 120 generates such a map c="object-based potential map".

[0080] Note that on-stage object placement plan data indicating the placement positions of objects from the start to the end of the live performance is generated and recorded in advance as C. on-stage object placement plan data 113 in the storage unit 110. In step S13, the individual potential map generation unit 120 acquires the planned on-stage object placement data 113 from the memory unit 110, and by referring to the acquired data, generates a time-series map c = "object-based potential map" in which each area (no-driving area, driving caution area, driving permitted area) is dynamically changed from the start to the end of the live performance (performance).

[0081] As described above, the individual potential map generating unit 120 shown in FIG. 4 generates the following three maps in steps S11 to S13. In step S11, a map a = a performer-based potential map is generated. In step S12, a map b=illumination-based potential map is generated. In step S13, a map c=an object-based potential map is generated.

[0082] These three individual potential maps generated by the individual potential map generating unit 120, namely, Map a = performer-based potential map, Map b = illumination-based potential map, map c = object-based potential map, These three maps are input to the potential map synthesis unit 130.

[0083] The potential map synthesis unit 130 executes synthesis processing of the three individual potential maps generated by the individual potential map generation unit 120 to generate a pre-generated potential map 150. The pre-generated potential map 150 is also a time-series map in which each area (no-driving area, driving caution area, and driving permitted area) changes dynamically from the start to the end of the live performance.

[0084] The pre-generated potential map 150 generated by the potential map synthesis unit 130 is a combination of the three individual potential maps generated by the individual potential map generation unit 120, namely: Map a = performer-based potential map, Map b = illumination-based potential map, Map c = Object-based potential map It is generated as synthetic data that reflects all three of these time series data.

[0085] FIG. 9 shows a specific example of the pre-generated potential map 150 generated by the potential map synthesis unit 130.

[0086] FIG. 9 shows an example of pre-generated potential maps 150 at four timings (t1 to t4) similar to the individual potential maps described with reference to FIGS.

[0087] The pre-generated potential maps for the four timings (t1 to t4) shown in FIG. 9 are maps generated by synthesizing the individual potential maps for the four same timings (t1 to t4) shown in FIGS. 6 to 8 for each timing unit.

[0088] The map (t1) in the upper left of Figure 9 is a "pre-generated potential map" at time t1. The "pre-generated potential map" at time t1 is The performer-based potential map at time t1 shown in Figure 6(t1) The illumination-based potential map at time t1 shown in FIG. 7(t1) The object-based potential map at time t1 shown in Figure 8(t1) This map is a composite of these three individual potential maps taken at the same timing (t1).

[0089] On the stage, two performers are positioned in the same positions as in Figure 6(t1), and objects such as speakers, monitors, and decorative objects are placed in the same positions as in Figure 8(t1). The lighting is set to darken the entire stage, just like in Figure 7(t1).

[0090] A specific sequence of the synthesis process will be described later, but for example, the synthesis process is performed by the following process. The areas of each individual potential map (no driving areas, driving caution areas, and driving permitted areas) are quantified. for example, No driving area = 10, Driving attention area = 5, Allowable travel area = 0 After such digitization, the numerical values ​​of each region of each individual potential map are added, and a pre-generated potential map is generated as a composite map based on the addition result.

[0091] For example, a composite map, i.e., a pre-generated potential map, is generated in which areas with an added value of 10 or more are set as no-driving areas, areas with an added value of 5 or more but less than 10 are set as caution areas, and areas with an added value of less than 5 are set as permitted areas.

[0092] The "pre-generated potential map" shown in Figure 9(t1) is The performer-based potential map at time t1 shown in Figure 6(t1) The illumination-based potential map at time t1 shown in FIG. 7(t1) The object-based potential map at time t1 shown in Figure 8(t1) This map was generated by quantifying and adding up each area (no driving area, driving caution area, and driving permitted area) of these three individual potential maps at the same timing (t1), and then dividing the areas based on the above-mentioned added values.

[0093] The "pre-generated potential map" is a map in which no-driving areas, caution-over-driving areas, and permitted driving areas are determined taking into consideration the performer's position, lighting conditions (lighting position, brightness, color), and object placement positions, and the determined areas are colored in the following different colors. No-travel areas are colored red. The driving caution area is set to yellow. The permitted travel area is set in blue.

[0094] The map (t2) in the upper right of FIG. 9 is an example of a composite map at time t2, a certain time after time t1, that is, a "pre-generated potential map." At time t2, the performer position, lighting conditions (lighting position, brightness, color), and object placement positions are set differently from those at time t1.

[0095] The "pre-generated potential map" at time t2 is The performer-based potential map at time t2 shown in Figure 6(t2) The illumination-based potential map at time t2 shown in FIG. 7(t2) The object-based potential map at time t2 shown in Figure 8(t2) This map was generated by quantifying and adding up each area (no driving area, driving caution area, and driving permitted area) of these three individual potential maps at the same timing (t2), and then dividing the area based on the added value.

[0096] The map (t3) at the bottom left of FIG. 9 is an example of a composite map at time t3, a certain time after time t2, that is, a "pre-generated potential map." At time t3, the performer position, lighting conditions (lighting position, brightness, color), and object placement positions are set differently from those at times t1 and t2.

[0097] The "pre-generated potential map" at time t3 is The performer-based potential map at time t3 shown in Figure 6(t3) The illumination-based potential map at time t3 shown in FIG. 7(t3) The object-based potential map at time t3 shown in Fig. 8(t3) This map was generated by quantifying and adding up each area (no driving area, driving caution area, and driving permitted area) of these three individual potential maps at the same timing (t3), and then dividing the areas based on the added values.

[0098] The map (t4) at the bottom right of FIG. 9 is an example of a composite map at time t4, a certain time after time t3, that is, a "pre-generated potential map." At time t4, the performer position, lighting conditions (lighting position, brightness, color), and object placement positions are set differently from those at times t1 to t3.

[0099] The "pre-generated potential map" at time t4 is The performer-based potential map at time t4 shown in Figure 6(t4) The illumination-based potential map at time t4 shown in FIG. 7(t4) The object-based potential map at time t4 shown in Fig. 8(t4) This map was generated by quantifying and adding up each area (no driving area, driving caution area, and driving permitted area) of these three individual potential maps at the same timing (t4), and then dividing the areas based on the added values.

[0100] In this way, the potential map synthesis unit 130 synthesizes the three individual potential maps generated by the individual potential map generation unit 120, i.e., Map a = performer-based potential map, Map b = illumination-based potential map, Map c = Object-based potential map A synthesis process is performed on these individual potential maps to generate the pre-generated potential map 150.

[0101] The potential map synthesis unit 130 digitizes and adds up each area (no-driving area, caution-driving area, permitted driving area) of multiple individual potential maps at the same time, and divides the areas based on the added values ​​to generate the pre-generated potential map 150.

[0102] The pre-generated potential map 150 generated by the potential map synthesis unit 130 is provided to the running route generation unit 160, which determines the running route of the image capturing robot 50 from the start to the end of the live performance based on the pre-generated potential map 150.

[0103] That is, as shown in FIG. 10, the travel route generation unit 160 inputs the pre-generated potential map 150 generated by the potential map synthesis unit 130, and generates travel route information 165 that sets a route for the image capturing robot 50 to travel from the start to the end of the live performance (live show) without colliding with performers or objects on the stage and by selecting areas that will not stand out due to lighting.

[0104] The driving route generation unit 160 generates a driving route that travels only within the permitted driving area in the pre-generated potential map 150, for example.

[0105] The generated travel route is provided to a travel control unit 170 that controls the travel of the image capturing robot 50, and the travel control unit 170 causes the image capturing robot 50 to travel in accordance with the generated travel route information 165. The traveling control unit 170 may be configured as an information processing device within the image capturing robot 50, or may be configured as an information processing device capable of communicating with robots outside the image capturing robot 50.

[0106] In this way, by making the image capturing robot 50 travel according to the travel route information 165 generated using the pre-generated potential map 150, it is possible to make the image capturing robot 50 travel from the start to the end of the live performance without colliding with performers or objects on the stage, and by selecting areas that will not be noticeable due to lighting.

[0107] An example of the travel route information 165 generated using the pre-generated potential map 150 will be described with reference to FIG. FIG. 11 shows an example of the travel route of the image capturing robot 50 from time (t3) to time (t4).

[0108] The travel route of the image capturing robot 50 from time (t3) to time (t4) shown in FIG. 11 is a travel route set so as to travel only within the travel-permitted area within the pre-generated potential map 150 generated by the potential map synthesis unit 130.

[0109] In this way, by making the image capturing robot 50 travel along a travel route that is set so that it travels only within the travel-permitted areas within the pre-generated potential map 150, it is possible to select areas where the robot will travel without colliding with performers or objects on the stage from the start to the end of the live performance, and where the robot will not stand out due to lighting.

[0110] 3. Sequence of processing executed by information processing device according to first embodiment of the present disclosure Next, a processing sequence executed by the information processing apparatus according to the first embodiment of the present disclosure will be described.

[0111] The flowcharts shown in FIG. 12 and subsequent figures are flowcharts illustrating the sequence of processes executed by information processing device 100 of the present disclosure, which was previously described with reference to FIG. The flowcharts shown in FIGS. 12 to 15 are sequences of processing executed by the individual potential map generating section 120 of the information processing device 100, and correspond to the generation sequences of the following three types of individual potential maps. (1) Figure 12 = Generation sequence of Map a "Performer-based Potential Map" (2) Figures 13-14 = Generation sequence of map b "lighting-based potential map" (3) Figure 15 = Map c "Object-based potential map" generation sequence

[0112] 16 is a process sequence for generating a "pre-generated potential map" which is a composite map executed by the potential map synthesis unit 130 of the information processing device 100 shown in FIG.

[0113] The processing according to the flow described below can be executed, for example, according to a program stored in a storage unit of an information processing device, and is executed under the control of a control unit having a program execution function such as a CPU. Details of the processing of the flow shown in Figure 14 will be explained below.

[0114] [3-(1) Map a: About the generation sequence of the "performer-based potential map"] First, the sequence for generating the map a "performer-based potential map" executed by the individual potential map generating unit 120 of the information processing device 100 will be described with reference to the flowchart shown in FIG.

[0115] The processing according to the flowchart shown in FIG. 12 corresponds to a detailed sequence of the processing for generating map a = performer-based potential map, which is the processing of step S11 executed by the individual potential map generating unit 120 described above with reference to FIG. 4.

[0116] In other words, this is a detailed sequence of the process for generating Map a = Performer-Based Potential Map, which is a time-series map in which each area (no-driving area, driving caution area, driving permitted area) is dynamically changed depending on the performer's position from the start to the end of the live performance.

[0117] The processing of each step in the flow shown in FIG. 12 will be explained below in order. (Step S101) First, in step S101, the data processing unit (individual potential map generating unit 120) of the information processing device 100 acquires the number of performers (=na) during the live performance period of the live performance for which the performer-based potential map is to be generated. This is performed, for example, as a process of obtaining performer action schedule data generated based on a pre-set live program, i.e., "A. Performer action schedule data 111" stored in the memory unit 110 of the information processing device 100 shown in Figure 4.

[0118] (Step S102) Next, in step S102, the data processing unit of the information processing device 100 selects one performer P to be analyzed.

[0119] (Step S103) Next, in step S103, the data processing unit of the information processing device 100 acquires time-series behavior data of the performer P to be analyzed from the start to the end of the live performance.

[0120] This process is also performed, for example, as a process of obtaining performer action schedule data generated based on a pre-set live program, i.e., "A. Performer action schedule data 111" stored in the memory unit 110 of the information processing device 100 shown in Figure 4.

[0121] (Step S104) Next, in step S104, the data processing unit of the information processing device 100 generates a potential map based on the time-series behavior data of the performer P to be analyzed from the start to the end of the live performance.

[0122] This is a process that is executed in accordance with the process previously described with reference to Figures 4 and 6, and determines three areas (no driving area, driving caution area, and driving permitted area) depending on the distance from the position of the performer P being analyzed at each time from the start to the end of the live performance, and assigns different colors to each determined area.

[0123] Specifically, the following area and color settings are made. Areas close to the position of performer P are set in red as no-driving areas. The mid-distance position of the performer P is set to yellow as a driving caution area. The far distance position of the performer P is set in blue as a travel-allowed area.

[0124] This process generates a performer-based potential map corresponding to one performer P to be analyzed.

[0125] (Step S105) Next, in step S105, the data processing unit of the information processing device 100 determines whether or not there are any performers who have not yet been analyzed. That is, it is determined whether or not the generation of all performer-based potential maps for the number na of performers acquired in step S101 has been completed.

[0126] If there are any unprocessed performers, the determination in step S105 is Yes, and in this case, the processing from step S102 onwards is executed for the unprocessed performers. On the other hand, if there are no unprocessed actors, that is, if it is determined that the generation of all actor-based potential maps for the number of actors na acquired in step S101 has been completed, the determination in step S105 is No. In this case, proceed to step S106.

[0127] (Step S106) When the generation of all performer-based potential maps is completed, the data processing unit of the information processing device 100 executes the processes from step S106 onwards.

[0128] First, in step S106, the set regions of each of the na potential maps corresponding to all the performers 1 to na are quantified. For example, No driving area = 10, Driving attention area = 5, Allowable travel area = 0 Such area units are quantified.

[0129] (Step S107) Next, in step S107, the data processing unit of the information processing device 100 adds up the numerical values ​​corresponding to the regions of the potential maps of all the performers 1 to na for each region to calculate an added value corresponding to the region.

[0130] For example, suppose the number of performers is 3, individual performer-based potential maps (m1 to m3) corresponding to the three performers have been generated, and the area settings for a certain stage position (x1, y1) at a certain time tx are as follows: Map m1 value = 5 (driving caution area) Map m2 value = 5 (driving caution area) Map m3 value = 0 (allowable driving area) In this case, the added value is 5+5+0=10. This addition process is performed for all maps, which are time-series data, i.e., for all stage positions on the maps at all times.

[0131] (Step S108) Next, in step S108, the data processing unit of the information processing device 100 resets the area divisions based on the sum values ​​corresponding to the areas of the potential maps of all the performers 1 to na.

[0132] For example, the area is reset according to the following rules: Addition value of 10 or more = No driving zone, Addition value of 5 points or more = driving caution area, Addition value less than 5 points = Driving allowable area, For example, the area is reset according to the above rules.

[0133] (Step S109) Next, in step S109, the data processing unit of the information processing device 100 outputs the potential map in which the region divisions have been reset in step S108 to the potential map synthesis unit 130 as a "performer-based potential map."

[0134] The above is the detailed sequence of the process of generating the map a (performer-based potential map) executed by the individual potential map generating unit 120. This process generates Map a (Performer-Based Potential Map), a time-series map in which each area (no driving area, driving caution area, driving permitted area) changes dynamically depending on the performer's position from the start to the end of the live performance.

[0135] [3-(2) Map b "Illumination-based Potential Map" Generation Sequence] Next, the generation sequence of map b "illumination-based potential map" executed by the individual potential map generating unit 120 of the information processing device 100 will be described with reference to the flowcharts shown in FIGS.

[0136] The processing according to the flowcharts shown in FIGS. 13 and 14 corresponds to a detailed sequence of the generation processing of map b=illumination-based potential map, which is the processing of step S12 executed by the individual potential map generation unit 120 described above with reference to FIG. 4.

[0137] In other words, this is a detailed sequence of the process for generating map b = lighting-based potential map, which is a time-series map in which each area (no-driving area, driving caution area, driving permitted area) is dynamically changed according to the lighting conditions (lighting position, brightness, color, etc.) from the start to the end of the live performance.

[0138] The processing of each step in the flow shown in FIGS. 13 and 14 will be explained below in order. (Step S121) First, in step S121, the data processing unit (individual potential map generating unit 120) of the information processing device 100 acquires the number of illumination position division regions (=nb) during the live performance period of the live performance for which an illumination-based potential map is to be generated.

[0139] This process is performed, for example, as a process of obtaining lighting control schedule data generated based on a pre-set live program, i.e., "B. Stage lighting control schedule data 112" stored in the memory unit 110 of the information processing device 100 shown in Figure 4.

[0140] (Step S122) Next, in step S122, the data processing unit of the information processing device 100 selects one illumination position division area Q to be analyzed.

[0141] (Step S123) Next, in step S123, the data processing unit of the information processing device 100 acquires time-series illumination brightness information from the start to the end of the live broadcast for one illumination position segment area Q selected as the analysis target.

[0142] This is also performed as a process of obtaining lighting control schedule data generated based on a pre-set live program, i.e., "B. Stage lighting control schedule data 112" stored in the memory unit 110 of the information processing device 100 shown in Figure 4.

[0143] (Step S124) Next, in step S124, the data processing unit of the information processing device 100 generates a potential map based on the time-series illumination brightness data from the start to the end of the live broadcast for one illumination position segment area Q selected as the analysis target.

[0144] This is a process that is executed in accordance with the process previously described with reference to Figures 4 and 7, and determines three areas (no driving area, driving caution area, and driving permitted area) depending on the brightness of the lighting position division area Q at each time from the start to the end of the live performance, and assigns different colors to each area depending on the determined area.

[0145] Specifically, the following area and color settings are made. If the brightness of the illumination position section area Q is equal to or greater than the specified threshold value Thd1, the area is set to red as a no-travel area. When the brightness of the illumination position section area Q is within the range of the specified threshold values ​​Thd1 to Thd2, it is set to yellow as a driving caution area. When the brightness of the illumination position section area Q is equal to or less than the specified threshold value Thd2, the area is set to blue as a travel-permitted area.

[0146] By this process, an illumination brightness-based potential map corresponding to one illumination position division region Q is generated.

[0147] (Step S125) Next, in step S125, the data processing unit of the information processing device 100 determines whether or not there is an illumination position division area that has not yet been analyzed. That is, it is determined whether or not generation of all illumination brightness based potential maps for the number nb of illumination position segmented regions acquired in step S121 has been completed.

[0148] If there are any unprocessed illumination position division areas, the determination in step S125 is Yes, and in this case, the processes from step S122 onward are executed for the unprocessed illumination position division areas. On the other hand, if there is no unprocessed illumination position division area, that is, if it is determined that the generation of all illumination brightness based potential maps of the illumination position division area nb acquired in step S121 has been completed, the determination in step S125 is No. In this case, proceed to step S126.

[0149] (Step S126) When the generation of all illumination brightness based potential maps is completed, the data processing unit of the information processing device 100 executes the processes from step S126 onwards.

[0150] First, in step S126, one illumination position division area Q to be analyzed is selected.

[0151] (Step S127) Next, in step S127, the data processing unit of the information processing device 100 acquires time-series illumination color information from the start to the end of the live broadcast in one illumination position division area Q selected as the analysis target.

[0152] This is also performed as a process of obtaining lighting control schedule data generated based on a pre-set live program, i.e., "B. Stage lighting control schedule data 112" stored in the memory unit 110 of the information processing device 100 shown in Figure 4.

[0153] (Step S128) Next, in step S128, the data processing unit of the information processing device 100 generates a potential map based on the time-series illumination light color data from the start to the end of the live performance for one illumination position segment area Q selected as the analysis target.

[0154] This is a process that is executed in accordance with the process previously described with reference to Figures 4 and 7, and determines three areas (no driving area, driving caution area, and driving permitted area) according to the lighting light color of the lighting position division area Q at each time from the start to the end of the live performance, and assigns different colors to each area according to the determined area.

[0155] Specifically, the following area and color settings are made. If the illumination color of the illumination position section area Q is different from that of the image capturing robot, it is set to red as a no-travel area. Even if the illumination color of the illumination position section area Q is different from that of the image capturing robot, if it is not a similar color, it is set to yellow as a driving caution area. If the illumination color of the illumination position section area Q is similar to that of the image capturing robot, the area is set to blue as a travel-allowed area.

[0156] By this process, an illumination color-based potential map corresponding to one illumination position division region Q is generated.

[0157] (Step S129) Next, in step S129, the data processing unit of the information processing device 100 determines whether or not there is an illumination position division area that has not yet been analyzed. That is, it is determined whether or not generation of all illumination color based potential maps for the number nb of illumination position division areas acquired in step S121 has been completed.

[0158] If there are any unprocessed illumination position division areas, the determination in step S129 is Yes, and in this case, the processes in step S126 and thereafter are executed for the unprocessed illumination position division areas. On the other hand, if there is no unprocessed illumination position division area, that is, if it is determined that the generation of all illumination color based potential maps of the illumination position division area nb acquired in step S121 has been completed, the determination in step S129 is No. In this case, proceed to step S131.

[0159] (Step S131) When the generation of all illumination color-based potential maps is completed, the data processing unit of the information processing device 100 executes the processes from step S131 onwards.

[0160] In step S131, the data processing unit of the information processing device 100 digitizes the set areas of the illumination brightness-based potential map and the illumination color-based potential map. For example, No driving area = 10, Driving attention area = 5, Allowable travel area = 0 Such area units are quantified.

[0161] (Step S132) Next, in step S132, the data processing unit of the information processing device 100 adds the numerical values ​​corresponding to the set regions of the illumination brightness-based potential map and the illumination color-based potential map for each region to calculate an added value corresponding to the region.

[0162] For example, suppose the region settings at a certain stage position (x1, y1) at a certain time tx are as follows: Lighting brightness based potential map m1 value = 5 (driving caution area) Lighting color based potential map m2 value = 0 (permitted driving area) In this case, the added value is 5+0=5. This addition process is performed for all maps, which are time-series data, i.e., for all stage positions on the maps at all times.

[0163] (Step S133) Next, in step S133, the data processing unit of the information processing device 100 resets the area divisions based on the added values ​​corresponding to the areas of the illumination brightness-based potential map and the illumination color-based potential map.

[0164] For example, the area is reset according to the following rules: Addition value of 10 or more = No driving zone, Addition value of 5 points or more = driving caution area, Addition value less than 5 points = Driving allowable area, For example, the area is reset according to the above rules.

[0165] (Step S134) Next, in step S134, the data processing unit of the information processing device 100 outputs the potential map whose area divisions have been reset in step S133 to the potential map synthesis unit 130 as an "illumination-based potential map."

[0166] The above is the detailed sequence of the process for generating the map b (illumination-based potential map) executed by the individual potential map generating unit 120. This process generates map b (lighting-based potential map), which is a time-series map in which each area (no-driving area, driving caution area, driving permitted area) changes dynamically depending on the lighting conditions (lighting position, brightness, color, etc.) from the start to the end of the live performance.

[0167] [3-(3) Map c "Object-based Potential Map" Generation Sequence] First, the generation sequence of map c "object-based potential map" executed by the individual potential map generating unit 120 of the information processing device 100 will be described with reference to the flowchart shown in FIG.

[0168] The processing according to the flowchart shown in FIG. 15 corresponds to a detailed sequence of the processing for generating map c=object-based potential map, which is the processing of step S13 executed by the individual potential map generating unit 120 described above with reference to FIG. 4.

[0169] In other words, this is a detailed sequence of the process for generating map c = object-based potential map, which is a time-series map in which each area (no-driving area, driving caution area, driving permitted area) is dynamically changed according to the position of objects on the stage from the start to the end of the live performance.

[0170] Note that the objects are objects that are placed on the stage, such as speakers, monitors, and decorative objects.

[0171] The processing of each step in the flow shown in FIG. 15 will be explained below in order. (Step S151) First, in step S151, the data processing unit (individual potential map generating unit 120) of the information processing device 100 acquires the number of objects (=nc) during the live performance period of the live performance for which the object-based potential map is to be generated. This is performed, for example, as a process of obtaining data on planned stage object placement generated based on a pre-set live program, i.e., "C. Planned stage object placement data 113" stored in the memory unit 110 of the information processing device 100 shown in Figure 4.

[0172] (Step S152) Next, in step S152, the data processing unit of the information processing device 100 selects one object Ob to be analyzed.

[0173] (Step S153) Next, in step S153, the data processing unit of the information processing device 100 acquires time-series arrangement data of the analysis target object Ob from the start to the end of the live performance.

[0174] This process is also executed as a process of obtaining data on planned stage object placement generated based on a pre-set live program, i.e., "C. Planned stage object placement data 113" stored in the memory unit 110 of the information processing device 100 shown in Figure 4.

[0175] (Step S154) Next, in step S154, the data processing section of the information processing device 100 generates a potential map based on the time-series arrangement data of the analysis target object Ob from the start to the end of the live performance.

[0176] This is a process that is executed in accordance with the process previously described with reference to Figures 4 and 8, and determines three areas (no-driving area, driving caution area, and driving permitted area) according to the distance from the placement position of the object Ob to be analyzed at each time from the start to the end of the live performance, and assigns different colors to each determined area.

[0177] Specifically, the following area and color settings are made. The area close to the placement position of the object Ob is set in red as a no-travel area. The mid-distance position of the object Ob is set to yellow as a driving caution area. The far-distance position of the object Ob is set to blue as a permissible travel area.

[0178] This process generates an object-based potential map corresponding to one analysis target object Ob.

[0179] (Step S155) Next, in step S155, the data processing unit of the information processing device 100 determines whether or not there is an unanalyzed object. That is, it is determined whether or not the generation of all object-based potential maps for the number of objects nc acquired in step S151 has been completed.

[0180] If there are any unprocessed objects, the determination in step S155 is Yes, and in this case, the processes from step S152 onwards are executed for the unprocessed objects. On the other hand, if there are no unprocessed objects, that is, if it is determined that generation of all object-based potential maps for the number of objects nc acquired in step S151 has been completed, the determination in step S155 is No. In this case, proceed to step S156.

[0181] (Step S156) When the generation of all object-based potential maps is completed, the data processing unit of the information processing device 100 executes the processes from step S156 onwards.

[0182] First, in step S156, the set regions of the nc potential maps corresponding to the individual objects 1 to nc are quantified. For example, No driving area = 10, Driving attention area = 5, Allowable travel area = 0 Such area units are quantified.

[0183] (Step S157) Next, in step S157, the data processing unit of the information processing device 100 adds up the numerical values ​​corresponding to the regions of the potential maps of all the objects 1 to nc for each region to calculate an added value corresponding to the region.

[0184] For example, suppose the number of objects is 3, individual object-based potential maps (m1 to m3) corresponding to the three objects have been generated, and the region settings at a certain stage position (x1, y1) at a certain time tx are as follows: Map m1 value = 5 (driving caution area) Map m2 value = 5 (driving caution area) Map m3 value = 0 (allowable driving area) In this case, the added value is 5+5+0=10. This addition process is performed for all maps, which are time-series data, i.e., for all stage positions on the maps at all times.

[0185] (Step S158) Next, in step S158, the data processing unit of the information processing device 100 resets the area divisions based on the added values ​​corresponding to the areas of the potential maps of all the objects 1 to nc.

[0186] For example, the area is reset according to the following rules: Addition value of 10 or more = No driving zone, Addition value of 5 points or more = driving caution area, Addition value less than 5 points = Driving allowable area, For example, the area is reset according to the above rules.

[0187] (Step S159) Next, in step S159, the data processing unit of the information processing device 100 outputs the potential map whose region divisions have been reset in step S158 to the potential map synthesis unit 130 as an "object-based potential map."

[0188] The above is the detailed sequence of the process for generating the map c (object-based potential map) executed by the individual potential map generating unit 120. This process generates map c (object-based potential map), which is a time-series map in which each area (no-driving area, driving caution area, driving permitted area) changes dynamically depending on the object placement position from the start to the end of the live performance.

[0189] [3-(4) Generation sequence of pre-generated potential map by combining individual potential maps] Next, a sequence for generating a pre-generated potential map executed by the potential map synthesis unit 130 of the information processing device 100 will be described with reference to the flowchart shown in FIG.

[0190] As previously described with reference to FIG. 4, the potential map synthesis unit 130 synthesizes the three individual potential maps generated by the individual potential map generation unit 120, namely, Map a = performer-based potential map, Map b = illumination-based potential map, Map c = Object-based potential map These three potential maps are synthesized to generate a pre-generated potential map, which is synthesized data that reflects all of the time-series data of these three potential maps.

[0191] The flowchart shown in Fig. 16 is a sequence for generating a pre-generated potential map executed by the potential map synthesis unit 130. The processing of each step of the flow shown in Fig. 16 will be explained below in order.

[0192] (Step S171) First, in step S171, the data processing unit (potential map synthesis unit 130) of the information processing device 100 shown in FIG. 4 synthesizes the three individual potential maps generated by the individual potential map generation unit 120, i.e., Map a = performer-based potential map, Map b = illumination-based potential map, Map c = Object-based potential map The set area of ​​each of these three potential maps is quantified. For example, No driving area = 10, Driving attention area = 5, Allowable travel area = 0 Such area units are quantified.

[0193] (Step S172) Next, in step S172, the data processing unit of the information processing device 100 generates three individual potential maps, namely, Map a = performer-based potential map, Map b = illumination-based potential map, Map c = Object-based potential map The numerical values ​​corresponding to the set regions of these three potential maps are added for each region to calculate the sum corresponding to the region.

[0194] For example, suppose the region settings at a certain stage position (x1, y1) at a certain time tx are as follows: Map a = performer-based potential map value = 5 (driving attention area) Map b = Lighting-based potential map value = 0 (allowable driving area) Map c = Object-based potential map value = 5 (driving attention area) In this case, the added value is 5+0+5=10. This addition process is performed for all maps, which are time-series data, i.e., for all stage positions on the maps at all times.

[0195] (Step S173) Next, in step S173, the data processing unit of the information processing device 100 generates three individual potential maps, namely, Map a = performer-based potential map, Map b = illumination-based potential map, Map c = Object-based potential map The area divisions are redefined based on the sum of the corresponding area values ​​of these three potential maps.

[0196] For example, the area is reset according to the following rules: Addition value of 10 or more = No driving zone, Addition value of 5 points or more = driving caution area, Addition value less than 5 points = Driving allowable area, For example, the area is reset according to the above rules.

[0197] (Step S174) Next, in step S174, the data processing unit of the information processing device 100 generates the potential map whose region divisions have been reset in step S173 as a composite map, that is, a "pre-generated potential map."

[0198] The above is the detailed sequence of the process for generating the pre-generated potential map 150 executed by the potential map synthesis unit 130. In this way, the potential map synthesis unit 130 synthesizes the three individual potential maps generated by the individual potential map generation unit 120, i.e., Map a = performer-based potential map, Map b = illumination-based potential map, Map c = Object-based potential map A synthesis process is performed on these individual potential maps to generate the pre-generated potential map 150.

[0199] The pre-generated potential map 150 generated by the potential map synthesis unit 130 is provided to the running route generation unit 160, which determines the running route of the image capturing robot 50 from the start to the end of the live performance based on the pre-generated potential map 150.

[0200] That is, as explained above with reference to FIG. 10, the travel route generation unit 160 inputs the pre-generated potential map 150 generated by the potential map synthesis unit 130, and generates travel route information 165 that sets a route for the image capturing robot 50 to travel from the start to the end of the live performance (live show) without colliding with performers or objects on the stage and by selecting areas that will not stand out due to lighting. For example, a driving route is generated that travels only within the permitted driving area in the pre-generated potential map 150.

[0201] The generated travel route is provided to a travel control unit 170 that controls the travel of the image capturing robot 50, and the travel control unit 170 causes the image capturing robot 50 to travel in accordance with the generated travel route information 165. In this way, by making the image capturing robot 50 travel according to the travel route information 165 generated using the pre-generated potential map 150, it is possible to make the image capturing robot 50 travel from the start to the end of the live performance without colliding with performers or objects on the stage, and by selecting areas that will not be noticeable due to lighting.

[0202] The travel route of the image capturing robot 50 generated using the pre-generated potential map 150 can be displayed as a simulation image on the display unit of the information processing device 100, for example.

[0203] The data processing unit of the information processing device 100 generates simulation data for displaying the planned traveling route of the image capturing robot 50 from the start to the end of the live performance, based on the traveling route information 165 generated using the generated pre-generated potential map 150. A specific display example of this simulation data is shown in FIG.

[0204] As shown in Figure 17, a user such as a robot control operator can check the running position of the image capturing robot 50 at each time from the start (ts) to the end (te) of the live broadcast by moving the slider left and right. FIG. 17 shows an example of display data indicating the traveling positions of the image capturing robot 50 at time t3 and time t4 as a display example of simulation data.

[0205] By referring to this simulation data, a user such as a robot control operator can check the driving route generated using the generated pre-generated potential map 150.

[0206] [4. (Example 2) Configuration and Processing Details of an Information Processing Device that Creates Maps Taking into Account the Viewpoints of Spectators and the Viewpoints of Television Cameras on the Spectator Seat Side] Next, as a second embodiment of the present disclosure, a configuration and processing details of an information processing device that creates a map taking into consideration the line of sight of spectators and the line of sight of a television camera on the spectator seat side will be described.

[0207] In the above-described first embodiment, when generating a potential map for determining a travel route of the image capturing robot 50, the following three individual maps are generated and used. Map a = performer-based potential map, Map b = illumination-based potential map, map c = object-based potential map,

[0208] In the above-described first embodiment, the three individual potential maps are generated separately, and then these are combined to generate a pre-generated potential map, and the generated pre-generated potential map is used to determine the travel route of the image capturing robot 50. For example, the configuration was such that a process was performed to select a travel-allowed area in a pre-generated potential map and determine the travel route of the image capturing robot 50.

[0209] The second embodiment described below is an embodiment in which, in addition to the processing of the first embodiment described above, a fourth individual potential map is created that takes into account the line of sight of the spectators and the line of sight of the television camera on the spectator seating side, and a pre-generated potential map is generated that also takes this fourth individual potential map into account.

[0210] The following describes the line of sight of the spectators and the line of sight of the television camera on the spectator seating side, which are taken into consideration in this second embodiment, with reference to FIG.

[0211] The image capturing robot 50 shown in Figure 18 is a mobile device equipped with a camera, i.e., a running robot, similar to the image capturing robot 50 shown in Figure 1 described above, and moves around the stage capturing the performance of the performer 20 from various angles.

[0212] However, as shown in FIG. 18, when the image capturing robot 50 captures an image of the performer 20 from the front, it may get between the performer 20 and the audience 30 or TV camera 31 on the auditorium side.

[0213] In this state, even if the audience 30 turns their gaze in the direction of the performer 20, the image capturing robot 50 will be in their field of vision, making it difficult for them to see the performer 20, which is a problem. The same is true for the TV camera 31 on the audience side; even if the TV camera 31 aims its shooting direction (line of sight) in the direction of the performer 20 to photograph him, the image-taking robot 50 will enter the captured image and interfere with the filming of the performer 20, which is a problem. The following second embodiment is an embodiment that solves this problem.

[0214] An example of a live concert venue where the processing of the second embodiment is performed is shown in FIG. As shown in FIG. 19, a stage 10 where a live performance actually takes place is equipped with speakers 12, monitors 13, and various decorative objects 14, and also has lighting 11 directed at performers 20.

[0215] As the performers 20 move around the stage in accordance with the progress of the live performance, the position, brightness, and color of the lights 11 change in accordance with their movements. The decorative objects 14 are also replaced in various ways.

[0216] As shown in FIG. 19, there are many spectators 30 on the auditorium side, and TV cameras 31 are also placed there to film the live performance of the performers 20. In the second embodiment, a potential map is generated taking into consideration the viewpoint positions of a portion of the spectators 30, that is, priority spectators 35 shown in the figure, and the viewpoint position of the TV camera 31.

[0217] If a potential map is generated taking into account the viewpoint positions of all spectators, the entire stage will be set as a no-travel area, leaving almost no area for the image-taking robot 50 to travel in. Therefore, processing is performed taking into account the viewpoint positions of some priority spectators 35.

[0218] The priority spectators 35 shown in the figure are, for example, seats reserved for related parties, and are reserved seats reserved for important guests. In this second embodiment, in addition to the processing of the first embodiment, a potential map is generated taking into consideration the viewpoint positions of the priority spectators 35 and the viewpoint position of the TV camera 31. The details of the second embodiment will be described with reference to FIG. 20 and subsequent figures.

[0219] FIG. 20 is a diagram illustrating a configuration example of an information processing apparatus 100b according to a second embodiment of the present disclosure. This information processing device 100b may be configured inside a mobile device that runs on the stage shown in Figures 1 to 3, i.e., an image-capturing robot 50 equipped with a camera, or it may be a device that can communicate with the image-capturing robot 50, or it may be a device independent of the image-capturing robot 50.

[0220] The configuration of an information processing device 100b shown in FIG. 20 will be described. As shown in FIG. 20, the information processing device 100b includes a storage unit 110, an individual potential map generating unit 120, a potential map combining unit 130, and a driving route generating unit 160. These basic configurations are the same as those of the information processing device 100 described above with reference to FIG.

[0221] The information processing device 100b shown in Figure 20 may also be configured inside the mobile device that runs on the stage shown in Figures 1 to 3, i.e., the image-capturing robot 50 equipped with a camera, or it may be a device independent of the image-capturing robot 50, such as a device (server) that can communicate with the image-capturing robot 50 as previously described with reference to Figure 5.

[0222] The storage unit 110 stores the following four pieces of data: A. Performer action schedule data 111, B. Stage lighting control schedule data 112, C. Planned object placement data on stage 113, D. Priority audience, TV camera viewpoint position data 114,

[0223] These four pieces of scheduled data are data that are prepared in advance before the start of the live performance and stored in the storage unit 110. In other words, they are scheduled data that are prepared in advance according to a program such as a live performance progress chart that is prepared before the start of the live performance.

[0224] Data A to C are the same data as those described above with reference to FIG. 4 in the first embodiment. A. The performer behavior schedule data 111 is time-series position data of the performers who move around on stage during the live performance, that is, time-series position data of the performers from the start to the end of the live performance.

[0225] B. Stage lighting control schedule data 112 is time-series data of lighting control information including lighting setting information such as lighting position, brightness, color, etc. of lighting from the start to the end of a live performance (live show). C. On-stage object placement plan data 113 is time-series data of on-stage object placement position information including the placement positions of on-stage objects from the start to the end of a live performance. The stage objects include speakers, monitors, decorative objects, and the like that are placed on the stage.

[0226] In this second embodiment, in addition to these data A to C, D. Priority audience, TV camera viewpoint position data 114, This data D is stored in the storage unit 110.

[0227] D. Priority audience and TV camera viewpoint position data 114 is viewpoint position data of the priority audience on the audience seat side and the TV camera from the start to the end of the live performance (performance).

[0228] Furthermore, if the viewpoint positions of the priority audience and the TV camera change to various positions from the start to the end of the live performance (performance), D. Priority audience, TV camera viewpoint position data 114 will be time-series data that changes dynamically over time. However, if the viewpoint position of the priority audience and the viewpoint position of the TV camera are in the same position from the start to the end of the live performance (performance), D. Priority audience, TV camera viewpoint position data 114 can be one fixed data.

[0229] The data A to D stored in the storage unit 110 are used in the individual potential map generating unit 120. The individual potential map generating section 120 generates the following four types of individual potential maps by individually using the four types of data A, B, C, and D described above. Map a = performer-based potential map, Map b = illumination-based potential map, map c = object-based potential map, Map d = Priority audience, TV camera viewpoint position based potential map,

[0230] Maps a to c are the individual potential maps previously described in the first embodiment. Map a = performer-based potential map is a time series map in which each area (no driving area, driving caution area, driving permitted area) changes dynamically depending on the performer's position from the start to the end of the live performance. Map b = lighting-based potential map is a time series map in which each area (no driving area, driving caution area, driving permitted area) changes dynamically depending on the lighting conditions (lighting position, brightness, color, etc.) from the start to the end of the live performance. Map c = Object-based potential map is a time-series map in which each area (no-driving area, driving caution area, driving permitted area) changes dynamically depending on the object placement position from the start to the end of the live performance.

[0231] Map d = Priority spectator, TV camera viewpoint position based potential map is a map in which each area (no driving area, driving caution area, driving permitted area) is set according to the priority spectator on the spectator seat side and the TV camera viewpoint position. Furthermore, this map is time-series data in which the set areas (no driving areas, driving caution areas, and driving permitted areas) change dynamically over time as the viewpoint positions of priority spectators and TV cameras change to various positions from the start to the end of the live performance.

[0232] However, if the viewpoint of the priority audience and the viewpoint of the TV camera are in the same position from the start to the end of the live performance, the set areas (no driving areas, driving caution areas, driving permitted areas) will become one map that does not change dynamically.

[0233] In this second embodiment, the individual potential map generating unit 120 generates the following four maps in steps S11 to S14 shown in FIG. In step S11, a map a = a performer-based potential map is generated. In step S12, a map b=illumination-based potential map is generated. In step S13, a map c=an object-based potential map is generated. In step S14, a potential map based on the map d (priority audience) and TV camera viewpoint positions is generated.

[0234] As shown in FIG. 20, in step S14, the individual potential map generating unit 120 divides the area from the priority audience viewpoint position and the short distance position to the long distance position of the straight line connecting the viewpoint position of the TV camera to the center position of the stage into a no-run area (red), a caution area (yellow), and a permitted area (blue). Map d = Potential map based on priority audience and TV camera viewpoint positions Generate.

[0235] FIG. 21 shows a specific example of the “map d=priority spectator, TV camera viewpoint position based potential map” generated by the individual potential map generating unit 120.

[0236] "Map d = Potential map based on priority spectators and TV camera viewpoint position" is a map in which various areas (no driving areas, driving caution areas, and driving permitted areas) are set according to the priority spectators on the spectator seating side and the TV camera viewpoint position. As shown in Figure 21, a map is generated that divides the area from the priority audience viewpoint and the short distance position to the long distance position of the straight line connecting the TV camera viewpoint to the center of the stage into no-driving areas (red), driving caution areas (yellow), and driving permitted areas (blue).

[0237] As mentioned above, the set areas (no driving areas, driving caution areas, driving permitted areas) of this "map d = priority spectator, TV camera viewpoint position based potential map" can be either a dynamically changing time-series map or a single fixed map where the set areas do not change.

[0238] In this way, the individual potential map generating section 120 generates the following four individual potential maps. Map a = performer-based potential map, Map b = illumination-based potential map, map c = object-based potential map, Map d = Priority audience, TV camera viewpoint position based potential map,

[0239] These four individual potential maps generated by the individual potential map generating section 120 are input to the potential map combining section 130.

[0240] The potential map synthesis unit 130 executes synthesis processing of the four individual potential maps generated by the individual potential map generation unit 120 to generate a pre-generated potential map b, 150b. The pre-generated potential map b, 150b is also a time-series map in which each area (no-driving area, driving caution area, and driving permitted area) is dynamically changed from the start to the end of the live performance.

[0241] The "pre-generated potential map b" generated in this Example 2 is a map in which no-driving areas, driving caution areas, and driving-allowed areas are determined taking into consideration the performer positions, lighting conditions (lighting position, brightness, color), object placement positions, and even the priority audience viewpoint and TV camera viewpoint, and the determined areas are colored in the following different colors. No-travel areas are colored red. The driving caution area is set to yellow. The permitted travel area is set in blue.

[0242] The pre-generated potential maps b, 150b generated by the potential map synthesis unit 130 are provided to the running route generation unit 160, and the running route generation unit 160 determines the running route of the image capturing robot 50 from the start to the end of the live performance based on the pre-generated potential maps b, 150b.

[0243] The travel route generation unit 160 inputs the pre-generated potential maps b, 150b generated by the potential map synthesis unit 130, and generates travel route information 165 that sets a route for the image capturing robot 50 to travel from the start to the end of the live performance (live show) without colliding with performers or objects on the stage, by selecting an area that will not be conspicuous by lighting, and by selecting an area that will not interfere with the line of sight of the priority spectators 35 or the filming by the TV camera 31.

[0244] The driving route generation unit 160 generates a driving route that travels only within the permitted driving area in the pre-generated potential map b, 150b, for example.

[0245] The generated travel route is provided to a travel control unit 170 that controls the travel of the image capturing robot 50, and the travel control unit 170 causes the image capturing robot 50 to travel in accordance with the generated travel route information 165.

[0246] In this way, by making the image capturing robot 50 travel according to the travel route information 165 generated using the pre-generated potential maps b, 150b generated in this Example 2, it becomes possible to make the image capturing robot 50 travel from the start to the end of the live performance without colliding with performers or objects on the stage, and to select areas that will not be conspicuous by lighting, and that will not interfere with the line of sight of the priority spectators 35 or the filming of the TV camera 31.

[0247] Next, the sequence for generating map d "priority spectator, TV camera viewpoint position-based potential map" executed by the individual potential map generating unit 120 of the information processing device 100 will be described with reference to the flow shown in FIG.

[0248] The processing according to the flowchart shown in FIG. 22 corresponds to a detailed sequence of the processing of step S14 executed by the individual potential map generating section 120 described above with reference to FIG.

[0249] In other words, this is a detailed sequence of the process for generating a map in which each area (no driving area, driving caution area, driving permitted area) is set according to the viewpoint positions of priority spectators and TV cameras from the start to the end of the live performance. The processing of each step in the flow shown in FIG. 22 will be explained below in order.

[0250] (Step S201) First, in step S201, the data processing unit (individual potential map generation unit 120) of the information processing device 100 acquires the number of priority spectators and TV camera viewpoints (nd) during the live performance period of the live event for which the priority spectators and TV camera viewpoint-based potential map are to be generated.

[0251] This is performed, for example, as a process of obtaining priority spectator and TV camera viewpoint position data generated based on a pre-set live program, i.e., "D. Priority spectator and TV camera viewpoint viewpoint position data 114" stored in the memory unit 110 of the information processing device 100 shown in Figure 20.

[0252] (Step S202) Next, in step S202, the data processing section of information processing device 100 selects one priority spectator viewpoint or TV camera viewpoint S to be analyzed.

[0253] (Step S203) Next, in step S203, the data processing section of the information processing device 100 acquires time-series position data from the start to the end of the live broadcast of the priority spectator viewpoint or TV camera viewpoint S to be analyzed.

[0254] This process is also performed, for example, as a process of obtaining priority spectator viewpoint and TV camera viewpoint position data generated based on a pre-set live program, i.e., "D. Priority spectator, TV camera viewpoint viewpoint position data 114" stored in the memory unit 110 of the information processing device 100 shown in Figure 4.

[0255] As mentioned above, the time-series position data from the priority spectator viewpoint or TV camera viewpoint S from the start to the end of the live broadcast may be data that changes dynamically over time, or it may be a single fixed piece of data that does not change.

[0256] (Step S204) Next, in step S204, the data processing section of the information processing device 100 generates a potential map based on the position data from the start to the end of the live performance of the priority spectator viewpoint or TV camera viewpoint S selected as the analysis target.

[0257] For example, a map such as that described above with reference to Figure 21 is generated, that is, a map in which each area (no driving area, driving caution area, driving permitted area) is set according to the priority spectators on the spectator seating side and the TV camera viewpoint position. As mentioned above, the set areas (no driving areas, driving caution areas, driving permitted areas) of this "map d = priority spectator, TV camera viewpoint position based potential map" can be either a dynamically changing time-series map or a single fixed map where the set areas do not change.

[0258] This process generates a priority spectator / TV camera viewpoint-based potential map corresponding to the one priority spectator viewpoint or TV camera viewpoint S that is the subject of analysis.

[0259] (Step S205) Next, in step S205, the data processing section of information processing device 100 determines whether there is a priority spectator viewpoint or a TV camera viewpoint that has not yet been analyzed. That is, it is determined whether or not the generation of potential maps based on all priority spectators and TV camera viewpoints of the number nd of priority spectators and TV camera viewpoints acquired in step S201 has been completed.

[0260] If there is an unprocessed priority spectator viewpoint or TV camera viewpoint, the determination in step S205 is Yes. In this case, the processing from step S202 onwards is executed for the unprocessed priority spectator viewpoint or TV camera viewpoint. On the other hand, if there are no unprocessed priority spectator viewpoints or TV camera viewpoints, that is, if it is determined that the generation of all priority spectator and TV camera viewpoint-based potential maps for the number nd of priority spectator and TV camera viewpoints acquired in step S201 has been completed, the determination in step S205 is No. In this case, proceed to step S206.

[0261] (Step S206) When the generation of all priority spectator and TV camera viewpoint-based potential maps is completed, the data processing unit of the information processing device 100 executes the processes from step S206 onwards.

[0262] First, in step S206, the set areas of each of the nd potential maps corresponding to the individual priority spectator viewpoints or TV camera viewpoints 1 to nd are quantified. For example, No driving area = 10, Driving attention area = 5, Allowable travel area = 0 Such area units are quantified.

[0263] (Step S207) Next, in step S207, the data processing unit of the information processing device 100 adds up the numerical values ​​corresponding to the areas of the potential map of all the priority spectator viewpoints or TV camera viewpoints 1 to nd for each area to calculate an added value corresponding to the area.

[0264] For example, suppose the total number of priority spectator and TV camera viewpoints is 3, three individual priority spectator and TV camera viewpoint-based potential maps (m1 to m3) corresponding to the priority spectator viewpoints or TV camera viewpoints have been generated, and the area setting for a certain stage position (x1, y1) at a certain time tx is as follows: Map m1 value = 5 (driving caution area) Map m2 value = 5 (driving caution area) Map m3 value = 0 (allowable driving area) In this case, the added value is 5+5+0=10. This addition process is carried out for all maps. If the maps are time-series data, it is carried out for all stage positions on the maps at all times.

[0265] (Step S208) Next, in step S208, the data processing section of the information processing device 100 resets the area divisions based on the sum of the areas corresponding to the potential maps of all the priority spectator viewpoints or TV camera viewpoints 1 to nd.

[0266] For example, the area is reset according to the following rules: Addition value of 10 or more = No driving zone, Addition value of 5 points or more = driving caution area, Addition value less than 5 points = Driving allowable area, For example, the area is reset according to the above rules.

[0267] (Step S209) Next, in step S209, the data processing unit of the information processing device 100 outputs the potential map in which the area divisions have been reset in step S208 to the potential map synthesis unit 130 as a "priority spectator, TV camera viewpoint-based potential map."

[0268] The above is the detailed sequence of the process executed by the individual potential map generating unit 120 to generate the map a (priority spectator, TV camera viewpoint-based potential map). This process generates a time-series map in which each area (no driving areas, driving caution areas, driving permitted areas) changes dynamically depending on the priority spectator viewpoint or TV camera viewpoint position from the start to the end of the live performance, or a map with no dynamic changes: Map a = priority spectator, TV camera viewpoint-based potential map.

[0269] Next, a sequence for generating a pre-generated potential map executed by the potential map synthesis unit 130 of the information processing device 100 will be described with reference to the flowchart shown in FIG.

[0270] As previously described with reference to FIG. 20, the potential map synthesis unit 130 synthesizes the four individual potential maps generated by the individual potential map generation unit 120, i.e., Map a = performer-based potential map, Map b = illumination-based potential map, Map c = Object-based potential map Map d = Priority audience, TV camera viewpoint position based potential map, These four potential maps are synthesized to generate a pre-generated potential map b, which is synthesized data that reflects all of these four potential maps.

[0271] The flowchart shown in Fig. 23 is a generation sequence of the pre-generated potential map b executed by the potential map synthesis unit 130. The processing of each step of the flow shown in Fig. 23 will be explained below in order.

[0272] (Step S221) First, in step S221, the data processing unit (potential map synthesis unit 130) of the information processing device 100 shown in FIG. 20 synthesizes the four individual potential maps generated by the individual potential map generation unit 120, i.e., Map a = performer-based potential map, Map b = illumination-based potential map, Map c = Object-based potential map Map d = Priority audience, TV camera viewpoint position based potential map, The set area of ​​each of these four potential maps is quantified. For example, No driving area = 10, Driving attention area = 5, Allowable travel area = 0 Such area units are quantified.

[0273] (Step S222) Next, in step S222, the data processing unit of the information processing device 100 generates four individual potential maps, namely, Map a = performer-based potential map, Map b = illumination-based potential map, Map c = Object-based potential map Map d = Priority audience, TV camera viewpoint position based potential map, The numerical values ​​corresponding to the set regions of these four potential maps are added for each region to calculate the sum corresponding to the region.

[0274] For example, suppose the region settings at a certain stage position (x1, y1) at a certain time tx are as follows: Map a = performer-based potential map value = 5 (driving attention area) Map b = Lighting-based potential map value = 0 (allowable driving area) Map c = Object-based potential map value = 5 (driving attention area) Map d = priority spectator, TV camera viewpoint position based potential map = 5 (driving attention area), In this case, the added value is 5+0+5+5=15. This addition process is performed for all maps, which are time-series data, i.e., for all stage positions on the maps at all times.

[0275] (Step S223) Next, in step S223, the data processing unit of the information processing device 100 generates four individual potential maps, namely, Map a = performer-based potential map, Map b = illumination-based potential map, Map c = Object-based potential map Map d = Potential map based on priority audience and TV camera viewpoint positions The area divisions are redefined based on the sum of the corresponding area values ​​of these four potential maps.

[0276] For example, the area is reset according to the following rules: Addition value of 10 or more = No driving zone, Addition value of 5 points or more = driving caution area, Addition value less than 5 points = Driving allowable area, For example, the area is reset according to the above rules.

[0277] (Step S224) Next, in step S224, the data processing unit of the information processing device 100 generates the potential map whose region divisions have been reset in step S223 as a composite map, that is, a "pre-generated potential map b."

[0278] The above is the detailed sequence of the process for generating the pre-generated potential maps b, 150b executed by the potential map synthesis unit 130. In this way, the potential map synthesis unit 130 synthesizes the four individual potential maps generated by the individual potential map generation unit 120, i.e., Map a = performer-based potential map, Map b = illumination-based potential map, Map c = Object-based potential map Map d = Potential map based on priority audience and TV camera viewpoint positions These individual potential maps are combined to generate a pre-generated potential map b, 150b.

[0279] The pre-generated potential maps b, 150b generated by the potential map synthesis unit 130 are provided to the running route generation unit 160, and the running route generation unit 160 determines the running route of the image capturing robot 50 from the start to the end of the live performance (performance) based on the pre-generated potential map 150.

[0280] That is, as explained above with reference to FIG. 10, the travel route generation unit 160 inputs the pre-generated potential maps b, 150b generated by the potential map synthesis unit 130, and generates travel route information 165 that sets a route along which the image capturing robot 50 will travel from the start to the end of the live performance (live show) without colliding with performers or objects on the stage, by selecting areas that will not be conspicuous by lighting, and by selecting areas that will not obstruct the view of priority spectators or TV cameras. For example, a driving route is generated that travels only within the permitted driving area in the pre-generated potential map 150.

[0281] The generated travel route is provided to a travel control unit 170 that controls the travel of the image capturing robot 50, and the travel control unit 170 causes the image capturing robot 50 to travel in accordance with the generated travel route information 165. In this way, by making the image capturing robot 50 travel according to the travel route information 165 generated using the pre-generated potential map 150, it is possible to make the image capturing robot 50 travel in areas from the start to the end of the live performance without colliding with performers or objects on the stage, and in areas that are not conspicuous by lighting and do not obstruct the view of priority spectators or TV cameras.

[0282] [5. (Example 3) Example of generating a potential map reflecting real-time data using information during live execution] Next, as a third embodiment, an embodiment in which a potential map reflecting real-time data is generated using information during live execution will be described.

[0283] In the above-mentioned Examples 1 and 2, the potential map was generated using data that could be obtained before the live performance on stage began, i.e., the following data stored in the memory unit 110 of the information processing device 100 shown in Figures 4 and 20. A. Performer action schedule data 111, B. Stage lighting control schedule data 112, C. Planned object placement data on stage 113, D. Priority audience, TV camera viewpoint position data 114,

[0284] The data A to D used in Examples 1 and 2 were generated on the assumption that, for example, performers would act according to the live program, lighting would be controlled, and objects would be positioned. However, once the live performance actually begins, the movements of the performers and lighting control may differ from those originally planned.

[0285] In such a case, if the image capturing robot 50 is driven using the pre-generated potential map generated based on the first and second embodiments, there is a possibility that the robot may come into contact with the performer. The third embodiment described below is intended to prevent such a situation, and is an embodiment in which real-time data is acquired from an actual live performance, i.e., the time when the performer is performing, and a potential map is generated.

[0286] FIG. 24 shows an example of the configuration of an information processing apparatus 200 according to the third embodiment. As shown in Figure 24, the information processing device 200 has a real-time stage information acquisition unit 201, a real-time spectator seat information acquisition unit 202, a real-time net information acquisition unit 203, a real-time stage information analysis unit 204, a memory unit 205, a pre-generated potential map correction unit 206, a real-time attention area analysis unit 207, and a real-time data-reflecting potential map generation unit 208.

[0287] The information processing device 200 shown in Figure 24 may be configured, for example, inside a mobile device that runs on the stage shown in Figures 1 to 3, i.e., an image-capturing robot 50 equipped with a camera, or it may be a device independent of the image-capturing robot 50, such as a device that can communicate with the image-capturing robot 50.

[0288] FIG. 25 shows an example of the configuration of an information processing system in which the information processing device having the configuration shown in FIG. 24 is an independent device from the image capturing robot 50. For example, as shown in FIG. 25, an information processing system 280 is constructed in which an information processing device (server) 200, a live venue information acquisition device 60, an image capturing robot 50 within the live venue, and an SNS server 290 are connected via a communication network. The information processing device (server) 100 has the configuration shown in FIG.

[0289] The live venue information acquisition device 60 is composed of a camera that captures images of the live venue, a microphone that acquires audio information from the live venue, a light meter that detects lighting conditions, a color analysis device, etc., and transmits information such as acquired images to the information processing device (server) 200 via a communication network.

[0290] The information processing device (server) 200 analyzes the information received from the live venue information acquisition device 60 and the information acquired from the SNS server 290, and performs processes such as generating a map (potential map) for determining the running route of the image capturing robot 50 within the live venue, determining the running route using the map, and generating running control information for the image capturing robot 50 according to the determined running route.

[0291] The SNS server 290 is a server that collects comment information such as tweets from users who are viewing the live show online. The information processing device (server) 200 analyzes the information acquired from the SNS server 290 to analyze areas of interest of users who are viewing the live show via the internet.

[0292] The information processing device (server) 200 further transmits the generated driving control information to the image capturing robot 50 via a communication network. The image capturing robot 50 moves on the stage in accordance with the movement control information received from the information processing device (server) 100. For example, it is possible to perform processing using such an information processing system.

[0293] The configuration and processing of the information processing device 100 shown in FIG. 24 will be described below. The real-time stage information acquisition unit 201 is specifically configured with, for example, a camera, a light meter, etc., and acquires information on the actual live performance, that is, information on the stage where the performers are performing, in real time.

[0294] As shown in the figure, the real-time stage information acquisition unit 201 acquires the following information as real-time data. Speaker position 211, lighting conditions 212, Object position 213 on stage, Image capture robot position 214, The lighting state includes information on the position of the stage illuminated by the light, the brightness of the light, and the color of the light.

[0295] The real-time stage information acquisition unit 201 continuously acquires information such as performer position 211, lighting condition 212, on-stage object position 213, and image capture robot position 214 as real-time data during the live performance period, and outputs the acquired data to the real-time stage information analysis unit 204.

[0296] The real-time stage information analysis unit 204 compares each piece of information input from the real-time stage information acquisition unit 201, such as performer position 211, lighting condition 212, on-stage object position 213, and image capture robot position 214, with planned data 231 based on the live program stored in advance in the memory unit 205, and with a pre-generated potential map 150 generated according to the previous Example 1 or Example 2.

[0297] The scheduled data 231 based on the live program stored in the storage unit 205 is the same data as the following information previously described with reference to FIGS. A. Performer action schedule data 111, B. Stage lighting control schedule data 112, C. Planned object placement data on stage 113, Each of these data is generated in advance based on the scheduled live program before the live performance begins.

[0298] The real-time stage information analysis unit 204 compares each piece of information input from the real-time stage information acquisition unit 201, such as performer position 211, lighting condition 212, on-stage object position 213, and image capture robot position 214, with planned data 231 based on the live program stored in advance in the memory unit 205, and checks whether there are any differences.

[0299] If there is no difference, the image capturing robot 50 can be made to travel along a route set using the pre-generated potential map 150 generated according to the previous Example 1 or Example 2 without colliding with performers or objects and without being conspicuous by lighting. However, if there is a difference, when the image capturing robot 50 is made to travel along a travel route set using the pre-generated potential map 150 generated according to the previous Example 1 or Example 2, there is a possibility that it may come into contact with a performer or an object, or that the travel may be conspicuous due to lighting.

[0300] If the real-time stage information analysis unit 204 detects a difference between the information input from the real-time stage information acquisition unit 201, such as the performer position 211, lighting conditions 212, on-stage object position 213, and image capture robot position 214, and the planned data 231 based on the live program pre-stored in the memory unit 205, it further determines whether there are any problems with traveling along the travel route in accordance with the pre-generated potential map 150 generated according to the previous Example 1 or Example 2.

[0301] If it is determined that there is a problem, that is, if it is determined that there is a possibility that the image capturing robot 50 may come into contact with a performer or an object, or that the robot may be traveling in a way that is conspicuous due to lighting, when the image capturing robot 50 is caused to travel along a route set using the pre-generated potential map 150, a request to modify the pre-generated potential map 150 is output to the pre-potential map modification unit 206.

[0302] When a request to modify the pre-generated potential map is input from the real-time stage information analysis unit 204, the pre-generated potential map modification unit 206 modifies the pre-generated potential map 150 so as to reduce the possibility of contact with performers or objects, and the possibility of running in a way that is conspicuous due to lighting. The pre-potential map corrector 206 corrects the pre-generated potential map 150 to generate a potential map that reflects real-time stage information.

[0303] The real-time stage information reflecting potential map is a map in which each area (no-travel area, caution area for travel, allowed area) is set by reflecting the real-time performer position 211, lighting condition 212, on-stage object position 213, and image capturing robot position 214 acquired by the real-time stage information acquisition unit 201.

[0304] A specific example of the process of generating a "real-time stage information reflected potential map" executed by the preliminary potential map corrector 206 will be described with reference to FIG. FIG. 26 shows the following two potential maps: (1) Pre-generated potential map (2) Real-time stage information reflected potential map

[0305] (1) The pre-generated potential map is the pre-generated potential map 150 generated according to the above-described first or second embodiment, and is a potential map stored in the storage unit 205 of the information processing device 200 shown in FIG.

[0306] (2) The real-time stage information reflecting potential map is a map generated by the pre-potential map correcting unit 206 by correcting the pre-generated potential map 150. In other words, it is a map in which each area (no-travel area, caution area for travel, permitted area) is reconfigured to reflect the real-time performer positions 211, lighting conditions 212, on-stage object positions 213, and image capturing robot positions 214 acquired by the real-time stage information acquiring unit 201.

[0307] Comparing (1) and (2) in Figure 26, for example, the performer position in (1) is slightly different from that in (2), with the performer position in (2) shifted slightly to the right. In this way, situations that differ from those planned may occur during an actual live performance. The pre-potential map correction unit 206 uses real-time stage information to correct the pre-generated potential map and generate a potential map that reflects real-time stage information.

[0308] The real-time stage information reflecting potential map generated by the advance potential map correction unit 206 is a map generated by reflecting the real-time performer positions, lighting conditions, and object positions, and is a map in which the following area settings are made, for example. Allowable travel area (blue) = Area far from the real-time performer position and object position, and determined to be the least noticeable based on real-time lighting conditions. No-driving area (red) = Areas close to the real-time performer and object positions and that are determined to be conspicuous based on real-time lighting conditions. Driving caution area (yellow) = The area between the above driving permitted area (blue) and driving prohibited area (red),

[0309] The real-time stage information reflected potential map generated by the preliminary potential map correction unit 206 is output to the real-time data reflected potential map generation unit 208.

[0310] The real-time audience seat information acquisition unit 202 is specifically configured with, for example, a camera, and acquires in real time information about the audience seating in front of the stage where the actual live performance, that is, the performers' performance, is taking place.

[0311] As shown in the figure, the real-time spectator seat information acquisition unit 202 acquires the following information as real-time data. Spectator line of sight 215, TV camera shooting direction 216 on the spectator side,

[0312] Audience viewpoint position 215 is the direction of the view of the audience in front of the stage where the live performance is being performed. Audience seat side TV camera shooting direction 216 is the shooting direction of the TV camera in front of the stage where the live performance is being performed. The real-time information acquired by the real-time spectator seat information acquisition unit 202 is input to the real-time attention area analysis unit 207 .

[0313] Furthermore, the real-time internet information acquisition unit 203 has a communication unit connected to a communication network such as the Internet, acquires so-called SNS information 217, such as tweets of impressions from many live viewers on the Internet, and inputs the acquired SNS information 217 to the real-time attention area analysis unit 207.

[0314] The real-time attention area analysis unit 207 analyzes the audience line of sight direction 215 input from the real-time audience seating information acquisition unit 202, the audience seat side TV camera shooting direction 216, and also the SNS information 217 input from the real-time internet information acquisition unit 203, and analyzes where on the stage the audience is looking, where on the stage the TV camera is shooting, and where on the stage online live viewers are paying attention. Based on these analysis results, the real-time attention area analysis unit 207 estimates the current attention area on the stage and outputs the estimated area to the real-time data reflected potential map generation unit 208 as real-time attention area information 221 shown in the figure.

[0315] The real-time data reflecting potential map generating unit 208 receives the following data. (1) The "real-time stage information reflected potential map" generated by the preliminary potential map correction unit 206 (2) Real-time attention area information 221 generated by the real-time attention area analysis unit 207 The real-time data reflecting potential map generating unit 208 inputs each of these data and generates a real-time data reflecting potential map 230 based on these input data.

[0316] The real-time data reflecting potential map generating unit 208 executes the following processes 1 and 2 in sequence. (Process 1) Based on the real-time attention area information 221 generated by the real-time attention area analysis unit 207, a real-time attention area reflecting potential map that reflects the real-time attention area is generated. (Process 2) The pre-potential map correction unit 206 generates a real-time data reflecting potential map 230 by combining a "real-time stage information reflecting potential map" that reflects the real-time performer position 211, lighting conditions 212, on-stage object position 213, and image capturing robot position 214 and sets each area (no-driving area, driving caution area, driving permitted area) with the real-time attention area reflecting potential map generated in process 1.

[0317] A specific example of these (Process 1) and (Process 2) will be described with reference to FIG. Figure 27 shows the following three potential maps. (2) Real-time stage information reflected potential map (3) Real-time potential map reflecting the area of ​​interest (4) Real-time data reflecting potential map 230

[0318] (2) The real-time stage information reflecting potential map is the map previously described with reference to FIG. 26, and is a “real-time stage information reflecting potential map” generated by the preliminary potential map correcting unit 206 using real-time stage information.

[0319] (3) The real-time attention area reflecting potential map is a map generated by the above (Process 1) executed by the real-time data reflecting potential map generating unit 208, and is a potential map generated to reflect the real-time attention area based on the real-time attention area information 221 generated by the real-time attention area analyzing unit 207.

[0320] The real-time attention area reflecting potential map is a map generated by reflecting real-time information from spectators, TV cameras, and online information, and is a map with the following area settings. Allowable driving area (blue) = Area with the least attention based on real-time spectators, TV cameras, and online information. No-driving zone (red) = The most popular area based on real-time spectators, TV cameras, and online information. Driving caution area (yellow) = The area between the above driving permitted area (blue) and driving prohibited area (red),

[0321] (4) The real-time data reflecting potential map 230 shown in FIG. (2) Real-time stage information reflected potential map (3) Real-time potential map reflecting the area of ​​interest This is a map generated by combining two potential maps.

[0322] This map synthesis process generates the map by converting the set areas of each map ((2) potential map reflecting real-time stage information) and (3) potential map reflecting real-time attention areas) into numerical values, adding the numerical values ​​for each area, and redefining the areas (no-driving areas, driving caution areas, and driving permitted areas) based on the addition results. A detailed example of the generation sequence of "(4) real-time data reflected potential map 230" based on this map synthesis process will be explained later with reference to a flowchart.

[0323] The real-time data reflecting potential map 230 generated by the real-time data reflecting potential map generating unit 208 is a map that is generated by reflecting real-time information on the stage, real-time information on the audience seats, and also real-time online information.

[0324] The real-time data reflecting potential map 230 generated by the real-time data reflecting potential map generation unit 208 is a map that is generated by reflecting all real-time performer positions, lighting conditions, object positions, audience, TV cameras, and online information, and is a map in which the following area settings are made. Allowable travel area (blue) = The least noticeable non-attention area based on real-time performer position, lighting conditions, object position, audience, TV camera, and internet information. No-driving area (red) = The most prominent area based on real-time performer position, lighting conditions, object position, audience, TV camera, and internet information. Driving caution area (yellow) = The area between the above driving permitted area (blue) and driving prohibited area (red),

[0325] The real-time data reflecting potential map 230 generated by the real-time data reflecting potential map generation unit 208 is provided to the driving route generation unit 240, and the driving route generation unit 240 determines the driving route of the image capturing robot 50 based on the real-time data reflecting potential map 230.

[0326] The travel route generation unit 240 inputs the real-time data reflecting potential map 230 and generates travel route information that sets a route for the image capturing robot 50 to travel without colliding with performers or objects on the stage, by selecting areas that are not conspicuous by lighting, and by selecting areas other than areas of interest to spectators, TV cameras, and online viewers.

[0327] The driving route generation unit 240 can set the above-described driving route by, for example, generating a driving route that travels only within the permitted driving area in the real-time data reflected potential map 230.

[0328] The generated travel route is provided to a travel control unit that controls the travel of the image capturing robot 50, and the travel control unit causes the image capturing robot 50 to travel in accordance with the generated travel route information.

[0329] In this way, by making the image capturing robot 50 travel according to the travel route information generated using the real-time data reflecting potential map 230 generated in this embodiment 3, it is possible to make the image capturing robot 50 travel without colliding with the real-time positions of performers or objects on the stage where a live performance is taking place, and to select areas that are not conspicuous by lighting, and areas other than those that are of interest to spectators, TV cameras, and online viewers.

[0330] Next, a process sequence for generating a real-time data-reflecting potential map executed by the information processing device 200 of the third embodiment will be described with reference to the flow shown in FIG. The processing of each step in the flow shown in FIG. 28 will be explained below in order.

[0331] (Step S301) First, in step S301, the information processing device 200 acquires real-time stage information (performer positions, lighting conditions, on-stage object positions, and image capturing robot positions).

[0332] This process is executed by the real-time stage information acquisition unit 201 of the information processing device 200 shown in FIG. The real-time stage information acquisition unit 201 is specifically composed of, for example, a camera, a light meter, etc., and acquires information about the actual live performance, i.e., the stage where the performers are performing, i.e., real-time stage information, such as the performer positions, lighting conditions, positions of objects on the stage, and the position of the image-taking robot.

[0333] (Step S302) Next, in step S302, the information processing device 200 corrects the previously generated pre-generated potential map based on the real-time stage information acquired in step S301, and generates a real-time stage information-reflecting potential map.

[0334] The process of step S302 is executed by the preliminary potential map correcting section 206 of the information processing device 200 shown in FIG.

[0335] Although omitted in the flow shown in Figure 28, after processing of step S301, the real-time stage information analysis unit 204 of the information processing device 200 shown in Figure 24 analyzes the difference between the real-time stage information (performer positions, lighting conditions, positions of objects on the stage, and image-capturing robot positions) input from the real-time stage information acquisition unit 201 and the scheduled data 231 based on the live program stored in advance in the memory unit 205, and analyzes any problems with traveling along the traveling route according to the pre-generated potential map 150.

[0336] If it is determined that there is a problem, that is, if it is determined that there is a possibility that the image capturing robot 50 may come into contact with a performer or an object, or that the robot may be traveling in a way that is conspicuous due to lighting, when the image capturing robot 50 is caused to travel along a route set using the pre-generated potential map 150, a request to modify the pre-generated potential map 150 is output to the pre-potential map modification unit 206.

[0337] The processing of step S302 is the subsequent processing, and when the pre-generated potential map correction unit 206 receives a request to correct the pre-generated potential map from the real-time stage information analysis unit 204, it corrects the pre-generated potential map 150 so as to reduce the possibility of contact with performers or objects, and the possibility of running in a way that is conspicuous by lighting, and generates a potential map that reflects the real-time stage information.

[0338] The real-time stage information reflecting potential map generated by the advance potential map correction unit 206 is a map generated by reflecting the real-time performer positions, lighting conditions, and object positions, and is a map in which the following area settings are made, for example. Allowable travel area (blue) = Area far from the real-time performer position and object position, and determined to be the least noticeable based on real-time lighting conditions. No-driving area (red) = Areas close to the real-time performer and object positions and that are determined to be conspicuous based on real-time lighting conditions. Driving caution area (yellow) = The area between the above driving permitted area (blue) and driving prohibited area (red),

[0339] (Step S303) Next, in step S303, the information processing device 200 acquires real-time spectator seat information and real-time net information.

[0340] These processes are executed by the real-time spectator seat information acquisition unit 202 and the real-time net information acquisition unit 203 of the information processing device 200 shown in FIG.

[0341] As previously described with reference to FIG. 24, the real-time spectator seat information acquisition unit 202 acquires the following information as real-time data. Spectator line of sight 215, TV camera shooting direction 216 on the spectator side, In addition, the real-time internet information acquisition unit 203 acquires so-called SNS information 217 such as tweets of impressions from many live viewers on the Internet, and inputs the acquired SNS information 217 to the real-time attention area analysis unit 207.

[0342] (Step S304) Next, in step S304, the information processing device 200 analyzes the real-time attention area based on the real-time spectator seat information and real-time net information acquired in step S303.

[0343] This process is executed by the real-time region-of-interest analysis unit 207 of the information processing device 200 shown in FIG.

[0344] The real-time attention area analysis unit 207 analyzes the audience line of sight direction 215 input from the real-time audience seating information acquisition unit 202, the audience seat side TV camera shooting direction 216, and also the SNS information 217 input from the real-time internet information acquisition unit 203, and analyzes where on the stage the audience is looking, where on the stage the TV camera is shooting, and where on the stage online live viewers are paying attention.

[0345] Based on these analysis results, the real-time attention area analysis unit 207 estimates the current attention area on the stage and outputs the estimated area to the real-time data reflected potential map generation unit 208 as real-time attention area information 221 shown in FIG.

[0346] (Step S305) Next, in step S305, the information processing device 200 generates a "potential map reflecting real-time attention area" that reflects the real-time attention area analyzed based on the real-time spectator seating information and real-time net information.

[0347] This process is executed by the real-time data reflected potential map generating unit 208 of the information processing device 200 shown in FIG.

[0348] The real-time data reflecting potential map generating unit 208 generates a "real-time attention area reflecting potential map" that reflects real-time spectator seating information and real-time attention areas analyzed based on real-time internet information.

[0349] The real-time attention area reflecting potential map is a map generated by reflecting real-time information from spectators, TV cameras, and online information, and is a map with the following area settings. Allowable driving area (blue) = Area with the least attention based on real-time spectators, TV cameras, and online information. No-driving zone (red) = The most popular area based on real-time spectators, TV cameras, and online information. Driving caution area (yellow) = The area between the above driving permitted area (blue) and driving prohibited area (red),

[0350] (Step S306) Next, in step S306, the information processing device 200 digitizes the "real-time stage information reflected potential map" and the "real-time attention area reflected potential map" and the set area of ​​each map.

[0351] This process is also executed by the real-time data reflected potential map generating unit 208 of the information processing device 200 shown in FIG. for example, No driving area = 10, Driving attention area = 5, Allowable travel area = 0 Such area units are quantified.

[0352] (Step S307) Next, in step S307, the information processing device 200 adds the numerical values ​​corresponding to the regions of the "real-time stage information reflected potential map" and the "real-time attention region reflected potential map" for each region to calculate the sum corresponding to the regions.

[0353] This process is also executed by the real-time data reflected potential map generating unit 208 of the information processing device 200 shown in FIG.

[0354] For example, suppose the area settings for the same stage position (x1, y1) in the "real-time stage information reflecting potential map" and the "real-time attention area reflecting potential map" are as follows: Real-time stage information reflected potential map value = 5 (driving caution area) Real-time attention area reflecting potential map value = 5 (driving attention area) In this case, the added value is 5+5=10. This addition process is performed for all stage areas within the map.

[0355] (Step S308) Next, in step S308, the data processing unit of the information processing device 200 resets the area divisions based on the area-corresponding sum calculated in step S307.

[0356] This process is also executed by the real-time data reflected potential map generating unit 208 of the information processing device 200 shown in FIG.

[0357] The real-time data reflecting potential map generating unit 208 of the information processing device 200 resets the region according to the following rules, for example. Addition value of 10 or more = No driving zone, Addition value of 5 points or more = driving caution area, Addition value less than 5 points = Driving allowable area, For example, the area is reset according to the above rules.

[0358] (Step S309) Next, in step S309, the data processing unit of the information processing device 200 outputs the potential map in which the region divisions have been reset in step S308 as a "real-time data reflected potential map."

[0359] This process is also executed by the real-time data reflected potential map generating unit 208 of the information processing device 200 shown in FIG.

[0360] The above is the detailed sequence of the process for generating the real-time data reflected potential map executed by the information processing device 200 of the third embodiment shown in FIG.

[0361] By making the image capturing robot 50 travel according to travel route information generated using the real-time data-reflecting potential map 230 generated by the information processing device 200 of Example 3 shown in Figure 24, it is possible to make the image capturing robot 50 travel without colliding with the real-time positions of performers or objects on the stage where a live performance is taking place, and to select areas that are not conspicuous by lighting, and areas other than those that are of interest to spectators, TV cameras, and online viewers.

[0362] In steps S301 and S302 of the flowchart shown in FIG. 28, the following processing is performed. The information processing device 200 shown in FIG. 24 performs a process of acquiring real-time stage information (performer positions, lighting conditions, positions of objects on the stage, and image capturing robot positions) by a real-time stage information acquisition unit 201, and a process of generating a potential map that reflects real-time stage information by correcting the pre-generated potential map by a pre-potential map correction unit 206.

[0363] After the processing of steps S301 to S302, the processing of steps S303 to S309 is executed, and finally in step S309, a real-time data-reflecting potential map 230 is generated, and based on the generated real-time data-reflecting potential map 230, the travel route of the image-capturing robot 50 is determined and travel control is performed.

[0364] However, it is not always possible to change the travel route of the image capturing robot 50 during the live performance. There are periods when the route can be changed (=travel route change permitted period) and periods when the route cannot be changed (=travel route change not permitted period). Therefore, it is preferable that the real-time stage information reflecting potential map generation process in steps S301 to S302 be configured to be executed only during a period in which the image capturing robot 50 is allowed to change its travel route.

[0365] A processing sequence in which the real-time stage information reflected potential map generation process in steps S301 to S302 of the flow shown in FIG. 28 is set to be executed only during a period in which the travel route of the image capturing robot 50 is allowed to be changed will be described with reference to the flowchart shown in FIG. 29.

[0366] The flowchart shown in FIG. 29 is a flow that can be executed by replacing the processing of steps S301 to S302 of the flow shown in FIG. The processing of each step in the flow shown in FIG. 29 will be explained below in order.

[0367] (Step S321) First, in step S321, the information processing device 200 shown in FIG. 24 determines whether it is time to acquire real-time stage information.

[0368] This process is executed by the real-time stage information acquisition unit 201 of the information processing device 200 shown in FIG. 24 or a control unit that controls the real-time stage information acquisition unit 201. The timing for acquiring real-time stage information is predetermined so that it is acquired at regular intervals, such as every 10 seconds.

[0369] In step S321, it is determined whether or not it is time to acquire real-time stage information in accordance with this rule. If it is determined that it is time to acquire real-time stage information, the process proceeds to step S322.

[0370] (Step S322) If it is determined in step S321 that it is time to acquire real-time stage information, the information processing device 200 acquires real-time stage information (performer positions, lighting conditions, on-stage object positions, image capturing robot positions) in step S322.

[0371] This process is executed by the real-time stage information acquisition unit 201 of the information processing device 200 shown in FIG. The real-time stage information acquisition unit 201 is specifically composed of, for example, a camera, a light meter, etc., and acquires information about the actual live performance, i.e., the stage where the performers are performing, i.e., real-time stage information, such as the performer positions, lighting conditions, positions of objects on the stage, and the position of the image-taking robot.

[0372] (Step S323) Next, in step S323, the information processing device 200 determines whether or not it is time for the image capturing robot to finish traveling.

[0373] Information on the travel end time of the image capturing robot is stored in advance in the storage unit 205 of the information processing device 200. For example, it is recorded in association with the planned travel route of the image capturing robot. The information processing device 200 refers to this recorded information and determines whether the current time is the end time of the image capturing robot's travel.

[0374] If the current time is the end time of the image capturing robot's travel, the process ends. On the other hand, if the current time is not the end time of the image capturing robot's travel, the process proceeds to step S324.

[0375] (Step S324) If it is determined in step S323 that the travel end time of the image capturing robot has not arrived, the information processing device 200 determines in step S324 whether or not it is the permitted time for changing the travel route of the image capturing robot.

[0376] Information on whether or not it is the allowable time for changing the travel route of the image capturing robot is stored in advance in the storage unit 205 of the information processing device 200. For example, it is recorded in association with the planned travel route of the image capturing robot. The information processing device 200 refers to this recorded information and determines whether the current time is within the permitted time for changing the travel route of the image capturing robot.

[0377] If the current time is not within the allowable time for changing the travel route of the image capturing robot, the process does not proceed to step S325, but returns to step S321, and the processes of steps S321 to S324 are repeated. Only when it is determined that the current time is within the allowable time for changing the travel route of the image capturing robot, the process proceeds to step S325.

[0378] (Step S325) If it is determined in step S324 that the current time is within the permitted time for changing the travel route of the image capturing robot, the process of step S325 is executed.

[0379] In this case, in step S325, the information processing device 200 performs a comparison process between the real-time stage information (performer positions, lighting conditions, on-stage object positions, and image capturing robot positions) acquired in step S321 and the pre-generated potential map that has already been generated.

[0380] The process of step S325 is executed by the real-time stage information analysis unit 204 of the information processing device 200 shown in FIG.

[0381] (Step S326) Next, in step S326, the information processing device 200 determines whether the travel route of the image capturing robot based on the pre-generated potential map is set to pass through a dangerous area estimated from the real-time stage information (performer positions, lighting conditions, positions of objects on the stage, and position of the image capturing robot) acquired in step S321.

[0382] The processing in step S326 is also executed by the real-time stage information analysis unit 204 of the information processing device 200 shown in FIG.

[0383] The real-time stage information analysis unit 204 analyzes the difference between the real-time stage information (performer positions, lighting conditions, positions of objects on the stage, and image-taking robot positions) input from the real-time stage information acquisition unit 201 and the planned data 231 based on the live program pre-stored in the memory unit 205, and analyzes the danger of the driving route according to the pre-generated potential map 150.

[0384] If it is determined that there is no danger, that is, if it is determined that there is no possibility of the image capturing robot 50 coming into contact with a performer or an object or running in a way that is conspicuous due to lighting when running along the running route set using the pre-generated potential map 150, the process returns to step S321 without proceeding to step S327, and the processes from step S321 onwards are repeated.

[0385] On the other hand, if it is determined that there is a risk, that is, if it is determined that there is a possibility that the image capturing robot 50 may come into contact with a performer or an object if it is driven along the driving route set using the pre-generated potential map 150, or that the driving may be conspicuous due to lighting, the process proceeds to step S327.

[0386] (Step S327) In step S326, if it is determined that the travel route set using the pre-generated potential map 150 poses a risk of contact with the performer, the information processing device executes the process of step S327.

[0387] In this case, in step S327, the information processing device 200 corrects the pre-generated potential map 150 stored in the storage unit 205, and generates a real-time stage information reflected potential map.

[0388] The processing of step S327 is executed by the pre-generated potential map correction unit 206 of the information processing device 200 shown in FIG.

[0389] The pre-potential map correction unit 206 corrects the pre-generated potential map 150 so as to reduce the possibility of contact with performers or objects and the possibility of running in a way that is conspicuous by lighting, and generates a potential map that reflects real-time stage information.

[0390] The real-time stage information reflecting potential map generated by the advance potential map correction unit 206 is a map generated by reflecting the real-time performer positions, lighting conditions, and object positions, and is a map in which the following area settings are made, for example. Allowable travel area (blue) = Area far from the real-time performer position and object position, and determined to be the least noticeable based on real-time lighting conditions. No-driving area (red) = Areas close to the real-time performer and object positions and that are determined to be conspicuous based on real-time lighting conditions. Driving caution area (yellow) = The area between the above driving permitted area (blue) and driving prohibited area (red),

[0391] After the processing of steps S321 to S327 shown in FIG. 29, the processing of step S303 and the subsequent steps in the flow shown in FIG. 28 is executed. That is, the processing of steps S303 to S309 is executed, and finally in step S309, a real-time data reflecting potential map 230 is generated, and based on the generated real-time data reflecting potential map 230, the travel route of the image capturing robot 50 is determined and travel control is performed.

[0392] By executing the processing of steps S321 to S327 shown in the flowchart of FIG. 29 instead of the processing of steps S301 to S302 shown in FIG. 28, it becomes possible to execute the real-time stage information reflecting potential map generation processing only during the period in which the travel route of the image capturing robot 50 is allowed to be changed.

[0393] [6. Examples of hardware configurations of information processing devices] Next, an example of the hardware configuration of an information processing device that executes the processing according to the above-described embodiment will be described with reference to FIG. The hardware shown in Figure 30 is an example of the hardware configuration of the information processing device 100 previously described with reference to Figure 4, the information processing device 100b described with reference to Figure 20, and the information processing device 200 described with reference to Figure 24. The hardware configuration shown in FIG. 30 will be described.

[0394] A CPU (Central Processing Unit) 301 functions as a data processing unit that executes various processes in accordance with programs stored in a ROM (Read Only Memory) 302 or a storage unit 308. For example, it executes processes in accordance with the sequences described in the above-mentioned embodiments. A RAM (Random Access Memory) 303 stores programs and data executed by the CPU 301. The CPU 301, ROM 302, and RAM 303 are interconnected by a bus 304.

[0395] The CPU 301 is connected to an input / output interface 305 via a bus 304, and the input / output interface 305 is connected to an input unit 306 consisting of various sensors, a camera, a switch, a keyboard, a mouse, a microphone, etc., and an output unit 307 consisting of a display, a speaker, etc.

[0396] A storage unit 308 connected to the input / output interface 305 is formed of, for example, a hard disk, and stores various data and programs executed by the CPU 301. A communication unit 309 functions as a transmitter / receiver for data communication via a network such as the Internet or a local area network, and communicates with external devices.

[0397] A drive 310 connected to the input / output interface 305 drives removable media 311 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory such as a memory card, and executes recording or reading of data.

[0398] 7. Summary of the Disclosure The embodiments of the present disclosure have been described in detail above with reference to specific examples. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of examples and should not be interpreted as being limited. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0399] The technology disclosed in this specification can be configured as follows. (1) A data processing unit is provided to generate a potential map that defines an allowable travel area of ​​an image capturing robot that moves on a stage and captures images; The data processing unit An information processing device acquires at least one of data on the planned behavior of performers on the stage or data on the planned placement of objects on the stage, and based on the acquired data, generates a potential map that defines an allowable driving area where the probability of collision with performers or objects on the stage is below a specified threshold.

[0400] (2) The data processing unit An information processing device as described in (1) that acquires time series data of planned behavior data of performers on the stage or planned placement data of objects on the stage, and generates time series data of a potential map that defines an allowable driving area where the possibility of collision with performers or objects on the stage is below a specified threshold based on the acquired data.

[0401] (3) The data processing unit An information processing device as described in (1) or (2) that acquires planned lighting control data on the stage and generates a potential map based on the acquired data, defining as a travel-allowed area any area other than an area where the lighting state value calculated based on the lighting irradiated on the stage is equal to or greater than a predetermined threshold value.

[0402] (4) The data processing unit An information processing device described in any one of (1) to (3) that acquires planned data for controlling the brightness of the lighting on the stage, and generates a potential map based on the acquired data, defining areas other than areas where the brightness of the lighting irradiated on the stage is equal to or greater than a predetermined threshold as allowed travel areas.

[0403] (5) The data processing unit An information processing device described in any one of (1) to (4) that acquires planned color control data for the lighting on the stage, and based on the acquired data, generates a potential map that defines areas other than the area where the lighting irradiated on the stage has a predetermined color value as a travel-allowed area.

[0404] (6) The data processing unit a permitted travel area where the probability of collision with a performer or object on the stage is below a specified threshold; The information processing device according to any one of (1) to (5) generates a potential map that defines a travel-prohibited area that is an area closer to a performer or an object on a stage than the travel-permitted area.

[0405] (7) The data processing unit The information processing device according to (6), further comprising: generating a potential map defining a driving caution area at an intermediate position between the driving permitted area and the driving prohibited area.

[0406] (8) The data processing unit (a) a performer-based potential map that defines a permissible driving area where the probability of collision with the performer on the stage is below a specified threshold, based on the action schedule data of the performer on the stage; (b) an object-based potential map that defines a permissible travel area in which the probability of collision with an object on the stage is equal to or less than a specified threshold, based on the planned placement data of the object on the stage; (c) an illumination-based potential map that defines, as a travel allowance area, an area other than an area where an illumination state value calculated based on the illumination irradiated on the stage is equal to or greater than a predetermined threshold value based on the lighting control schedule data on the stage; Generate the three types of individual potential maps (a) to (c) above, The information processing device according to any one of (1) to (7) further generates a pre-generated potential map by combining the three types of individual potential maps (a) to (c) above.

[0407] (9) The pre-generated potential map is An information processing device as described in (8) in which an area other than an area where the possibility of collision with performers and objects on stage is below a specified threshold and the lighting state value calculated based on the lighting is above a predetermined threshold is defined as a travel-permitted area.

[0408] (10) The data processing unit In the synthesis process of the three types of individual potential maps (a) to (c), An information processing device according to (8) or (9), which performs a region-by-region quantification process for each of the three types of individual potential maps (a) to (c), the driving-permitted region, the driving caution region, and the driving-prohibited region, and calculates an added value by adding the numerical values ​​of each individual potential map after the quantification process, and performs region reconfiguration of the driving-permitted region, the driving caution region, and the driving-prohibited region based on the calculated added value, thereby generating the pre-generated potential map.

[0409] (11) The data processing unit An information processing device described in any of (1) to (10) that acquires viewpoint position data of at least one of the spectator's viewpoint position data on the audience seating side watching the stage and the camera's viewpoint position data, and generates a potential map based on the acquired data that defines positions far from a line connecting the viewpoint position of the spectator or the camera to the center position of the stage as an allowable travel area.

[0410] (12) The data processing unit (a) a performer-based potential map that defines a permissible driving area where the probability of collision with the performer on the stage is below a specified threshold, based on the action schedule data of the performer on the stage; (b) an object-based potential map that defines a permissible travel area in which the probability of collision with an object on the stage is equal to or less than a specified threshold, based on the planned placement data of the object on the stage; (c) an illumination-based potential map that defines, as a travel allowance area, an area other than an area where an illumination state value calculated based on the illumination irradiated on the stage is equal to or greater than a predetermined threshold value based on the lighting control schedule data on the stage; (d) an audience and camera-based potential map that defines, based on at least one of the planned viewpoint data of audience members on the audience seating side watching the stage and the planned viewpoint data of a camera, positions far from a line connecting the planned viewpoint positions of the audience members or the camera to the center position of the stage as a permitted travel area; The four individual potential maps (a) to (d) above are generated, The information processing device according to any one of (1) to (11) further generates a pre-generated potential map by combining the four types of individual potential maps (a) to (d) above.

[0411] (13) The data processing unit An information processing device as described in any one of (1) to (12), which acquires real-time data of at least one of the performer's behavior data and the object placement data on the stage as real-time data during the performance by the performer on the stage, and generates a real-time data reflecting potential map based on the acquired real-time data, which defines an allowable driving area where the possibility of collision with the performer or object on the stage is below a specified threshold.

[0412] (14) The data processing unit As real-time data during the performance by the performer on the stage, The information processing device described in (13) acquires data on at least one of the line of sight direction of the spectators in the audience seats watching the stage and the shooting direction of the camera, analyzes the area of ​​interest based on the acquired data, and generates a real-time data-reflecting potential map that defines areas other than the area of ​​interest as permitted driving areas.

[0413] (15) The data processing unit further An information processing device as described in (13) or (14) that acquires user comments viewing the performance by the performer on the stage via the Internet, analyzes the attention areas of the users viewing the Internet based on the acquired comments, and generates a real-time data-reflecting potential map that defines areas other than the attention areas as permissible driving areas.

[0414] (16) The data processing unit The information processing device according to any one of (1) to (15) generates a route for the image capturing robot to travel based on the potential map.

[0415] (17) The data processing unit The information processing device according to (16), wherein a travel route is generated in which the image capturing robot is set to travel within a travel allowable area defined in the potential map.

[0416] (18) The data processing unit The information processing device according to any one of (1) to (17) generates simulation data for displaying on a display unit a travel route of the image capturing robot generated based on the potential map.

[0417] (19) A memory unit storing travel route information generated based on a potential map, which is a map generated based on at least one of planned behavior data of performers on a stage and planned placement data of objects on the stage, and which defines an allowable travel area where the probability of collision with performers or objects on the stage is below a specified threshold value; or a communication unit that acquires the travel route information from an external device, A mobile device that executes a travel process in accordance with either the travel route information acquired from the storage unit or the travel route information acquired via the communication unit.

[0418] (20) An information processing system having an image capturing robot and a server, the image capturing robot is an image capturing robot that moves on a stage and captures images, The server a data processing unit that generates a potential map that defines an allowable travel area for the image capturing robot; The data processing unit acquiring at least one of planned behavior data of performers on the stage and planned placement data of objects on the stage, and generating a potential map based on the acquired data that defines an allowable travel area where the probability of collision with performers or objects on the stage is below a specified threshold; The image capturing robot An information processing system that travels according to a travel route determined based on the potential map generated by the server.

[0419] Furthermore, the series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in the memory of a computer incorporated in dedicated hardware, or the program can be installed and executed on a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing the program on a computer from the recording medium, the program can also be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as an internal hard disk.

[0420] The various processes described in this specification may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capabilities of the devices executing the processes or as needed. Furthermore, in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are all located in the same housing. [Industrial Applicability]

[0421] As described above, according to the configuration of one embodiment of the present disclosure, it is possible to generate a map that determines a safe driving route that will avoid colliding with performers or objects on stage, and to make an image-capturing robot drive along the route determined based on the map. Specifically, for example, a potential map is generated that defines the allowable travel area for an image capturing robot that moves around a stage and captures images. The data processing unit acquires planned data on the actions of performers on the stage, planned data on the placement of objects, and planned data on lighting control on the stage, and generates a potential map that defines, as the allowable travel area, an area that will not collide with performers or objects and will not be noticeable due to lighting, based on the acquired data. Furthermore, the robot's travel route is determined based on the generated map, and the robot is made to travel. This configuration makes it possible to generate a map that determines a safe driving route that will avoid colliding with performers or objects on the stage, and to make the image-taking robot drive along the route determined based on the map. [Explanation of symbols]

[0422] 10 stages 11. Lighting 12 speakers 13 Monitor 14 Decorative Objects 20 Performers 30 Audience 31 TV camera 35 Priority Spectators 50 Image-taking robot 100 Information processing device 110 Storage section 111 Performer Activity Schedule Data 112 Stage lighting control schedule data 113 Planned placement data for objects on stage 114 Priority spectator and TV camera viewpoint position data 120 Individual potential map generation unit 130 Potential map synthesis unit 150 Pre-generated Potential Maps 160 Driving route determination unit 165 Driving route information 170 Travel control unit 180 Information Processing Systems 200 Information processing device 201 Real-time stage information acquisition unit 202 Real-time seat information acquisition unit 203 Real-time Internet Information Acquisition Department 204 Real-time Stage Information Analysis Department 205 Storage section 206 Pre-generated potential map correction unit 207 Real-time region of interest analysis unit 208 Real-time data reflecting potential map generation unit 240 Driving route generation unit 280 Information Processing Systems 290 SNS Server 301 CPU 302 ROM 303 RAM 304 Bus 305 Input / Output Interface 306 Input section 307 Output section 308 Storage section 309 Communications Department 310 Drive 311 Removable Media

Claims

1. a data processing unit that generates a potential map that defines an allowable travel area for an image capturing robot that moves on the stage and captures images; The data processing unit An information processing device acquires at least one of data on the planned behavior of performers on the stage or data on the planned placement of objects on the stage, and based on the acquired data, generates a potential map that defines an allowable driving area where the probability of collision with performers or objects on the stage is below a specified threshold.

2. The data processing unit An information processing device as described in claim 1, which acquires time series data of planned behavior data of performers on the stage or planned placement data of objects on the stage, and generates time series data of a potential map based on the acquired data, which defines an allowable driving area where the probability of collision with performers or objects on the stage is below a specified threshold.

3. The data processing unit The information processing device according to claim 1, wherein planned lighting control data on the stage is acquired, and a potential map is generated based on the acquired data, in which areas other than areas where the lighting state value calculated based on the lighting irradiated on the stage is equal to or greater than a predetermined threshold are defined as allowed travel areas.

4. The data processing unit 2. The information processing device according to claim 1, wherein planned data for controlling the brightness of the lighting on the stage is acquired, and a potential map is generated based on the acquired data, in which areas other than areas where the brightness of the lighting irradiated on the stage is equal to or greater than a predetermined threshold are defined as allowed travel areas.

5. The data processing unit 2. The information processing device according to claim 1, wherein planned color control data for the lighting on the stage is acquired, and a potential map is generated based on the acquired data, in which an area other than an area where the lighting irradiated on the stage has a predetermined color value is defined as a travel-allowed area.

6. The data processing unit a permitted travel area where the probability of collision with a performer or object on the stage is below a specified threshold; The information processing device according to claim 1 , wherein a potential map is generated that defines a no-travel area that is an area closer to a performer or an object on the stage than the permitted travel area.

7. The data processing unit The information processing device according to claim 6 , wherein a potential map is generated that defines a cautionary driving area at an intermediate position between the permitted driving area and the prohibited driving area.

8. The data processing unit (a) a performer-based potential map that defines a permissible travel area in which the probability of collision with the performer on the stage is below a specified threshold, based on the behavioral schedule data of the performer on the stage; (b) an object-based potential map that defines a travel allowance area in which the probability of collision with an object on the stage is equal to or less than a specified threshold value, based on the planned placement data of the object on the stage; (c) an illumination-based potential map that defines, as a travel allowance region, a region other than a region where an illumination state value calculated based on the illumination irradiated on the stage is equal to or greater than a predetermined threshold value based on the illumination control schedule data on the stage; Generate the three types of individual potential maps (a) to (c) above, 2. The information processing apparatus according to claim 1, further comprising: a pre-generated potential map that is generated by combining the three types of individual potential maps (a) to (c).

9. The pre-generated potential map is The information processing device of claim 8, wherein the permissible travel area is defined as an area other than an area where the probability of collision with performers and objects on the stage is below a specified threshold and where the lighting state value calculated based on the lighting is above a predetermined threshold.

10. The data processing unit In the synthesis process of the three types of individual potential maps (a) to (c), 9. The information processing device according to claim 8, wherein a digitization process is performed on a region-by-region basis for each of the three types of individual potential maps (a) to (c), the digitization process is performed on each of the three types of individual potential maps, the digitization process is performed on each of the three types of individual potential maps, the digitization process on each of the three types of individual potential maps is added together to calculate an added value, and the digitization process on each of the three types of individual potential maps is performed ..., and the digitization process on each of the

11. The data processing unit An information processing device as described in claim 1, which acquires viewpoint position data of at least one of the spectator's viewpoint position data on the audience seating side watching the stage and the camera's viewpoint position data, and based on the acquired data, generates a potential map that defines positions far from a line connecting the spectator's or camera's viewpoint position to the center position of the stage as a travel-allowed area.

12. The data processing unit (a) a performer-based potential map that defines a permissible travel area in which the probability of collision with the performer on the stage is below a specified threshold, based on the behavioral schedule data of the performer on the stage; (b) an object-based potential map that defines a travel allowance area in which the probability of collision with an object on the stage is equal to or less than a specified threshold value, based on the planned placement data of the object on the stage; (c) an illumination-based potential map that defines, as a travel allowance region, a region other than a region where an illumination state value calculated based on the illumination irradiated on the stage is equal to or greater than a predetermined threshold value based on the illumination control schedule data on the stage; (d) an audience and camera-based potential map that defines, based on at least one of scheduled viewpoint position data of audience members on the audience seating side watching the stage and scheduled viewpoint position data of a camera, positions far from a line connecting the scheduled viewpoint positions of the audience members or the camera to the center position of the stage as permitted travel areas; Generate the four types of individual potential maps (a) to (d) above, 2. The information processing apparatus according to claim 1, further comprising: a pre-generated potential map that is generated by combining the four types of individual potential maps (a) to (d).

13. The data processing unit An information processing device as described in claim 1, which acquires real-time data during a performance by the performer on the stage, at least one of real-time data of the performer's behavior data or data of the placement of objects on the stage, and generates a real-time data reflecting potential map based on the acquired real-time data, which defines an allowable driving area where the possibility of collision with the performer or object on the stage is below a specified threshold.

14. The data processing unit As real-time data during the performance by the performer on the stage, An information processing device as described in claim 13, which acquires data on at least one of the line of sight direction of spectators in the audience seats watching the stage and the camera's shooting direction, analyzes the area of ​​interest based on the acquired data, and generates a real-time data-reflecting potential map that defines areas other than the area of ​​interest as allowed driving areas.

15. The data processing unit further An information processing device as described in claim 13, which acquires user comments viewing the performance by the performer on the stage via the Internet, analyzes the attention areas of the users viewing the Internet based on the acquired comments, and generates a real-time data-reflecting potential map that defines areas other than the attention areas as allowable driving areas.

16. The data processing unit The information processing device according to claim 1 , wherein a route for the image capturing robot to travel is generated based on the potential map.

17. The data processing unit The information processing device according to claim 16, wherein a travel route is generated in which the image capturing robot is set to travel within an allowable travel area defined in the potential map.

18. The data processing unit The information processing device according to claim 1 , wherein simulation data is generated for displaying on a display unit a travel route of the image capturing robot generated based on the potential map.

19. a memory unit storing travel route information generated based on a potential map, which is a map generated based on at least one of planned behavior data for performers on stage and planned placement data for objects on the stage, and which defines an allowable travel area where the probability of collision with performers or objects on the stage is below a specified threshold; or a communication unit that acquires the travel route information from an external device, A mobile device that executes a travel process in accordance with either the travel route information acquired from the storage unit or the travel route information acquired via the communication unit.

20. An information processing system having an image capturing robot and a server, the image capturing robot is an image capturing robot that moves on a stage and captures images, The server a data processing unit that generates a potential map that defines an allowable travel area for the image capturing robot; The data processing unit acquiring at least one of planned behavior data of performers on the stage and planned placement data of objects on the stage, and generating a potential map based on the acquired data that defines an allowable travel area where the probability of collision with performers or objects on the stage is below a specified threshold; The image capturing robot An information processing system that travels according to a travel route determined based on the potential map generated by the server.

Images