Information processing method, information processing device, and computer program

Abstract

[Problem] To make it possible to easily produce a planned route of a moving body that is to be moved together with a moving object. [Solution] In an information processing system S according to one embodiment, executed are: an object position information acquisition step (step S405) for acquiring at least information of the lateral-direction position and information of the depth-direction position of an artist 40, which is a moving object; a position expression image generation step (step S407) for generating, on the basis of the information acquired in the object position information acquisition step, position expression images that express at least the lateral-direction position and the depth-direction position of the artist 40 together with a lateral axis X and a depth axis Y at a plurality of points in time; and an image display step (step S408) for displaying the position expression images from at least two points in time side by side in the vertical direction together with a time axis Z extending in the vertical direction.

Description

Information processing method, information processing device, and computer program

[0001] The present disclosure relates to an information processing method, an information processing device, and a computer program.

[0002] In recent years, there has been a growing need for unmanned video capture of moving objects, such as performance footage of an artist captured by a robot or drone.

[0003] The performance of an artist is basically performed by moving and directing according to a predetermined pattern. When a pattern including movement is predetermined in this way, it is believed that desirable filming can be carried out by checking the pattern related to the movement of an object such as an artist in advance and creating a planned route for the moving object in advance. In other words, it is believed that desirable filming can be carried out by controlling the moving object based on the created planned route.

[0004] However, it is not necessarily easy to generate a planned route as electronic information while sequentially specifying the desired positions of a moving object relative to an object whose position changes over time. In other words, there is currently no known user interface that can easily generate such a planned route.

[0005] Various technologies for controlling moving objects such as vehicles and robots based on a pre-planned route have been proposed. These technologies include a technology for changing the planned route of a moving object in response to changes in the environment during the moving object's travel. Specifically, for example, a technology for changing the planned route to avoid a collision with an obstacle when the obstacle appears during travel based on the planned route, and a technology for presenting congestion forecast information to a user while the vehicle is traveling based on the planned route, prompting the user to change the planned route. However, none of these conventional route planning technologies take into account the movement states of objects that can be recognized in advance when creating a planned route for a moving object, and therefore cannot be said to fully contribute to creating such a planned route.

[0006] JP 2009-187343 A JP 2017-96769 A

[0007] The present disclosure has been made in consideration of the above circumstances, and provides an information processing method, an information processing device, and a computer program that make it possible to easily create a planned route for a moving body to move together with a moving object.

[0008] The information processing method of the present disclosure includes an object position information acquisition step of acquiring information on at least the lateral position and depth position of a moving object; a position representation image generation step of generating position representation images for multiple points in time that represent at least the lateral position and the depth position of the object along a horizontal axis and a depth axis based on the information acquired by the object position information acquisition step; and an image display step of displaying the position representation images for at least two points in time generated by the position representation image generation step side by side in the vertical direction along a time axis extending in the vertical direction.

[0009] The information processing device of the present disclosure includes an object position information acquisition unit that acquires information on at least the lateral position and depth position of a moving object; a position representation image generation unit that generates position representation images for multiple points in time that represent at least the lateral position and the depth position of the object along a horizontal axis and a depth axis based on the information acquired by the object position information acquisition unit; and an image display unit that displays the position representation images for at least two points in time generated by the position representation image generation unit in a vertical arrangement along a time axis extending in the vertical direction.

[0010] The computer program of the present disclosure causes a computer to execute the following steps: an object position information acquisition step for acquiring information on at least the lateral position and depth position of a moving object; a position representation image generation step for generating position representation images for multiple points in time that represent at least the lateral position and the depth position of the object along a horizontal axis and a depth axis based on the information acquired by the object position information acquisition step; and an image display step for displaying the position representation images for at least two points in time generated by the position representation image generation step side by side in the vertical direction along a time axis extending in the vertical direction.

[0011] 6 is a diagram showing an information processing system according to an embodiment. FIG. 6 is a block diagram showing the functional configuration of an information processing system according to an embodiment. FIG. 6 is a diagram showing a user interface for creating a planned path for a robot, which is displayed by an information processing system according to an embodiment. A flowchart illustrating an example of processing by an information processing system according to an embodiment. FIG. 6 is a diagram illustrating operations that can be performed in the user interface shown in FIG. 4. FIG. 6 is a diagram showing an example display of a planned path for a robot (mobile body), which is displayed by the user interface shown in FIG. 4. A flowchart illustrating an example of processing by an information processing system according to a first modified example. FIG. 6 is a diagram showing a user interface displayed by an information processing system according to a second modified example. FIG. 6 is a diagram illustrating an example of how to use the user interface shown in FIG. 6. FIG. 6 is a diagram showing a user interface displayed by an information processing system according to a third modified example. FIG. 6 is a diagram illustrating a user interface displayed by an information processing system according to a fourth modified example and operations thereon. FIG. 6 is a diagram showing an example of the hardware configuration of an information processing device that can constitute an operator terminal and a robot (mobile body) according to an embodiment.

[0012] Hereinafter, an information processing method, an apparatus, etc. according to an embodiment of the present disclosure will be described with reference to the drawings.

[0013] <Information Processing System> Fig. 1 shows an information processing system S according to one embodiment. The information processing system S shown in Fig. 1 includes an operator terminal 10, a robot 20, and an environment-installed sensor 30.

[0014] The operator terminal 10 creates a planned route for the robot 20. The robot 20 moves based on the planned route created by the operator terminal 10. The operator terminal 10 acquires various information about the environment in which the robot 20 moves from the robot 20 and the environmental sensors 30, and uses the information to create the planned route.

[0015] As an example, the operator terminal 10 in this embodiment creates a planned path for the robot 20 when the robot 20 moves and films a performance by the artist 40. In this embodiment, the robot 20 is a cart-type robot that travels on the stage 41, but is not particularly limited to this. The robot 20 may also be an articulated robot or an air vehicle such as a drone.

[0016] 1 shows a situation in which an artist 40 is performing on a stage 41. Reference numeral 42 denotes a microphone stand used by the artist 40. Reference numeral 43 denotes a first obstacle installed on the stage 41. Reference numeral 44 denotes a second obstacle installed on the stage 41. The microphone stand 42 is classified as a dynamic obstacle because it is moved by the artist 40. In this example, the first obstacle 43 and the second obstacle 44 are classified as static obstacles, and are specifically decorative structures (so-called sets).

[0017] The robot 20 and the environmental sensor 30 capture images of the performance (e.g., rehearsal) of the artist 40 that is performed before the path plan is created on the operator terminal 10, and perform object detection on the stage 41 during the performance. As a result, the robot 20 and the environmental sensor 30 acquire information such as the position of the artist 40 and the positions of obstacles, which is various information related to the environment in which the robot 20 moves. In this example, the environmental sensor 30 is placed above the stage 41. As a result, the environmental sensor 30 can capture and / or detect the overall state of the stage 41 and the state of the artist 40 and obstacles on the stage 41.

[0018] In the information processing system S, the operator terminal 10 generates a user interface for creating a planned route for the robot 20 based on the various types of information as described above provided from the robot 20 and the environmental sensors 30. Then, the operator terminal 10 creates a planned route that reflects the operator's intentions based on operations from the operator on the user interface. The created planned route is provided to the robot 20. Thereafter, as the artist 40 begins performing, the robot 20 begins moving based on the planned route, and the robot 20 takes a photograph of the artist 40. Below, each component of the information processing system S will be described in detail.

[0019] (Configuration of Each Part of Information Processing System) Fig. 2 is a block diagram showing the functional configuration of the information processing system. The operator terminal 10, the robot 20, and the environmental sensor 30 will be described in detail below.

[0020] [Operator Terminal] As shown in FIG. 2, the operator terminal 10 includes an information input / output unit 11, an information acquisition unit 12, an information processing unit 13, and an information presentation unit 14.

[0021] The information input / output unit 11 is a module having a function of inputting and outputting information. In this example, the information input / output unit 11 includes an external information input unit 111, an instruction information input unit 112, and an operation instruction output unit 113.

[0022] The external information input unit 111 is a part that inputs information provided from external devices or recording media such as the robot 20, the environmental sensors 30, a mobile terminal, a computer, a server, or a cloud via wireless or wired communication. As described above, the robot 20 and the environmental sensors 30 capture images of the performance (e.g., rehearsal) of the artist 40 that takes place before a path plan is created by the operator terminal 10, and detect objects on the stage 41 during the performance. The external information input unit 111 receives information about the performance from, for example, the robot 20 and the environmental sensors 30, and inputs the information into the operator terminal 10.

[0023] The robot 20 and the environmental sensor 30 can also acquire information about the performance when the robot 20 is moving along a route plan and capturing images of the performance of the artist 40. The various pieces of information acquired in this case are also provided to the external information input unit 111.

[0024] Furthermore, if the performance-related information provided by the robot 20 and the environmental sensors 30 includes audio information, the external information input unit 111 can input the audio information into the operator terminal 10 simultaneously with the video and / or object detection information. Alternatively, such audio information may be input to the operator terminal 10 from an external source, such as a mobile terminal or recording medium, as audio information separate from the information provided by the robot 20 and the environmental sensors 30. In this case, if the audio information includes metadata including additional information indicating constituent sections such as an intro, chorus, and ending, such additional information can be incorporated and effectively utilized when creating a planned route. In this embodiment, audio information related to the artist 40's performance is input from an external source.

[0025] The external information input unit 111 also acquires map information specifying the coordinates and size of the stage 41 from an external device and inputs the map information into the operator terminal 10. The map information may be created by measuring the stage 41 and its surrounding environment in advance, or may be created by cutting out the map information from existing map information. The map information may be two-dimensional map information consisting of a horizontal coordinate plane, or may be three-dimensional map information.

[0026] The instruction information input unit 112 is a part that inputs information instructed by operating means such as a mouse, keyboard, or touch panel on a display (not shown) that are provided in the operator terminal 10. As described above, the operator terminal 10 generates a user interface for creating a planned path for the robot 20. For example, instructions given by the operator to this interface via a touch panel, mouse, keyboard, or the like are input into the operator terminal 10 from the instruction information input unit 112. Figure 3 shows an example of a user interface displayed by the operator terminal 10. In the following description, reference will be made to Figure 3 as necessary.

[0027] The operation instruction output unit 113 is a part that outputs information to the robot 20 via wireless communication. As described above, the operator terminal 10 creates a planned route for the robot 20. The operation instruction output unit 113 outputs the planned route created in the operator terminal 10 together with operation instructions to the robot 20. In detail, the planned route is created by a moving object position determination unit 133 in the information processing unit 13, which will be described later.

[0028] The information acquisition unit 12 is a module that acquires information input from the information input / output unit 11 for internal processing, and at least partially processes and stores the information as necessary. In this example, the information acquisition unit 12 includes a map information acquisition unit 121, an object position information acquisition unit 122, an obstacle position information acquisition unit 123, and a voice information acquisition unit 124.

[0029] The map information acquisition unit 121 is a part that acquires and stores map information of the environment in which the robot 20 moves. As described above, the external information input unit 111 acquires map information of the stage 41 from an external device and inputs it into the operator terminal 10. In this example, the map information acquisition unit 121 acquires and stores the map information from the external information input unit 111.

[0030] In this embodiment, two-dimensional map information of the stage 41 is used when creating a planned path for the robot 20. Therefore, it is sufficient for the map information acquisition unit 121 to be able to store two-dimensional map information. When three-dimensional map information of the stage 41 is provided from the external information input unit 111, the map information acquisition unit 121 may extract the two-dimensional map information from the three-dimensional map information and store it. Furthermore, when the shape of the stage 41 can be detected from the environmental sensor 30, the map information acquisition unit 121 may generate and store map information based on information from the environmental sensor 30.

[0031] The object position information acquisition unit 122 is a part that acquires object position information related to the position of an object that is the target of processing by the robot 20. In this embodiment, the artist 40 that is the target of photography processing by the robot 20 corresponds to the object. The object position information includes at least information on the lateral position and depth position of the object. The object position information acquisition unit 122 acquires and stores object position information that includes information on at least the lateral position and depth position of the artist 40, which is the object, based on information on the performance of the artist 40 that is provided from the robot 20 and the environmental sensors 30 via the external information input unit 111 described above.

[0032] As described above, the robot 20 and the environmental sensors 30 capture images of the artist 40's performance before the operator terminal 10 creates a path plan, and perform object detection on the stage 41 during the performance. When the robot 20 and the environmental sensors 30 capture video of the artist 40 and / or perform object detection of the artist 40, the position and orientation (face orientation) of the artist 40, who moves over time, can be identified based on this information. Therefore, the object position information acquisition unit 122 can acquire the object position information that identifies the position of the artist 40, based on information about the artist 40's performance provided by the robot 20 and the environmental sensors 30.

[0033] In this embodiment, the object position information acquisition unit 122 acquires information on the lateral position and depth position of the artist 40 as object position information, and information on the facial direction of the artist 40, and stores the information linked to time. In more detail, the object position information acquisition unit 122 acquires information on the lateral position and depth position of the artist 40 based on information from the environment-installed sensor 30. Meanwhile, the object position information acquisition unit 122 acquires information on the facial direction of the artist 40 based on information from the robot 20.

[0034] When the environmental sensor 30 captures images, information on the lateral position, depth position, and facial orientation of the artist 40 can be identified by analyzing the images captured by the environmental sensor 30. However, because the environmental sensor 30 is installed at a position relatively far from the artist 40, it may not be possible to ensure sufficient accuracy in identifying the facial orientation of the artist 40. Therefore, in this embodiment, for example, information on the lateral and depth positions of the artist 40 is obtained based on the images from the environmental sensor 30, and information on the facial orientation of the artist 40 is obtained based on the images from the robot 20.

[0035] However, the processing of the object position information acquisition unit 122 as described above is not particularly limited. For example, the environmental sensor 30 may detect the position of the artist 40 by tracking radio waves or light emitted from a marker attached to the artist 40. In this case, the object position information acquisition unit 122 may acquire the position information of the artist 40 without analyzing the video. Furthermore, if information about the orientation of the object is not used when creating the planned route, the object position information acquisition unit 122 does not need to acquire the orientation of the object.

[0036] Specifically, the object position information is a data set that includes at least a plurality of sets of time points and at least horizontal position information and depth position information of the artist 40 corresponding to each time point. The time interval between two consecutive time points among the plurality of time points is not particularly limited. Also, for example, if the artist is a group of multiple people and all of them are being photographed, all of the artists may become objects, and there may be multiple objects. In this case, the object position information acquisition unit 122 may acquire multiple sets of object position information.

[0037] The obstacle position information acquisition unit 123 is a part that acquires position information of obstacles. In this embodiment, the microphone stand 42, the first obstacle 43, and the second obstacle 44 described above correspond to obstacles. The obstacle position information acquisition unit 123 acquires and stores obstacle position information based on information about the performance of the artist 40 provided by the environmental sensors 30, etc. The obstacle position information includes information that identifies at least the position of the obstacle and the area occupied by the obstacle.

[0038] The obstacle position information acquisition unit 123 stores obstacle position information linked to time. The first obstacle 43 and the second obstacle 44 are static obstacles that do not move over time, and therefore their position information does not change over time. On the other hand, the microphone stand 42 can be moved by the artist 40. Since the microphone stand 42 is usually located in front of the artist 40, it can become an obstacle during filming. Therefore, the position information of the microphone stand 42 can be effectively used in creating a planned route.

[0039] When the environmental sensor 30 captures images, the position of the microphone stand 42, which is a dynamic obstacle, can be identified by analyzing the images captured by the environmental sensor 30. In this embodiment, as described above, the object position information acquisition unit 122 acquires information about the lateral and depth positions of the artist 40 based on the images from the environmental sensor 30. At the same time, the obstacle position information acquisition unit 123 acquires position information of the microphone stand 42 based on the images from the environmental sensor 30. Furthermore, the obstacle position information acquisition unit 123 also acquires the positions of the first obstacle 43 and the second obstacle 44 based on the images from the environmental sensor 30.

[0040] However, the processing of the obstacle position information acquisition unit 123 as described above is not particularly limited. For example, the environmental sensor 30 can detect the position of the microphone stand 42 by tracking radio waves or light emitted from a marker attached to the microphone stand 42. The position of the microphone stand 42 detected in this manner may be acquired by the obstacle position information acquisition unit 123. On the other hand, in the case of such tracking, the positions of the static first obstacle 43 and second obstacle 44 are not detected. In this case, the positions of the static first obstacle 43 and second obstacle 44 may be identified by acquiring a three-dimensional model created in advance from the outside or by creating it inside the operator terminal 10.

[0041] The audio information acquisition unit 124 is a part that acquires audio information. As described above, if audio information is included in the information about the artist 40's performance provided by the robot 20 and the environmental sensors 30, the external information input unit 111 can input the audio information together with the video and / or object detection information. In this case, the audio information acquisition unit 124 acquires and stores the audio information input by the external information input unit 111 together with the video and / or object detection information.

[0042] On the other hand, as described above, the voice information may be input from an external source such as a mobile terminal or a recording medium as voice information separate from the information provided by the robot 20 and the environmental sensor 30. In this case, the voice information acquisition unit 124 acquires and stores the voice information input from the external source.

[0043] The audio information acquisition unit 124 can add additional information to audio information in response to a user operation. The additional information is, for example, information indicating a composition section such as an intro, chorus, or ending. Audio information input from the outside may contain metadata including information indicating the composition section. In this case, the audio information acquisition unit 124 may also store such metadata.

[0044] The information processing unit 13 is a module that generates a user interface for creating a planned path for the robot 20 based on the information acquired by the information acquisition unit 12. In this example, the information processing unit 13 includes a position representation image generation unit 131, a priority determination unit 132, and a moving object position determination unit 133.

[0045] The position representation image generation unit 131 is a part that generates a position representation image that constitutes a user interface for creating a planned path for the robot 20. The position representation image is an image that represents at least the lateral position and the depth position of the artist 40, who is an object, along with a horizontal axis and a depth axis.

[0046] The position representation image generation unit 131 generates position representation images for at least two time points during the period of the artist 40's performance, in other words, during the period during which the object that is the artist 40 moves, based on the information acquired by the object position information acquisition unit 122. In more detail, the position representation generation unit 131 in this embodiment generates position representation images for each predetermined time interval during the period during which the artist 40 performs.

[0047] In this embodiment, the position representation image generated by the position representation image generator 131 is an image showing the position of the object (artist 40) in a bird's-eye view image that uses a projection method to represent a horizontal coordinate plane defined by a horizontal axis and a depth axis. Figure 3 shows position representation images Ims, Imm, and Ime, which are part of the position representation image generated by the position representation image generator 131. In the position representation images Ims, Imm, and Ime, the axis indicated by the symbol X represents the horizontal axis, and the axis indicated by the symbol Y represents the depth axis.

[0048] The priority determination unit 132 is a part that determines the position priority, which is information indicating the importance of the position of the robot 20 that is moved together with the artist 40 during a performance, and also determines the time priority, which is information indicating the importance of the movement of the artist 40 during the performance period. The priority determination unit 132 determines the position priority and the time priority automatically or in response to the operation of the operator.

[0049] The position priority is automatically determined at predetermined time intervals during the performance, for example, depending on the relative positions of the artist 40 and the robot 20. Specifically, in this embodiment, the closer a position on the stage 41 is to the position where the artist 40 and the robot 20 face each other, or more precisely, the closer a position is to the position where the artist 40 and the camera 24 (described later) mounted on the robot 20 face each other, the higher the position priority is set. Even when the artist 40 and the robot 20 face each other, it is not necessarily desirable for them to be too close or too far apart. Taking this into consideration, in this embodiment, the highest position priority is assigned to a position where the artist 40 and the robot 20 face each other and where the distance between the artist 40 and the robot 20 is a predetermined value, and the position priority is set lower as the position moves away from this position. Furthermore, a lower position priority is set to a position where an obstacle is present. Note that the above-described manner of setting the position priority is merely an example, and the position priority may be set in other ways or arbitrarily by the operator.

[0050] As will be described in detail later, when a position priority is set, the display manner of the position representation images Ims, Imm, and Ime included in the user interface for creating a planned route for the robot 20 shown in FIG. 3 is changed according to the position priority. Specifically, the area of ​​the position representation image is color-coded according to the position priority, and at least the position of the artist 40 is indicated on the position representation image. By referring to the position representation image that is color-coded in this manner, the operator can easily recognize the position that should be set for the robot 20 that is to be moved together with the artist 40.

[0051] On the other hand, the time priority may be set, for example, according to the structural section of the music piece. Specifically, for example, a high time priority may be set for a structural section corresponding to an intro, and a low time priority may be set for other structural sections. As described above, the audio information acquisition unit 124 can assign and retain information indicating structural sections such as an intro, chorus, and outro to the audio information. The operator may, for example, refer to such additional information and set different time priorities for different time periods during the performance period. Furthermore, when information indicating structural sections such as an intro, chorus, and outro is retained in the audio information, the priority determination unit 132 may automatically set time priorities for each time period corresponding to the structural section.

[0052] As will be described in detail later, when a time priority is set, the display mode of the user interface for creating a planned route for the robot 20 is changed according to the time priority. Specifically, as shown in Fig. 3, the display mode of the time axis Z included in the user interface is changed according to the time priority. By being able to visually recognize such time priority, the operator can easily recognize the position to be set for the robot 20 that is to be moved together with the artist 40.

[0053] The moving object position determination unit 133 is a part that determines the planned position of the robot 20 that is moved together with the artist 40 automatically or in response to an operation by the operator. When automatically determining the planned position, the moving object position determination unit 133 automatically determines, as the planned position, the position of the robot 20 whose relative position with respect to the artist 40 satisfies a predetermined condition, for example. The predetermined condition may be, for example, a position where the artist 40 and the robot 20 face each other and where the distance between the artist 40 and the robot 20 is a predetermined value or range. The moving object position determination unit 133 may determine the planned position based on the position priority determined by the priority determination unit 132.

[0054] As will be described later, in this embodiment, the operator can specify the movement range (traveling rails) of the robot 20. When this specification is made, the moving body position determination unit 133 automatically determines (generates) as the planned position the position of the robot 20 within the specified movement range, where the relative position with respect to the artist 40 satisfies a predetermined condition.

[0055] The planned position is information that constitutes the planned path of the robot 20. In other words, the planned path of the robot 20 is constructed by connecting planned positions set at multiple points in time during the performance period. Furthermore, the moving object position determination unit 133 can change the planned position determined as described above in response to an operation by the operator. By enabling such an operation, it becomes possible to determine a planned path of the robot 20 that reflects the operator's intentions.

[0056] On the other hand, when the planned position is determined in accordance with the operation of the operator without automatically determining the planned position as described above, the moving object position determination unit 133 receives an input of the position of the robot 20 from the operator and determines the input position as the planned position. Whether the planned position is determined automatically by the moving object position determination unit 133 or in accordance with the operation of the operator may be selected by the operator operating the operator terminal 10. As will be described later, in this embodiment, after the planned position is automatically determined by the moving object position determination unit 133, the operator can change the planned position (planned route) as necessary.

[0057] The position representation image, position priority, time priority, and planned position described above are provided to the information presentation unit 14. The information presentation unit 14 is a module that displays a user interface for creating a planned path for the robot 20 and changes the display of the user interface in response to operations performed on the user interface. In this example, the information presentation unit 14 includes an image display unit 141 and a planned path display unit 142.

[0058] The image display unit 141 is a unit that receives position representation images from the position representation image generation unit 131 and displays a user interface including the position representation images on a display (not shown). Specifically, the image display unit 141 displays position representation images for at least two points in time during the artist 40's performance in a three-dimensional manner, arranging them vertically along a time axis Z extending vertically. As shown in FIG. 3 , the time axis Z is coupled to each position representation image so as to be perpendicular to both the horizontal axis X and the depth axis Y of each position representation image, which constitutes a bird's-eye view image expressed using a projection method. As a result, the multiple position representation images are displayed aligned in the direction of the time axis in a three-dimensional Cartesian coordinate space consisting of the horizontal axis, the depth axis, and the time axis.

[0059] In this example, a user interface including three position representation images Ims, Imm, and Ime and a time axis Z, as shown in FIG. 3, is displayed. The three position representation images Ims, Imm, and Ime are arranged from bottom to top in chronological order. In the example shown in FIG. 3, the position representation image Ims located at the bottom corresponds to the position representation image at the start of the artist 40's performance. The position representation image Imm located in the center corresponds to the position representation image at the midpoint of the performance, more specifically, the position representation image at the chorus of the song being performed. The position representation image Ime located at the top corresponds to the position representation image at the start of the performance. While the position of the artist 40 is shown as a three-dimensional model in FIG. 3, it may also be represented in two dimensions (e.g., as a highlight on a horizontal coordinate plane or as a point). Furthermore, the number of position representation images included in the user interface is not particularly limited.

[0060] Furthermore, when the priority determination unit 132 determines a position priority and / or a time priority, the image display unit 141 is provided with the position priority and / or the time priority and reflects them in the position representation image. As shown in FIG. 3 , in this embodiment, the areas of the position representation image Ims, Imm, and Ime are color-coded according to their position priority. In this example, the area indicated by a has the highest position priority, followed by the area indicated by b and the area indicated by c, in that order. Furthermore, an arrow F extends from the portion of the position representation image corresponding to the artist 40, and the arrow F indicates the direction of the artist 40's face. The area a in the position representation image is set to be an area located in the direction of the arrow F relative to the artist 40. In other words, the position where it is possible to face the artist 40 has a high position priority.

[0061] 3, the display mode of the time axis Z is changed according to the time priority. Specifically, a highlight is added to the time axis Z, with the area indicated by a' in the added highlight having the highest time priority, followed by the areas indicated by b' and c' in descending order of time priority. In FIG. 3, a highlight representing the area a' is added to the time period on the time axis Z corresponding to the intro in the song.

[0062] The planned path display unit 142 is a part that is provided with the planned position of the robot 20 determined by the moving object position determination unit 133, and displays the planned path of the robot 20 based on the planned position. In detail, the planned path display unit 142 displays the planned path of the robot 20 by connecting points corresponding to the planned positions in at least two position representation images displayed by the image display unit 141 with lines extending between the at least two position representation images.

[0063] Although details will be described later, Fig. 6 shows the planned route Rp of the robot 20 displayed by the planned route display unit 142. Such a planned route Rp makes it easier to grasp the position of the robot 20 over time. Furthermore, the greater the slope between the planned route Rp and the time axis Z, the greater the speed of the robot 20. Checking the state of such a slope makes it easier to grasp the speed profile of the robot 20.

[0064] In the user interface provided by the information presentation unit 14 as described above, the operator can change the shape of the planned route Rp, for example, by operating on the screen. The shape of the planned route Rp is changed by the moving object position determination unit 133 changing the planned position in response to the operator's operation. When the operator then performs an operation to confirm the planned route, the action instruction output unit 113 outputs an action instruction to the robot 20 along with the planned route at the time of the confirmation operation. The planned route also includes directional information about the orientation of the artist 40, and the robot 20 can determine the desired orientation of the camera 24 based on the directional information.

[0065] 2, the robot 20 has an information input / output unit 21, an information processing unit 22, a sensor 23, a camera 24, and a movement mechanism 25. As described above, the robot 20 is a cart-type robot, and is configured by holding the sensor 23 and the camera 24 on the cart portion. The movement mechanism 25 includes, for example, wheels and a motor for rotating the wheels, but is not particularly limited to these.

[0066] The information input / output unit 21 is a module having a function of inputting and outputting information to and from the outside via wireless or wired communication. The information input / output unit 21 inputs, for example, information on the planned route of the robot 20 and operation instructions according to the planned route, which are provided from the operator terminal 10. In addition, for example, the information input / output unit 21 outputs information acquired by the robot 20 to the operator terminal 10.

[0067] As described above, the robot 20 captures footage of the artist 40's performance (e.g., rehearsal) before creating a path plan on the operator terminal 10, and performs object detection on the stage 41 during the performance. Specifically, the information input / output unit 21 outputs to the operator terminal 10 the video of the performance captured by the camera 24 and the detection results obtained by detecting objects on the stage 41 using the sensor 23.

[0068] Furthermore, as described above, even when the robot 20 is filming the performance of the artist 40 while moving according to a route plan, the robot 20 outputs the acquired information (video and object detection results) to the operator terminal 10 via the information input / output unit 21.

[0069] The information processing unit 22 is a module that controls the overall operation of the robot 20. In this example, the information processing unit 22 includes an operation control unit 221 and an imaging control unit 222.

[0070] The movement control unit 221 is a part that controls the movement mechanism 25. In detail, the movement control unit 221 controls the robot 20 so that the robot 20 moves together with the movement of the artist 40, which is an object, based on the planned route provided by the information input / output unit 21.

[0071] The imaging control unit 222 is a part that controls the operation of the camera 24. More specifically, the imaging control unit 222 controls the orientation of the camera 24 in accordance with the movement of the robot 20, based on the planned route provided by the information input / output unit 21. As described above, the planned route also includes directional information about the orientation of the artist 40. The robot 20 determines the desired orientation of the camera 24 using the imaging control unit 222, based on the directional information.

[0072] As described above, the sensor 23 is a part that performs object detection on the stage 41. The sensor 23 acquires the positions of objects and obstacles, i.e., in this example, the artist 40, the microphone stand 42, the first obstacle 43, and the second obstacle 44, by object detection. The sensor 23 may be any one of an RGB-D camera, a Lidar sensor, a ToF sensor, an infrared sensor, and an ultrasonic sensor, or a combination of two or more of these.

[0073] The camera 24 is a part that takes pictures under the control of the imaging control unit 222. The camera 24 may be supported so as to be rotatable around three axes, for example, and the orientation (image capture direction) may be controlled by being rotated by the imaging control unit 222. The camera 24 may be, for example, an RGB camera or an RGB-D camera.

[0074] As described above, the movement mechanism 25 includes, for example, wheels and motors for rotating the wheels, and is a part that moves the robot 20. The movement mechanism 25 is controlled by the operation control unit 221 to, for example, adjust the driving state of the motor. Such a movement mechanism 25 can take various forms depending on the configuration of the robot 20. The robot 20 may be an articulated robot as exemplified above, an air vehicle such as a drone, or may be one that moves on physical running rails, or one that moves along the running rails while suspended from the physical running rails. The movement mechanism 25 adopts an appropriate configuration depending on the configuration of such a robot 20.

[0075] [Environmental Sensor] The environmental sensor 30 is provided to acquire the movement trajectory of the artist 40, which is an object, and the movement trajectory or position of an obstacle, and outputs the acquired information to the operator terminal 10. As described above, the environmental sensor 30 captures footage of the artist 40's performance (e.g., a rehearsal) that takes place before the operator terminal 10 creates a path plan, and performs object detection on the stage 41 during the performance. Specifically, the environmental sensor 30 outputs video of the performance and / or the results of object detection performed on the stage 41 to the operator terminal 10.

[0076] The environmental sensor 30 may be an RGB camera, an RGB-D camera, a Lidar sensor, a Time of Flight sensor, an infrared sensor, an ultrasonic sensor, or a UWB sensor, or a combination of two or more of these. If the environmental sensor 30 includes a UWB sensor, the position of the artist 40 can be detected by receiving radio waves transmitted from a marker attached to the artist 40 using the UWB method.

[0077] (Operation) Next, an example of processing (information processing method) of the information processing system S will be described with reference to the flowchart shown in Fig. 4. Specifically, the operation described below begins when, before a path plan is created on the operator terminal 10, the robot 20 and the environmental sensor 30 film a performance (e.g., a rehearsal) of the artist 40 and perform object detection on the stage 41 during the performance, and then provide the video and object detection results to the operator terminal 10, and then the operator issues a command to display a user interface on the operator terminal 10.

[0078] When the operator issues the display instruction, first, in step S401, a map of the environment in which the robot 20 moves is created. More specifically, a map of the stage 41 is created. In this embodiment, the external information input unit 111 acquires map information specifying the coordinates and size of the stage 41 from an external device and inputs the map information into the operator terminal 10. The map of the stage 41 is created based on this map information.

[0079] Next, in step S402, the time for which the robot 20 will move, in other words, the period for which the robot 20 will move, is set. The time for which the robot 20 will move is at least equal to or longer than the performance time of the artist 40. The time set here is set in order to link the position information of the artist 40 and obstacles with time.

[0080] Next, in step S403, position information of static obstacles is acquired. In this embodiment, specifically, position information of the first obstacle 43 and the second obstacle 44 is acquired. The position information of the first obstacle 43 and the second obstacle 44 includes information on their positions and the areas they occupy. The processing in step S403 corresponds to the processing of the obstacle position information acquisition unit 123. In other words, in step S403, the obstacle position information acquisition unit 123 acquires and stores the position information of the first obstacle 43 and the second obstacle 44 based on information about the performance of the artist 40 provided by the environmental sensors 30, etc.

[0081] Next, in step S404, position information of a moving obstacle is acquired. In this embodiment, specifically, position information of the microphone stand 42 is acquired. The position information of the microphone stand 42 includes information about the position of the microphone stand 42, which changes over time, and the area it occupies. By identifying the position of the microphone stand 42, which changes over time, the trajectory of the microphone stand 42 is determined. The processing in step S404 corresponds to the processing of the obstacle position information acquisition unit 123. In other words, in step S404, the obstacle position information acquisition unit 123 acquires and stores position information of the microphone stand 42 based on information about the performance of the artist 40, etc., provided by the environmental sensors 30. Here, the position information of the moving obstacle is stored in association with time.

[0082] Next, in step S405, object position information is acquired, which is position information of the artist 40, who is an object and a subject of the image. The object position information includes at least information on the lateral position and depth position of the artist 40, which change over time, and information on the facial orientation of the artist 40. By identifying the position of the artist 40, which changes over time, the trajectory of the artist 40 is determined. The processing in step S405 corresponds to the processing of the object position information acquisition unit 122. That is, in step S405, the object position information acquisition unit 122 acquires and stores object position information including information on at least the lateral position and depth position of the artist 40, who is an object, based on information on the performance of the artist 40 provided by the robot 20 and the environmental sensors 30. Here, the position information of the artist 40 is stored in association with time.

[0083] Next, in step S406, the priority determination unit 132 determines a position priority and a time priority. In this embodiment, the position priority is automatically determined according to, for example, the relative positions of the artist 40 and the robot 20. In detail, in this embodiment, the closer a position on the stage 41 is to the position where the artist 40 and the robot 20 face each other, or more precisely, the closer a position is to the position where the artist 40 and a camera 24 (described below) provided on the robot 20 face each other, the higher the position priority is set. Such determination of the position priority can be made based on the object position information (particularly the facial orientation of the artist 40) acquired in step S405.

[0084] In this embodiment, time priorities are also determined automatically. In this embodiment, audio information related to the performance of the artist 40 includes information indicating composition sections such as an intro, a chorus, and an outro. The priority determination unit 132 automatically sets time priorities for each time period corresponding to a composition section. Specifically, a high time priority is automatically assigned to the chorus.

[0085] Next, in step S407, a position representation image is generated. As described above, the position representation image is an image that represents at least the lateral position and the depth position of the artist 40 along the horizontal axis and the depth axis. In this embodiment, the position representation image generation unit 131 generates a position representation image for each predetermined time interval during the period in which the artist 40 performs. Furthermore, in this embodiment, the position priority determined in step S406 is reflected in the position representation image. The position representation image can be generated based on the map created in step S401, the object position information acquired in step S405, and the position priority determined in step S406.

[0086] Next, in step S408, the position representation images generated in step S407 are displayed three-dimensionally, arranged vertically along the time axis Z extending in the vertical direction. The processing of step S408 displays the user interface shown in Fig. 3. As shown in Fig. 3, the position representation images indicate at least the horizontal and depth positions of the artist 40, who is an object, and the facial orientation of the artist 40. The processing of step S408 corresponds to the processing of the image display unit 141.

[0087] After the user interface is displayed in step S408, in this embodiment, step S409 accepts designation of a travel rail (movement range) for the robot 20. Whether or not to accept designation of a travel rail for the robot 20 can be switched according to an operation by the operator.

[0088] Next, in step S410, the planned route for the robot 20 is displayed on the user interface. In this example, the travel rail is specified in step S409, and the planned route within the range of the travel rail is displayed. The processing in step S410 corresponds to processing by the moving object position determination unit 133 and the planned route display unit 142. The moving object position determination unit 133 automatically determines (generates) as a planned position the position of the robot 20 within the travel rail specified in step S409, whose relative position with respect to the artist 40 satisfies a predetermined condition. The planned route display unit 142 displays the planned route by connecting the planned positions at multiple points in time determined by the moving object position determination unit 133.

[0089] Next, in step S411, the system accepts adjustments made by the operator to the planned route displayed in step S410. In the user interface displayed in step S408, the operator can change the shape of the planned route displayed in step S410, for example, by operating on the screen.

[0090] If an operation is performed by the operator in step S411, the mobile object position determination unit 133 recalculates and modifies the planned route in step S412 in response to the operation by the operator. At this time, the speed profile is changed along with the change in the planned position. In this embodiment, the speed profile is changed so that the displayed planned route has a smooth curve.

[0091] Thereafter, in step S413, the planned route is confirmed and input of a driving instruction is accepted. When the planned route is confirmed and a driving instruction is input, the operation instruction output unit 113 outputs an operation instruction to the robot 20 along with the planned route at the time of the confirmation operation. Note that if the planned route is not adjusted in step S411, the process proceeds from step S411 to step S413.

[0092] (User Interface) The user interface shown in FIG. 3 , which is displayed by the processing of step S408 in FIG. 4 , and examples of its operation will be described in detail below with reference to FIGS. 3 to 7 . As described above, the user interface according to this embodiment displays position representation images (Ims, Imm, Ime) at at least two points in time (three points in this example) during the artist 40's performance in a three-dimensional manner, arranged vertically along a time axis Z extending vertically. The time axis Z is coupled to each position representation image so as to be perpendicular to both the horizontal axis X and the depth axis Y of each position representation image, which forms a bird's-eye view image represented by a projection method. As a result, the multiple position representation images are displayed aligned in the direction of the time axis in a three-dimensional Cartesian coordinate space consisting of the horizontal axis, depth axis, and time axis.

[0093] 3, the position representation image in the user interface indicates at least the lateral and depth positions of the artist 40, as well as the facial orientation of the artist 40 (arrow F). This allows the operator to easily grasp the position and state of the artist 40 over time by checking the position representation image along the time axis Z. This allows the operator to easily grasp the position of the robot 20 that will allow the operator to properly photograph the artist 40.

[0094] Furthermore, in this embodiment, the areas of the position representation image Ims, Imm, and Ime are color-coded according to the position priority, which indicates the importance of the position of the robot 20 in relation to the artist 40. In Figure 3, the order of position priority is c → b → a. Area a, which has high position priority in the position representation image in Figure 3, is a position or its vicinity where it is possible to face the artist 40, and by referring to this color coding, the operator can easily grasp the position of the robot 20 where it can properly photograph the artist 40.

[0095] Highlighting is also added to the time axis Z, and the added highlighting indicates that the time priority is highest in the order of area c' → b' → a'. In FIG. 3 , a highlighting is added to the time period on the time axis Z corresponding to the intro in the song, which is area a'. By referring to such highlighting, the operator can easily grasp important time periods in the shooting, and can, for example, thoroughly consider adjustments to the planned route. In addition, in FIG. 3 , the time axis Z is also provided with text indicating the compositional sections of the song (time attribute information such as intro, chorus, and outro). By displaying such additional information, the user interface according to this embodiment becomes easier to use.

[0096] 3 also displays a scroll bar 50 for adjusting the display state of the position representation image. The scroll bar 50 has a rail bar 51 extending parallel to the time axis Z and a slider 52 movable along the rail bar 51. In this example, the position of the slider 52 corresponds to the position representation image Imm at an intermediate point in the performance. The operator can change the display of the position representation image Imm to a position representation image at another time point by, for example, changing the position of the slider 52 with the pointer 70. The position of the slider 52 indicates the designation of an arbitrary point on the time axis Z. In this embodiment, a position representation image at a time point corresponding to an arbitrary point on the time axis Z designated via the slider 52 can be displayed in a pop-up area 60 separate from the area where the position representation image is displayed (corresponding to the pop-up display step).

[0097] 3 shows a plan view of the position representation image displayed in the pop-up area 60. The display mode of the position representation image displayed in the pop-up area may be changed from the display mode of the position representation images displayed in a vertically aligned manner as in the example of FIG. 3. On the other hand, the display mode of the position representation image displayed in the pop-up area may be the same as the display mode of the position representation images displayed in a vertically aligned manner as shown in parentheses below the pop-up area 60.

[0098] In the process described with reference to Fig. 4, it has been explained that the designation of the travel rail (movement range) of the robot 20 is received in step S409. Fig. 5 is a diagram illustrating the operation of receiving the travel rail. In the example of Fig. 5, the travel rail 80 is designated by moving the pointer 70 laterally on the position representation image Ims. In the example of Fig. 5, the linear travel rail 80 is designated as the movement range, but the movement range may be set by a curved line, or may be designated as a rectangular range, etc.

[0099] FIG. 6 shows an example of a planned route Rp displayed on a user interface. In this embodiment, a planned route Rp within the range of the traveling rail 80 is generated and displayed. As described above, in this embodiment, the planned route Rp is automatically generated by connecting the planned movement positions of the robot 20 at multiple points in time that satisfy predetermined conditions. It can be seen from FIG. 6 that the planned route Rp is generated so as to pass through areas with high position priority as much as possible. Also, in FIG. 6, auxiliary lines 81 are displayed extending parallel to the time axis Z from both ends of the traveling rail 80. By referring to these auxiliary lines 81, the range of the traveling rail 80 can be confirmed in each position representation image.

[0100] In the process described with reference to FIG. 4 , adjustments made by the operator to the displayed planned route are accepted in step S411. FIG. 7 is a diagram illustrating an adjustment operation of the planned route Rp made by the operator. In the example of FIG. 7 , an arbitrary point P1 on the planned route Rp is designated using the pointer 70, and a movement operation of the arbitrary point P1 is accepted, as shown from the left to the right diagram. When such a movement operation is accepted, in this example, the shape of the planned route Rp is changed so that it passes through point P2, which has been moved by the movement operation, as shown in the right diagram of FIG. 7 (see the transition from the two-dot chain line to the solid line). When the shape of the planned route Rp is changed, the speed profile is also changed in addition to changes in the scheduled positions constituting the planned route Rp. As shown in FIG. 7 , the speed profile is changed so that the displayed planned route becomes a smooth curve. The operation of changing the planned route is not limited to the above, and may be performed by changing numerical values, etc.

[0101] The information processing system S according to the embodiment described above performs an object position information acquisition step (step S405) of acquiring object position information including information on at least the horizontal position and depth position of the artist 40, who is a moving object; a position representation image generation step (step S407) of generating position representation images for multiple points in time based on the object position information, which represent at least the horizontal position and depth position of the artist 40 along a horizontal axis X and a depth axis Y; and an image display step (step S408) of displaying the position representation images for at least two points in time side by side in the vertical direction along the time axis Z extending in the vertical direction.

[0102] In the user interface displayed in the image display step (step S408) and including position representation images at at least two points in time aligned along the time axis Z, the position of the artist 40, which changes over time, can be easily grasped by referring to the time axis Z extending in the vertical direction. This makes it easier to grasp the desired position of the robot 20 to be moved together with the artist 40. Therefore, it becomes possible to easily create a planned route for the robot 20 to be moved together with the artist 40.

[0103] Furthermore, in this embodiment, when planned positions at multiple time points constituting the planned path of the robot 20 are determined, the planned path of the robot 20 is displayed by connecting points corresponding to the planned positions in at least two displayed position representation images with lines extending between the at least two position representation images (see the planned path Rp in FIG. 6 ). The planned path Rp displayed in this manner makes it easier to grasp the position of the robot 20 over time. Furthermore, the greater the inclination between the planned path and the time axis Z, the greater the speed of the robot 20. Checking the state of this inclination makes it easier to grasp the speed profile of the robot 20. Therefore, it is easy to check whether the displayed planned path matches the operator's image. Furthermore, by checking the planned path and adjusting it, it becomes easier to create a desired planned path.

[0104] Furthermore, in this embodiment, it is possible to set a position priority indicating the importance of the position of the robot 20 in relation to the artist 40, and a time priority indicating the importance of the actions of the artist 40 during the artist 40's performance period. The display mode of the position representation image can be changed according to the position priority, and the display mode of the time axis Z can be changed according to the time priority (see FIG. 3). In this way, by reflecting information indicating importance in the position representation image, the operator can more easily create a planned route that reflects his or her own intentions.

[0105] <Modifications> Modifications of the above-described embodiment will be described below. Duplicate descriptions of the configurations and functions of the modifications described below that are similar to those of the above-described embodiment will be omitted. Furthermore, similar configurations will be denoted by the same reference numerals.

[0106] (First Modification) This is a flowchart illustrating an example of processing by an information processing system according to a first modification. In the first modification, before generating a position representation image, a reference image is input by an operator (step S801). Then, a process of generating a planned route based on the reference image is performed (step S802).

[0107] The process of step S801 is performed by the operator inputting a reference video to the operator terminal 10. The operator inputs the reference video with the intention of requesting the operator terminal 10 to propose a planned route that allows for shooting similar to the reference video, for example.

[0108] Then, in step S802, the planned route generated based on the reference video is displayed on a user interface including a position representation image. In this example, the processing of step S802 is performed by inputting the object position information acquired in step S405 and the reference video input in step S801 as variables to a trained model that has learned videos of various artists and the positional relationships between the artists and the camera at that time. As a result, the trained model derives a planned route that allows for filming similar to that of the reference video.

[0109] The trained model in this variation is a trained model that has learned the video and the positional relationship between the artist and the camera at that time. However, in this case, classifications related to the video, such as music genre, number of people in the group, and target age group, may also be included in the training data. When training is performed in this manner, a planned route for capturing the desired video by the operator can be derived even when classification information, such as music genre, number of people in the group, and target age group, is input instead of the reference video in step S801. Furthermore, in deriving the planned route, a simple trained model that has learned information on the positional relationship (positional relationship on the horizontal coordinate plane) between other objects that change position over time and other moving bodies that move together with the other objects (the positional relationship between the artist and the camera) may be used. In such a trained model, a planned route that reflects the learning tendency can be derived by inputting at least information on the lateral position and depth position of the object as variables.

[0110] When a planned route is derived using machine learning as described above, a planned route that reflects the operator's intentions can be created more efficiently.

[0111] (Second Modification) In the second modification, a user interface different from that of the above-described embodiment is displayed, as shown in FIG. 9 . In the user interface shown in FIG. 9 , the aspects of the position representation images Ims′, Imm′, and Ime′ differ from those of the above-described embodiment. Specifically, the position representation images Ims′, Imm′, and Ime′ displayed in the second modification show the position of the artist 40 in a bird's-eye view image that uses a projection method to represent a three-dimensional space defined by a horizontal axis X, a depth axis Y, and a portion of a time axis Z. The position representation images Ims′, Imm′, and Ime′ as a three-dimensional space are a three-dimensional Cartesian coordinate space defined by a horizontal axis X, a depth axis Y, and a vertical axis indicating the up-down direction, and the vertical axis and the time axis Z are integrated. The time axis Z connects the multiple position representation images Ims′, Imm′, and Ime′.

[0112] 9, the areas of the three-dimensional space are color-coded according to the position priority. Also, a position representation image corresponding to the position of the slider 52 of the scroll bar 50 is displayed in a pop-up.

[0113] The user interface displayed in the second modified example is useful, for example, when creating a planned route for flying a drone 90, as shown in FIG.

[0114] (Third Modification) In the third modification, a user interface different from that of the above-described embodiment is displayed, as shown in FIG. 11 . In the user interface shown in FIG. 11 , the aspects of the position representation images Ims'', Imm'', and Ime'' are different from those of the above-described embodiment. In detail, the position representation images Ims'', Imm'', and Ime'' displayed in the third modification show the position of the artist 40 in a plan view of a coordinate plane consisting of a horizontal axis X and a depth axis Y. FIG. 11 also shows the traveling rail 80, the robot 20, and the planned route Rp′. The position and speed profile of the robot 20 can be grasped in FIG. 11 as well.

[0115] (Fourth Modification) In the above-described embodiment, the planned path of the robot 20 is automatically displayed. In contrast, the fourth modification provides a function that enables the operator to manually create a planned path in a user interface.

[0116] As shown in Figures 12(A) and 12(B), the user interface in the fourth modified example accepts input of the position (Pa, Pb) of the robot 20 in each of at least two position representation images (Ims, Imm). The moving object position determination unit 133 determines these input positions as the planned position of the robot 20. Thereafter, the planned positions are connected by lines as shown in Figure 12(C), thereby displaying the planned route R1. With this configuration of the fourth modified example, the operator's intentions can be reflected in detail in the planned route.

[0117] <Hardware Configuration> Fig. 13 shows an example of the hardware configuration of the operator terminal 10 and the robot 20. The operator terminal 10 and the robot 20 as information processing devices may be configured by a computer device 600. The computer device 600 includes a CPU 601, an input interface 602, an external interface 603, a communication device 604, a main storage device 605, and an external storage device 606, which are interconnected by a bus. The operator terminal 10 and the robot 20 may not include at least one of these elements.

[0118] The CPU (Central Processing Unit) 601 executes a computer program on the main storage device 605. The computer program is a program that realizes the above-mentioned functional configurations of the operator terminal 10 and the robot 20. The computer program may be realized by a combination of multiple programs or scripts, rather than by a single program. The CPU 601 executes the computer program to realize the functional configurations.

[0119] The input interface 602 is a circuit for inputting operation signals from input devices such as a keyboard, a mouse, and a touch panel to the operator terminal 10 and the robot 20. The instruction information input unit 112 of the information input / output unit 11 described above can be realized by the input interface 602, for example.

[0120] The external interface 603 displays, on a display device, for example, data stored in the operator terminal 10 and the robot 20 or data calculated by the operator terminal 10 and the robot 20. The external interface 603 may be connected to, for example, an LCD (liquid crystal display) or an organic electroluminescence display. The information presentation unit 14 described above can be realized by, for example, the external interface 603.

[0121] The communication device 604 is a circuit that enables the operator terminal 10 and the robot 20 to communicate with external devices wirelessly or via a cable. Data used by the operator terminal 10 and the robot 20 can be input from the external device via the communication device 604. The communication device 604 may include an antenna. Data input from the external device can be stored in the main memory device 605 or the external memory device 606. The external information input unit 111 and the operation instruction output unit 113 of the information input / output unit 11 described above can be realized by, for example, the communication device 604.

[0122] The main memory device 605 stores computer programs, data necessary for executing the computer programs, data generated by executing the computer programs, etc. The computer programs are deployed and executed on the main memory device 605. The main memory device 605 is, for example, RAM, DRAM, or SRAM, but is not limited to these.

[0123] The external storage device 606 stores computer programs, data required for executing the computer programs, data generated by executing the computer programs, etc. These computer programs and data are read into the main storage device 605 when the computer programs are executed. The external storage device 606 is, for example, but is not limited to, a hard disk, an optical disk, a flash memory, or a magnetic tape.

[0124] The computer program may be pre-installed in the computer device 600, or may be stored on a storage medium such as a CD-ROM. The computer program may also be uploaded to the Internet.

[0125] Furthermore, the computer device 600 may be configured as a single device, or may be configured as a system made up of multiple computer devices connected to each other.

[0126] The above-described embodiment shows an example for realizing the present disclosure, and the present disclosure can be implemented in various other forms. For example, various modifications, substitutions, omissions, or combinations thereof are possible without departing from the spirit of the present disclosure. Such modifications, substitutions, omissions, etc. are also included within the scope of the present disclosure, as well as the scope of the inventions described in the claims and their equivalents.

[0127] For example, in the above embodiment, an example is shown in which a planned path is created for the robot 20 photographing the artist 40. However, the technology of the present disclosure is an effective technology for creating a planned path for a moving body that moves together with an object that moves in a predetermined pattern, for example, and can be used for various purposes.

[0128] Furthermore, the effects of the present disclosure described in this specification are merely examples, and other effects may also be present.

[0129] The present disclosure may also be configured as follows: [Item 1] An information processing method comprising: an object position information acquisition step of acquiring information on at least a lateral position and a depth position of a moving object; a position representation image generation step of generating, for a plurality of time points, position representation images that represent at least the lateral position and the depth position of the object along horizontal axes and depth axes based on the information acquired in the object position information acquisition step; and an image display step of displaying the position representation images for at least two time points generated by the position representation image generation step side by side in the vertical direction along a time axis extending in the vertical direction.

[0130] [Item 2] The information processing method according to Item 1, further comprising: a moving object position determination step of determining a planned position of a moving object to be moved together with the object; and a planned route display step of displaying a planned route of the moving object by connecting points corresponding to the planned positions in at least two of the position representation images displayed by the image display step with lines extending between the at least two of the position representation images. [Item 3] The information processing method according to Item 2, wherein the moving object position determination step determines, as the planned position, a position of the moving object whose relative position with respect to the object satisfies a predetermined condition. [Item 4] The information processing method according to Item 2 or 3, wherein the moving object position determination step receives input of a position of the moving object in each of the at least two position representation images displayed by the image display step, and determines the input position as the planned position. [Item 5] The information processing method according to any one of items 2 to 4, further comprising a movement range designation step of accepting designation of a movement range of the moving object on a horizontal coordinate plane defined by the horizontal axis and the depth axis, wherein the moving object position determination step generates the planned position within the movement range when the designation of the movement range is accepted in the movement range designation step. [Item 6] The information processing method according to any one of items 2 to 5, wherein the moving object position determination step determines the planned position by inputting at least information on the lateral position and information on the depth position of the object as variables to a trained model that has learned, as training data, information on the positional relationship between at least another object whose position changes over time and another moving object moved together with the other object. [Item 7] The information processing method according to any one of items 2 to 6, further comprising a change acceptance step of accepting designation of an arbitrary point on the planned route and accepting an operation to move the designated point, wherein, when the movement operation is accepted in the change acceptance step, the shape of the planned route is changed so as to pass through the point moved by the movement operation.[Item 8] The information processing method according to any one of items 1 to 7, further comprising a position priority determination step of determining a position priority indicating the importance of the position of a moving body that moves together with the object, wherein the position representation image generation step changes the display mode of the position representation image according to the position priority. [Item 9] The information processing method according to item 8, wherein the object position information acquisition step further acquires information about the orientation of the object, and the position priority determination step increases the position priority as the position of the moving body approaches a position where the moving body and the object face each other. [Item 10] The information processing method according to any one of items 1 to 9, further comprising a time priority determination step of determining a time priority indicating the importance of an action of the object during a period in which the object moves, wherein the image display step changes the display mode of the time axis according to the time priority. [Item 11] The information processing method according to item 10, wherein the object moves together with a piece of music, and the time priority determination step determines the time priority according to a structural section of the music. [Item 12] The information processing method according to any one of items 1 to 11, further comprising a pop-up display step of, in response to designation of an arbitrary point on the time axis, displaying the position representation image at a time point corresponding to the designated point in an area different from at least two of the position representation images displayed by the image display step. [Item 13] The information processing method according to any one of items 2 to 12, wherein the moving body is equipped with a camera for capturing an image of the object. [Item 14] The information processing method according to any one of items 1 to 13, wherein the position representation image is an image showing at least the position of the object in a bird's-eye view image that represents, by projection, a horizontal coordinate plane defined by the horizontal axis and the depth axis. [Item 15] The information processing method according to any one of items 1 to 13, wherein the position representation image is an image showing at least the position of the object in a bird's-eye view image that represents, by projection, a three-dimensional space defined by the horizontal axis, the depth axis, and a portion of the time axis.[Item 16] The information processing method according to any one of Items 2 to 15, further comprising an operation control step of moving the moving object based on the planned route displayed in the planned route display step. [Item 17] An information processing device comprising: an object position information acquisition unit that acquires information on at least a lateral position and a depth position of a moving object, a position representation image generation unit that generates, for a plurality of time points, position representation images that represent at least the lateral position and the depth position of the object along horizontal axes and depth axes based on the information acquired by the object position information acquisition unit, and an image display unit that displays the position representation images for at least two time points generated by the position representation image generation unit side by side in the vertical direction along a time axis extending in the vertical direction. [Item 18] A computer program for causing a computer to execute the following steps: an object position information acquisition step for acquiring information on at least the lateral position and the depth position of a moving object; a position representation image generation step for generating, for a plurality of time points, position representation images that represent at least the lateral position and the depth position of the object along horizontal and depth axes based on the information acquired by the object position information acquisition step; and an image display step for displaying the position representation images for at least two time points generated by the position representation image generation step side by side in the vertical direction along a time axis extending in the vertical direction.

[0131] S... Information processing system 10... Operator terminal 11... Information input / output unit 111... External information input unit 112... Instruction information input unit 113... Operation instruction output unit 12... Information acquisition unit 121... Map information acquisition unit 122... Object position information acquisition unit 123... Obstacle position information acquisition unit 124... Audio information acquisition unit 13... Information processing unit 131... Position representation image generation unit 132... Priority determination unit 133... Planned route determination unit 14... Information presentation unit 141... Image display unit 142... Planned route display unit 20... Robot 21... Information input / output unit 22... Information processing unit 221... Operation control unit 222... Imaging control unit 23... Sensor 24... Camera 25... Moving mechanism 30... Environmentally installed sensor 40... Artist 41... Stage 42... Microphone stand 43... First obstacle 44... Second obstacle 50... Scroll bar 51: Rail bar 52: Slider 60: Pop-up area 70: Pointer 80: Traveling rail 81: Auxiliary line X: Horizontal axis Y: Depth axis Z: Time axis Ims, Imm, Ime: Position representation image Rp: Planned route

Claims

1. An information processing method comprising: an object position information acquisition step for acquiring information on at least the lateral position and depth position of a moving object; a position representation image generation step for generating position representation images for a plurality of points in time based on the information acquired by the object position information acquisition step, the position representation images representing at least the lateral position and the depth position of the object along a horizontal axis and a depth axis; and an image display step for displaying the position representation images for at least two points in time generated by the position representation image generation step side-by-side in the vertical direction along a time axis extending in the vertical direction.

2. The information processing method of claim 1, further comprising: a moving body position determination step for determining a planned position of a moving body to be moved together with the object; and a planned route display step for displaying a planned route of the moving body by connecting points corresponding to the planned positions in at least two of the position representation images displayed by the image display step with a line extending between the at least two of the position representation images.

3. The information processing method according to claim 2, wherein said moving body position determination step determines, as said expected position, the position of said moving body whose relative position with respect to said object satisfies a predetermined condition.

4. An information processing method as described in claim 2, wherein the moving body position determination step accepts input of the position of the moving body in each of at least two position representation images displayed by the image display step, and determines the input position as the planned position.

5. An information processing method as described in claim 2, further comprising a movement range designation step of accepting designation of a movement range of the moving body on a horizontal coordinate plane defined by the horizontal axis and the depth axis, wherein the moving body position determination step generates the planned position within the movement range when the designation of the movement range is accepted in the movement range designation step.

6. The information processing method of claim 2, wherein the moving object position determination step determines the planned position by inputting at least information on the lateral position and information on the depth position of the object as variables to a trained model that has learned, as training data, information on the positional relationship between at least other objects that change position over time and other moving objects that have been moved together with the other objects.

7. An information processing method as described in claim 2, further comprising a change acceptance step of accepting the designation of an arbitrary point on the planned route and accepting an operation to move the designated point, wherein when the movement operation is accepted in the change acceptance step, the shape of the planned route is changed so as to pass through the point moved by the movement operation.

8. An information processing method as described in claim 1, further comprising a position priority determination step of determining a position priority indicating the importance of the position of a moving body to be moved together with the object, and wherein the position representation image generation step changes the display mode of the position representation image in accordance with the position priority.

9. The information processing method according to claim 8, wherein the object position information acquisition step further acquires information on the orientation of the object, and the position priority determination step sets the position priority to be higher the closer the position of the moving body is to a position where the moving body and the object face each other.

10. An information processing method as described in claim 1, further comprising a time priority determination step of determining a time priority indicating the importance of the action of the object during the period in which the object moves, and wherein the image display step changes the display mode of the time axis in accordance with the time priority.

11. The information processing method according to claim 10, wherein the object moves together with the music piece, and the time priority determination step determines the time priority according to a constituent section of the music piece.

12. An information processing method as described in claim 1, further comprising a pop-up display step for displaying, in response to the designation of an arbitrary point on the time axis, the position representation image at the time corresponding to the designated point in an area different from the at least two position representation images displayed by the image display step.

13. The information processing method according to claim 2, wherein the moving body is equipped with a camera for capturing an image of the object.

14. The information processing method according to claim 1, wherein the position representation image is an image showing at least the position of the object on a bird's-eye view image that represents, by projection, a horizontal coordinate plane defined by the horizontal axis and the depth axis.

15. An information processing method as described in claim 1, wherein the position representation image is an image showing at least the position of the object in a bird's-eye view image that represents, by projection, a three-dimensional space defined by the horizontal axis, the depth axis and a portion of the time axis.

16. The information processing method according to claim 2, further comprising an operation control step of moving the mobile object based on the planned route displayed in the planned route display step.

17. An information processing device comprising: an object position information acquisition unit that acquires information on at least the lateral position and depth position of a moving object; a position representation image generation unit that generates position representation images for multiple points in time based on the information acquired by the object position information acquisition unit, the position representation images representing at least the lateral position and the depth position of the object along a horizontal axis and a depth axis; and an image display unit that displays the position representation images for at least two points in time generated by the position representation image generation unit in a vertical arrangement along a time axis extending in the vertical direction.

18. A computer program for causing a computer to execute the following steps: an object position information acquisition step for acquiring information on at least the lateral position and depth position of a moving object; a position representation image generation step for generating position representation images for a plurality of points in time based on the information acquired by the object position information acquisition step, the position representation images representing at least the lateral position and the depth position of the object along a horizontal axis and a depth axis; and an image display step for displaying the position representation images for at least two points in time generated by the position representation image generation step side-by-side in the vertical direction along a time axis extending in the vertical direction.

Images