Information processing device, information processing method, and program

Abstract

This system detects actions that may violate the given rules of a competition. [Solution] The information processing device 100 according to this disclosure acquires three-dimensional shape data corresponding to each of a plurality of objects, which is generated based on captured images obtained by imaging by a plurality of imaging devices, acquires trajectory information indicating the trajectory of each of the plurality of objects based on the three-dimensional shape data, and detects violations of a given rule based on the trajectory information.

Description

Technical Field

[0004] , , , , , , ,

[0001] The present disclosure relates to a technique for detecting rule-violating acts.

Background Art

[0002] There is a technique for generating a virtual viewpoint image corresponding to an arbitrary virtual viewpoint using a plurality of captured images obtained by synchronized imaging from a plurality of different positions. Patent Document 1 discloses a technique for determining the position of a virtual viewpoint and the direction of the line of sight at the virtual viewpoint (the position and orientation of a virtual camera) based on a scene, a composition scenario, and an object selected by a user. Specifically, in the technique disclosed in Patent Document 1, information regarding a plurality of scenes is given in advance as event information, and the user selects information regarding a desired scene, and further selects a composition scenario and an object to be focused on. Thereby, in the technique disclosed in Patent Document 1, a continuous transition (virtual camera path) of the position and orientation of a virtual camera according to the composition scenario selected by the user in the scene selected by the user is automatically determined, and a virtual viewpoint image is generated based on the determined virtual camera path. In the disclosure of Patent Document 1, a scene is, for example, a goal scene, a shoot scene, a pass scene, a foul scene, etc. in a sport such as a ball game.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, Patent Document 1 provides pre-prepared event information regarding foul scenes, etc., and does not disclose any methods for detecting foul scenes, etc. This disclosure aims to provide a technology that can detect actions suspected of violating given rules in a competition, for example, without requiring any effort from users such as referees. [Means for solving the problem]

[0005] The information processing device according to this disclosure includes: shape acquisition means for acquiring three-dimensional shape data corresponding to each of a plurality of objects, which is generated based on captured images obtained by imaging by a plurality of imaging devices; trajectory acquisition means for acquiring trajectory information indicating the trajectory of each of the plurality of objects based on the three-dimensional shape data; and detection means for detecting a violation of a given rule based on the trajectory information. [Effects of the Invention]

[0006] According to this disclosure, it is possible to detect conduct that is suspected of violating given rules in a competition. [Brief explanation of the drawing]

[0007] [Figure 1] This is a block diagram showing an example of the configuration of an information processing system according to Embodiment 1. [Figure 2] This figure shows an example of the arrangement of the imaging device and the object to be imaged according to Embodiment 1, as well as an example of a coordinate system relating to the virtual camera and the object to be imaged. [Figure 3] This is a block diagram showing an example of the hardware configuration of an information processing device according to Embodiment 1. [Figure 4] This is a block diagram showing an example of the functional configuration of the information processing device according to Embodiment 1. [Figure 5] This flowchart shows an example of the processing flow of the information processing device according to Embodiment 1. [Figure 6] This is a diagram illustrating an example of the trajectory of an object according to Embodiment 1. [Figure 7] This diagram illustrates an example of the trajectory of an object suspected of committing a violation according to Embodiment 1. [Figure 8] This figure shows an example of the transition of the virtual viewpoint position when the type of violation candidate according to Embodiment 1 is "oblique movement". [Figure 9] This figure shows an example of the transition of the virtual viewpoint position when the type of violation candidate according to Embodiment 1 is "pressure". [Figure 10] This figure shows an example of the transition of the virtual viewpoint position when the type of violation candidate according to Embodiment 1 is "inside overtaking". [Figure 11] This figure shows an example of a virtual viewpoint image corresponding to each type of potential violation in Embodiment 1. [Figure 12] This figure shows an example of a display image according to Embodiment 1. [Figure 13] This is a block diagram showing an example of the functional configuration of the information processing device according to Embodiment 2. [Figure 14] This flowchart shows an example of the processing flow of the information processing device according to Embodiment 2. [Figure 15] This is a diagram illustrating the object contact detection process according to Embodiment 2. [Figure 16] This figure shows an example of a display image according to Embodiment 2. [Modes for carrying out the invention]

[0008] Embodiments of the technology of this disclosure will be described in detail below with reference to the attached drawings. Note that the following embodiments are not limiting to the technology of this disclosure. Furthermore, not all combinations of features described in the following embodiments are essential for solving the problems related to the technology of this disclosure. In the following descriptions of embodiments, identical components will be given the same reference numerals to avoid redundant explanations.

[0009] [Embodiment 1] <Configuration of the Information Processing System> Figure 1 is a block diagram showing an example of the configuration of an information processing system 1 according to Embodiment 1. The information processing system 1 has a plurality of imaging devices 11 and an information processing device 100, and the information processing device 100 is connected to each of the plurality of imaging devices 11 so as to be able to communicate with each other. Figure 2 is a diagram showing an example of the arrangement of imaging devices 11 and the target to be imaged, as well as an example of a coordinate system relating to a virtual camera 21 and the target to be imaged, according to Embodiment 1. Specifically, Figure 2(a) shows an example of the arrangement of a plurality of imaging devices 11 installed on a bank 20, which is a track where bicycle track races are held, and is an example of a target to be imaged according to Embodiment 1. The bank 20 is a circular track consisting of a track that includes straight sections and curved sections. The bank 20 is configured such that the track is inclined inward so that bicycles can travel at high speeds, with the inclination angle being maximum in the curved sections and gradually becoming gentler towards the straight sections.

[0010] As shown in Figure 2(a), the multiple imaging devices 11 are installed at different positions from each other so as to surround the bank 20 from the outside. Specifically, each of the multiple imaging devices 11 is positioned to image part or all of the bank 20, and the field of view of any one imaging device 11 overlaps with the field of view of another imaging device 11, and they are connected to each other so as to be able to communicate with each other, for example, via a transmission cable. In addition, each imaging device 11 is installed facing a predetermined point of fixation. Each point of fixation is imaged from different directions by two or more imaging devices 11. The images obtained by imaging by each imaging device 11 are transmitted to the information processing device 100. The images obtained by imaging by each imaging device 11 may be still images, moving images, or both still images and moving images. In the following description, unless otherwise specified, the term "image" includes the meanings of both still images and moving images.

[0011] Fig. 2(b) shows an example of the definition of the coordinate system (camera coordinate system) for the virtual camera 21, and Fig. 2(c) shows an example of the definition of the coordinate system of the imaging target. Hereinafter, a world coordinate system with the center of the bank 20 as the origin, the long side direction of the bank 20 as the X-axis, the short side direction of the bank 20 as the Y-axis, and the direction orthogonal to the X-axis and the Y-axis as the Z-axis is used as the coordinate system of the imaging target, and the X-axis, Y-axis, and Z-axis of the world coordinate system are denoted as Xw, Yw, and Zw in order for explanation. In Fig. 2(c), it is assumed that the direction of the arrow of each axis of Xw, Yw, and Zw represents the positive direction of each axis. Further, hereinafter, a camera coordinate system with the optical center of the virtual camera 21 as the origin, the optical axis direction as the Z-axis, the horizontal direction (left-right direction) of the virtual camera 21 as the X-axis, and the vertical direction (up-down direction) of the virtual camera 21 as the Y-axis is described as the coordinate system for the virtual camera 21. Furthermore, the X-axis, Y-axis, and Z-axis of the camera coordinate system are denoted as Xc, Yc, and Zc in order for explanation. In Fig. 2(b), it is assumed that the direction of the arrow of each axis of Xc, Yc, and Zc represents the positive direction of each axis.

[0012] The direction of the virtual camera 21 is represented by rotation about the Xc axis (pan P), rotation about the Yc axis (tilt T), and rotation about the Zc axis (roll R). That is, the pan P is a parameter related to the turning of the virtual camera 21 in the left-right direction of the optical axis, and is the rotation angle of the virtual camera 21 with the Xc axis as the rotation axis. The tilt T is a parameter related to the turning of the virtual camera 21 in the up-down direction of the optical axis, and is the rotation angle of the virtual camera 21 with the Yc axis as the rotation axis. The roll R is a parameter indicating the rotation angle with the optical axis (Zc axis) of the virtual camera 21 as the rotation axis. Note that the definitions of the above-described world coordinate system and camera coordinate system are examples, and they may be defined by other methods. Hereinafter, two white lines not shown in Fig. 2 are drawn along the course on the bank 20, and the inner white line is referred to as the inner circle line, and the white line 70 centimeters outside the inner circle line is referred to as the outer band line for explanation.

[0013] The information processing apparatus 100 is configured by a personal computer, a server apparatus, or the like, and generates a virtual viewpoint image based on a plurality of captured images obtained by capturing an object from a plurality of directions by a plurality of imaging apparatuses 11. The virtual viewpoint image, also called a free viewpoint video, is an image corresponding to the view from a virtual viewpoint (virtual viewpoint) freely or arbitrarily specified by a user in a three-dimensional space corresponding to the imaging target. The information processing apparatus 100 acquires frames as captured images captured by the plurality of imaging apparatuses 11 installed in the bank 20 at a predetermined frame rate. Hereinafter, as an example, the frame rate will be described as 60 frames / second, but it is not limited thereto. Note that the information processing apparatus 100 acquires a plurality of captured images from the plurality of imaging apparatuses 11 by a wired or wireless communication module or an image transmission module such as SDI.

[0014] The information processing device 100 uses multiple captured images acquired from multiple imaging devices 11 to identify objects traveling on the bank 20 and detect objects suspected of committing violations. The following explanation is based on the rules of Keirin as an example of the rules of two-wheeled racing. Violations in Keirin include "diagonal riding," "pushing," and "inside overtaking." "Diagonal riding" is the act of changing course from a straight line to ride diagonally and obstructing the path of other riders. "Pushing" is the act of applying impact or pressure to another rider while riding, pushing them towards the inside of the track. "Inside overtaking" is the act of overtaking a rider who is riding ahead inside the outer lane line from the inside of the track. The information processing device 100 generates information indicating the position and orientation of the virtual camera 21 (hereinafter referred to as "virtual viewpoint information") according to the type of act detected as a suspected violation (hereinafter referred to as "candidate violation"). The virtual viewpoint information includes a data set in which the position and orientation of the virtual camera 21 are represented in chronological order. The position of the virtual camera 21, i.e., the position of the virtual viewpoint, is expressed using the world coordinate system, and the orientation of the virtual camera 21, i.e., the direction of the line of sight in the virtual viewpoint, is expressed using the camera coordinate system. In this embodiment, "oblique driving," "pressing," and "inside overtaking" are described as examples of violations, but the violations are not limited to these and may include other violations.

[0015] The information processing device 100 generates an image (virtual viewpoint image) corresponding to the view from a virtual viewpoint identified by the virtual viewpoint information, using virtual viewpoint information generated based on the type of candidate violation and multiple captured images acquired from multiple imaging devices 11. The virtual viewpoint image is generated, for example, using MBR (Model-Based Rendering). MBR is a method for generating a virtual viewpoint image using three-dimensional shape data (hereinafter also referred to as "3D model") that shows the three-dimensional shape of an object, which is generated based on multiple captured images obtained by imaging the object from multiple directions. MBR generates a virtual viewpoint image of the view of the scene from a virtual viewpoint, for example, by using a 3D model corresponding to an object obtained by a three-dimensional shape reconstruction method such as the viewing volume cross-eyed method. Alternatively, the information processing device 100 may generate information that can generate a virtual viewpoint image, such as information on the 3D model of the object and a texture image (hereinafter referred to as "texture") for mapping to the 3D model.

[0016] <Hardware configuration of the information processing device> The hardware configuration of the information processing device 100 will be described with reference to Figure 3. Figure 3 is a block diagram showing an example of the hardware configuration of the information processing device 100 according to Embodiment 1. The information processing device 100 has a controller unit 300, an operation unit 309, and a display device 310 as its hardware configuration. The controller unit 300 includes a CPU 301, ROM 302, RAM 303, HDD 304, operation I / F (interface) 305, display I / F 306, and communication I / F 307. The various parts included in the controller unit 300 are connected to each other via a system bus 308 so that they can communicate with one another.

[0017] The CPU 301 controls the operation of the ROM 302, RAM 303, HDD 304, operation I / F 305, display I / F 306, and communication I / F 307 via the system bus 308. The CPU 301 starts the OS (operating system) using the boot program stored in the ROM 302. On the started OS, the CPU 301 executes application programs stored in the HDD 304, etc. By executing application programs, the CPU 301 realizes various functions of the information processing device 100. The RAM 303 is used as the main memory and temporary storage area such as the work area of ​​the CPU 301. The HDD 304 stores application programs, etc., as described above. The CPU 301 may be composed of one processor or multiple processors.

[0018] The operation I / F 305 is a communication interface with the operation unit 309. The operation I / F 305 sends user input information from the operation unit 309 to the CPU 301. The operation unit 309 includes an input device capable of accepting user input, such as a mouse, keyboard, or touch panel. The display I / F 306 is a communication interface with the display device 310. The display I / F 306 outputs image data to be displayed on the display device 310, for example. The display device 310 may be composed of a liquid crystal display or the like, or it may be composed of a touch panel including an input device. If the display device 310 is composed of a touch panel or the like, the display device 310 can also perform the functions of the operation unit 309.

[0019] The communication interface 307 is a communication interface for communication such as Ethernet (registered trademark). The communication interface 307 includes connectors for connecting transmission cables. The communication interface 307 transmits and receives information with external devices such as the imaging device 11 via the transmission cable. The communication interface 307 may also be a wireless communication circuit such as a baseband circuit or RF circuit, or an antenna. The controller unit 300 can also control the display of an image on an external display device 310 connected via a transmission cable or communication network. In this case, the controller unit 300 performs display control by outputting display data to the external display device 310.

[0020] Note that the hardware configuration shown in Figure 3 is just an example, and some parts may be omitted, configurations not shown may be added, or the configurations shown in Figure 3 may be combined. For example, the information processing device 100 does not have to have a display device 310. For example, in the above description, the controller unit 300 was described as having a CPU 301, but instead of the CPU 301, or in addition to the CPU 301, it may have a processing circuit such as an ASIC, DSP, or FPGA. In this case, the processing circuit such as an ASIC or DSP may perform some or all of the processing that the CPU 301 would perform.

[0021] <Functional Configuration of Information Processing Devices> Figure 4 is a block diagram showing an example of the functional configuration of the information processing device 100 according to Embodiment 1. The information processing device 100 has the following functional configurations: an image acquisition unit 400, a model acquisition unit 401, a trajectory acquisition unit 402, a detection unit 403, a viewpoint generation unit 404, an image generation unit 405, and a display control unit 406. Each unit of the information processing device 100's functional configuration is realized, for example, by the CPU 301 executing an application program stored in the HDD 304 or the like using the RAM 303 as work memory. The image acquisition unit 400 acquires captured images (frames) output from multiple imaging devices 11. The captured image data acquired by the image acquisition unit 400 is stored in the RAM 303 or the HDD 304 or the like and used for processing by the model acquisition unit 401, etc.

[0022] The model acquisition unit 401 acquires 3D model information for each object. For example, the model acquisition unit 401 acquires this information by generating 3D model information for each object using multiple captured images obtained through synchronized imaging and the camera parameters of each imaging device 11. Here, camera parameters include information such as the position, orientation, and focal length of the imaging device 11, as well as information such as the distortion coefficient of the optical system, such as the lens, of the imaging device 11. Hereinafter, the model acquisition unit 401 will be described as acquiring 3D model information in accordance with the frame rate of the multiple imaging devices 11, that is, each time multiple captured images are acquired from the multiple imaging devices 11. However, the timing of the acquisition of 3D model information by the model acquisition unit 401 is not limited to this.

[0023] The 3D model information includes information about the three-dimensional shape of the object represented by voxels (hereinafter referred to as "object shape"). The 3D model information also includes information about the texture mapped to the object shape, and information about the position of the object shape represented in a world coordinate system (hereinafter referred to as "object position"). The 3D model information acquired by the model acquisition unit 401 is stored in RAM 303 or HDD 304, etc., and used for processing by the trajectory acquisition unit 402, etc.

[0024] For example, the model acquisition unit 401 generates a silhouette image corresponding to each object from multiple captured images, and uses the generated silhouette image and the camera parameters of each imaging device 11 to generate object shape information for each object. Specifically, for example, the model acquisition unit 401 uses the silhouette image and the camera parameters of each imaging device 11 to estimate the three-dimensional shape of each object using the viewing volume cross-eyed method to acquire object shape information for each object. The model acquisition unit 401 also acquires information by generating texture information to map to each object shape using the acquired object shape information, multiple captured images, and the camera parameters of each imaging device 11. Furthermore, the model acquisition unit 401 acquires object position information by identifying the position of each object shape in the world coordinate system based on the acquired object shape information.

[0025] In this embodiment, the camera parameters of each imaging device 11 are output from each imaging device 11 along with the data of the captured image captured by each imaging device 11 and are acquired by the image acquisition unit 400. However, if the camera parameters of each imaging device 11 do not change over time and are stored in advance in the HDD 304 or the like, the model acquisition unit 401 may acquire the camera parameters by reading them from the HDD 304 or the like.

[0026] Furthermore, in this embodiment, the information processing device 100 is described as acquiring captured images (frames) output from a plurality of imaging devices 11 and generating silhouette images corresponding to each object from the acquired captured images, but it is not limited to this. For example, each imaging device 11 may generate silhouette images corresponding to each object using the captured images, and the information processing device 100 may acquire the silhouette images output from the imaging devices 11 and generate information for the 3D model of each object. Also, in this embodiment, the information processing device 100 is described as acquiring information for the 3D model of each object by generating it, but it is not limited to this. For example, the information processing device 100 may acquire information for the 3D model of each object generated and output by an external device (not shown in Figure 1) using a plurality of captured images (frames).

[0027] The trajectory acquisition unit 402 acquires information indicating the trajectory of each object (hereinafter referred to as "trajectory information"). Specifically, the trajectory acquisition unit 402 acquires this information by generating trajectory information for each object using the object position information included in the 3D model information acquired by the model acquisition unit 401. For example, the trajectory acquisition unit 402 stores the information of each object position over a predetermined period and acquires the trajectory of the changes in the object position of each object during that period as trajectory information. The trajectory information is updated each time the model acquisition unit 401 acquires new 3D model information, that is, in accordance with the frame rate of the multiple imaging devices 11. The trajectory information acquired by the trajectory acquisition unit 402 is stored in the RAM 303 or HDD 304, etc., and used for processing by the detection unit 403, etc. Note that the trajectory information may be expressed in a two-dimensional coordinate system using Xw coordinates and Yw coordinates from the object position information expressed in the world coordinate system.

[0028] The detection unit 403 uses the trajectory information of each object acquired by the trajectory acquisition unit 402 to detect an object suspected of having committed a violation. In this embodiment, as an example, the object is a bicycle traveling on bank 20, and the detection unit 403 will be described as detecting at least one of the following violation candidates based on the trajectory information of each object: "diagonal travel," "pressure," and "inside overtaking." For example, the detection unit 403 detects that object A, which was traveling in a straight line, suddenly changed course, and determines that a violation related to "diagonal travel" may have occurred if there is another object B traveling behind object A after the course change. When the detection unit 403 detects a violation candidate, it associates the type of violation candidate with the trajectory information of the perpetrator (the object that obstructed the travel) and the victim (the object whose travel was obstructed), and stores them in RAM 303 or HDD 304, etc. In this case, the detection unit 403 stores identification information that can uniquely identify the perpetrator and the victim in the RAM 303 or HDD 304, etc., associating it with the perpetrator's trajectory information and the victim's trajectory information.

[0029] The viewpoint generation unit 404 generates virtual viewpoint information based on the type of violation candidate and trajectory information stored in the RAM 303 or HDD 304, etc., by the detection unit 403. For example, if the type of violation candidate is "oblique," the viewpoint generation unit 404 determines the position and orientation of the virtual camera, i.e., the position of the virtual viewpoint and the direction of the line of sight at the virtual viewpoint, as follows. Specifically, in this case, the viewpoint generation unit 404 determines the position of the virtual viewpoint and the direction of the line of sight at the virtual viewpoint so that the position and direction of the line of sight at the virtual viewpoint are such that the three-dimensional shapes of objects A and B can be viewed from directly above. In this case, the direction of the line of sight at the virtual viewpoint is determined to be in the negative direction of the Zx axis, and the height of the virtual viewpoint is determined so that both the perpetrator and the victim are within the field of view from the virtual viewpoint, i.e., within the angle of view of the virtual camera 21. Furthermore, the position of the virtual viewpoint is determined to move over time to follow the object positions of the perpetrator and the victim based on the trajectory information. In this way, the viewpoint generation unit 404 generates virtual viewpoint information that defines the position of the virtual viewpoint and the direction of the line of sight at the virtual viewpoint, i.e., the position and orientation of the virtual camera 21 in a time series. The generated virtual viewpoint information is stored in the RAM 303 or HDD 304, etc., and used for processing by the image generation unit 405.

[0030] The image generation unit 405 generates a virtual viewpoint image based on the 3D model information of each object acquired by the model acquisition unit 401 and the virtual viewpoint information generated by the viewpoint generation unit 404. The generated virtual viewpoint image is stored in RAM 303 or HDD 304, etc., and used for processing by the display control unit 406, etc. The display control unit 406 places the virtual viewpoint image generated by the image generation unit 405 at a predetermined position in the display image, transmits the placed display image to the display device 310 via the display I / F 306, and displays the display image on the display device 310.

[0031] For example, when a virtual viewpoint image is generated by the image generation unit 405, the display control unit 406 switches the display so that the display image containing the virtual viewpoint image is displayed on the display device 310. Examples of display image layouts include single-screen display, multi-split display, and PIP (Picture-in-Picture) display. For example, when a virtual viewpoint image is generated while the display control unit 406 is controlling the display of a display image containing an captured image, it changes the display image to a layout such as a two-screen display or PIP display containing both the captured image and the virtual viewpoint image. Furthermore, when a candidate for a violation is detected by the detection unit 403, the display control unit 406 may place a warning image on the display image indicating that a candidate for a violation has been detected to notify the user that a violation may have occurred. In this case, for example, the display control unit 406 controls the display so that the display image containing the virtual viewpoint image is displayed on the display device 310 based on user operation.

[0032] <Operation of the Information Processing Device> Figure 5 is a flowchart showing an example of the processing flow of the information processing device 100 according to Embodiment 1. The processing shown in the flowchart in Figure 5 is realized by the CPU 301 loading an application program stored in the HDD 304 or the like into the RAM 303 and executing it. The flowchart shown in Figure 5 is started when the user initiates the image capture process. In the following description, the symbol "S" represents a step. First, at S500, the image acquisition unit 400 acquires multiple frames obtained by synchronized imaging by multiple imaging devices 11.

[0033] Next, in S501, the model acquisition unit 401 acquires 3D model information for each object based on the multiple frames acquired in S500. For example, if the frame rate of the multiple imaging devices 11 is 60 frames / second, the model acquisition unit 401 acquires 3D model information for each object approximately every 16.67 ms (milliseconds) in accordance with that frame rate. Hereafter, the 3D model information acquired in S501 will be described as information for the 3D model corresponding to each of the objects traveling in bank 20. Furthermore, hereafter, the 3D model information for each object will be described as being managed, for example, by identification information assigned to each object. Note that the timing of the acquisition of 3D model information by the model acquisition unit 401 is not limited to approximately every 16.67 ms, nor is it limited to timing in accordance with the frame rate of the multiple imaging devices 11. The model acquisition unit 401 may acquire 3D model information for each object at any time interval.

[0034] Next, in S502, the trajectory acquisition unit 402 acquires trajectory information for each object based on the object position information included in the 3D model information of each object acquired in S501.

[0035] Figure 6 is a diagram illustrating an example of the trajectory of an object according to Embodiment 1. Figure 6 is represented using the Xw and Yw coordinates of the world coordinate system. In Figure 6, positions 601 to 607, indicated by black circles (●), represent the position of the three-dimensional shape corresponding to the object (object position) in each frame. Position 601 indicates the object position at time T1, and position 602 indicates the object position at time T2. Similarly, positions 603 to 607 indicate the object positions at times T3 to T7, respectively. The difference between time Tn (where n is an integer greater than or equal to 1) and time Tn+1 is, for example, 16.67 ms. That is, Figure 6 shows how the position of the three-dimensional shape corresponding to the object (object position) changes as time progresses.

[0036] The trajectory 60 is a line that sequentially connects positions 602 to 606. For example, the trajectory acquisition unit 402 stores object position information corresponding to each object for 5 frames using a FIFO (First-In, First-Out) method, and the object's trajectory 60 is created by connecting these 5 frames with a line. The object position information held by the trajectory acquisition unit 402 is updated each time a new 3D model is acquired in S601, as the processing in S500 to S508 in the flowchart shown in Figure 6 is repeated. Specifically, the object position information included in the 3D model information acquired in S501 is newly stored, and the oldest object position information is deleted. The trajectory 60 is generated for each object. In this embodiment, the trajectory acquisition unit 402 is described as holding object position information for 5 frames, but the object positions held by the trajectory acquisition unit 402 are not limited to this and can be 2 frames or more.

[0037] Following S502, in S503, the detection unit 403 detects an object suspected of having committed a violation based on the trajectory 60 acquired in S502.

[0038] Figure 7 is a diagram illustrating an example of the trajectory of an object suspected of committing a violation according to Embodiment 1. Similar to Figure 6, Figure 7 is represented using the Xw and Yw coordinates of the world coordinate system. In Figure 7, object positions are represented by black circles or white circles (〇). Hereinafter, black circles represent the object position of the perpetrator (the object that obstructed the vehicle's movement), and white circles represent the object position of the victim (the object whose movement was obstructed). Figures 7(a) to (c) each show the trajectories of the perpetrator and the victim, which are close to each other. Furthermore, in Figures 7(a) to (c), the last digit of the code attached to each object position being the same indicates the object position at the same time. For example, position 701 and position 711 represent the object position corresponding to the perpetrator and the object position corresponding to the victim at the same time.

[0039] Figure 7(a) shows an example of a trajectory in which a candidate violation is classified as "diagonal movement". In Figure 7(a), the perpetrator's trajectory 70 shows that the perpetrator moved while accelerating in the positive direction of the Yw axis from position 701 to position 704, and then changed course diagonally forward to the right from position 704 to position 705. Also in Figure 7(a), the victim's trajectory 71 shows that the victim moved in the positive direction of the Yw axis from position 711 to position 715. Furthermore, in Figure 7(a), trajectories 70 and 71 show that at a certain time, the perpetrator changed course and moved in front of the direction of travel of the victim, who was at position 715.

[0040] The detection unit 403 analyzes the trajectories 70 and 71 shown in Figure 7(a) to detect sudden changes in the direction of travel by the perpetrator and obstruction of the victim's path. For example, in the case of trajectories 70 and 71 as shown in Figure 7(a), the detection unit 403 determines that the perpetrator may have obstructed the victim's path by "diagonal movement" and identifies the perpetrator as an object suspected of committing a violation classified as "diagonal movement".

[0041] Figure 7(b) shows an example of a trajectory where the proposed violation is classified as "pressure." In Figure 7(b), the perpetrator's trajectory 73 shows that the perpetrator moved in the positive direction of the Yw axis from position 731 to position 732, and then moved to position 735, gradually curving sharply to the left from position 732. Also in Figure 7(b), the victim's trajectory 72 shows that the victim moved in the positive direction of the Yw axis from position 721 to position 723, and then moved to position 725, gradually curving to the left from position 723. Furthermore, in Figure 7(b), trajectories 72 and 73 show that the perpetrator is moving in a way that overlaps the victim's path.

[0042] The detection unit 403 analyzes the trajectories 72 and 73 shown in Figure 7(b) and determines that both the perpetrator and the victim are moving in a curved manner to the left, and that the perpetrator is moving in a way that causes them to lean over the victim, based on their relative positions. For example, in the case of trajectories 72 and 73 as shown in Figure 7(b), the detection unit 403 determines, based on this determination, that the perpetrator may have obstructed the victim's movement by "pressing". The detection unit 403 also identifies the perpetrator as an object suspected of having committed a violation classified as "pressing".

[0043] Figure 7(c) shows an example of a trajectory where the violation is classified as "inside overtaking". The left side of the outer lane line 76 shown in Figure 7(c) is the inside of the course. In Figure 7(b), the offender's trajectory 74 shows that the offender moved while accelerating in the positive Yw direction from position 741 to position 745, and then moved to position 745. Also, in Figure 7(c), the victim's trajectory 75 shows that the victim moved at approximately a constant speed in the positive Yw direction from position 751 to position 755, and then moved to position 755. Furthermore, in Figure 7(c), the victim was driving inside the outer lane line 76, and the offender was driving inside the victim, the offender's speed was faster than the victim's speed, and the offender overtook the victim between position 744 and position 745.

[0044] The detection unit 403 analyzes the trajectories 74 and 75 shown in Figure 7(c) to determine that the perpetrator overtook the victim, who was traveling inside the outer lane 76, from the inside of the victim. For example, in the case of trajectories 74 and 75 as shown in Figure 7(c), the detection unit 403 determines, based on this determination, that the perpetrator may have obstructed the victim's driving by "inside overtaking". The detection unit then identifies the perpetrator as an object suspected of having committed a violation classified as "inside overtaking".

[0045] Following S503, in S504, the detection unit 403 determines whether or not it has detected an object suspected of having committed a violation based on the results of S503. If it is determined in S504 that an object suspected of having committed a violation has been detected, the information processing device 100 executes the process in S505. On the other hand, if it is determined in S504 that no object suspected of having committed a violation has been detected, the information processing device 100 executes the process in S508. In S505, the viewpoint generation unit 404 generates virtual viewpoint information based on the type of violation candidate identified in S503 and the trajectory information used in the detection of the violation candidate.

[0046] Figure 8 shows an example of the transition of the virtual viewpoint position indicated by the virtual viewpoint information generated by the viewpoint generation unit 404 according to Embodiment 1 when the type of violation candidate is "oblique". Figure 8 is represented using the Xw and Yw coordinates of the world coordinate system, similar to Figure 7(a). In Figure 8, positions 801 to 805 indicate the position of the virtual viewpoint at each time point.

[0047] If the type of violation candidate is "oblique movement," the direction of the line of sight in the virtual viewpoint, i.e., the orientation of the virtual camera 21, is determined to be, for example, in the negative direction of the Zw axis, i.e., downward in the vertical direction. The vertical position of the virtual viewpoint, i.e., the Zw coordinate, is determined so that, for example, both the perpetrator and the victim are within the field of view from the virtual viewpoint, i.e., the angle of view of the virtual camera. The position of the virtual viewpoint on the horizontal plane, i.e., the Xw and Yw coordinates, is determined so that, for example, it is midway between the perpetrator's trajectory 70 and the victim's trajectory 71. Furthermore, the Xw and Yw coordinates of the virtual viewpoint at each time point are determined so that, for example, they follow the movement of the perpetrator. The position of the virtual viewpoint and the direction of the line of sight 810 at each time point, determined in this way, are stored as virtual viewpoint information in RAM 303, etc.

[0048] Figure 9 shows an example of the transition of the virtual viewpoint position indicated by the virtual viewpoint information generated by the viewpoint generation unit 404 according to Embodiment 1 when the type of violation candidate is "pressure". Figure 9 is represented using the Xw and Yw coordinates of the world coordinate system, similar to Figure 7(b). In Figure 9, positions 901 to 905 indicate the position of the virtual viewpoint at each time point.

[0049] If the type of violation candidate is "pressure," the vertical position of the virtual viewpoint, i.e., the Zw coordinate, is determined to be, for example, at the eye level of the perpetrator or victim. The position of the virtual viewpoint on the horizontal plane, i.e., the Xw and Yw coordinates, is determined to be, for example, in front of the perpetrator and victim, and so that both the perpetrator and victim are within the field of view from the virtual viewpoint, i.e., within the angle of view of the virtual camera 21. The direction of the line of sight in the virtual viewpoint, i.e., the orientation of the virtual camera 21, is determined to be, for example, pointing in the direction of a position midway between the perpetrator and victim from the determined Xw and Yw coordinate positions. Furthermore, the Xw and Yw coordinates of the virtual viewpoint at each time point are determined to follow the movement of the perpetrator. The position of the virtual viewpoint and the direction of the line of sight 910 at each time point, as determined in this way, are stored as virtual viewpoint information in RAM 303, etc.

[0050] Figure 10 shows an example of the transition of the virtual viewpoint position indicated by the virtual viewpoint information generated by the viewpoint generation unit 404 according to Embodiment 1 when the type of violation candidate is "inside overtaking". Figure 10 is represented using the Xw and Yw coordinates of the world coordinate system, similar to Figure 7(c). In Figure 10, positions 1001 to 1005 indicate the position of the virtual viewpoint at each time point.

[0051] If the type of violation candidate is "inside overtaking," the position of the virtual viewpoint on the horizontal plane, i.e., the Xw and Yw coordinates, is determined to be, for example, the following position. Specifically, the position is determined to be on the inner side of the track relative to the position where the offender is driving, and the entire front wheel of the bicycle the offender is riding is within the field of view of the virtual camera 21. The vertical position of the virtual viewpoint, i.e., the Zw coordinate, is determined to be, for example, equal to the height of the center of the wheel. The direction of the line of sight in the virtual viewpoint, i.e., the orientation of the virtual camera 21, is determined to be, for example, pointed in the direction of the offender's position from the determined Xw and Yw coordinates. Furthermore, the Xw and Yw coordinates of the virtual viewpoint at each time point are determined to follow the movement of the offender, for example. The position of the virtual viewpoint and the direction of the line of sight 1010 at each time point, as determined in this way, are stored as virtual viewpoint information in the RAM 303, etc.

[0052] The method for determining the position of the virtual viewpoint and the direction of the line of sight in the virtual viewpoint, according to the type of potential violation, as described above, is merely an example, and the method of determination is not limited to this. Furthermore, multiple virtual viewpoint pieces of information may be generated for a single type of potential violation. In that case, the multiple virtual viewpoint pieces of information may be generated such that, for example, they have different virtual viewpoint positions or directions of sight.

[0053] Following S505, in S506, the image generation unit 405 generates a virtual viewpoint image based on the 3D model information acquired in S501 and the virtual viewpoint information generated in S505. Figure 11 is a diagram showing an example of a virtual viewpoint image corresponding to each type of violation candidate generated by the image generation unit 405 according to Embodiment 1. The virtual viewpoint image shown in Figure 11 is generated based on virtual viewpoint information corresponding to an arbitrary time. The virtual viewpoint image shown in Figure 11 can be used, for example, as an image to assist in a judge's determination of whether or not a violation has occurred.

[0054] Figure 11(a) shows an example of a virtual viewpoint image generated by the image generation unit 405 when the type of violation candidate is "oblique driving". Specifically, Figure 11(a) shows an example of a virtual viewpoint image generated based on the position of the virtual viewpoint at each time point shown in Figure 8, and the direction of the line of sight 810 in the virtual viewpoint. The virtual viewpoint image shown in Figure 11(a) is an overhead view from above the bank 20, and corresponds to the view from a virtual viewpoint that includes both the perpetrator (object on the left) and the victim (object on the right) in its field of view. When the type of violation candidate is "oblique driving", using the virtual viewpoint image shown in Figure 11(a) makes it easier for judges to determine whether there was a sudden change of course and whether the perpetrator obstructed the victim's driving.

[0055] Figure 11(b) shows an example of a virtual viewpoint image generated by the image generation unit 405 when the type of violation candidate is "pressure". Specifically, Figure 11(b) shows an example of a virtual viewpoint image generated based on the position of the virtual viewpoint at each time point shown in Figure 9, and the direction of the line of sight 910 at the virtual viewpoint. The virtual viewpoint image shown in Figure 11(b) is an image that corresponds to what is seen from a virtual viewpoint positioned in front of both the perpetrator (object on the left) and the victim (object on the right). When the type of violation candidate is "pressure", using the virtual viewpoint image shown in Figure 11(b) makes it easier for judges, etc., to determine whether the perpetrator applied impact or pressure to the victim and pushed them inward into the truck.

[0056] Figure 11(c) shows an example of a virtual viewpoint image generated by the image generation unit 405 when the type of violation candidate is "inside overtaking". Specifically, Figure 11(c) shows an example of a virtual viewpoint image generated based on the position of the virtual viewpoint at each time shown in Figure 10, and the direction of the line of sight 1010 at the virtual viewpoint. The virtual viewpoint image shown in Figure 11(c) is an image corresponding to the view from a virtual viewpoint positioned directly beside both the perpetrator (the object on the near side) and the victim (the object on the far side). When the type of violation candidate is "inside overtaking", using the virtual viewpoint image shown in Figure 11(c) makes it easier for judges, etc., to determine whether the perpetrator overtook the victim from the inside.

[0057] After S506, in S507, the display control unit 406 places the virtual viewpoint image generated in S506 at a predetermined position in the display image and displays the placed display image on the display device 310 via the display I / F 306. Figure 12 is a diagram showing an example of a display image 1200 that the display control unit 406 according to Embodiment 1 displays on the display device 310. Specifically, the display image 1200 shown in Figure 12 shows an example of a two-screen display layout. Figure 12 shows a display image 1200 in which the captured image 1201 obtained by imaging by the imaging device 11 is placed in the left area, and the virtual viewpoint image 1202 is placed in the right area. The virtual viewpoint image 1202 placed in the right area is a virtual viewpoint image generated in S506. In this case, for example, the captured image 1201 from the imaging device 11, which performs imaging from a direction similar to the position and direction of line of sight of the virtual viewpoint corresponding to the displayed virtual viewpoint image 1202, is placed in the left area of ​​the two-screen display layout. For example, under normal circumstances, a single-screen display layout image consisting only of the captured image is displayed. If there is suspicion that a violation has occurred, the display switches to a two-screen display layout image 1200, which consists of the captured image 1201 and the virtual viewpoint image 1202.

[0058] While a two-screen display layout including an captured image and a virtual viewpoint image was described as an example of a display image, the layout of the display image is not limited to this. For example, if multiple virtual viewpoint information is generated in S505 for the detection of a single potential violation, the display image may show a multi-screen display layout in which multiple different virtual viewpoint images are arranged.

[0059] Furthermore, a display image may be shown in a two-screen display layout in which an image captured by an imaging device different from the imaging device 11 installed in bank 20 and a virtual viewpoint image are arranged. In this case, for example, the image captured is a real-time image used in television broadcasting, etc. Also, the display image is not limited to a two-screen display layout, but may be a multi-screen display layout of three or more screens or a PIP display, etc.

[0060] In S508, the information processing device 100 determines whether or not the user has performed an operation to terminate the imaging. If it is determined that the user has not performed an operation to terminate the imaging, the information processing device 100 returns to the process in S500 and repeatedly executes the process from S500 to S508 until it is determined in S508 that the user has performed an operation to terminate the imaging. If it is determined that the user has performed an operation to terminate the imaging, the information processing device 100 terminates the process shown in the flowchart in Figure 5.

[0061] According to the information processing device 100 of Embodiment 1, it is possible to generate and display virtual viewpoint images that assist in determining suspected violations such as obstruction of riding during a competition, without requiring any intervention from users such as referees. Furthermore, although this embodiment was described using keirin (bicycle racing) as an example of a two-wheeled competition, the applicable two-wheeled competitions are not limited to this. For example, violations in auto racing, motocross competitions, and bicycle motocross (BMX) competitions can also be detected.

[0062] [Embodiment 2] <Configuration of the information processing device> Embodiment 2 will be described with reference to Figures 13 to 15. Figure 13 is a block diagram showing an example of the functional configuration of the information processing device 100 according to Embodiment 2 (hereinafter simply referred to as "information processing device 100"). The functional configuration of the information processing device 100 is the same as that of the information processing device 100 according to Embodiment 1 shown as an example in Figure 4, with the addition of a speed calculation unit 1301 and a contact determination unit 1302, and the detection unit 403 being changed to a detection unit 1303. The same configuration as Embodiment 1 will be omitted or simplified in the description. The hardware configuration of the information processing device 100 is the same as that of the information processing device 100 according to Embodiment 1 shown as an example in Figure 3, so the description will be omitted. That is, each part of the functional configuration of the information processing device 100 shown as an example in Figure 13 is realized, for example, by the CPU 301 executing an application program stored in the HDD 304 or the like using the RAM 303 as work memory.

[0063] The velocity calculation unit 1301 calculates the velocity of an object (hereinafter referred to as "object velocity") using the object position information contained in the 3D model information acquired by the model acquisition unit 401. For example, the object velocity is calculated by dividing the difference in object position between frames by the time interval between frames. The time interval between frames is determined by the frame rate of the multiple imaging devices 11. For example, if the frame rate of the multiple imaging devices 11 is 60 frames / second, the time interval between frames will be approximately 16.67 ms. The object velocity calculated for each object is stored as velocity information in the RAM 303, etc., and used for processing by the detection unit 1303, etc.

[0064] The contact detection unit 1302 uses the object shape information contained in the 3D model information acquired by the model acquisition unit 401 to determine whether or not objects are in contact with each other. The contact detection process is performed, for example, using an OBB (Oriented Bounding Box). An OBB is a method in which each side of a rectangular prism enclosing the object shape is configured to match the front, back, left, and right orientations of the object. The contact detection unit 1302 determines whether or not objects are in contact by determining whether or not the rectangular prisms for each object shape overlap. Note that the contact detection process is not limited to OBB and may be performed using other methods such as an AABB (Axis-Aligned Bounding Box). While contact detection using AABB has lower contact detection accuracy compared to OBB, it is expected to reduce the amount of computation. The contact detection result for each object by the contact detection unit 1302 is stored as contact detection information in the RAM 303, etc., and used in the processing of the detection unit 1303, etc.

[0065] The detection unit 1303 uses trajectory information acquired by the trajectory acquisition unit 402, speed information calculated by the speed calculation unit 1301, and contact determination information determined by the contact determination unit 1302 to detect objects suspected of committing a violation. When the detection unit 1303 detects a candidate for a violation, it associates the type of the candidate violation with the trajectory information of the perpetrator (the object that obstructed the movement) and the victim (the object whose movement was obstructed) and stores them in the RAM 303 or HDD 304, etc. In this case, the detection unit 1303 also associates identification information that can uniquely identify the perpetrator and the victim with the trajectory information of the perpetrator and the trajectory information of the victim and stores it in the RAM 303 or HDD 304, etc.

[0066] <Operation of the Information Processing Device> Figure 14 is a flowchart showing an example of the processing flow of the information processing device 100 according to Embodiment 2. The flowchart shown in Figure 14 is a modified version of the information processing device 100 according to Embodiment 1 shown as an example in Figure 5, with S1401 and S1402 added and S503 changed to S1403.

[0067] The flowchart shown in Figure 14 is executed by the CPU 301 loading the application program stored in the HDD 304 or the like into the RAM 303 and executing it. The flowchart shown in Figure 14 is started when the user initiates the image capture process. In the following, processing steps similar to those shown in the flowchart of Figure 5 are denoted by the same reference numerals, and their explanations are omitted or simplified. First, the information processing device 100 executes the processes from S500 to S502.

[0068] Following S502, in S1401, the velocity calculation unit 1301 calculates the object velocity based on the object position information included in the 3D model information acquired in S501. For example, the object velocity is calculated by dividing the difference in object position between frames by the time interval between frames. Specifically, if the frame rate of the multiple imaging devices 11 is 60 frames / second and the difference in object position between frames is 0.3 m (meters), the object velocity will be 17.996 m / second, or 64.787 km (kilometers) / hour. The object velocity is calculated for all objects. The calculation result of the object velocity corresponding to each object is stored as velocity information in RAM 303, etc. Next, in S1402, the contact determination unit 1302 uses the object shape information and object position information included in the 3D model information acquired in S501 to determine whether or not objects are in contact with each other.

[0069] Figure 15 is a diagram illustrating the object contact determination process by the contact determination unit 1302 according to Embodiment 2. Specifically, Figure 15 is a diagram illustrating the object contact determination process using the OBB method. Object shapes 1501 and 1502 are placed at the positions of each other, i.e., at the object positions. Rectangular parallelepipeds 1511 and 1512 are, in order, rectangular parallelepipeds that circumscribe object shapes 1501 and 1502. Contact between rectangular parallelepiped 1511 and rectangular parallelepiped 1512 is determined using separation axis determination, a known technique. Note that contact between rectangular parallelepipeds is not limited to separation axis determination and may be performed by other methods. Also, the shape enclosing the object shapes is not limited to rectangular parallelepipeds and may be other shapes such as capsule shapes or spherical shapes. Contact between object shapes is determined for all objects, and the result of the determination is stored as contact determination information in RAM 303, etc.

[0070] In S1403, the detection unit 1303 detects an object suspected of having committed a violation based on the trajectory information acquired in S502, the object velocity calculated in S1401, and the contact determination result between object shapes determined in S1402. Specifically, first, the detection unit 1303 executes the process in S503 shown in Figure 5, which was explained in Embodiment 1. Subsequently, the detection unit 1303 detects an object suspected of having committed a violation by performing a final detection process using the results of the detection process in S503, the object velocity calculation result in S1401, and the contact determination result between objects in S1402. Below, as an example, the cases in which the provisional detection result from the process in S503 in S1403 is "diagonal movement," "pressing," and "inside overtaking" will be explained.

[0071] If the provisional detection result is "diagonal," the detection unit 1303 determines the final detection result to be "diagonal" if the contact determination information determines that the perpetrator and the victim are in contact. If the detection unit 1303 determines that the perpetrator and the victim are not in contact, it observes the change in the victim's speed, as indicated by the speed information, over time since the frame in which the provisional detection result was obtained. If the victim's speed has decreased, the detection unit 1303 determines the final detection result to be "diagonal." If the victim's speed has increased or has not changed, the final detection result is determined to be "no violation." If the provisional detection result is "pressure," the detection unit 1303 determines the final detection result to be "diagonal" if the contact determination information determines that the perpetrator and the victim are in contact. If the detection unit 1303 determines that the perpetrator and the victim are not in contact, it determines the final detection result to be "no violation." If the provisional detection result is "inside overtaking," the detection unit 1303 determines the provisional detection result, "inside overtaking," as the final detection result.

[0072] After S1403, the information processing device 100 appropriately executes the processes from S504 to S508. In the process of S507, the display control unit 406 places the virtual viewpoint image generated in S506 at a predetermined position in the display image, as described in Embodiment 1, and displays the placed display image on the display device 310 via the display I / F 306. In this case, the display control unit 406 may superimpose the victim's speed calculated in S1401 onto the virtual viewpoint image in the display image.

[0073] Figure 16 shows an example of a display image that the display control unit 406 according to Embodiment 2 displays on the display device 310. In the display image shown in Figure 16, region 1601 displays the victim's speed, which is included as an image in the display image, superimposed on the virtual viewpoint image. The number in parentheses in region 1601 indicates the difference between the victim's speed at the time of the frame used to generate the virtual viewpoint image and the victim's speed immediately before the obstruction. By checking the number in parentheses, judges and others can confirm how much the victim's speed decreased as a result of the obstruction.

[0074] According to the information processing device 100 of Embodiment 2, by detecting violations using speed information and contact determination information in addition to trajectory information, the detection accuracy of suspected violations such as obstruction of driving in a competition can be improved.

[0075] [Other embodiments] In the embodiments described above, the case in which motorcycle racing was used as the imaging target was explained, but the imaging target is not limited to motorcycle racing, and may be, for example, horse racing or track and field.

[0076] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more of the functions.

[0077] Within the scope of this disclosure, it is possible to freely combine the embodiments, modify any component of each embodiment, or omit any component in each embodiment.

[0078] [Technical Features of This Disclosure] This disclosure includes the following configurations, methods, and programs.

[0079] <Configuration 1> A shape acquisition means for acquiring three-dimensional shape data corresponding to each of multiple objects, which is generated based on captured images obtained by imaging with multiple imaging devices, Trajectory acquisition means that acquires trajectory information showing the trajectory of each of the multiple objects based on the three-dimensional shape data, A detection means for detecting a violation of a given rule based on the aforementioned trajectory information, An information processing device characterized by having the following features.

[0080] <Configuration 2> The three-dimensional shape data includes shape information indicating the three-dimensional shape of the object and position information indicating the position of the object. An information processing device as described in Configuration 1, characterized by the above.

[0081] <Structure 3> The trajectory acquisition means generates the trajectory information based on the position information. An information processing device as described in configuration 2, characterized by the above.

[0082] <Structure 4> A contact determination means that determines whether or not the objects are in contact with each other based on the three-dimensional shape data. It further possesses, The detection means detects the violation based on the determination result of whether or not the object that obstructed the movement and the object that was obstructed came into contact. An information processing device according to any one of configurations 1 to 3 characterized by the above.

[0083] <Composition 5> A velocity calculation means that calculates the velocity of the object based on the three-dimensional shape data, It further possesses, The detection means detects the violation based on whether or not the speed of the object that was obstructed has decreased. An information processing device according to any one of configurations 1 to 4 characterized by the above.

[0084] <Composition 6> A viewpoint generation means that generates virtual viewpoint information relating to a virtual viewpoint based on the type of violation detected, Image generation means for generating a virtual viewpoint image based on the three-dimensional shape data and the virtual viewpoint information, Having further, An information processing device according to any one of configurations 1 to 5 characterized by the above.

[0085] <Composition 7> The virtual viewpoint information includes information regarding the position of the virtual viewpoint and the direction of the line of sight at the virtual viewpoint, which change over time. An information processing device according to configuration 6, characterized by the above.

[0086] <Structure 8> The viewpoint generation means determines the position of the virtual viewpoint and the direction of the line of sight in the virtual viewpoint, which change over time, so as to provide an overhead view of both the object that obstructed the movement and the object that was obstructed. An information processing device according to configuration 7, characterized by the above.

[0087] <Composition 9> The viewpoint generation means determines the position of the virtual viewpoint and the direction of the line of sight in the virtual viewpoint, which change over time, so that the portion where the object that obstructed movement and the object that was obstructed movement came into contact with each other is included as an image in the virtual viewpoint image. An information processing apparatus according to configuration 7 or 8, characterized by the above.

[0088] <Composition 10> The viewpoint generation means determines the position of the virtual viewpoint and the direction of the line of sight in the virtual viewpoint, which change over time, so that the process of the object that obstructed the movement overtaking the object that was obstructed is included as an image in the virtual viewpoint image. An information processing device according to any one of configurations 7 to 9 characterized by the above.

[0089] <Composition 11> Display control means for controlling the display of the virtual viewpoint image, Having further, An information processing device according to any one of configurations 6 to 10 characterized by the above.

[0090] <Composition 12> The display control means controls the display of a display image in which the velocity of the object is superimposed on the virtual viewpoint image. An information processing device according to configuration 11, characterized by the above.

[0091] <Method> A shape acquisition process that acquires three-dimensional shape data corresponding to each of multiple objects, which is generated based on captured images obtained by imaging with multiple imaging devices, A trajectory acquisition step, which acquires trajectory information showing the trajectory of each of the multiple objects based on the three-dimensional shape data, A detection step that detects a violation of a given rule based on the aforementioned trajectory information, An information processing method characterized by including

[0092] <Program> A program for causing a computer to function as an information processing device described in any one of configurations 1 through 12. [Explanation of Symbols]

[0093] 100 Information Processing Devices 401 Model Acquisition Section 402 Trajectory acquisition part 403 Detection Unit

Claims

1. A shape acquisition means for acquiring three-dimensional shape data corresponding to each of multiple objects, which is generated based on captured images obtained by imaging with multiple imaging devices, Trajectory acquisition means that acquires trajectory information showing the trajectory of each of the multiple objects based on the three-dimensional shape data, A detection means for detecting a violation of a given rule based on the aforementioned trajectory information, An information processing device characterized by having the following features.

2. The three-dimensional shape data includes shape information indicating the three-dimensional shape of the object and position information indicating the position of the object. The information processing apparatus according to claim 1, characterized by the following:

3. The trajectory acquisition means generates the trajectory information based on the position information. The information processing apparatus according to claim 2, characterized in that

4. A contact determination means that determines whether or not the objects are in contact with each other based on the three-dimensional shape data. It further possesses, The detection means detects the violation based on the determination result of whether or not the object that obstructed the movement and the object that was obstructed came into contact. The information processing apparatus according to claim 1, characterized by the following:

5. A velocity calculation means that calculates the velocity of the object based on the three-dimensional shape data, It further possesses, The detection means detects the violation based on whether or not the speed of the object that was obstructed has decreased. The information processing apparatus according to claim 1, characterized by the following:

6. A viewpoint generation means that generates virtual viewpoint information relating to a virtual viewpoint based on the type of violation detected, Image generation means for generating a virtual viewpoint image based on the three-dimensional shape data and the virtual viewpoint information, Having further, The information processing apparatus according to claim 1, characterized by the following:

7. The virtual viewpoint information includes information regarding the position of the virtual viewpoint and the direction of the line of sight at the virtual viewpoint, which change over time. The information processing apparatus according to claim 6, characterized by the following:

8. The viewpoint generation means determines the position of the virtual viewpoint and the direction of the line of sight in the virtual viewpoint, which change over time, so as to provide an overhead view of both the object that obstructed the movement and the object that was obstructed. The information processing apparatus according to claim 7, characterized by the following:

9. The viewpoint generation means determines the position of the virtual viewpoint and the direction of the line of sight in the virtual viewpoint, which change over time, so that the portion where the object that obstructed movement and the object that was obstructed movement came into contact with each other is included as an image in the virtual viewpoint image. The information processing apparatus according to claim 7, characterized by the following:

10. The viewpoint generation means determines the position of the virtual viewpoint and the direction of the line of sight in the virtual viewpoint, which change over time, so that the process of the object that obstructed the movement overtaking the object that was obstructed is included as an image in the virtual viewpoint image. The information processing apparatus according to claim 7, characterized by the following:

11. Display control means for controlling the display of the virtual viewpoint image, Having further, The information processing apparatus according to claim 6, characterized by the following:

12. The display control means controls the display of a display image in which the velocity of the object is superimposed on the virtual viewpoint image. The information processing apparatus according to claim 11, characterized by the above.

13. A shape acquisition process that acquires three-dimensional shape data corresponding to each of multiple objects, which is generated based on captured images obtained by imaging with multiple imaging devices, A trajectory acquisition step, which acquires trajectory information showing the trajectory of each of the multiple objects based on the three-dimensional shape data, A detection step that detects a violation of a given rule based on the aforementioned trajectory information, An information processing method characterized by including

14. A program for causing a computer to function as an information processing device according to any one of claims 1 to 12.

Images