Image processing system, image processing method, and storage medium

Abstract

To generate a highly accurate virtual viewpoint image when performing image processing in a plurality of devices in a distributed manner.SOLUTION: An image processing device 120 that performs processing in cooperation with another image processing device 120 in an image processing system, the image processing device 120 acquiring a captured image output from an imaging device and performing first image processing on the acquired captured image in a first operation mode, outputting information related to a result of the first image processing, acquiring a captured image from a storage device, outputting the acquired captured image to the other image processing apparatus, acquiring a captured image output from the other image processing apparatus in the second operation mode, performing second image processing on the acquired captured image, and outputting information related to a result of the second image processing.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to techniques for generating virtual viewpoint images. [Background technology]

[0002] There is a technology that generates an image corresponding to the view from an arbitrarily specified virtual viewpoint (hereinafter referred to as a "virtual viewpoint") using multiple captured images (hereinafter referred to as a "multi-viewpoint image") obtained by installing multiple imaging devices at different positions and capturing images in synchronization with each other. Hereinafter, such an image will be referred to as a "virtual viewpoint image." Services that use such virtual viewpoint images allow users to view a specific scene, such as a fine play in baseball or soccer, from various angles, thereby providing a greater sense of realism to users compared to viewing conventional captured images.

[0003] An image generating device configured with a server device or the like first acquires images captured by a plurality of imaging devices capturing images of specific positions within a space to be imaged as multi-viewpoint images. Next, the image generating device generates a virtual viewpoint image by using the acquired multi-viewpoint images to generate three-dimensional shape data corresponding to objects present within the space and to render the generated three-dimensional shape data. In this case, to reduce the processing load of the image generating device, each imaging device or an image processing device connected to each imaging device may perform image processing, such as separating the image area of ​​the captured image into a foreground area and a background area, which is part of the processing performed by the image generating device.

[0004] The multiple imaging devices are installed so as to capture the entire space of the imaging target. For example, if the imaging target is baseball, each imaging device is installed so as to capture the entire playing field, including the infield, outfield, and foul zone. The imaging environment in which each imaging device is installed in this manner can also be used to capture sports competitions other than baseball, or performances such as concerts. When imaging something other than baseball, the space to be imaged is localized, so some of the installed imaging devices may not be used.

[0005] Incidentally, Patent Document 1 discloses a technique for improving performance in terms of processing speed in image processing by distributing predetermined image processing among a plurality of imaging devices connected on a ring topology network. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Special Publication No. 2016-526340 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the technology disclosed in Patent Document 1 simply distributes predetermined image processing among a plurality of imaging devices, and therefore the results of the image processing may not reach the desired accuracy. Therefore, even if the technology disclosed in Patent Document 1 is applied to generating the above-mentioned virtual viewpoint image, it may not be possible to generate a highly accurate virtual viewpoint image.

[0008] An object of the present disclosure is to provide a technology capable of generating highly accurate virtual viewpoint images when image processing is distributed among multiple devices. [Means for solving the problem]

[0009] An image processing device in an image processing system according to the present disclosure that performs processing in cooperation with other image processing devices, the image processing device having: a first image acquisition means that operates in a first operating mode and acquires a captured image output from an imaging device; a first image processing means that performs first image processing on the captured image acquired by the first image acquisition means; a first output means that outputs information relating to a result of the first image processing by the first image processing means; a second image acquisition means that acquires the captured image from a storage device; a second output means that outputs the captured image acquired by the second image acquisition means to the other image processing device; a third image acquisition means that operates in a second operating mode and acquires the captured image output from the second output means of the other image processing device; a second image processing means that performs second image processing on the captured image acquired by the third image acquisition means; and a third output means that outputs information relating to the result of the second image processing by the second image processing means. [Effects of the Invention]

[0010] According to the present disclosure, when image processing is distributed among a plurality of devices, a highly accurate virtual viewpoint image can be generated. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an image processing system according to a first embodiment. [Figure 2] 1 is a block diagram showing an example of the functional configuration of an image processing device, an image generating device, and an information processing device according to a first embodiment. [Figure 3] 1 is a block diagram showing an example of the hardware configuration of an image processing device, an image generating device, and an information processing device according to a first embodiment. [Figure 4] FIG. 10 is a diagram showing an example of information included in a reprocessing request acquired by the image processing apparatus that is the transmission source according to the first embodiment. [Figure 5] FIG. 10 is a diagram showing an example of information included in a reprocessing request acquired by a destination image processing apparatus according to the first embodiment. [Figure 6] FIG. 2 is a sequence diagram showing an example of a processing sequence in the image processing system according to the first embodiment. [Figure 7] FIG. 2 is a sequence diagram showing an example of a processing sequence in the image processing system according to the first embodiment. [Figure 8] 10 is a diagram for explaining an example of a process for determining an image processing device as a transmission destination in a determination unit of the information processing device according to the first embodiment. FIG. [Figure 9] FIG. 10 is a block diagram showing an example of the functional configuration of an image processing device and an information processing device according to a second embodiment. [Figure 10] FIG. 10 is a sequence diagram showing an example of a processing sequence in an image processing system according to a second embodiment. [Figure 11] FIG. 10 is a diagram showing an example of a silhouette image according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, the present embodiment will be described with reference to the drawings. Note that the following embodiment does not necessarily limit the means for solving the problems according to the present disclosure. Furthermore, not all of the combinations of features described in the present embodiment are necessarily essential to the means for solving the problems according to the present disclosure.

[0013] [Embodiment 1] <Image processing system configuration> FIG. 1 is a diagram showing an example of the configuration of an image processing system according to this embodiment. As shown in FIG. 1(a) as an example, the image processing system includes a plurality of camera units 102, an image generating device 105, and an information processing device 106. The plurality of camera units 102, the image generating device 105, and the information processing device 106 are communicatively connected to one another via a transmission cable 103. The transmission cable 103 is a cable line such as a wired local area network (LAN) or an optical fiber cable. As shown in FIG. 1(b) as an example, each camera unit 102 includes an image capturing device 110 and an image processing device 120. The image capturing device 110 and the image processing device 120 included in each camera unit 102 are communicatively connected to one another. The playing field 101 is a ball field such as a baseball stadium including an infield, an outfield, and a foul zone.

[0014] Multiple camera units 102 are installed to surround the competition field 101. The imaging device 110 included in each camera unit 102 captures at least a portion of the competition field 101, and is installed so that a portion of the angle of view of any one imaging device 110 overlaps a portion of the angle of view of one or more other imaging devices 110. Each imaging device 110 is fixed so that its attitude (also referred to as the "direction of the optical axis") and position do not shift over time. The image processing device 120 included in each camera unit 102 is connected to the image processing devices 120 included in the other camera units 102 via a transmission cable 103 so that they can communicate with each other. Images captured by the imaging devices included in each camera unit 102 can be transmitted to the other image processing devices 120, image generation devices 105, information processing devices 106, etc. via the image processing device 120 and the transmission cable 103. Furthermore, the image obtained as a result of image processing in the image processing device 120 can be transmitted via the transmission cable 103 to other image processing devices 120, the image generation device 105, the information processing device 106, and the like.

[0015] By permanently installing multiple camera units 102 and transmission cables 103 on the competition field 101, the image processing system shown as an example in FIG. 1(a) can capture images of baseball games and the like throughout the season. The image processing system shown as an example in FIG. 1(a) can also capture images of other sports competitions or performances such as concerts, in addition to baseball. Specifically, the image processing system shown as an example in FIG. 1(a) can capture images of a concert held on a stage 104 installed in the outfield of the competition field 101. In this type of imaging, the space to be imaged is limited to a portion of the space including the stage 104. Therefore, an imaging device 110 whose imaging range falls outside the stage 104 becomes a standby imaging device 110 that does not capture images (hereinafter referred to as a "standby imaging device 110").

[0016] Hereinafter, the camera unit 102 including the standby imaging device 110 will be referred to as the "standby camera unit 102," and the image processing device 120 included in the standby camera unit 102 will be referred to as the "standby image processing device 120." Meanwhile, the imaging device 110 capturing an image will be referred to as the "operating imaging device 110." Furthermore, the camera unit 102 including the operating imaging device 110 will be referred to as the "operating camera unit 102," and the image processing device 120 included in the operating camera unit 102 will be referred to as the "operating image processing device 120."

[0017] The standby image processing device 120 is used to improve the image quality of a virtual viewpoint image when generating the virtual viewpoint image for a replay, a highlight scene, or the like. Each imaging device 110 may be configured to capture a still image or a moving image. Each imaging device 110 may also be configured to capture both a still image and a moving image. In this embodiment, unless otherwise specified, the term "image" includes the meaning of both a still image and a moving image.

[0018] <Configuration of each device> FIG. 2 is a block diagram showing an example of the functional configuration of the image processing device 120, the image generation device 105, and the information processing device 106 according to the first embodiment. Specifically, FIG. 2(a) shows an example of the functional configuration of the image processing device 120, FIG. 2(b) shows an example of the functional configuration of the image generation device 105, and FIG. 2(c) shows an example of the functional configuration of the information processing device 106. The image processing device 120 includes a first image acquisition unit 221, a first separation unit 222, a request acquisition unit 223, a second image acquisition unit 224, a third image acquisition unit 225, a second separation unit 226, and an output unit 229. The image generation device 105 includes an acquisition unit 251, a generation unit 252, and an output unit 253. The information processing device 106 includes an acquisition unit 261, an evaluation unit 262, an identification unit 263, a determination unit 264, and an issuance unit 265. The functional configurations of the image processing device 120, the image generating device 105, and the information processing device 106 will be described in detail later.

[0019] 3 is a block diagram showing an example of the hardware configuration of the image processing device 120, image generation device 105, and information processing device 106 according to the first embodiment. Each of the image processing device 120, image generation device 105, and information processing device 106 has a controller unit 300, an operation input device 309, and a display device 310. The controller unit 300 includes a CPU 301, a ROM 302, a RAM 303, an HDD 304, an operation unit I / F 305, a display unit I / F 306, and a communication I / F 307. The hardware components included in the controller unit 300 are connected to each other via a system bus 308 so as to be able to communicate with each other. Note that CPU is an acronym for Central Processing Unit, HDD is an acronym for Hard Disk Drive, and I / F is an abbreviation for interface.

[0020] The CPU 301 controls the entire device by controlling the operation of the ROM 302, RAM 303, HDD 304, operation unit I / F 305, display unit I / F 306, and communication I / F 307 via a system bus 308. The CPU 301 starts up an operating system (OS) by executing a boot program stored in the ROM 302. The CPU 301 executes application programs stored in the HDD 304, for example, on this OS. The CPU 301 executes the application programs to realize the processing of each functional configuration of each device. The RAM 303 is used, for example, to store various types of temporary information and as a working area for the CPU 301. The HDD 304 stores the application programs and various data used in executing the application programs, as described above.

[0021] Note that some or all of the processing executed by CPU 301 may be executed by one or more GPUs (Graphics Processing Units) or processing circuits not shown in Fig. 3. Examples of processing circuits include ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), and FPGAs (Field Programmable Gate Arrays). Furthermore, HDD 304 is not limited to a hard disk drive, and may be realized by a magnetic disk, optical disk, semiconductor memory, or the like, as long as it is capable of storing application programs and various data.

[0022] The operation unit I / F 305 is a communication interface with the operation input device 309. The operation unit I / F 305 transfers information related to user operations accepted by the operation input device 309 to the CPU 301. The operation input device 309 includes an input device capable of accepting user operations, such as a mouse, a keyboard, or a touch panel. The display unit I / F 306 is a communication interface with the display device 310, and outputs image data to be displayed on the display device 310 to the display device 310. The display device 310 includes a display device such as a liquid crystal display. The communication I / F 307 is a communication interface for communication via Ethernet (registered trademark) or the like, and includes, for example, a connector for accepting connection of a transmission cable. Note that the communication I / F 307 may be a wireless communication interface and may include, for example, a baseband circuit, an RF circuit, and an antenna. The communication I / F 307 inputs and outputs information to and from an external device.

[0023] Each device may perform display control to display an image on a display device 310, which is an external device connected via a transmission cable or a network. In this case, the device performs display control to output display data to the display device 310. The configuration in Fig. 3 is an example, and some of the components may be omitted, components not shown may be added, or the illustrated components may be combined.

[0024] The imaging device 110 of the camera unit 102 has an imaging section including an optical lens and an imaging element, an image generation section, and an output section. The imaging section 201 uses the optical lens to focus external light on the imaging element and converts the light focused on the imaging element into an electrical signal. The converted electrical signal is converted into a digital signal by an A / D converter, and the image generation section generates a captured image based on the digital signal. The generated captured image is output from the output section to the image processing device 120 as captured image data. The imaging element is a photoelectric conversion element that converts an optical signal based on an image focused on a light-receiving surface into an electrical signal for each light-receiving pixel at a corresponding position.

[0025] <Functional configuration of image processing device> The image processing device 120 acquires a captured image, generates a foreground image and a background image corresponding to the acquired captured image, and outputs them. Hereinafter, with reference to FIGS. 2(a) and 3, the processing of each unit included in the image processing device 120 as a functional configuration will be described. The first image acquisition unit 221 acquires a captured image output by the imaging device 110 included in the same camera unit 102. The captured image acquired by the first image acquisition unit 221 is temporarily stored in a frame memory. Note that the frame memory may be a predetermined area in the RAM 303, or may be a volatile memory not shown in FIG. 3. The frame memory is divided into several storage areas, and for example, a usable functional configuration is assigned to each storage area. The first image acquisition unit 221 stores the captured image acquired from the imaging device 110 in a predetermined storage area in the frame memory.

[0026] Further, the first image acquisition unit 221 associates the acquired captured image with information indicating the capture time, which is the time when the captured image was captured, generated, or acquired, and stores the captured image in the HDD 304. Hereinafter, the image processing device 120 will be described as storing the captured image in the HDD 304, but the storage destination of the captured image by the image processing device 120 is not limited to the HDD 304. For example, the image processing device 120 may store the captured image in a storage device (not shown in FIG. 3 ) that is an external device connected to the image processing device 120 via a cable or a network. In this embodiment, the first image acquisition unit 221 will be described as storing the acquired captured image in a storage device such as the HDD 304, but the storage destination is not limited thereto. For example, the image capturing device 110 may output the captured image to a storage device (not shown) that is also accessible from the image processing device 120, and store the captured image in the storage device. Furthermore, the storage device may be a NAS (Network-Attached Storage) having an HDD or the like.

[0027] The first separation unit 222 reads out the captured image acquired by the first image acquisition unit 221 and stored in the frame memory, and separates the image region in the read captured image into a foreground region and a background region. The process of separating the image region in an image into a foreground region and a background region is also called foreground-background separation. Furthermore, the first separation unit 222 generates a foreground image indicating the foreground region and a background image indicating the background region based on the foreground region and background region separated by foreground-background separation. The generated foreground image and background image are output to the output unit 229. The output unit 229 receives the foreground image and background image output from the first separation unit 222 and outputs the foreground image and background image to the image generation device 105.

[0028] Here, the foreground region is a region in a captured image that includes an image of an object that can be in the foreground, such as a person or a ball, and the background region is a region in the captured image other than the foreground region. The foreground image is an image in which an area corresponding to the foreground region is extracted from the captured image, and the background image is an image in which an area corresponding to the background region is extracted from the captured image. The foreground image and the background image include texture information such as color information. The foreground object is a moving object whose absolute position or shape may change when captured from the same direction in chronological order. Specifically, for example, the foreground object is a person, such as a player or a referee, who is present on the playing field 101, or an object, such as a ball used in a ball game. If the subject of the image capture is a concert or entertainment event, the foreground object is a person, such as a singer, musician, performer, or presenter, or an object held by the person.

[0029] On the other hand, an object contained as an image in the background region, i.e., an object that can be the background, is an object that remains stationary or substantially stationary when, for example, images are taken from the same direction in chronological order. Specifically, for example, objects that can be the background include a stage for a concert or the like, a stadium where an event such as a competition is held, structures such as goals used in ball games, and the floor of a field or the like. However, the background region is a region different from the foreground region that includes at least the image of the object that will be the foreground. Note that the imaged subject may include other objects in addition to the objects that can be the foreground and background.

[0030] Foreground-background separation in the first separation unit 222 is realized by, for example, a trained model configured by a CNN (Convolutional Neural Network) or the like obtained as a result of learning by machine learning or the like. The first separation unit 222 may separate the foreground region and the background region in the captured image using a well-known foreground-background separation method such as background subtraction. The foreground-background separation in the first separation unit 222 will be described as being realized by a trained model. Furthermore, although the foreground-background separation in the first separation unit 222 will be described below as being realized by a trained CNN, the trained model is not limited to a trained CNN as long as it is capable of separating the foreground region and the background region in the input image.

[0031] CNN is a model (hereinafter referred to as "CNN model") that mimics the nerve cells (neurons) in the human brain, and is mainly composed of three layers: a convolutional layer, a pooling layer, and a fully connected layer. Generally, a CNN model has a structure in which multiple convolutional layers and pooling layers are alternately stacked, followed by several fully connected layers. The convolutional layers and pooling layers repeatedly extract features from the input image, and by alternately stacking multiple convolutional layers and pooling layers, complex features in the input image can be extracted. In the fully connected layer, the extracted complex features are treated as multiple feature quantities, and inferences such as prediction or classification can be made by combining multiple feature quantities. By increasing the number of these layers, a CNN model can improve its inference performance such as prediction or classification.

[0032] However, increasing the number of layers results in a huge number of parameters for the CNN model, which increases memory usage and the amount of calculation. Therefore, the more powerful the CNN model, the longer it takes to obtain an inference result, making it difficult to obtain the inference result in real time. For example, if the image capture frame rate of the image capture device 110 is 60 fps (frames per second), the CNN model needs to output an inference result within 16.6 ms (milliseconds). However, if the performance of the CNN model is improved, it becomes difficult to achieve this time constraint.

[0033] One possible solution to this problem is, for example, optimizing the CNN model. Optimizing a CNN model is a technique for reducing memory usage and speeding up processing time while maintaining the accuracy of the inference results of the CNN model as much as possible. Typical optimization techniques include pruning, quantization, and distillation. The first separation unit 222 performs foreground / background separation of the captured image using the optimized CNN model so that the processing is completed within a predetermined time, such as within 16.6 ms. In the following description, the CNN model before optimization will be referred to as the original CNN model. The original CNN model is a model that places the greatest importance on inference accuracy.

[0034] First separation unit 222 may compress the generated foreground image and background image using lossless compression, and output the compressed foreground image and background image to output unit 229. In this case, output unit 229 receives the compressed foreground image and background image output from first separation unit 222, and outputs the compressed foreground image and background image to image generation device 105. By compressing the foreground image and background image using lossless compression, even if the bandwidth of transmission cable 103 is not sufficiently wide, it becomes possible to transmit the foreground image and background image at high speed without degrading the quality of the decoded foreground image and background image.

[0035] Specifically, when the foreground image and background image are compressed using lossless compression, the first separation unit 222 losslessly compresses the foreground image based on, for example, a foreground ratio, which is a performance index of the compression process. The foreground ratio is the ratio of the size of the foreground region to the size of the original image before foreground / background separation. For example, if the foreground ratio is 30% and the size of the original image is 4K (4096 x 2160 pixels), the size of the foreground region will be 2,654,208 pixels. Here, the foreground ratio is set to a value that allows the compression process to be completed within a predetermined processing time. For example, if the frame rate of the image capture is 60 fps, the allowable processing time for the compression process per frame is 16.6 ms or less. Therefore, the foreground ratio is set to a value that allows the first separation unit 222 to complete the compression process within this allowable processing time.

[0036] The first separation unit 222 selects and compresses one or more foreground images corresponding to each of the multiple foreground regions included in the captured image, within the range that falls within the foreground ratio. Specifically, for example, the first separation unit 222 sorts the foreground images corresponding to each of the multiple foreground regions included in the captured image by image size and adds up the number of pixels of the foreground images in descending order of image size. In this case, each time the first separation unit 222 adds up the number of pixels, it compares the ratio of the total number of pixels to the number of pixels of the captured image with the foreground ratio, and if the ratio is equal to or less than the foreground ratio, selects the added foreground image as the image to be compressed. If the ratio exceeds the foreground ratio, the added foreground image and the subsequent foreground images are excluded from the compression process. The foreground images that are not the image to be compressed are discarded.

[0037] The request acquisition unit 223 acquires a reprocessing request issued from the information processing device 106. Details of the reprocessing request issued from the information processing device 106 will be described later. The second image acquisition unit 224 acquires the captured image stored in the HDD 304 by the first image acquisition unit 221, based on the reprocessing request from the information processing device 106 acquired by the request acquisition unit 223, by reading out the captured image. The captured image acquired by the second image acquisition unit 224 is output to the output unit 229. Based on the reprocessing request from the information processing device 106 acquired by the request acquisition unit 223, the output unit 229 outputs the captured image output from the second image acquisition unit 224 to another image processing device 120, specifically, a standby image processing device 120, via the transmission cable 103.

[0038] Note that output unit 229 may output the foreground image and background image obtained as a result of image processing by first separation unit 222 and the captured image acquired by second image acquisition unit 224 via different transmission cables 130. With this configuration, it is possible to prevent the transmission of the captured image acquired by second image acquisition unit 224 from interfering with the transmission of the foreground image and background image obtained as a result of image processing by first separation unit 222.

[0039] The third image acquisition unit 225 acquires, based on a reprocessing request from the information processing device 106 acquired by the request acquisition unit 223, the captured image acquired by the second image acquisition unit 224 of the other image processing device 120 and output from the output unit 229 via the transmission cable 103. The captured image acquired by the second image acquisition unit 224 is temporarily stored in the frame memory.

[0040] The second separation unit 226 reads out the captured image acquired by the third image acquisition unit 225 and stored in the frame memory, and separates the image region in the read captured image into a foreground region and a background region. That is, the second separation unit 226 performs foreground / background separation on the captured image. Furthermore, the second separation unit 226 generates a foreground image indicating the foreground region and a background image indicating the background region based on the foreground region and background region separated by foreground / background separation. The generated foreground image and background image are output to the output unit 229. The output unit 229 receives the foreground image and background image output from the second separation unit 226 and outputs the foreground image and background image to the image generation device 105.

[0041] Note that output unit 229 may output the foreground image and background image obtained as a result of the image processing by first separation unit 222 and the foreground image and background image obtained as a result of the image processing by second separation unit 226 via different transmission cables 130. With this configuration, it is possible to prevent the transmission of the foreground image and background image obtained as a result of the image processing by first separation unit 222 from being hindered by the transmission of the foreground image and background image obtained as a result of the image processing by second separation unit 226. Furthermore, second separation unit 226 may compress the generated foreground image and background image using lossless compression, similar to first separation unit 222, and output the compressed foreground image and background image to output unit 229. In this case, output unit 229 receives the compressed foreground image and background image output from second separation unit 226, and outputs the compressed foreground image and background image to image generation device 105.

[0042] Here, the accuracy of foreground / background separation performed by the second separation unit 226 differs from the accuracy of foreground / background separation performed by the first separation unit 222. Specifically, the foreground / background separation performed by the first separation unit 222 prioritizes processing speed, while the foreground / background separation performed by the second separation unit 226 prioritizes the accuracy of extraction of the foreground region. For example, the foreground / background separation performed by the second separation unit 226 is performed using an original CNN model. The foreground / background separation performed by the second separation unit 226 is not limited to using an original CNN model, and may be any method that can perform foreground / background separation with at least higher accuracy than the foreground / background separation performed by the first separation unit 222.

[0043] Furthermore, when both first separation unit 222 and second separation unit 226 compress foreground images, the foreground images may be compressed based on different foreground ratios. Specifically, the foreground ratio in second separation unit 226 is preferably a value greater than the foreground ratio in first separation unit 222, such as 100%. This is because the foreground / background separation in second separation unit 226 prioritizes the extraction accuracy of foreground regions, and therefore it is desirable to output foreground images corresponding to as many foreground regions as possible to image generation device 105. Note that when the foreground ratio is 100%, foreground images corresponding to all foreground regions separated by foreground / background separation are compressed, and the compressed foreground images are output to image generation device 105.

[0044] The following describes the functional configuration that functions when the image processing device 120 operates as an active image processing device 120 and when the image processing device 120 operates as a standby image processing device 120. When the image processing device 120 operates as an active image processing device 120, the first image acquisition unit 221, the first separation unit 222, the request acquisition unit 223, the second image acquisition unit 224, and the output unit 229 function as the functional configuration of the image processing device 120. On the other hand, when the image processing device 120 operates as a standby image processing device 120, the request acquisition unit 223, the third image acquisition unit 225, the second separation unit 226, and the output unit 229 function as the functional configuration of the image processing device 120.

[0045] <Functional configuration of the image generation device> The image generating device 105 acquires the foreground image and background image output from the image processing device 120 via the transmission cable 103, and generates a virtual viewpoint image using the acquired foreground image and background image. Hereinafter, the processing of each unit that the image generating device 105 has as a functional configuration will be described with reference to Fig. 2(b) and Fig. 3.

[0046] The acquisition unit 251 acquires foreground images and background images output from each of the image processing devices 120 included in the multiple camera units 102. The acquisition unit 251 stores the acquired foreground images and background images in the HDD 304 of the image generation device 105. The acquisition unit 251 also acquires viewpoint information indicating the position of a virtual viewpoint and the direction of the line of sight at the virtual viewpoint (hereinafter referred to as the "direction of the virtual viewpoint").

[0047] The viewpoint information is a parameter set including a parameter representing the three-dimensional position of the virtual viewpoint and a parameter representing the direction of a virtual line of sight at the virtual viewpoint, which may be expressed by the directions of pan, tilt, and roll. Note that the content of the viewpoint information is not limited to the above. For example, the parameter set serving as viewpoint information may include a parameter representing the size of the field of view (angle of view) at the virtual viewpoint. The viewpoint information may also have multiple parameter sets. For example, the viewpoint information may have multiple parameter sets corresponding to frames constituting a virtual viewpoint image as a moving image, and the viewpoint information may represent the position and direction of the virtual viewpoint at each of multiple consecutive points in time. For example, the user inputs the viewpoint information by operating the operation input device 309 of the image generation device 105. The operation input device 309 outputs the input viewpoint information to the controller unit 300 via the operation unit I / F 305, and the acquisition unit 251 acquires the same.

[0048] The generation unit 252 generates a virtual viewpoint image using the viewpoint information acquired by the acquisition unit 251 and the foreground image and background image stored in the HDD 304 of the image generation device 105. The virtual viewpoint image is generated, for example, by the following method. First, the generation unit 252 reads out the foreground image and background image stored in the HDD 304 of the image generation device 105, and acquires the foreground image and background image. The foreground image and background image have texture information such as color information. If the foreground image is compressed, the generation unit 252 decompresses it.

[0049] Next, the generation unit 252 uses the foreground image to generate a foreground model representing the three-dimensional shape of the object and texture data for coloring the foreground model. The generation unit 252 also uses the background image to generate texture data for coloring a background model representing the three-dimensional shape of the object that forms the background of a stadium or the like. The generation unit 252 then generates a virtual viewpoint image by mapping the texture data to the foreground model and the background model and performing rendering according to the position and direction of the virtual viewpoint indicated by the viewpoint information. The above-described method for generating a virtual viewpoint image is merely an example and is not limited thereto. For example, the generation unit 252 can use various methods for generating a virtual viewpoint image, such as generating a virtual viewpoint image by projective transformation of a captured image without using a foreground model or a background model.

[0050] The output unit 253 outputs the virtual viewpoint image generated by the generation unit 252 to the information processing device 106. Furthermore, the output unit 253 may output the virtual viewpoint image to the information processing device 106 and also to the display device 310 of the image generation device 105, causing the display device 310 to display the virtual viewpoint image. In this case, the user may change the position or direction of the virtual viewpoint by operating the operation input device 309 while viewing the virtual viewpoint image displayed on the display device 310. The output destination of the virtual viewpoint image by the output unit 253 is not limited to those described above. For example, the output unit 253 may output the virtual viewpoint image to the HDD 304 of the image generation device 105 and store the virtual viewpoint image in the HDD 304. Furthermore, for example, the output unit 253 may output the virtual viewpoint image to an external device at a remote location via the Internet or the like.

[0051] <Functional configuration of information processing device> The information processing device 106 acquires the virtual viewpoint image output from the image generation device 105, and issues a reprocessing request to the image processing device 120 based on the acquired virtual viewpoint image. The information processing device 106 may also control the image capture devices 110 and image processing devices 120 included in all of the camera units 102, and monitor the statuses of the image capture devices 110 and image processing devices 120 included in all of the camera units 102. Hereinafter, the processing of each unit that the information processing device 106 has as a functional configuration will be described with reference to Fig. 2(c) and Fig. 3.

[0052] The acquisition unit 261 acquires a virtual viewpoint image output from the image generation device 105. The evaluation unit 262 evaluates the image quality of the virtual viewpoint image acquired by the acquisition unit 261 and determines whether or not there is an abnormality in the image quality. The evaluation unit 262 outputs the result of the evaluation as an evaluation result of the image quality of the virtual viewpoint image. Specifically, the evaluation unit 262 extracts a foreground region from the virtual viewpoint image and determines whether or not there is a missing part of the image of a potential foreground object included in the foreground region based on the shape of the extracted foreground region, thereby determining whether or not there is an abnormality in the image quality of the virtual viewpoint image. For example, if the image of a potential foreground object included in the foreground region is an image of a natural person, the evaluation unit 262 determines whether or not there is a missing part of the image corresponding to a part of the natural person's body, such as the head, hands, feet, or torso. For example, the evaluation unit 262 determines whether or not there is an abnormality in the image quality of the virtual viewpoint image using a trained model or the like obtained as a result of learning by machine learning or the like.

[0053] The method for determining the image quality of the virtual viewpoint image in the evaluation unit 262 is not limited to the method using a trained model. For example, the evaluation unit 262 may determine the image quality of the virtual viewpoint image by inputting feature amounts extracted from the virtual viewpoint image into an algorithm based on predetermined rules. Furthermore, for example, the evaluation unit 262 may determine the image quality of the virtual viewpoint image based on an operation input from a user. Specifically, the user monitors the virtual viewpoint image displayed on the display device 310 of the image generation device 105. If the user determines that the image quality of the virtual viewpoint image is lower than the quality desired by the user, the user uses the operation input device 309 of the information processing device 106 to input information indicating that there is an abnormality in the image quality of the virtual viewpoint image. If the evaluation unit 262 acquires the information, it determines that there is an abnormality in the image quality of the virtual viewpoint image.

[0054] When the evaluation unit 262 determines that there is an abnormality in the image quality of the virtual viewpoint image, the identification unit 263 identifies the camera unit 102 (operating camera unit 102) including the operating image capture device 110 that captured the captured image causing the image quality abnormality. Specifically, first, the identification unit 263 identifies the three-dimensional position of an area where an object included as an image in the virtual viewpoint image determined to be abnormal exists. For example, the identification unit 263 identifies the three-dimensional position of an area where the object exists using a foreground model generated as an intermediate product when the image generation device 105 generates the virtual viewpoint image. Next, the identification unit 263 identifies the operating image capture device 110 whose imaging range includes at least a part of the three-dimensional position of the area where the identified object exists. Next, the identification unit 263 identifies the camera unit 102 (operating camera unit 102) including the identified image capture device 110 as the camera unit 102 causing the image quality abnormality of the virtual viewpoint image.

[0055] The imaging device 110 is arranged so that the angle of view of the imaging device 110 overlaps with the angle of view of at least one other imaging device 110. Therefore, in the identification process of the identification unit 263, at least two imaging devices 110 are identified, and at least two sets of camera units 102 are identified. Hereinafter, the operating camera unit 102 identified by the identification unit 263 will be referred to as the "transmission source camera unit 102," and the image processing device 120 included in the transmission source camera unit 102 will be referred to as the "transmission source image processing device 120."

[0056] The determination unit 264 determines, from among one or more standby image processing devices 120, an image processing device 120 that will execute foreground / background separation reprocessing instead of the source image processing device 120. Hereinafter, the standby image processing device 120 determined by the determination unit 264 will be referred to as the "destination image processing device 120." Details of how the determination unit 264 determines the destination image processing device 120 will be described later with reference to FIG. 8. The issuing unit 265 issues a reprocessing request to the source image processing device 120 and the destination image processing device 120.

[0057] <Reprocessing request> FIG. 4 is a diagram showing an example of information included in a reprocessing request issued by the information processing device 106 according to the first embodiment to the image processing device 120 at the transmission source (hereinafter referred to as a "reprocessing request for the transmission source"). As shown as an example in FIG. 4, the reprocessing request for the transmission source includes the camera ID of the transmission destination, a start time code, and an end time code. The camera ID of the transmission destination is an identification symbol assigned to uniquely identify the camera unit 102 including the image processing device 120 at the transmission destination. The camera ID is composed of, for example, a combination of one or more letters or numbers. The start time code indicates the time when an abnormality occurred in the image quality of the virtual viewpoint image. The end time code indicates the time when the abnormality in the image quality of the virtual viewpoint image was resolved. The start time code and the end time code are composed of, for example, HH (hours):MM (minutes):SS (seconds):FF (frames).

[0058] 5 is a diagram showing an example of information included in a reprocessing request issued by the information processing device 106 according to the first embodiment to the image processing device 120 at the transmission destination (hereinafter referred to as a "reprocessing request for transmission destination"). As shown as an example in FIG. 5, the reprocessing request for transmission destination includes the camera ID of the transmission source, a start time code, and an end time code. The camera ID of the transmission source is an identification code assigned to uniquely identify the camera unit 102 that includes the image processing device 120 at the transmission source. The start time code and end time code are the same as the start time code and end time code included in the reprocessing request for transmission source, and therefore description thereof will be omitted.

[0059] The reprocessing request for the destination may include information regarding foreground / background separation settings and compression settings, as shown as an example in FIG. 5. The foreground / background separation settings are setting values ​​for specifying the foreground / background separation method to be performed in the destination image processing device 120. For example, if the foreground / background separation settings in the reprocessing request for the destination specify the use of the original CNN model, which is the CNN model before optimization, the second separation unit 226 of the destination image processing device 120 performs foreground / background separation using the original CNN model. If the foreground / background separation method is not specified in the foreground / background separation settings in the reprocessing request for the destination, the second separation unit 226 of the destination image processing device 120 performs foreground / background separation using a predetermined method.

[0060] The compression setting is a setting value for specifying a performance index for the compression process performed in the destination image processing device 120. For example, if a foreground ratio of 100%, meaning that all foreground images are compressed, is specified in the compression setting of the destination reprocessing request, the second separation unit 226 of the destination image processing device 120 performs compression processing on all foreground images. If no foreground / background separation method is specified in the foreground / background separation setting of the destination reprocessing request, the second separation unit 226 of the destination image processing device 120 performs compression processing based on a predetermined foreground ratio.

[0061] <Image processing system operation> The operation of the image processing system will be described with reference to Figs. 6 and 7. Figs. 6 and 7 are sequence diagrams showing an example of a processing sequence of the image processing system according to the first embodiment. Specifically, the sequence diagram shown in Fig. 6 shows a processing sequence of the image processing system from when the image capture device 110 outputs a captured image until the image processing device 120 acquires a reprocessing request. Furthermore, the sequence diagram shown in Fig. 7 shows a processing sequence of the image processing system from when the image processing device 120 acquires a reprocessing request until the image generation device 105 generates a new virtual viewpoint image. In the following description, the image capture device 110 and image processing device 120 included in the operating camera unit 102 will be referred to as the image capture device 110a and image processing device 120a, and the image processing device 120 included in the standby camera unit 102 will be referred to as the image processing device 120b.

[0062] First, in S601, the imaging device 110a outputs captured images obtained by imaging to the image processing device 120a included in the same camera unit 102. The imaging device 110a repeatedly outputs the captured images at time intervals corresponding to a predetermined frame rate such as 60 fps, and the image processing system repeatedly executes the processing sequence shown in FIG. 6 each time the imaging device 110a outputs a captured image. After S601, in S602, the first image acquisition unit 221 of the image processing device 120a acquires the captured images output from the imaging device 110a in S601. The captured images acquired in S602 are temporarily stored in, for example, a frame memory of the image processing device 120a. Next, in S603, the first image acquisition unit 221 of the image processing device 120a saves the captured images acquired in S602 in a storage device such as the HDD 304.

[0063] Next, in S604, the first separation unit 222 of the image processing device 120a separates the image region in the captured image acquired in S602 into a foreground region and a background region to generate a foreground image and a background image. For example, the first separation unit 222 of the image processing device 120a acquires the captured image temporarily stored in the frame memory of the image processing device 120a in S602 by reading it from the frame memory, and separates the image region in the acquired captured image into a foreground region and a background region. After generating the foreground image and the background image, the first separation unit 222 of the image processing device 120a may perform a compression process using lossless compression to compress the generated foreground image and background image.

[0064] In order to complete the process within a predetermined period of time, the first separation unit 222 of the image processing device 120a performs foreground / background separation using, for example, an optimized CNN model that prioritizes processing speed. When compressing a foreground image, the first separation unit 222 of the image processing device 120a compresses the foreground image after setting the foreground ratio to a relatively small value, such as 30%, in order to reduce the amount of information transmitted. Hereinafter, the first separation unit 222 of the image processing device 120a will be described as compressing the generated foreground and background images.

[0065] Next, in S605, the output unit 229 of the image processing device 120a outputs the foreground image and background image generated and compressed in S604 to the image generation device 105. Specifically, in addition to the foreground image and background image, the output unit 229 of the image processing device 120a also outputs to the image generation device 105 supplementary information including the camera ID corresponding to the image processing device 120a and information indicating the image capture time of the captured image acquired in S602.

[0066] After S605, in S606, the acquisition unit 251 of the image generation device 105 acquires the foreground image, background image, and additional information output from the image processing device 120a in S605. If the foreground image and background image are compressed, for example, the acquisition unit 251 of the image generation device 105 decompresses the compressed foreground image and background image. Next, in S607, the acquisition unit 251 of the image generation device 105 stores the foreground image, background image, and additional information acquired in S606 in a storage device such as the HDD 304. Next, in S608, the acquisition unit 251 of the image generation device 105 acquires viewpoint information. Next, in S609, the generation unit 252 of the image generation device 105 generates a virtual viewpoint image using the foreground image, background image, and additional information acquired in S606 and the viewpoint information acquired in S608. The virtual viewpoint image generated in S609 is output to the display device 310 by the output unit 253 of the image generation device 105 and displayed thereon, for example.

[0067] Next, in S610, the output unit 253 of the image generating device 105 outputs the virtual viewpoint image generated in S609 and the viewpoint information acquired in S608 to the information processing device 106. The processing of S610 may be executed each time a series of processes from S601 to S609 is executed, or each time the series of processes is repeatedly executed a predetermined number of times, or each time the series of processes is repeatedly executed over a predetermined period of time.

[0068] After S610, in S611, the acquisition unit 261 of the information processing device 106 acquires the virtual viewpoint image and viewpoint information output from the image generation device 105 in S610. Next, in S612, the evaluation unit 262 of the information processing device 106 evaluates the virtual viewpoint image acquired in S611 and determines whether or not there is an abnormality in the image quality of the virtual viewpoint image. If it is determined in the evaluation of S612 that there is an abnormality in the image quality of the virtual viewpoint image, in S613, the identification unit 263 of the information processing device 106 identifies the camera unit 102 including the imaging device 110a that captured the captured image that is causing the abnormality in the image quality of the virtual viewpoint image.

[0069] After S613, in S614, the determination unit 264 of the information processing device 106 determines the image processing device 120a included in the camera unit 102 identified in S613 as the source image processing device 120. Also, in S614, the determination unit 264 of the information processing device 106 determines, from among one or more standby image processing devices 120b, the image processing device 120 that will perform foreground / background separation reprocessing instead of the source image processing device 120, as the destination image processing device 120b. Details of how the determination unit 264 of the information processing device 106 determines the destination image processing device 120b will be described later with reference to FIG. 8. After S614, in S615, the issuing unit 265 of the information processing device 106 issues a source reprocessing request to the image processing device 120a determined as the source image processing device 120 in S614. In addition, in S615, the issuing unit 265 of the information processing device 106 issues a reprocessing request for the destination to the image processing device 120b that was determined in S614 as the image processing device 120 of the destination.

[0070] After S615, in S616, the request acquisition unit 223 of the image processing device 120a that is the transmission source determined in S614 acquires the reprocessing request for the transmission source issued in S615 from the information processing device 106. Also, after S615, in S617, the request acquisition unit 223 of the image processing device 120b that is the transmission destination determined in S614 acquires the reprocessing request for the transmission destination issued in S615 from the information processing device 106.

[0071] After S616 and S617, in S701, the second image acquisition unit 224 of the image processing device 120a of the transmission source determined in S614 acquires the captured images saved in the storage device in S603 by, for example, reading them from the storage device. Specifically, the reprocessing request for the transmission source includes information on the start time code and the end time code. The second image acquisition unit 224 of the image processing device 120a of the transmission source refers to the reprocessing request for the transmission source acquired in S616 and acquires captured images corresponding to the period from the start time code to the end time code from the storage device. The second image acquisition unit 224 of the image processing device 120a of the transmission source temporarily stores the acquired captured images in a frame memory.

[0072] Next, in S702, the output unit 229 of the source image processing device 120a reads out the captured image stored in the frame memory and outputs the read-out captured image to the destination image processing device 120b. The source reprocessing request shown in Fig. 4 includes camera ID information that uniquely identifies the camera unit 102 that includes the destination image processing device 120b. The output unit 229 of the source image processing device 120a references the source reprocessing request and outputs the read-out captured image to the destination image processing device 120b included in the camera unit 102 that corresponds to the destination camera ID.

[0073] After S702, in S703, the third image acquisition unit 225 of the destination image processing device 120b acquires the captured image output from the source image processing device 120a in S702. The captured image acquired in S703 is temporarily stored in the frame memory of the destination image processing device 120b. Next, in S704, the second separation unit 226 of the destination image processing device 120b separates the image region in the captured image acquired in S703 into a foreground region and a background region to generate a foreground image and a background image. For example, the second separation unit 226 of the destination image processing device 120b reads out the captured image temporarily stored in the frame memory of the destination image processing device 120b in S703 from the frame memory, and separates the image region in the captured image into a foreground region and a background region. The second separation unit 226 of the destination image processing device 120b may perform a lossless compression process after generating the foreground image and the background image to compress the generated foreground image and background image.

[0074] In order to obtain a foreground image with higher accuracy, the second separation unit 226 of the destination image processing device 120b performs foreground-background separation using, for example, the original CNN model before optimization, which prioritizes processing accuracy. When compressing a foreground image, the second separation unit 226 of the destination image processing device 120b sets the foreground ratio to a relatively large value, such as 100%, in order to compress the foreground image corresponding to a larger foreground area. In the following description, the second separation unit 226 of the destination image processing device 120b will be described as compressing the generated foreground image and background image.

[0075] The reprocessing request for the transmission destination shown in Fig. 5 includes information specifying a foreground / background separation method. The second separation unit 226 of the image processing device 120b at the transmission destination may determine a foreground / background separation method by referring to the reprocessing request for the transmission destination acquired in S617, and may separate the image region in the captured image acquired in S703 into a foreground region and a background region based on the determination. The reprocessing request for the transmission destination shown in Fig. 5 also includes information specifying a foreground ratio to be used in the compression process using lossless compression. The second separation unit 226 of the image processing device 120b at the transmission destination may determine a foreground ratio to be used in the compression process using lossless compression by referring to the reprocessing request for the transmission destination acquired in S617, and may compress the foreground image based on the determination.

[0076] After S704, in S705, the output unit 229 of the destination image processing device 120b outputs the foreground image and background image generated and compressed in S704 to the image generation device 105. Specifically, the output unit 229 of the destination image processing device 120b outputs, in addition to the foreground image and background image, additional information including a camera ID and a time code to the image generation device 105. The additional information output in S705 is generated based on the reprocessing request for the destination acquired in S617. The camera ID of the additional information is set to the camera ID corresponding to the source image processing device 120a. Furthermore, the time code of the additional information is set to a value calculated in accordance with the number of frames read from the frame memory and the frame rate based on the start time code.

[0077] After S705, in S706, the acquisition unit 251 of the image generation device 105 acquires the foreground image and background image and the associated information output from the destination image processing device 120b in S705. If the foreground image and background image are compressed, for example, the acquisition unit 251 of the image generation device 105 decompresses the compressed foreground image and background image. Next, in S707, the acquisition unit 251 of the image generation device 105 stores the foreground image and background image acquired in S706 in a storage device such as the HDD 304 of the image generation device 105 based on the associated information acquired in S706. Specifically, the storage device contains the foreground image and background image previously saved in S607. The acquisition unit 251 of the image generation device 105 overwrites the already saved foreground image and background image, thereby replacing the foreground image and background image with the foreground image and background image acquired in S706.

[0078] Next, in S708, the acquisition unit 251 of the image generation device 105 acquires, by reading from the storage device, foreground and background images that are identical or substantially identical to the foreground and background images acquired in S706. Next, in S709, the acquisition unit 251 of the image generation device 105 acquires viewpoint information. Next, in S710, the generation unit 252 of the image generation device 105 generates a virtual viewpoint image using the foreground and background images acquired in S706, the foreground and background images acquired in S708, and the viewpoint information acquired in S609. The virtual viewpoint image generated in S710 is output to the display device 310 by the output unit 253 of the image generation device 105 and displayed, for example.

[0079] After S710, the image processing system ends the processing sequence shown in Fig. 7, and executes the processing sequence every time the processing of S615 to S617 is executed. Note that the processing sequence shown in Fig. 6 is repeatedly executed at predetermined time intervals even when the processing sequence shown in Fig. 7 is being executed.

[0080] <Method for determining the destination image processing device> FIG. 8 is a diagram illustrating an example of the process of determining the destination image processing device 120 by the determination unit 264 of the information processing device 106 according to the first embodiment, i.e., the process of S614 shown in FIG. 6. FIG. 8(a) is a diagram illustrating an example of a basic method of determining the destination image processing device 120. For example, in the case of the connection configuration shown in FIG. 8(a), the determination unit 264 determines the destination image processing device 120b as follows. Specifically, first, the determination unit 264 identifies the standby image processing device 120b that is closest to the source image processing device 120a among the three standby image processing devices 120b,x,z connected by the transmission cable 103. Next, the determination unit 264 determines the identified standby image processing device 120b as the destination image processing device 120.

[0081] FIG. 8(b) is a diagram illustrating a method for determining the destination image processing device 120b when the lane configuration has multiple lanes. A lane is a series of image processing devices 105 connected in a daisy chain. When the number of image processing devices 105 that can be connected to one lane is limited due to restrictions such as the bandwidth of the transmission cable 103, a lane configuration divided into multiple lanes may be adopted. For example, in the case of the connection configuration shown in FIG. 8(a), the determination unit 264 determines the destination image processing device 120b as follows. Specifically, first, the determination unit 264 identifies the standby image processing device 120b connected to the same lane as the source image processing device 120a from among the four standby image processing devices 120b,u,v,w. Next, the determination unit 264 determines the identified standby image processing device 120b as the destination image processing device 120b.

[0082] When there are multiple standby image processing devices 120 connected to the same lane as the image processing device 120a, the determination unit 264 identifies one image processing device 120b from among the multiple standby image processing devices 120 as follows. Specifically, the determination unit 264 identifies the standby image processing device 120b that is closest to the source image processing device 120a from among the multiple identified standby image processing devices 120. Then, the determination unit 264 determines the identified standby image processing device 120b as the destination image processing device 120. Furthermore, when there is no standby image processing device 120 in the same lane, the determination unit 264 determines one of three standby image processing devices 120u, v, w existing in different lanes as the destination image processing device 120.

[0083] The number of image processing devices 120 as the transmission destination determined by the determination unit 264 is not limited to one. When there are multiple standby image processing devices 120, the determination unit 264 may determine each of two or more standby image processing devices 120 among the multiple standby image processing devices 120 as the transmission destination image processing device 120. Determining two or more image processing devices 120 as the transmission destination image processing device 120 is effective when the number of frames to be reprocessed (the number of captured images) is extremely large, for example.

[0084] FIG. 8(c) is a diagram illustrating, as an example, a method for determining two of a plurality of standby image processing devices 120 as the destination image processing devices 120. FIG. 8(c) illustrates, as an example, a case where there are 200 frames to be reprocessed. That is, FIG. 8(c) illustrates a case where the source image processing device 120a acquires the 200 frames requiring reprocessing from a storage device and outputs them to the destination image processing device 120. In this case, the determination unit 264 determines the destination image processing device 120 as follows. Specifically, first, the determination unit 264 identifies two standby image processing devices 120b and 120c that are closest to the destination image processing device 120a among the plurality of standby image processing devices 120b and 120c. Next, the determination unit 264 determines each of the identified two standby image processing devices 120b and 120c as the destination image processing device 120. In this case, the image processing device 120a at the transmission destination outputs a frame group including 100 frames different from each other to each of the two image processing devices 120b and 120c at the transmission destination.

[0085] By determining multiple destination image processing devices 120 and outputting frames in this manner, distributed processing of foreground / background separation can be performed using multiple standby image processing devices 120. The determination unit 264 can determine the number of standby image processing devices 120 that will perform distributed processing according to the number of frames that require reprocessing, i.e., the number of destination image processing devices 120. The determination unit 264 can also determine all standby image processing devices 120 that are not performing or are not scheduled to perform reprocessing as destination image processing devices 120, and cause them to perform distributed processing of foreground / background separation.

[0086] Note that, if the determination unit 264 of the information processing device 106 determines multiple standby image processing devices 120 as destination image processing devices 120 in S614 shown in Fig. 6, the issuing unit 265 of the image processing device 120 executes the following process in S615 shown in Fig. 6. Specifically, in this case, in S615, the issuing unit 265 of the image processing device 120 issues multiple source and destination reprocessing requests according to the number of destination image processing devices 120 determined in S614.

[0087] For example, if the determination unit 264 of the information processing device 106 determines two standby image processing devices 120 as destination image processing devices 120 in S614, the issuing unit 265 of the image processing device 120 executes the following process in S615. Specifically, in this case, the issuing unit 265 of the image processing device 120 issues two source reprocessing requests to the source image processing device 120 in S615, for causing each of the two destination image processing devices 120 to output a captured image. Also, in S615, the issuing unit 265 of the image processing device 120 issues destination reprocessing requests to each of the two destination image processing devices 120, i.e., issues two destination reprocessing requests.

[0088] <Effects of image processing system> As described above, the image generation device 105 generates a virtual viewpoint image corresponding to the virtual viewpoint operated by the user using the operation input device 309. The virtual viewpoint image generated by the image generation device 105 is displayed, for example, on a large monitor installed on a back screen of the competition field 101. In this case, the virtual viewpoint image displayed on the monitor is generated with priority given to processing speed, i.e., the time required for display. Therefore, the image quality of the virtual viewpoint image is not guaranteed, and it may not be possible to display a highly accurate virtual viewpoint image. Specifically, for example, the virtual viewpoint image may lack part of the image of an object, or may contain noise if multiple objects are closely spaced.

[0089] According to the image processing system of the first embodiment, when there is a problem with image quality as described above, a reprocessing request is issued to the image processing devices 120 at the transmission source and the transmission destination, and data such as a foreground image and a background image stored in the image generation device 105 is replaced. After replacing the data, the image generation device 105 can generate a virtual viewpoint image with improved image quality by regenerating the virtual viewpoint image again. The virtual viewpoint image with improved image quality is generated when a replay, a highlight scene, or the like is displayed. When a replay, a highlight scene, or the like is displayed, a high-precision foreground image and background image have already been generated by the image processing device 120b at the transmission destination, and the data stored in the image generation device 105 has been replaced. Therefore, when a replay, a highlight scene, or the like is displayed, the image generation device 105 can instantly generate a high-precision virtual viewpoint image.

[0090] [Embodiment 2] The second embodiment will be described with reference to FIGS. 9 to 11. The configuration of the image processing system according to the second embodiment is the same as the configuration shown in FIG. 1. However, the functional configurations of the image processing device 120 and the information processing device 106 according to the second embodiment are different from those of the first embodiment. FIG. 9 is a block diagram showing an example of the functional configuration of the image processing device 120 and the information processing device 106 according to the second embodiment. Specifically, FIG. 9(a) shows an example of the functional configuration of the image processing device 120 according to the second embodiment, and FIG. 9(b) shows an example of the functional configuration of the information processing device 106 according to the second embodiment. The functional configuration of the image generation device 105 according to the second embodiment is the same as that of the first embodiment, and therefore a detailed description thereof will be omitted. Hereinafter, unless otherwise specified, the image processing device 120, the image generation device 105, and the information processing device 106 according to the second embodiment will be simply referred to as the image processing device 120, the image generation device 105, and the information processing device 106. Furthermore, in the second embodiment, the same configurations and processing steps as those of the first embodiment will be denoted by the same reference numerals, and redundant description will be omitted or simplified.

[0091] <Functional configuration of image processing device> The image processing device 120 has a first image acquisition unit 221, a first separation unit 222, an evaluation unit 921, a request acquisition unit 223, a second image acquisition unit 224, a third image acquisition unit 225, a second separation unit 226, and an output unit 929. Each functional configuration of the image processing device 120 is realized, for example, by a CPU 301, which the image processing device 120 has as a hardware configuration, executing an application program stored in an HDD 304 or the like, as shown as an example in FIG.

[0092] The evaluation unit 921 evaluates the foreground image generated by the first separation unit 222 or an intermediate generated image such as a silhouette image generated when the foreground image is generated. Specifically, the evaluation unit 921 evaluates the foreground image or the intermediate generated image to estimate whether or not the processing result of the first separation unit 222 will have an adverse effect on the image quality of the virtual viewpoint image generated by the image generation device 105, thereby determining whether or not to perform reprocessing of the foreground / background separation. Specifically, if it is estimated that the processing result of the first separation unit 222 will have an adverse effect on the image quality of the virtual viewpoint image generated by the image generation device 105, the evaluation unit 921 determines to perform reprocessing of the foreground / background separation.

[0093] In the following description, as an example, the evaluation unit 921 acquires a silhouette image corresponding to the captured image from the first separation unit 222 and evaluates it. A silhouette image is, for example, a 1-bit image in which an area corresponding to the foreground area in the captured image is expressed in white, and an area other than the foreground area, i.e., an area corresponding to the background area, is expressed in black. Details of a method for evaluating a silhouette image by the evaluation unit 921 will be described later with reference to FIG. 11.

[0094] If the evaluation unit 921 determines not to perform reprocessing, the output unit 929 outputs the foreground image, the background image, the supplementary information, etc., similar to the output unit 229 according to embodiment 1. On the other hand, if the evaluation unit 921 determines to perform reprocessing, the output unit 929 outputs the foreground image, the background image, the supplementary information, etc., and issues a reprocessing request issuance request (hereinafter simply referred to as an "issuance request") to the information processing device 106.

[0095] <Functional configuration of information processing device> The information processing device 106 has an acquisition unit 961, a determination unit 964, and an issuing unit 265. Each functional configuration of the information processing device 106 is realized by executing an application program stored in the HDD 304 or the like by a CPU 301, which is included in the hardware configuration of the information processing device 106, as shown in FIG. 3 as an example. The acquisition unit 961 acquires an issuance request issued from the image processing device 120.

[0096] The determination unit 964 determines the image processing device 120 to which the foreground / background separation reprocessing is to be performed from among one or more standby image processing devices 120b, based on the issuance request acquired by the acquisition unit 961. Furthermore, based on a camera ID included as additional information in the issuance request acquired by the acquisition unit 961 from the image processing device 120, the determination unit 964 determines the image processing device 120 included in the camera unit 102 indicated by the camera ID as the image processing device 120 from which the image processing device 120 is to be transmitted.

[0097] Note that, since it is obvious that the image processing device 120 that issued the issuance request will be the image processing device 120 of the transmission source, the information processing device 106 does not need to specify the image processing device 120 of the transmission source as in embodiment 1. Furthermore, when determining the image processing device 120 of the transmission destination, the determination unit 964 excludes image processing devices 120 that are currently executing foreground / background separation reprocessing based on another reprocessing request that was issued earlier, as in embodiment 1. Therefore, even if the information processing device 106 may simultaneously receive issuance requests from multiple image processing devices 120, it will not determine the same standby image processing device 120b as the image processing device 120 of the transmission destination.

[0098] The issuing unit 265 issues a reprocessing request to each of the source image processing device 120 and the destination image processing device 120. The issuing unit 265 according to the second embodiment generates a reprocessing request for the source image processing device and a reprocessing request for the destination image processing device based on the additional information included in the issuance request acquired by the acquiring unit 961.

[0099] <Image processing system operation> The operation of the image processing system will be described with reference to Fig. 10. Fig. 10 is a sequence diagram showing an example of a processing sequence of the image processing system according to the second embodiment. Specifically, the sequence diagram shown in Fig. 10 shows the processing sequence of the image processing system from when the imaging device 110 outputs a captured image to when the image processing device 120 acquires a reprocessing request. First, the imaging device 110a executes the processing of S601. Note that the imaging device 110a repeatedly outputs captured images at predetermined time intervals, and the image processing system repeatedly executes the processing sequence shown in Fig. 10 every time the imaging device 110a outputs a captured image. Next, the image processing device 120a executes the processing from S602 to S605. Next, the image generation device 105 executes the processing from S606 to S609.

[0100] Next, in S1011, the evaluation unit 921 of the image processing device 120a evaluates the silhouette image generated as an intermediate product in the process of S604 to determine whether or not to reprocess foreground / background separation. If it is determined in the evaluation of S1011 that foreground / background separation is to be reprocessed, in S1102, the output unit 929 of the image processing device 120a issues an issuance request to the information processing device 106. Note that in this embodiment, the processes of S1011 and S1012 are described as being executed after the process of S609, but the processes of S1011 and S1012 and the processes from S606 to S609 may be executed in parallel. Also, in this embodiment, the processes of S1011 and S1012 are described as being executed after the process of S605, but the process of S1011 may be executed before the process of S605 or in parallel with the process of S605 as long as it is executed after the process of S604. Furthermore, the process of S1012 may be executed before the process of S605, provided that it is executed after S1011.

[0101] Next, in S1013, the acquisition unit 961 of the information processing device 106 acquires the issuance request issued in S1012. Next, in S1014, the determination unit 964 of the information processing device 106 determines the image processing device 120 as the sender and the image processing device 120 as the destination, based on the issuance request acquired in S1013. After S1014, the issuer 265 of the information processing device 106 executes the process of S615. Next, the image processing device 120a executes the process of S616, and the image processing device 120b executes the process of S617. The processing sequence of the image processing system from when the image processing device 120 acquires the reprocessing request to when the image generation device 105 newly generates and outputs a virtual viewpoint image is the same as that of the image processing system according to the first embodiment shown as an example in FIG. 7, and therefore description thereof will be omitted.

[0102] <Silhouette image evaluation method> 11A and 11B are diagrams showing an example of a silhouette image generated as an intermediate product of foreground / background separation in the first separation unit 222. Specifically, FIG. 11A shows an example of a silhouette image in which multiple objects exist densely. FIG. 11B shows an example of a silhouette image in which multiple objects exist discretely. In FIG. 11, white areas represent areas corresponding to the foreground area in the captured image, and black areas represent areas corresponding to the background area in the captured image. An example of a method for evaluating a silhouette image in the evaluation unit 921 will be described with reference to FIG. 11.

[0103] When multiple objects are densely packed, as shown in FIG. 11(a) as an example, the foreground regions corresponding to the multiple objects overlap, resulting in a larger foreground region than the foreground region corresponding to a single object. When the foreground regions corresponding to multiple objects overlap, it is difficult to accurately separate and identify the foreground regions for each object, which can adversely affect the generation of a foreground model in the image generating device 105. If the image generating device 105 does not generate a high-precision foreground model, the image quality of the virtual viewpoint image will deteriorate. Therefore, for example, if the size of the white region in the silhouette image is larger than a predetermined threshold, the evaluation unit 921 determines that the foreground regions corresponding to the multiple objects overlap and determines that foreground-background separation needs to be reprocessed. Here, the threshold can be determined based on, for example, the size of a single object that could be in the foreground.

[0104] As described in the first embodiment, the first separation unit 222 may perform lossless compression on the foreground image based on a predetermined foreground ratio, such as 30%. The following description will be made assuming that the first separation unit 222 performs lossless compression on the foreground image based on the predetermined foreground ratio. In this case, if the ratio of white areas to the image area in the silhouette image exceeds 30%, as shown in FIG. 11B as an example, at least a portion of the foreground image will not be output to the image generation device 105. This will adversely affect the generation of foreground models corresponding to some objects in the image generation device 105. Therefore, for example, if the ratio of white areas to the image area in the silhouette image is greater than the predetermined foreground ratio, the evaluation unit 921 determines that some of the foreground image will not be output to the image generation device 105 and that foreground / background separation needs to be reprocessed. Here, when performing the foreground / background separation reprocessing, the foreground ratio during lossless compression is set to a larger value, such as 100%, so that more or all of the foreground image will be output to the image generation device 105.

[0105] <Effects of image processing system> As described above, the operating image processing device 120a evaluates the silhouette image or foreground image to determine whether or not to reprocess foreground / background separation in the standby image processing device 120b. Specifically, the operating image processing device 120a estimates the impact of the silhouette image or foreground image on the image quality of the virtual viewpoint image generated by the image generation device 105. This allows the operating image processing device 120a to determine whether or not to request the information processing device 106 to issue a reprocessing request for reprocessing foreground / background separation in the standby image processing device 120b. According to the image processing system configured in this manner, the impact on the image quality of a virtual viewpoint image that may be generated is estimated before the virtual viewpoint image is generated, so the reprocessing request can be issued more quickly compared to when the image quality of the virtual viewpoint image is evaluated. As a result, the time required to replace the foreground image and the background image is shortened, and a high-quality virtual viewpoint image can be displayed as a replay, a highlight scene, or the like, at an earlier timing.

[0106] [Modification of the second embodiment] In the second embodiment, the image generating device 105 evaluates the foreground image or the silhouette image. However, the evaluation of the foreground image or the silhouette image may be performed by the image generating device 105 or the information processing device 106. For example, when the image generating device 105 evaluates the foreground image, the image generating device 105 evaluates the foreground image output from the image processing device 120a in operation. If the image generating device 105 determines, as a result of the evaluation, to perform foreground / background separation reprocessing, the image generating device 105 may issue an issuance request to the information processing device 106. Also, for example, when the image generating device 105 evaluates the silhouette image, the image processing device 120a in operation outputs the silhouette image to the image generating device 105 together with the foreground image and the background image. The image generating device 105 evaluates the silhouette image output from the image processing device 120a in operation. If the image generating device 105 determines, as a result of the evaluation, to perform foreground / background separation reprocessing, the image generating device 105 may issue an issuance request to the information processing device 106.

[0107] Furthermore, for example, when the information processing device 106 evaluates a foreground image or a silhouette image, the operating image processing device 120a outputs the foreground image and background image to the information processing device 106, and also outputs the foreground image or silhouette image to the information processing device 106. The information processing device 106 evaluates the foreground image or silhouette image output from the operating image processing device 120a. If, as a result of this evaluation, the information processing device 106 determines that foreground / background separation should be reprocessed, the information processing device 106 may issue a reprocessing request. In this case, it is not necessary to issue a reprocessing request to the information processing device 106.

[0108] [Other embodiments] In the above embodiment, the imaging device 110 and the image processing device 120 included in the camera unit 102 are described as being different devices, but the aspect of the camera unit 102 is not limited to this. For example, the imaging device 110 may have some or all of the functions of the image processing device 120. Furthermore, for example, the imaging device 110 may have the image processing device 120 built therein.

[0109] Furthermore, in the above-described embodiment, the camera unit 102 has been described as including one imaging device 110 and one image processing device 120, but the configuration of the camera unit 102 is not limited to this. For example, the camera unit 102 may include a plurality of imaging devices 110, and may include a plurality of image processing devices 120. In this case, each of the one or more imaging devices 110 included in the camera unit 102 and each of the one or more image processing devices 120 included in the camera unit 102 are connected to each other so that they can communicate with each other.

[0110] Furthermore, in the above-described embodiment, the image generation device 105 and the information processing device 106 are described as being different devices, but the aspects of the image generation device 105 and the information processing device 106 are not limited to this. For example, the image generation device 105 or the information processing device 106 may be realized as a single device having both the functions of the image generation device 105 and the functions of the information processing device 106.

[0111] The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0112] It should be noted that within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0113] [Configuration of the present disclosure] <Configuration 1> An image processing device that performs processing in cooperation with other image processing devices in an image processing system, operating in a first mode of operation; a first image acquisition means for acquiring a captured image output from an imaging device; a first image processing means for performing first image processing on the captured image acquired by the first image acquisition means; a first output means for outputting information relating to a result of the first image processing by the first image processing means; a second image acquisition means for acquiring the captured image from a storage device; a second output means for outputting the captured image acquired by the second image acquisition means to the other image processing device; operating in a second operating mode; a third image acquisition means for acquiring the captured image output from the second output means of the other image processing device; a second image processing means for performing second image processing on the captured image acquired by the third image acquisition means; a third output means for outputting information relating to a result of the second image processing by the second image processing means; 1. An image processing device comprising:

[0114] <Configuration 2> the first operation mode is an operation mode that is selected when the imaging device is in an operating state, the second operation mode is an operation mode selected when the imaging device is in a standby state; 2. The image processing device according to configuration 1,

[0115] <Configuration 3> the second output means outputs the captured image to the other image processing device operating in the second operation mode, the third image acquisition means acquires the captured image output from the other image processing device operating in the first operation mode; 3. The image processing device according to configuration 1 or 2, characterized in that:

[0116] <Configuration 4> a request acquisition means for acquiring a reprocessing request for the first image processing issued from an external device; and the second output means determines the other image processing device as an output destination of the captured image based on the reprocessing request acquired by the request acquisition means; 4. The image processing device according to any one of configurations 1 to 3, characterized in that:

[0117] <Configuration 5> the second image acquisition means determines the captured image to be acquired based on the reprocessing request acquired by the request acquisition means; 5. The image processing device according to configuration 4,

[0118] <Configuration 6> the second image processing means determines a method of the second image processing based on the reprocessing request acquired by the request acquisition means; 6. The image processing device according to configuration 4 or 5,

[0119] <Configuration 7> the reprocessing request is issued based on information about the result of the first image processing output from the first output means; 7. The image processing device according to any one of configurations 4 to 6,

[0120] <Configuration 8> the reprocessing request is issued based on an evaluation result of a virtual viewpoint image generated using information about the result of the first image processing output from the first output means; 8. The image processing device according to configuration 7,

[0121] <Configuration 9> the reprocessing request is issued based on an evaluation result of a size or shape of a foreground image indicating a foreground region in the captured image, which is included in information on the result of the first image processing output from the first output means; 8. The image processing device according to configuration 7,

[0122] <Configuration 10> the first output means outputs information relating to a result of the first image processing and an intermediate product of the first image processing; the reprocessing request is issued based on an evaluation result of the intermediate product in the first image processing; 7. The image processing device according to any one of configurations 4 to 6,

[0123] <Configuration 11> an evaluation means, operating in the first operation mode, for evaluating a result of the first image processing or an intermediate product of the first image processing; and the first output means outputs information relating to a result of the first image processing and information relating to an evaluation result by the evaluation means; the reprocessing request is a request issued based on information about the evaluation result output from the first output means; 7. The image processing device according to any one of configurations 4 to 6,

[0124] <Configuration 12> the intermediate product of the first image processing is a silhouette image showing a foreground region in the captured image; 12. The image processing device according to configuration 10 or 11,

[0125] <Configuration 13> the evaluation of the intermediate product is an evaluation of the size or shape of the foreground region shown in the silhouette image; 13. The image processing device according to configuration 12,

[0126] <Configuration 14> The other image processing device to which the captured image is output by the second output means is determined based on the location of the image processing device itself on a network. 14. The image processing device according to any one of configurations 1 to 13,

[0127] <Configuration 15> the second output means outputs the captured image to the other image processing device that is closer to a location of the image processing device itself on the network; 15. The image processing device according to configuration 14,

[0128] <Configuration 16> the second output means, when the network is divided into a plurality of lanes and the other image processing device is present on the same lane as the lane in which the image processing device of the own device is present, outputs the captured image to the other image processing device on the same lane; 16. The image processing device according to configuration 14 or 15,

[0129] <Configuration 17> the second output means outputs the captured image to the other image processing device that is closer to a position of the image processing device of the own device on the same lane; 17. The image processing device according to configuration 16,

[0130] <Configuration 18> when there are a plurality of other image processing devices capable of outputting the captured image and there are a plurality of frames in the captured image to be output, the second output means divides the plurality of frames to be output into two or more frame groups each including one or more frames, and outputs each of the two or more frame groups to a different one of the other image processing devices; 18. The image processing device according to any one of configurations 1 to 17,

[0131] <Configuration 19> the second output means divides the plurality of frames into the two or more frame groups, each including one or more frames, based on the number of the plurality of frames, and outputs each of the two or more frame groups to the other image processing device different from each other; 19. The image processing device according to configuration 18,

[0132] <Configuration 20> a transmission path used by the first output means to transmit information and a transmission path used by the second output means to transmit information are different from each other; 20. The image processing device according to any one of configurations 1 to 19, characterized in that:

[0133] <Configuration 21> the second image processing is image processing with higher accuracy than the first image processing; 21. The image processing device according to any one of configurations 1 to 20,

[0134] <Configuration 22> the first image processing includes processing for generating a foreground image corresponding to a foreground region in the captured image; 22. The image processing device according to any one of configurations 1 to 21,

[0135] <Configuration 23> the first image processing includes a compression process of a foreground image generated in a generation process included in the first image processing; 23. The image processing device according to configuration 22,

[0136] <Configuration 24> An image processing system including a plurality of image processing devices, Each of the plurality of image processing devices operating in a first mode of operation; a first image acquisition means for acquiring a captured image output from an imaging device; a first image processing means for performing first image processing on the captured image acquired by the first image acquisition means; a first output means for outputting information relating to a result of the first image processing by the first image processing means; a second image acquisition means for acquiring the captured image from a storage device; a second output means for outputting the captured image acquired by the second image acquisition means to another image processing device; operating in a second operating mode; a third image acquisition means for acquiring the captured image output from the second output means of the other image processing device; a second image processing means for performing second image processing on the captured image acquired by the third image acquisition means; a third output means for outputting information relating to a result of the second image processing by the second image processing means; having An image processing system comprising:

[0137] <Configuration 25> Further including an information processing device, Each of the plurality of image processing devices a request acquisition means for acquiring a reprocessing request for the first image processing issued from the information processing device; and the second output means determines the other image processing device as an output destination of the captured image based on the reprocessing request acquired by the request acquisition means; 25. The image processing system according to configuration 24,

[0138] <Method> A control method for an image processing device that performs processing in cooperation with other image processing devices in an image processing system, comprising: performed in a first mode of operation, a first image acquisition step of acquiring a captured image output from an imaging device; a first image processing step of performing first image processing on the captured image acquired in the first image acquisition step; a first output step of outputting information relating to a result of the first image processing by the first image processing step; a second image acquisition step of acquiring the captured image from a storage device; a second output step of outputting the captured image acquired in the second image acquisition step to the other image processing device; performed in a second operating mode, a third image acquisition step of acquiring the captured image output from the second output step of the other image processing device; a second image processing step of performing second image processing on the captured image acquired in the third image acquisition step; a third output step of outputting information relating to a result of the second image processing by the second image processing step; A control method comprising:

[0139] <Program> A program for causing a computer to function as the image processing device according to any one of configurations 1 to 23. [Explanation of symbols]

[0140] 110 Imaging device 120 Image Processing Device 106 Information processing equipment 304 HDD 221 First Image Acquisition Unit 222 1st separation section 224 Second Image Acquisition Unit 225 Third Image Acquisition Unit 226 2nd separation section 229 Output Section

Claims

1. An image processing device that performs processing in cooperation with other image processing devices in an image processing system, operating in a first mode of operation; a first image acquisition means for acquiring a captured image output from an imaging device; a first image processing means for performing first image processing on the captured image acquired by the first image acquisition means; a first output means for outputting information relating to a result of the first image processing by the first image processing means; a second image acquisition means for acquiring the captured image from a storage device; a second output means for outputting the captured image acquired by the second image acquisition means to the other image processing device; operating in a second mode of operation; a third image acquisition means for acquiring the captured image output from the second output means of the other image processing device; a second image processing means for performing second image processing on the captured image acquired by the third image acquisition means; a third output means for outputting information relating to a result of the second image processing by the second image processing means; 1. An image processing device comprising:

2. the first operation mode is an operation mode selected when the imaging device is in an operating state; the second operation mode is an operation mode selected when the imaging device is in a standby state; 2. The image processing device according to claim 1, wherein:

3. the second output means outputs the captured image to the other image processing device operating in the second operation mode, the third image acquisition means acquires the captured image output from the other image processing device operating in the first operation mode; 2. The image processing device according to claim 1, wherein:

4. a request acquisition means for acquiring a reprocessing request for the first image processing issued from an external device; and the second output means determines the other image processing device as an output destination of the captured image based on the reprocessing request acquired by the request acquisition means; 2. The image processing device according to claim 1, wherein:

5. the second image acquisition means determines the captured image to be acquired based on the reprocessing request acquired by the request acquisition means; 5. The image processing device according to claim 4, wherein:

6. the second image processing means determines a method of the second image processing based on the reprocessing request acquired by the request acquisition means; 5. The image processing device according to claim 4, wherein:

7. the reprocessing request is issued based on information about the result of the first image processing output from the first output means; 5. The image processing device according to claim 4, wherein:

8. the reprocessing request is issued based on an evaluation result of a virtual viewpoint image generated using information about the result of the first image processing output from the first output means; 8. The image processing device according to claim 7,

9. the reprocessing request is issued based on an evaluation result of a size or shape of a foreground image indicating a foreground region in the captured image, the evaluation result being included in information on the result of the first image processing output from the first output means; 8. The image processing device according to claim 7,

10. the first output means outputs information relating to a result of the first image processing and an intermediate product of the first image processing; the reprocessing request is issued based on an evaluation result of the intermediate product in the first image processing; 5. The image processing device according to claim 4, wherein:

11. an evaluation means for evaluating a result of the first image processing or an intermediate product of the first image processing, the evaluation means operating in the first operation mode; and the first output means outputs information relating to a result of the first image processing and information relating to an evaluation result by the evaluation means; the reprocessing request is a request issued based on information about the evaluation result output from the first output means; 5. The image processing device according to claim 4, wherein:

12. the intermediate product of the first image processing is a silhouette image showing a foreground region in the captured image; The image processing device according to claim 10,

13. the evaluation of the intermediate product is an evaluation of the size or shape of the foreground region shown in the silhouette image; The image processing device according to claim 12,

14. The other image processing device to which the captured image is output by the second output means is determined based on the location of the image processing device itself on a network.

2. The image processing device according to claim 1, wherein:

15. the second output means outputs the captured image to the other image processing device that is closer to a location of the image processing device itself on the network; The image processing device according to claim 14,

16. the second output means, when the network is divided into a plurality of lanes and the other image processing device is present on the same lane as the image processing device of the own device, outputs the captured image to the other image processing device on the same lane; The image processing device according to claim 14,

17. the second output means outputs the captured image to the other image processing device that is closer to a position of the image processing device of the own device on the same lane; The image processing device according to claim 16,

18. the second output means, when there are a plurality of other image processing devices capable of outputting the captured image and there are a plurality of frames in the captured image to be output, divides the plurality of frames to be output into two or more frame groups, each including one or more frames, and outputs each of the two or more frame groups to a different one of the other image processing devices; 2. The image processing device according to claim 1, wherein:

19. the second output means divides the plurality of frames into the two or more frame groups, each including one or more frames, based on the number of the plurality of frames, and outputs each of the two or more frame groups to the other image processing device different from each other; The image processing device according to claim 18,

20. a transmission path used by the first output means for transmitting information and a transmission path used by the second output means for transmitting information are different from each other; 2. The image processing device according to claim 1, wherein:

21. the second image processing is image processing with higher accuracy than the first image processing; 2. The image processing device according to claim 1, wherein:

22. the first image processing includes processing for generating a foreground image corresponding to a foreground region in the captured image; 2. The image processing device according to claim 1, wherein:

23. the first image processing includes a compression process of a foreground image generated in a generation process included in the first image processing; The image processing device according to claim 22,

24. An image processing system including a plurality of image processing devices, Each of the plurality of image processing devices operating in a first mode of operation; a first image acquisition means for acquiring a captured image output from an imaging device; a first image processing means for performing first image processing on the captured image acquired by the first image acquisition means; a first output means for outputting information relating to a result of the first image processing by the first image processing means; a second image acquisition means for acquiring the captured image from a storage device; a second output means for outputting the captured image acquired by the second image acquisition means to another image processing device; performed in a second mode of operation, a third image acquisition means for acquiring the captured image output from the second output means of the other image processing device; a second image processing means for performing second image processing on the captured image acquired by the third image acquisition means; a third output means for outputting information relating to a result of the second image processing by the second image processing means; having An image processing system comprising:

25. Further including an information processing device, Each of the plurality of image processing devices a request acquisition unit that acquires a reprocessing request for the first image processing issued by the information processing device; and the second output means determines the other image processing device as an output destination of the captured image based on the reprocessing request acquired by the request acquisition means; 25. The image processing system according to claim 24,

26. A control method for an image processing device that performs processing in cooperation with other image processing devices in an image processing system, comprising: performed in a first mode of operation, a first image acquisition step of acquiring a captured image output from an imaging device; a first image processing step of performing first image processing on the captured image acquired in the first image acquisition step; a first output step of outputting information relating to a result of the first image processing by the first image processing step; a second image acquisition step of acquiring the captured image from a storage device; a second output step of outputting the captured image acquired in the second image acquisition step to the other image processing device; operating in a second mode of operation; a third image acquisition step of acquiring the captured image output from the second output step of the other image processing device; a second image processing step of performing second image processing on the captured image acquired in the third image acquisition step; a third output step of outputting information relating to a result of the second image processing by the second image processing step; A control method comprising:

27. A program for causing a computer to function as the image processing device according to any one of claims 1 to 23.

Images