Image processing device, image processing method, program, and image processing system

Abstract

To provide an image processing device, program, and image processing system, which reduce the image transmission load.SOLUTION: A method of computing a circumscribing rectangle comprises detecting a foreground area 301 from an input image. A circumscribing rectangle 304 is computed for each of a plurality of divided areas 303 obtained by dividing one foreground area. Data on the input image within the circumscribing rectangles is sent to the outside.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image processing method, a program, and an image processing system, and in particular to a technology for generating a virtual viewpoint image. [Background technology]

[0002] A technology that generates a virtual viewpoint image from a specified virtual viewpoint using multiple images captured by multiple imaging devices has been attracting attention. For example, Patent Document 1 describes that multiple cameras are installed at different positions and that images captured by each camera are input to a virtual viewpoint image generating device. Patent Document 1 also describes that a virtual viewpoint image is generated by restoring the three-dimensional shape of an object based on the input images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-45920 Summary of the Invention [Problem to be solved by the invention]

[0004] To generate a high-quality virtual viewpoint image, large, high-resolution digital images are handled. Therefore, to generate a high-quality virtual viewpoint image using the technology described in Patent Document 1, the transmission load of the images from the cameras to the virtual viewpoint image generation device becomes heavy. In particular, if the transmission load increases further due to the large number of cameras, there is a possibility that the transmission itself will become impossible.

[0005] The present disclosure aims to reduce the load of image transmission. [Means for solving the problem]

[0006] An image processing apparatus according to an embodiment of the present disclosure has the following configuration. From the input image The scenery A detecting means for detecting before scenery The foreground mask image corresponding to of , into a plurality of divided regions at a first interval in the horizontal direction and a second interval in the vertical direction. Split A dividing means; The aforementioned Multiple division regions Of of For a divided region including a part of the foreground, a region circumscribing the part of the foreground is A calculation means for calculating a circumscribing rectangle; the circumscribed rectangle corresponds to data Send a transmitting means for receiving the It has. [Effects of the Invention]

[0007] The image transmission load can be reduced. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image processing system according to an embodiment. [Figure 2] FIG. 1 is a diagram showing an example of the hardware configuration of an image processing apparatus according to an embodiment. [Figure 3] FIG. 10 is a diagram showing an example of a method for calculating a circumscribing rectangle. [Figure 4] 1 is a flowchart of an image processing method according to an embodiment. [Figure 5] 1 is a flowchart of an image processing method according to an embodiment. [Figure 6] 1 is a flowchart of an image processing method according to an embodiment. [Figure 7] FIG. 1 is a diagram showing an example of the configuration of an image processing system according to an embodiment. [Figure 8] FIG. 10 is a diagram showing an example of a method for calculating a circumscribing rectangle. [Figure 9] FIG. 1 is a diagram showing an example of the configuration of an image processing system according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claims. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] [Image processing system overview] FIG. 1 is a diagram illustrating an example of the configuration of an image processing system according to an embodiment. The image processing system illustrated in FIG. 1 can generate a virtual viewpoint image. The virtual viewpoint image generated by this image processing system is also called a free viewpoint image. By generating a virtual viewpoint image, it is possible to monitor an image corresponding to a viewpoint freely (arbitrarily) designated by the user. The virtual viewpoint image may be an image corresponding to a viewpoint selected by the user from a limited number of virtual viewpoint candidates. The virtual viewpoint may be designated by a user operation or automatically. The automatic designation of a virtual viewpoint image can be performed based on the results of image analysis using, for example, AI (machine learning). The image processing system may generate still images or may create videos. Such videos (virtual viewpoint videos) are represented by multiple virtual viewpoint images.

[0011] The virtual viewpoint image is generated, for example, by the following method. First, multiple imaging devices 150 (cameras) capture images of an imaging area including a subject from multiple directions. The imaging area may be, for example, an area at any height extending upward from the field of a stadium. The imaging area may also be any area, such as a concert venue or an imaging studio. The imaging area may correspond to a three-dimensional space from which the three-dimensional shape of the subject is to be estimated. This three-dimensional space may be the entire imaging area or a part of the imaging area. The image processing system estimates the three-dimensional shape of the subject located in such a three-dimensional space.

[0012] The multiple imaging devices 150 are installed at different positions surrounding the imaging area and capture images in synchronization. The multiple imaging devices 150 can be installed so that they each have a different imaging direction (attitude). The multiple imaging devices 150 can be installed, for example, around the entire periphery of the imaging area. On the other hand, the area in which the multiple imaging devices 150 are installed may be located in a limited direction relative to the imaging area. When the installation location is limited, the multiple imaging devices 150 can be arranged in this manner. The multiple imaging devices 150 are synchronized according to a common time. Furthermore, imaging time information is added to each frame of image captured by the imaging device 150.

[0013] The number of imaging devices 150 is not particularly limited. For example, if the imaging area is a rugby stadium, tens to hundreds of imaging devices 150 can be installed around the stadium. Furthermore, the multiple imaging devices 150 may include cameras with different angles of view. For example, the imaging devices 150 may include a wide-angle camera and a telephoto camera. Capturing high-resolution images of players using a telephoto camera can improve the resolution of the generated virtual viewpoint image. On the other hand, capturing images using a wide-angle camera can reduce the number of cameras. In particular, when filming ball games, using a wide-angle camera is effective because the ball moves over a wide range. Furthermore, combining the imaging area captured by a wide-angle camera and the imaging area captured by a telephoto camera improves the flexibility of the installation location of the imaging devices 150.

[0014] The image processing device 100 detects a foreground region from an input image. In this example, the input image is a captured image obtained by the imaging device 150. That is, the imaging device 150 inputs the captured image to the image processing device 100. Note that the imaging device 150 may capture a moving image. In this case, the input image is an image of one frame of the moving image.

[0015] The foreground region is the region of the subject in the input image. In this embodiment, the subject refers to a dynamic subject (moving object) that moves (its position and shape may change) when images are captured from the same direction in a time series. The subject is, for example, a person. For example, in a sporting event, the subject includes people such as players or referees on the field. In a concert or entertainment event, the subject includes singers, musicians, performers, or presenters. Furthermore, the subject can include an object such as a ball. For example, the subject can include an object used in sports (a ball in a ball game). Furthermore, the image processing device 100 can extract a foreground image that includes the foreground region from the input image.

[0016] The image processing device 100 can further extract a background image in a background region from the input image. The background region is a region other than the foreground region. A background image is an image of at least a region different from the foreground subject. For example, a background image can be obtained by removing the foreground subject from a captured image. The background region of such an input image includes the background. The background refers to an imaged object that remains stationary or nearly stationary when images are captured from the same direction in chronological order. Examples of such imaged objects include a stage for a concert, a stadium where an event such as a sport is held, a structure such as a goal used in a ball game, or a field.

[0017] It should be noted that the image capture target may include other objects in addition to the subject and background. The foreground image and background image also have texture information (color information, etc.).

[0018] The generating device 200 generates a virtual viewpoint image based on the foreground image extracted by the image processing device 100. In this embodiment, the generating device 200 generates a foreground model representing the three-dimensional shape of the subject and texture data used to color the foreground model based on the foreground image. The generating device 200 then maps the texture data to the foreground model. Furthermore, the generating device 200 renders the foreground model according to the virtual viewpoint indicated by the virtual viewpoint information. In this manner, a virtual viewpoint image is generated. Note that the method for generating a virtual viewpoint image is not limited to this method. For example, for each position in the subject area on the virtual viewpoint image, a corresponding three-dimensional position on the subject can be determined. Then, the color of the foreground image corresponding to this three-dimensional position can be extracted as the color of each position in the subject area.

[0019] The generation device 200 can also generate texture data for coloring a background model based on the background image. The background model can represent a three-dimensional shape of a background, such as a stadium. In this case, the generation device 200 can generate a virtual viewpoint image by rendering the foreground model and the background model.

[0020] The configuration of the image processing system, for example, the number and arrangement of the imaging devices 150, image processing devices 100, and generation devices 200, are not particularly limited. For example, the imaging devices 150, image processing devices 100, and generation devices 200 may be built into the same housing or may be realized as physically separate devices. For example, the imaging devices 150, image processing devices 100, and generation devices 200 may be separate devices connected to each other via signal paths. Alternatively, a corresponding image processing device 100 may be provided for each of the multiple imaging devices 150. In this case, the image processing device 100 can extract a foreground image from a captured image obtained by a corresponding one of the imaging devices 150 and transmit the foreground image to the generation device 200. In this case, the image processing device 100 may be included in the corresponding one of the imaging devices 150. As another example, one image processing device 100 may extract a foreground image from captured images obtained by some or all of the multiple imaging devices 150.

[0021] The image processing device 100 can be realized by a computer including a processor and a memory. For example, as will be described later with reference to FIG. 2, a processor included in a computer included in an image processing system can execute a program stored in a memory serving as a storage medium. In this way, the functions of the units shown in FIG. 1 and the like and the processes shown in FIG. 4 and the like can be realized. Alternatively, some or all of the functions of the image processing device 100 may be realized using hardware. The hardware may be a dedicated circuit (e.g., an ASIC or FPGA) or a processor (e.g., a reconfigurable processor or DSP). Similarly, the functions of the imaging device 150 and the generating device 200 can be realized by a program, hardware, or a combination thereof. Furthermore, at least one of the imaging device 150, the image processing device 100, and the generating device 200 may be configured by a plurality of information processing devices connected via a network, for example.

[0022] Next, an example of the hardware configuration of the image processing device 100 will be described with reference to Fig. 2. The image processing device 100 includes a CPU 111, a ROM 112, a RAM 113, an auxiliary storage device 114, a display unit 115, an operation unit 116, a communication I / F 117, and a bus 118.

[0023] The CPU 111 controls the entire image processing device 100 using computer programs or data stored in the ROM 112 or RAM 113, thereby realizing each function of the image processing device 100 shown in FIG. 1. The ROM 112 stores programs that do not require modification. The RAM 113 temporarily stores programs or data supplied from the auxiliary storage device 114, and data supplied from the outside via the communication I / F 117. The auxiliary storage device 114 stores various data such as image data or audio data. The auxiliary storage device 114 is, for example, a hard disk drive.

[0024] The display unit 115 displays various images or information to the user. The display unit 115 can display, for example, a GUI (Graphical User Interface) that the user uses to operate the image processing device 100. The display unit 115 is, for example, a liquid crystal display or an LED. The operation unit 116 inputs various instructions to the CPU 111 in accordance with operations by the user. The operation unit 116 is, for example, a keyboard, a mouse, a joystick, or a touch panel. In this example, the CPU 111 operates as a display control unit that controls the display unit 115 and an operation control unit that controls the operation unit 116.

[0025] The communication I / F 117 is an interface for communication with an external device of the image processing device 100. For example, if the image processing device 100 is connected to the external device via a wired connection, a communication cable is connected to the communication I / F 117. Furthermore, if the image processing device 100 has a function for wireless communication with the external device, the communication I / F 117 is equipped with an antenna. The bus 118 connects the various parts of the image processing device 100 and transmits information between the various parts.

[0026] 2, the image processing device 100 has a display unit 115 and an operation unit 116. However, at least one of the display unit 115 and the operation unit 116 may exist as a separate device outside the image processing device 100.

[0027] [Embodiment 1] The configuration of an image processing device 100 according to an embodiment will be described with reference to Fig. 1. The image processing device 100 includes a foreground separation unit 101, a division unit 102, a rectangle calculation unit 103, a transmission unit 104, and a control unit 105.

[0028] The foreground separation unit 101 detects a foreground region from an input image. In this way, the foreground separation unit 101 separates a background region in the input image from a foreground region corresponding to a predetermined subject. The method for detecting the foreground region is not particularly limited. For example, the foreground separation unit 101 can generate a background image using a sequential background update method. Specifically, for multiple input images acquired by the same imaging device 150, the foreground separation unit 101 can determine an area in which there is change in the image as a foreground region and an area in which there is no change in the image for a certain period of time as a background region. The foreground separation unit 101 can also extract only the background region of the input image. The foreground separation unit 101 can then generate a background image using the extracted image. For example, the foreground separation unit 101 can generate a background image using images extracted from the background regions of each of the multiple input images.

[0029] The foreground separation unit 101 can then detect a foreground region from the input image using the background image. For example, the foreground separation unit 101 can detect a foreground region using a background subtraction method. The foreground separation unit 101 can also generate information indicating the detected foreground region. For example, the foreground separation unit 101 can generate a binary image (foreground mask image) indicating the foreground region. The foreground separation unit 101 can generate the foreground mask image using the input image and the background image. Specifically, the foreground separation unit 101 can generate the foreground mask image by binarizing the difference image, obtained by subtracting the background image from the input image, using a predetermined threshold. In this embodiment, a pixel value of "1" in the foreground mask image indicates that the pixel is in the foreground region, and a pixel value of "0" indicates that the pixel is in the background region. These binary values ​​may be reversed.

[0030] The method for detecting the foreground region and the method for generating the background image and foreground mask image are not limited to the above methods. For example, an image captured in advance by the image capture device 150 when no subject is present can be stored in a frame memory. The foreground separation unit 101 can read this image and use it as the background image. Also, instead of using background subtraction, AI may be used to distinguish between the foreground region and the background region. In this case, the binary image output by the AI ​​can be used as the foreground mask image.

[0031] The dividing unit 102 and the rectangle calculation unit 103 calculate a circumscribing rectangle for each of a plurality of divided regions obtained by dividing a single foreground region. In this specification, this single foreground region is a group of foreground regions represented by a group of adjacent pixels, and can be referred to as a continuous foreground region. According to the method of this embodiment, when two or more foreground regions are detected from an input image, each foreground region can be divided. For example, when people are in the foreground, if the captured image contains three people without overlapping, there will be three foreground regions, and as a result, the division process described below will be performed on each foreground region.

[0032] In this embodiment, the dividing unit 102 divides a foreground mask image to divide one foreground region. The dividing unit 102 can divide the foreground mask image into regions of a predetermined size. Furthermore, the rectangle calculation unit 103 calculates a circumscribing rectangle for each of the divided regions according to each partial image of the foreground mask image obtained by the division. Specifically, the rectangle calculation unit 103 can obtain a circumscribing rectangle for the foreground region in each of the divided foreground mask images. For example, the rectangle calculation unit 103 can calculate the vertex coordinates of the circumscribing rectangle. In this way, the rectangle calculation unit 103 can calculate a circumscribing rectangle for each of the divided regions obtained by dividing the foreground region. These processes will be described later with reference to FIGS. 3 and 4.

[0033] In this embodiment, the multiple divided regions are obtained by dividing the foreground region horizontally, vertically, or in a grid pattern. To this end, the dividing unit 102 can divide the foreground mask image into a grid pattern, as will be described with reference to FIGS. 3(C) and 3(D). The dividing unit 102 may also divide the foreground mask image horizontally or vertically. In this embodiment, the multiple divided regions are obtained by dividing the foreground region according to a predetermined interval. For example, the dividing unit 102 can divide the foreground mask image according to a predetermined interval, as will be described with reference to FIGS. 3(C) and 3(D).

[0034] The transmitter 104 transmits data of the input image within the circumscribing rectangle to the outside. As described above, the rectangle calculation unit 103 can calculate multiple circumscribing rectangles. The transmitter 104 can transmit data of partial images (called foreground images, which represent textures) of the input image defined by each circumscribing rectangle to the generation device 200. The transmitter 104 can also transmit data of a foreground mask image to the outside. Here, the transmitter 104 can transmit data of the foreground mask image within the circumscribing rectangle to the outside, similar to the data of the input image. For example, the transmitter 104 can transmit data of partial images of the foreground mask image defined by each circumscribing rectangle to the generation device 200.

[0035] As described above, in this embodiment, the transmission target is limited to information within the circumscribing rectangle. That is, the transmission unit 104 can exclude data of the input image outside the circumscribing rectangle and data of the foreground mask image from the data to be transmitted to the outside. As described above, the input image may be an image of one frame of a moving image. In this case, the transmission unit 104 transmits data of the input image within the circumscribing rectangle for each frame to the outside. On the other hand, the transmission unit 104 can exclude data of the input image outside the circumscribing rectangle for each frame from the data to be transmitted to the outside. Note that the transmission unit 104 may also transmit background data (e.g., the background image described above) to the outside. Such background data may be generated from the input image. On the other hand, such background data may be common to multiple frames. That is, it is not necessary to transmit separate background data corresponding to each frame to the outside.

[0036] The transmitter 104 may compress and transmit this image data. Furthermore, if the area of ​​a circumscribing rectangle is less than a threshold, the transmitter 104 may exclude the input image data within the circumscribing rectangle from being transmitted externally. That is, if the area of ​​a certain circumscribing rectangle is equal to or less than a threshold, the transmitter 104 may not transmit the input image and foreground mask image data within the circumscribing rectangle. For example, when using background subtraction, noise may be detected as a foreground region. Because foreground regions detected due to noise are often small, this configuration can determine that a small circumscribing rectangle is due to noise and suppress transmission of the image data within the circumscribing rectangle. This reduces the amount of data transmitted from the image processing device 100 to the generation device 200. Furthermore, it is possible to suppress the occurrence of artifacts that do not correspond to the subject in the foreground model generated by the generation device 200.

[0037] Alternatively, the transmission unit 104 may transmit the image data with the highest ranking in order of the area of ​​the circumscribing rectangle, and not transmit any other image data. Furthermore, the transmission unit 104 may identify the image data up to the ranking in order of the area of ​​the circumscribing rectangle, compress the identified image data, and control the transmission of the image data based on the total data volume of the compressed data. That is, if the total data volume after compression exceeds a threshold, the transmission unit 104 may not transmit the image data, and if it is below the threshold, the transmission unit 104 may transmit the image data. This prevents congestion of the transmission bandwidth. In this way, in one embodiment, the transmission unit 104 can transmit data of the input image within a circumscribing rectangle selected from the multiple circumscribing rectangles. In this case, the selection of a circumscribing rectangle from the multiple circumscribing rectangles can be performed based on the area of ​​the circumscribing rectangle.

[0038] The control unit 105 can control the operation of the image processing device 100 .

[0039] Next, the processing performed by the image processing device 100 will be described with reference to Fig. 3 and Fig. 4. Fig. 4 is a flowchart of the processing performed by the image processing device 100 in this embodiment. This processing can be started when the operation unit 116 receives a user operation to start the processing.

[0040] In S401, the image processing device 100 acquires an input image from the imaging device 150. In S402, the foreground separation unit 101 generates a foreground mask image as described above. Fig. 3A shows an example of the foreground mask image. Fig. 3A shows a foreground region 301.

[0041] In S403 and S404, the division unit 102 divides the foreground region. In this embodiment, the division unit 102 divides the foreground region by dividing the foreground mask image. The method of dividing the foreground region and the foreground mask image is not particularly limited. For example, the division unit 102 can divide the foreground mask image so that the size of each divided region is equal to or smaller than a predetermined size. Furthermore, the division unit 102 can divide the foreground mask image into a grid pattern. In this case, the division unit 102 can divide the foreground mask image into a grid pattern at regular intervals. In this embodiment, the foreground mask image is divided using a pattern image as shown below.

[0042] In S403, the dividing unit 102 generates a pattern used to divide the foreground mask image. The dividing unit 102 can generate, for example, a grid-shaped pattern image as shown in FIG. 3(C). This pattern image is a binary image having the same resolution as the foreground mask image. The grid is represented by lines with a width of one dot. These lines are drawn at arbitrary or constant intervals in the horizontal and / or vertical directions. These intervals may be determined by a user operating the image processing system. Furthermore, these intervals may be determined according to the communication speed between the image processing device 100 and the generation device 200. In this pattern image, the pixel values ​​of the pixels that make up the lines are "0," and the pixel values ​​of the other pixels are "1." However, these binary values ​​may be reversed.

[0043] The dividing unit 102 can generate such a pattern image according to the values ​​of the horizontal counter and the vertical counter. The horizontal counter counts so that the horizontal spacing of the grid is at its maximum value. The vertical counter counts so that the vertical spacing of the grid is at its maximum value. In this example, the effective image area of ​​one frame of input image is raster scanned. Hereinafter, the scanned pixel will be referred to as the pixel of interest. These counters are incremented at clock edges of this effective image area. For example, in raster scanning, the horizontal counter is incremented as the pixel of interest position moves horizontally, and the vertical counter is incremented as the pixel of interest position moves vertically. These counters are reset when either of the following two reset conditions is met. The first reset condition is that the pixel of interest position is at the starting position of the effective image area in the horizontal direction (or vertical direction). The second reset condition is that the value of each counter reaches its maximum value minus 1 (the grid spacing). The reset value of the counter is 0. Based on these reset and increment conditions, these counters repeatedly increment within the effective image area from 0 to the maximum value minus 1. If one or both counter values ​​are 0, the pixel value of the pixel of interest is set to "0" to draw the grid lines. In all other cases, the pixel value of the pixel of interest is set to "1."

[0044] The method for acquiring the pattern image is not limited to the above method. For example, the pattern image may be generated in advance. For example, the pattern image may be stored in a frame memory. The dividing unit 102 can read out such a pattern image.

[0045] In S404, division unit 102 divides the foreground mask image using the pattern image obtained in S403. In this embodiment, division unit 102 divides the foreground mask image by calculating the logical product of pixel values ​​of pixels at the same coordinates between the foreground mask image and the pattern image. Equation (1) shows the logical product formula. F(x, y) represents the pixel value of the foreground mask image. L(x, y) represents the pixel value of the pattern image. D(x, y) represents the pixel value of the foreground mask image after division. x represents the X coordinate. y represents the Y coordinate. D(x,y)=F(x,y)&L(x,y) ··· Equation (1)

[0046] Fig. 3(D) shows the result of dividing the foreground mask image shown in Fig. 3(A) according to the pattern image shown in Fig. 3(C). Fig. 3(D) shows divided regions 303 obtained by dividing the foreground region. In this way, by dividing the foreground mask image, divided regions that are the division results of the foreground region can be obtained.

[0047] In S405, the rectangle calculation unit 103 calculates a circumscribing rectangle for each of the divided regions obtained by the division. In this embodiment, the rectangle calculation unit 103 calculates the coordinates of the circumscribing rectangle of the foreground region indicated by the divided foreground mask image obtained in S404. In the divided foreground mask image, the foreground region (region with pixel value "1") is divided by lines with pixel value "0". Therefore, by using a method for determining the circumscribing rectangle for each of the multiple regions present in the image, the circumscribing rectangle of the foreground region in each divided mask image can be calculated. FIG. 3(E) shows the result of superimposing the circumscribing rectangle calculated from the divided foreground mask image shown in FIG. 3(D) on the foreground mask image shown in FIG. 3(A). FIG. 3(E) shows the circumscribing rectangle 304 of the divided region. Note that in this embodiment, the rectangle calculation unit 103 performs a process of expanding the circumscribing rectangle of the divided region thus calculated by one dot. That is, the X and Y coordinates of each vertex of the circumscribing rectangle are moved outward by one dot. By performing this process, it is possible to fill in the gaps of one dot between the divided areas.

[0048] Here, the advantages of dividing the foreground region before calculating the circumscribing rectangle will be explained. FIG. 3B shows the result of superimposing the circumscribing rectangle of the foreground region calculated from the foreground mask image shown in FIG. 3A on FIG. 3A. FIG. 3B shows the circumscribing rectangle 302 of the (undivided) foreground region. Comparing the area of ​​the circumscribing rectangle 302 shown in FIG. 3B with the sum of the area of ​​the circumscribing rectangle 304 shown in FIG. 3E, the latter is smaller. Thus, according to the method of this embodiment, the amount of data transmitted in S406 is reduced. In particular, the greater the proportion of the foreground in the angle of view, the less wasted area (area other than the subject) in the circumscribing rectangle. This further reduces the transfer load of the foreground image in S406.

[0049] In S406, the transmission unit 104 transmits the data of the input image within the circumscribing rectangle calculated in S405 to the generation device 200 as described above. The transmission unit 104 may further transmit a background image to the generation device 200.

[0050] In S407, the control unit 105 determines whether to continue the processing. For example, if the user has not performed an operation to end the image processing shown in FIG. 4 using the operation unit 116, it is determined that the processing should be continued, and the processing returns to S401. In this case, it is possible to calculate a circumscribing rectangle for the input image of the next frame. Also, if the user has performed an operation to end the image processing, the image processing shown in FIG. 4 ends.

[0051] As described above, according to this embodiment, a bounding box is calculated for each of a plurality of divided regions obtained by dividing one foreground region. Then, the input image data within the bounding box is transmitted to an external device. This configuration can reduce the transmission load, as shown in Figures 3(B) and 3(E).

[0052] (Variation) As described above, the grid spacing used to divide the foreground region may be determined by the user. Alternatively, the grid spacing may be determined automatically. In the following modified example, the control unit 105 determines the method for dividing the foreground region. For example, as described above, multiple divided regions can be obtained by dividing the foreground region according to a predetermined spacing. Here, the control unit 105 can control this predetermined spacing. For example, the control unit 105 can determine the grid spacing (e.g., horizontal spacing and / or vertical spacing). Furthermore, the control unit 105 can determine the area of ​​each rectangle that constitutes the grid.

[0053] Specifically, the control unit 105 can control the predetermined interval based on the total area of ​​the circumscribing rectangles calculated according to the predetermined interval. In the example described below, the control unit 105 can generate a grid-like pattern image according to the transmission rate of the foreground image defined below. The control unit 105 can also control the predetermined interval according to the communication speed of the image processing device 100. For example, the control unit 105 can generate a grid-like pattern image according to the communication speed of the image processing device 100.

[0054] A method for controlling the grid spacing will be described below with reference to Fig. 5. Fig. 5 is a flowchart showing the processing of the image processing device 100 in this modified example. Note that the processing of S401 to S407 is as described above, and a description thereof will be omitted.

[0055] In S501, control unit 105 initializes the grid spacing. For example, control unit 105 can set the grid spacing to the same value as the resolution (i.e., the number of pixels) of the foreground mask image. Specifically, control unit 105 can set the horizontal grid spacing (Hperiod) to the horizontal resolution (Htotal) of the foreground mask image. Control unit 105 can also set the vertical grid spacing (Vperiod) to the vertical resolution (Vtotal) of the foreground mask image. Furthermore, control unit 105 can set the number of grid divisions (DIV) to 1. In this case, the foreground mask image is not divided. However, the initialized grid spacing is not limited to the above. For example, the grid spacing may be initialized so that division is performed.

[0056] In S502, control unit 105 calculates the transmission rate of the foreground image. The transmission rate indicates the ratio of the sum of the areas of the circumscribing rectangles of the foreground region to the area of ​​the input image of one frame. In this example, the area can be expressed by the number of pixels. The transmission rate can be calculated according to equation (2). In equation (2), T is the transmission rate. Htotal is the horizontal resolution. Vtotal is the vertical resolution. Hwidth(n) is the width of the nth circumscribing rectangle. Vhigh(n) is the height of the nth circumscribing rectangle. N is the total number of circumscribing rectangles in the input image of one frame.

number

[0057] If the transmission rate is equal to or less than the upper limit, control unit 105 determines that the foreground image can be transmitted, and the process proceeds to S505. If the transmission rate is greater than the upper limit, control unit 105 determines that the foreground image cannot be transmitted, and the process proceeds to S503.

[0058] In S503, the control unit 105 updates the grid spacing so that the grid spacing becomes narrower. For example, the control unit 105 can first increment the number of divisions (DIV). Then, the control unit 105 can calculate the grid spacing based on equations (3) and (4). Hperiod = Htotal / DIV...Equation (3) Vperiod = Vtotal / DIV...Equation (4)

[0059] A grid-shaped pattern image used to divide the foreground region of the input image of the next frame is generated according to the updated interval. Note that the method for calculating the grid interval is not limited to the above method. For example, it is not necessary to use the same number of divisions (DIV) in equations (3) and (4). That is, the number of divisions (DIV) used in equations (3) and (4) may be different. For example, the number of divisions (DIV) used in equations (3) and (4) may be controlled according to the ratio between the horizontal resolution (Htotal) and the vertical resolution (Vtotal) of the foreground mask image. For example, when Htotal is approximately twice Vtotal, the number of divisions (DIV) used in equation (3) may be twice the number of divisions (DIV) used in equation (4). With this configuration, the grid shape becomes approximately square.

[0060] An upper limit may also be set for the number of divisions (DIV). With this configuration, an increase in the number of divisions can cause the area of ​​the circumscribing rectangle to become extremely small, which can prevent the transmitter 104 from determining that the circumscribing rectangle is derived from noise and excluding it from transmission. Furthermore, with this configuration, an increase in transmission load caused by an extremely large number of circumscribing rectangles can be prevented.

[0061] In S504, transmission unit 104 transmits null data to generation device 200. This configuration prevents the transmission of a foreground image even when the transmission rate exceeds the upper limit. This also prevents generation device 200 from performing processing based on a foreground image that was not completely transmitted due to a limit in the transfer speed.

[0062] In S505, control unit 105 calculates the difference between the transmission rate calculated in S502 and the upper limit of the transmission rate. Control unit 105 then compares this difference with a margin. This margin may be predetermined in the image processing system. If the difference is equal to or greater than the margin, control unit 105 determines that there is a margin for the transmission rate, and the process proceeds to S506. If the difference is smaller than the margin, control unit 105 determines that there is not a margin for the transmission rate, but that the foreground image can be transmitted, and the process proceeds to S406.

[0063] In S506, the control unit 105 updates the grid spacing so that it becomes wider. For example, the control unit 105 can first decrement the number of divisions (DIV). Then, the control unit 105 can calculate the grid spacing based on the above-mentioned equations (3) and (4).

[0064] According to the above-described modified example, it is possible to automatically narrow or widen the grid spacing. The transmission rate changes depending on the installation position and angle of view of the image capture device 150, and the position of the subject. Therefore, the optimal grid spacing also differs for each image capture device 150. Therefore, it is a heavy burden for the user to sequentially set the grid spacing for each of the multiple image capture devices 150. According to this modified example, it is possible to automatically determine the grid spacing for each of the multiple image capture devices 150.

[0065] The method by which the control unit 105 determines how to divide the foreground region or foreground mask region is not limited to controlling the grid spacing as described above. For example, the control unit 105 may shift the position of the grid. Specifically, the control unit 105 may control the start position of the grid. For example, the control unit 105 may control the conditions under which the dividing unit 102 draws grid lines as described above. In the first embodiment, grid lines are drawn at the pixel of interest when the counter value is "0." However, grid lines may be drawn at the pixel of interest when the counter value is a specific value other than "0." The start position of the grid can be controlled by changing this specific value. For example, this specific value may be incremented for each frame, like scrolling the grid. Alternatively, this specific value may be randomly determined for each frame. If a grid line is located near the edge of the foreground region, the area of ​​the circumscribing rectangle may become extremely small. This may cause the transmitting unit 104 to determine that the circumscribing rectangle is derived from noise and therefore not to be transmitted. However, by shifting the starting position of the grid in this way, it is possible to prevent image data for a part of the foreground region from being continuously excluded from transmission. Note that, as will be described in the second embodiment, if the area of ​​the circumscribing rectangle for a divided region adjacent to another divided region is less than a threshold, the rectangle calculation unit 103 may enlarge the circumscribing rectangle.

[0066] [Embodiment 2] In the first embodiment, the foreground mask image is divided into a grid pattern, and a circumscribing rectangle is calculated from each region. However, the method for calculating a circumscribing rectangle for each divided region obtained by dividing the foreground region is not limited to this method. For example, the foreground region can be divided by any method, and a circumscribing rectangle can be calculated for each divided region. Furthermore, if the area of ​​a circumscribing rectangle is small, adjacent circumscribing rectangles can be merged. In the second embodiment, an example of such a method will be described.

[0067] The configuration of an image processing device 100 according to this embodiment will be described with reference to Fig. 7. The image processing device 100 has a rectangle integration unit 106 in addition to a foreground separation unit 101, a rectangle calculation unit 103, a transmission unit 104, and a control unit 105. Differences from the first embodiment will be described below.

[0068] In this embodiment, the rectangle calculation unit 103 divides the foreground region into multiple regions. For example, as preprocessing for calculating the circumscribing rectangle of the foreground region, the rectangle calculation unit 103 can calculate multiple rectangles that are part of the circumscribing rectangle and are adjacent to each other. A set of these multiple rectangles can encompass one foreground region. In this embodiment, the multiple rectangles calculated by the rectangle calculation unit 103 are rectangles that circumscribing each of the regions obtained by dividing the foreground region. Each rectangle can be searched and calculated according to a raster scan.

[0069] The rectangle integration unit 106 sets one divided area by integrating a group of adjacent areas that are part of the multiple areas. In this embodiment, the rectangle integration unit 106 integrates adjacent rectangles sequentially. Integration refers to calculating a circumscribing rectangle for two adjacent rectangles. In other words, the rectangle integration unit 106 can calculate a circumscribing rectangle for a group of rectangles as a result of integrating the group of rectangles. In this way, the rectangle integration unit 106 can calculate a circumscribing rectangle for one divided area by integrating a group of adjacent rectangles that are part of the rectangles circumscribing each of the multiple areas.

[0070] Here, the rectangle integration unit 106 can set the divided regions so that the area of ​​the circumscribing rectangle for one divided region is equal to or greater than a threshold. For example, the rectangle integration unit 106 can determine whether to integrate additional rectangles into the group of rectangles based on the area of ​​the circumscribing rectangle calculated as the integration result of the group of rectangles. Specifically, the rectangle integration unit 106 can calculate the area of ​​the circumscribing rectangle calculated as the integration result and compare this area with a threshold. Then, based on the comparison result, the rectangle integration unit 106 can determine whether to integrate additional rectangles into the circumscribing rectangle calculated as the integration result.

[0071] When calculating a bounding rectangle for a foreground area having a complex shape using hardware in accordance with the raster scan method, it is not easy to calculate the bounding rectangle all at once (within one frame period) due to restrictions on circuit size and processing time. For this reason, in this embodiment, after the rectangle calculation unit 103 calculates multiple rectangles as preprocessing, the rectangle integration unit 106 integrates the rectangles to calculate a bounding rectangle for each divided area.

[0072] The control unit 105 controls the area threshold referenced by the rectangle integration unit 106. The control unit 105 can control the threshold in accordance with the total area of ​​the circumscribing rectangles calculated in accordance with the threshold. In this embodiment, the control unit 105 controls the threshold in accordance with the transmission rate of the foreground image. The control unit 105 can also control the threshold in accordance with the communication speed of the image processing device 100.

[0073] The processing performed by the image processing device 100 in this embodiment will be described below with reference to Figures 6 and 8. Figure 6 is a flowchart of the processing performed by the image processing device 100 in this embodiment.

[0074] In S601, the control unit 105 initializes the area threshold and the number of divisions (DIV) to be referred to in S607. In this example, the area threshold is initialized to the "approximate area of ​​the circumscribing rectangle of the foreground area." The number of divisions is also initialized to "1." The approximate area of ​​the circumscribing rectangle can be determined by the user depending on the angle of view of the camera and the size of the subject being photographed. However, the initial values ​​are not limited to those described above. For example, the number of divisions may be initialized so that division is performed.

[0075] Steps S602 and S603 are performed in the same manner as steps S401 and S402. Fig. 8A shows an example of the foreground mask image generated in step S603. Fig. 8A shows a foreground region 801.

[0076] In S604, as preprocessing for calculating the circumscribing rectangle of the foreground region, the rectangle calculation unit 103 calculates information on multiple rectangles that are part of the circumscribing rectangle and are adjacent to each other. As described above, the rectangle calculation unit 103 can search for and calculate each rectangle according to raster scanning. FIG. 8B shows multiple rectangles 802 calculated in this manner. Each rectangle is labeled with a number to identify it in the order in which it was calculated. Furthermore, numbers indicating adjacent rectangles are recorded for each rectangle as information indicating the adjacency relationship between the rectangles. The rectangle calculation unit 103 can calculate the vertex coordinates, numbers, and information indicating the adjacency relationship of each rectangle as information for each rectangle. In this example, multiple rectangles adjacent in the vertical direction (or horizontal direction) are calculated for one foreground region.

[0077] The rectangle calculation unit 103 can detect rectangles formed by pixels in the contiguous foreground region for each row of the foreground mask image. Furthermore, if a rectangle in an upper row is adjacent to a rectangle in a row adjacent below, and the rectangle in the upper row is expanded downward, if the rectangle in the row adjacent below is included in the expanded rectangle, the rectangle in the upper row and the rectangle in the row adjacent below are merged. In this way, multiple rectangles 802 shown in FIG. 8(B) can be calculated.

[0078] In S605, the rectangle integration unit 106 integrates the rectangles calculated in S604. The rectangle integration unit 106 can integrate the two selected rectangles. The rectangle integration unit 106 can generate, as information about the rectangle obtained by integration, the vertex coordinates of the circumscribing rectangles of the two rectangles, the numbers of the two rectangles, and information indicating the adjacency relationship between the two rectangles.

[0079] In S605, the rectangle integration unit 106 can integrate an integrated rectangle (N) with its adjacent rectangles in accordance with the adjacency information. The integrated rectangle refers to the rectangle obtained by the integration process in the loop from S605 to S608. The integrated rectangle (N) refers to the rectangle into which the rectangle is integrated in the (N+1)th loop from S605 to S611. In the loop from S605 to S608, rectangles adjacent to the integrated rectangle (N) are integrated one by one. However, in the first loop from S605 to S608, the integrated rectangle (N) does not yet exist. Therefore, in the first loop, the integrated rectangle (N) is treated as a NULL value.

[0080] For example, when N=0, in S605 of the first loop of S605 to S608, the NULL value is integrated with the rectangle 803 with the smallest number shown in FIG. 8(B), and integrated rectangle (0) becomes rectangle 803. Then, from the second loop onwards, adjacent rectangles immediately below rectangle 803 are integrated one after another. Finally, rectangle 805 shown in FIG. 8(C) is obtained as the completed integrated rectangle (0).

[0081] Also, when N=1, in S605 of the first loop of S605 to S608, the NULL value and the unintegrated rectangle 804 with the smallest number are integrated, and integrated rectangle (1) becomes rectangle 804. Then, from the second loop onwards, adjacent rectangles directly below rectangle 804 are integrated one after another. Finally, rectangle 806 shown in FIG. 8C is obtained as the completed integrated rectangle (1).

[0082] In S606, the rectangle integration unit 106 calculates the area of ​​the integrated rectangle (N) obtained by the integration in S605. The rectangle integration unit 106 can calculate the area from the vertex coordinates.

[0083] In S607, the rectangle integration unit 106 compares the area calculated in S606 with the area threshold set by the control unit 105. If this area is smaller than the threshold, the process proceeds to S608. If this area is equal to or greater than the threshold, the generation of the integrated rectangle (N) is completed. That is, the vertex coordinates of the integrated rectangle (N) are determined. Then, the process proceeds to S610.

[0084] In S608, the rectangle integration unit 106 references the adjacency information for the integrated rectangle (N) obtained in S605 and determines whether there is a rectangle adjacent to the integrated rectangle (N) that should be integrated next. If there is an adjacent rectangle, the process returns to S605. The rectangles adjacent to the integrated rectangle (N) are then integrated into the integrated rectangle (N). If there is no adjacent rectangle, this means that the last rectangle among multiple adjacent rectangles has been integrated and the area of ​​the integrated rectangle (N) is less than the threshold value. In this case, the integrated rectangle (N) is determined to be a fractional rectangle. Figure 8(C) shows such a fractional rectangle 807. The process then proceeds to S609.

[0085] In S609, the rectangle integration unit 106 enlarges the integrated rectangle (N). In this example, the area of ​​the integrated rectangle (N) is less than the threshold. In this way, when the area of ​​the circumscribing rectangle for a divided area adjacent to another divided area is less than the threshold, the rectangle integration unit 106 can enlarge the circumscribing rectangle. For example, the rectangle integration unit 106 can move the vertex coordinates of the integrated rectangle (N). Specifically, the rectangle integration unit 106 can increase the Y coordinates of the lower left and lower right vertex coordinates of the integrated rectangle (N) so that the area of ​​the integrated rectangle (N) becomes equal to the area threshold set by the control unit 105. FIG. 8(D) shows a rectangle 808 enlarged in this way. The rectangle integration unit 106 may also move the X coordinates of the vertices. If the coordinates after expansion exceed the maximum coordinates of the vertical or horizontal coordinates, the coordinates after expansion can be clipped to the maximum coordinates.

[0086] In S610, the rectangle integration unit 106 increments N. In S611, the rectangle integration unit 106 determines whether integration processing has been completed for all rectangles. For example, the rectangle integration unit 106 determines whether the number of the rectangle last integrated in S605 is equal to the number of the rectangle last labeled in S604. If these numbers are equal, the rectangle integration unit 106 determines that integration processing has been completed for all rectangles. Then, the process proceeds to S612. If these numbers are not equal, the rectangle integration unit 106 determines that rectangles that have not yet been subjected to integration processing remain. Then, the process returns to S605. Then, in the next loop of S605 to S611, the next integrated rectangle (N) is generated.

[0087] Each integrated rectangle shown in FIG. 8(C) thus obtained represents a circumscribing rectangle for each of the divided regions obtained by dividing one foreground region. Also, in FIG. 8(D), each integrated rectangle other than rectangle 808 obtained last represents a circumscribing rectangle for each of the divided regions obtained by dividing one foreground region. In this example, rectangle 807 is a circumscribing rectangle for a portion of one foreground region, and is therefore unlikely to be derived from noise. To prevent image data for rectangle 807 from being excluded from transmission, rectangle 807 is enlarged to rectangle 808. However, enlarging a circumscribing rectangle smaller than the threshold is not essential.

[0088] In S612, control unit 105 compares the transmission rate of the foreground image with the upper limit, as in S502. In this embodiment, the transmission rate indicates the ratio of the total area of ​​the integrated rectangle to the area of ​​the input image of one frame. If the transmission rate is equal to or less than the upper limit, the process proceeds to S615. If the transmission rate is greater than the upper limit, the process proceeds to S613.

[0089] In S613, the control unit 105 updates the area threshold to decrease it. For example, the control unit 105 can increment the number of divisions (DIV). Then, the control unit 105 calculates the threshold according to equation (5). In equation (5), TH represents the area threshold. Furthermore, THini represents the initial value of the area set in S601. Note that, as in the modified example of the first embodiment, an upper limit may be set for the number of divisions (DIV). TH = THini / DIV ··· Equation (5)

[0090] S614 is performed in the same manner as S504.

[0091] In S615, the control unit 105 compares the difference between the transmission rate calculated in S612 and the upper limit with the margin, as in S505. If the difference is equal to or greater than the margin, the process proceeds to S616. If the difference is smaller than the margin, the process proceeds to S617.

[0092] In S616, the control unit 105 updates the area threshold to a larger value. For example, the control unit 105 can first decrement the number of divisions (DIV). Then, the control unit 105 can calculate the area threshold based on the above-mentioned formula (5).

[0093] S617 to S618 are performed in the same manner as S406 to S407. In this embodiment, the transmission targets of the foreground image and the foreground mask image in S406 are limited to the inside of each integrated rectangle. Also, if it is determined in S407 that the processing is to continue, the processing returns to S601.

[0094] As described above, according to this embodiment, a circumscribing rectangle is calculated for each of a plurality of divided regions obtained by dividing one foreground region. Then, data of the input image within the circumscribing rectangle is transmitted to the outside. This reduces the transmission load. In particular, according to this embodiment, the circumscribing rectangle is calculated so as to have a constant area. This prevents the calculation of an extremely small circumscribing rectangle that may be determined to be derived from noise.

[0095] (Other embodiments) The present disclosure can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0096] Furthermore, each of the image processing devices 100 downstream of the multiple imaging devices 150 may be connected to the generation device 200, or multiple image processing devices 100 may be cascade-connected. FIG. 9 shows an example configuration of an image processing system in which multiple image processing devices 100 are cascade-connected. The image processing system may further include multiple imaging devices 150 and a generation device 200. That is, in FIG. 9, data output from the transmission unit 104 of the upstream image processing device 100 is input to the transmission unit 104 of the downstream image processing device 100 and further transmitted to the downstream image processing device 100. That is, data processed by the upstream image processing device 100 is received by the downstream image processing device 100. The downstream image processing device 100 transmits data received from the upstream image processing device 100 and data processed by itself to a further downstream image processing device 100, and then transmits the data to the generation device 200 via that image processing device 100. Therefore, according to this cascade connection, data processed by each image processing device 100 is transmitted to the generation device 200. In such a cascade connection configuration, the transmission bandwidth becomes more constrained due to the amount of data, so by applying the above-described embodiment, the congestion of the transmission bandwidth is suppressed. Note that all of the multiple image processing devices 100 used in this system may be connected in a single cascade connection, or the multiple image processing devices 100 may be classified into several groups and each group may be cascade-connected.

[0097] As described above, an image processing system according to an embodiment may include a plurality of image processing devices 100 and a generation device 200. The plurality of image processing devices 100 may include a first image processing device and a second image processing device. Here, the second image processing device may receive data processed by the first image processing device. For example, the first image processing device may calculate a bounding rectangle of a foreground region from an input image captured by an imaging device 150 connected to the first image processing device as described above. The first image processing device may then transmit data of the input image within the calculated bounding rectangle to the second image processing device according to the method described above. The second image processing device may also transmit data received from the first image processing device and data processed by the second image processing device to the generation device 200. For example, the second image processing device may also calculate a bounding rectangle of a foreground region from an input image captured by an imaging device 150 connected to the second image processing device as described above. The second image processing device may then transmit data of the input image within the calculated bounding rectangle to the generation device 200 together with data received from the first image processing device according to the method described above. At this time, the second image processing device can transmit the data to the generating device 200 via another image processing device 100.

[0098] The disclosure of this specification includes the following image processing device, image processing method, program, and image processing system. (Item 1) a detection means for detecting a foreground region from an input image; a calculation means for calculating a circumscribing rectangle for each of a plurality of divided regions obtained by dividing one foreground region; a transmitting means for transmitting data of the input image within the circumscribing rectangle to an external device; 1. An image processing device comprising: (Item 2) 2. The image processing device according to item 1, wherein the detection means generates a foreground mask image indicating the foreground region. (Item 3) Item 3. The image processing device according to item 2, wherein the calculation means divides the foreground mask image and calculates a circumscribing rectangle for each of the plurality of divided areas according to each partial image of the foreground mask image obtained by the division. (Item 4) 4. The image processing device according to any one of items 1 to 3, wherein the plurality of divided regions are obtained by dividing the foreground region horizontally, vertically, or in a grid pattern. (Item 5) 5. The image processing device according to any one of items 1 to 4, wherein the plurality of divided regions are obtained by dividing the foreground region according to a predetermined interval. (Item 6) 6. The image processing device according to item 5, further comprising a control means for controlling the predetermined interval. (Item 7) 7. The image processing device according to item 6, wherein the control means controls the predetermined interval based on the total area of ​​the circumscribing rectangle calculated according to the predetermined interval. (Item 8) The calculation means means for dividing the foreground region into a plurality of regions; a means for setting one of the divided regions by integrating a group of adjacent regions that are part of the plurality of regions; 8. The image processing device according to any one of items 1 to 7, comprising: (Item 9) Item 9. The image processing device according to item 8, wherein the calculation means sets the divided areas so that the area of ​​a circumscribing rectangle for one of the divided areas is equal to or greater than a threshold value. (Item 10) 10. The image processing device according to item 9, further comprising a control means for controlling the threshold value. (Item 11) Item 11. The image processing device according to item 10, wherein the control means controls the threshold value in accordance with the total area of ​​the circumscribing rectangle calculated in accordance with the threshold value. (Item 12) The calculation means means for calculating a rectangle circumscribing each of a plurality of regions obtained by dividing the foreground region; a means for calculating the circumscribing rectangle for one of the divided regions by integrating a group of adjacent rectangles that are part of the rectangles circumscribing each of the plurality of regions; 12. The image processing device according to any one of items 1 to 11, comprising: (Item 13) Item 13. The image processing device according to item 12, characterized in that the calculation means calculates a circumscribing rectangle of the group of rectangles as a result of integrating the group of rectangles, and determines whether or not to integrate further rectangles into the group of rectangles depending on the area of ​​the calculated circumscribing rectangle. (Item 14) 14. The image processing device according to any one of items 1 to 13, wherein the calculation means enlarges the circumscribing rectangle for a divided area adjacent to another divided area when the area of ​​the circumscribing rectangle is less than a threshold value. (Item 15) 15. The image processing device according to any one of items 2 to 14, which cites item 2, characterized in that the transmitting means further transmits data of the foreground mask image within the circumscribing rectangle to an external device. (Item 16) the input image is an image of one frame of a video; 16. The image processing device according to any one of items 1 to 15, wherein the transmitting means transmits data of the input image within the circumscribing rectangle for each frame to the outside, and excludes data of the input image outside the circumscribing rectangle for each frame from being transmitted to the outside. (Item 17) 17. The image processing device according to any one of items 1 to 16, wherein the transmission means further transmits background data to the outside. (Item 18) 18. The image processing device according to any one of items 1 to 17, wherein the transmitting means excludes data of the input image within the circumscribing rectangle from data to be transmitted to the outside if the area of ​​the circumscribing rectangle is less than a threshold value. (Item 19) An image processing method performed by an image processing device, detecting foreground regions from the input image; calculating a circumscribing rectangle for each of a plurality of divided regions obtained by dividing one foreground region; transmitting data of the input image within the circumscribing rectangle to an external device; An image processing method comprising: (Item 20) 19. A program for causing a computer to function as the image processing device according to any one of items 1 to 18. (Item 21) A system including a plurality of image processing devices and a generating device that generates a virtual viewpoint image, the plurality of image processing devices include a first image processing device and a second image processing device; the second image processing device receives data to be processed by the first image processing device, and transmits the data received from the first image processing device and the data to be processed by the second image processing device to the generation device; The first image processing device a detection means for detecting a foreground region from an input image; a calculation means for calculating a circumscribing rectangle for each of a plurality of divided regions obtained by dividing one foreground region; a transmitting unit for transmitting data of the input image within the circumscribing rectangle to an external device.

[0099] The present disclosure is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the present disclosure. Accordingly, the following claims are appended to clarify the scope of the present disclosure. [Explanation of symbols]

[0100] 100: Image processing device, 101: Foreground separation unit, 102: Division unit, 103: Rectangle calculation unit, 104: Transmission unit, 105: Control unit, 106: Rectangle integration unit, 150: Imaging device, 200: Generation device

Claims

1. a detection means for detecting a foreground from an input image; a dividing means for dividing a foreground mask image corresponding to the foreground into a plurality of divided regions at a first interval in the horizontal direction and a second interval in the vertical direction; a calculation means for calculating a circumscribing rectangle that circumscribes a portion of the foreground for a divided region that includes the portion of the foreground among the plurality of divided regions; a transmitting means for transmitting data corresponding to the circumscribing rectangle; 1. An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, further comprising a control means for controlling the first interval and the second interval.

3. 3. The image processing device according to claim 2, wherein the control means controls the first interval and the second interval based on a total area of ​​the circumscribing rectangle calculated according to the first interval and the second interval.

4. 2. The image processing device according to claim 1, wherein the calculation means combines a group of adjacent regions that are part of the plurality of divided regions, and calculates a circumscribing rectangle that circumscribes part of the foreground for the combined group of adjacent regions.

5. 5. The image processing device according to claim 4, wherein the calculation means integrates the adjacent group of regions so that the area of ​​the circumscribing rectangle that circumscribes a portion of the foreground for the adjacent group of regions after integration is equal to or greater than a threshold.

6. 6. The image processing device according to claim 5, further comprising a control means for controlling the threshold value.

7. 7. The image processing device according to claim 6, wherein said control means controls said threshold value in accordance with the total area of ​​said circumscribing rectangles.

8. The image processing device according to claim 1 , wherein the calculation means integrates a group of adjacent circumscribing rectangles that are part of the plurality of circumscribing rectangles.

9. 9. The image processing device according to claim 8, wherein the calculation means calculates a circumscribing rectangle of the circumscribing rectangle group as a result of integrating the circumscribing rectangle group, and determines whether to integrate further circumscribing rectangles into the circumscribing rectangle group depending on the area of ​​the calculated circumscribing rectangle.

10. The image processing device according to claim 1 , wherein the calculation means enlarges the circumscribing rectangle when the area of ​​the circumscribing rectangle for a divided area adjacent to another divided area is less than a threshold value.

11. the input image is an image of one frame of a video; 2. The image processing device according to claim 1, wherein said transmission means transmits data corresponding to the circumscribing rectangle for each frame, and excludes data not corresponding to the circumscribing rectangle for each frame from data to be transmitted.

12. 2. The image processing device according to claim 1, wherein said transmission means excludes data corresponding to said circumscribing rectangle from data to be transmitted if the area of ​​said circumscribing rectangle is less than a threshold value.

13. An image processing method performed by an image processing device, Dividing a foreground mask image corresponding to the foreground into a plurality of divided regions at a first interval in the horizontal direction and a second interval in the vertical direction; calculating a circumscribing rectangle that circumscribes a portion of the foreground for a divided region that includes the portion of the foreground among the plurality of divided regions; transmitting data corresponding to the bounding rectangle; An image processing method comprising:

14. A program for causing a computer to function as the image processing device according to any one of claims 1 to 12.

15. A system including a plurality of image processing devices and a generating device that generates a virtual viewpoint image, the plurality of image processing devices include a first image processing device and a second image processing device; the second image processing device receives data to be processed by the first image processing device, and transmits the data received from the first image processing device and the data to be processed by the second image processing device to the generation device; The first image processing device a detection means for detecting a foreground from an input image; a dividing means for dividing a foreground mask image corresponding to the foreground into a plurality of divided regions at a first interval in the horizontal direction and a second interval in the vertical direction; a calculation means for calculating a circumscribing rectangle that circumscribes a portion of the foreground for a divided region that includes the portion of the foreground among the plurality of divided regions; and a transmitting means for transmitting data corresponding to the circumscribing rectangle.

Images