Image processing device, image processing method, program, and image processing system

Abstract

To provide an image processing device, program, and image processing system, which reduce the image transmission load.SOLUTION: A method of computing a circumscribing rectangle comprises detecting a foreground area 301 from an input image. A circumscribing rectangle 304 is computed for each of a plurality of divided areas 303 obtained by dividing one foreground area. Data on the input image within the circumscribing rectangles is sent to the outside.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, a program, and an image processing system, and in particular to a technology for generating a virtual viewpoint image. [Background technology]

[0002] A technology that generates a virtual viewpoint image from a specified virtual viewpoint using a plurality of images captured by a plurality of imaging devices has been attracting attention. For example, Patent Document 1 describes that a plurality of cameras are installed at different positions, and that the images captured by each camera are input to a virtual viewpoint image generating device. Patent Document 1 also describes that a virtual viewpoint image is generated by restoring the three-dimensional shape of an object based on the input images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-45920 A Summary of the Invention [Problem to be solved by the invention]

[0004] To generate a high-quality virtual viewpoint image, large-sized, high-resolution digital images are used. Therefore, to generate a high-quality virtual viewpoint image using the technology described in Patent Document 1, the transmission load of the images from the cameras to the virtual viewpoint video generation device becomes heavy. In particular, if the transmission load increases due to the large number of cameras, there is a possibility that the transmission itself will not be possible.

[0005] The present disclosure aims to reduce the transmission load of images. [Means for solving the problem]

[0006] An image processing apparatus according to an embodiment of the present disclosure has the following configuration. A detection means for detecting a foreground region from an input image; A calculation means for calculating a circumscribing rectangle for each of a plurality of divided regions obtained by dividing one foreground region; a transmitting means for transmitting data of the input image within the circumscribing rectangle to an external device; has. Effect of the Invention

[0007] The image transmission load can be reduced. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the arrangement of an image processing system according to an embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of the hardware configuration of an image processing apparatus according to an embodiment. [Diagram 3] FIG. 11 is a diagram showing an example of a method for calculating a circumscribing rectangle. [Figure 4] 1 is a flowchart of an image processing method according to an embodiment. [Diagram 5] 1 is a flowchart of an image processing method according to an embodiment. [Figure 6] 1 is a flowchart of an image processing method according to an embodiment. [Figure 7] FIG. 1 is a diagram showing an example of the arrangement of an image processing system according to an embodiment. [Figure 8] FIG. 11 is a diagram showing an example of a method for calculating a circumscribing rectangle. [Figure 9] FIG. 1 is a diagram showing an example of the arrangement of an image processing system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claims. Although the embodiments describe a plurality of features, not all of these features are necessarily essential, and the plurality of features may be combined in any manner. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] [Image processing system overview] FIG. 1 is a diagram showing an example of the configuration of an image processing system according to an embodiment. The image processing system shown in FIG. 1 can generate a virtual viewpoint image. The virtual viewpoint image generated by this image processing system is also called a free viewpoint image. By generating a virtual viewpoint image, it is possible to monitor an image corresponding to a viewpoint freely (arbitrarily) specified by a user. The virtual viewpoint image may be an image corresponding to a viewpoint selected by the user from a limited number of virtual viewpoint candidates. The virtual viewpoint may be specified by a user operation or automatically. The automatic specification of the virtual viewpoint image may be performed based on the result of image analysis using, for example, AI (machine learning). The image processing system may generate a still image or may create a video. Such a video (virtual viewpoint video) is represented by a plurality of virtual viewpoint images.

[0011] The virtual viewpoint image is generated, for example, by the following method. First, a plurality of imaging devices 150 (cameras) capture images of an imaging area including a subject from a plurality of directions. The imaging area may be, for example, an area at any height extending upward from a field of a stadium. The imaging area may also be any area such as a concert hall or an imaging studio. The imaging area may correspond to a three-dimensional space from which the three-dimensional shape of the subject is to be estimated. This three-dimensional space may be the entire imaging area or a part of the imaging area. The image processing system estimates the three-dimensional shape of the subject located in such a three-dimensional space.

[0012] The multiple imaging devices 150 are installed at different positions so as to surround the imaging area, and capture images in synchronization. The multiple imaging devices 150 can be installed so as to have different imaging directions (attitudes). The multiple imaging devices 150 can be installed, for example, all around the imaging area. On the other hand, the area in which the multiple imaging devices 150 are arranged may exist in a limited direction relative to the imaging area. When the installation location is limited, the multiple imaging devices 150 can be arranged in this manner. Note that the multiple imaging devices 150 are synchronized according to a common time. Furthermore, imaging time information is added to each frame of image obtained by imaging by the imaging device 150.

[0013] The number of the imaging devices 150 is not particularly limited. For example, when the imaging area is a rugby stadium, several tens to several hundreds of imaging devices 150 can be installed around the stadium. In addition, the multiple imaging devices 150 may include cameras having different angles of view. For example, the imaging device 150 may include a wide-angle camera and a telephoto camera. By capturing high-resolution images of the players using a telephoto camera, the resolution of the generated virtual viewpoint image can be improved. On the other hand, by capturing images using a wide-angle camera, the number of cameras can be reduced. In particular, when capturing images of ball games, it is effective to use a wide-angle camera because the ball has a wide range of movement. Furthermore, by combining the imaging area captured by the wide-angle camera and the imaging area captured by the telephoto camera, the degree of freedom of the installation position of the imaging device 150 is improved.

[0014] The image processing device 100 detects a foreground region from an input image. In this example, the input image is a captured image obtained by the imaging device 150. That is, the imaging device 150 inputs the captured image to the image processing device 100. Note that the imaging device 150 may capture a moving image. In this case, the input image is an image of one frame of the moving image.

[0015] The foreground region is the region of the subject in the input image. In this embodiment, the subject refers to a dynamic subject (moving object) that moves (whose position or shape may change) when images are captured from the same direction in a time series. The subject is, for example, a person. For example, in a competition, the subject includes people such as players or referees on the field. Also, in a concert or entertainment, the subject includes singers, musicians, performers, or presenters. Furthermore, the subject can include objects such as a ball. For example, the subject can include objects used in sports (a ball in a ball game). Also, the image processing device 100 can extract a foreground image including the foreground region from the input image.

[0016] The image processing device 100 can further extract a background image in a background region from the input image. The background region is a region other than the foreground region. A background image is an image of at least a region different from a foreground subject. For example, a background image is obtained by removing a foreground subject from a captured image. Such a background region of an input image includes a background. The background refers to an imaged object that is stationary or continues to be in a nearly stationary state when images are captured from the same direction in a chronological order. Such imaged objects include, for example, a stage for a concert, a stadium for an event such as a competition, a structure such as a goal used in a ball game, or a field.

[0017] Note that the imaging target may include other objects in addition to the subject and background. Furthermore, the foreground image and the background image have texture information (color information, etc.).

[0018] The generating device 200 generates a virtual viewpoint image based on the foreground image extracted by the image processing device 100. In this embodiment, the generating device 200 generates a foreground model representing a three-dimensional shape of the subject and texture data used to color the foreground model based on the foreground image. Then, the generating device 200 maps the texture data to the foreground model. Furthermore, the generating device 200 renders the foreground model according to the virtual viewpoint indicated by the virtual viewpoint information. In this way, a virtual viewpoint image is generated. Note that the method of generating the virtual viewpoint image is not limited to this method. For example, for each position of the subject area on the virtual viewpoint image, a corresponding three-dimensional position on the subject can be determined. Then, as the color of each position of the subject area, the color of the foreground image corresponding to this three-dimensional position can be extracted.

[0019] The generating device 200 can also generate texture data for coloring the background model based on the background image. The background model can represent a three-dimensional shape of the background, such as a stadium. In this case, the generating device 200 can generate a virtual viewpoint image by rendering the foreground model and the background model.

[0020] The configuration of the image processing system, for example, the number and arrangement of each of the imaging device 150, the image processing device 100, and the generation device 200, are not particularly limited. For example, the imaging device 150, the image processing device 100, and the generation device 200 may be built into the same housing, or may be realized as physically different devices. For example, the imaging device 150, the image processing device 100, and the generation device 200 may be different devices connected to each other via signal paths. Also, an image processing device 100 corresponding to each of the multiple imaging devices 150 may be provided. In this case, the image processing device 100 can extract a foreground image from a captured image obtained by one corresponding imaging device 150 and transmit the foreground image to the generation device 200. In this case, the image processing device 100 may be included in one corresponding imaging device 150. As another example, one image processing device 100 may extract a foreground image from a captured image obtained by some or all of the multiple imaging devices 150.

[0021] The image processing device 100 can be realized by a computer including a processor and a memory. For example, as described later with reference to FIG. 2, a processor included in a computer included in an image processing system can execute a program stored in a memory as a storage medium. In this way, the functions of each unit shown in FIG. 1 and the like and the processing shown in FIG. 4 and the like can be realized. On the other hand, a part or all of the functions of the image processing device 100 may be realized using hardware. As the hardware, a dedicated circuit (e.g., ASIC or FPGA) or a processor (e.g., a reconfigurable processor or DSP) can be used. Similarly, the functions of the imaging device 150 and the generating device 200 can also be realized by a program, hardware, or a combination of these. In addition, at least one of the imaging device 150, the image processing device 100, and the generating device 200 may be configured by a plurality of information processing devices connected via a network, for example.

[0022] Next, an example of the hardware configuration of the image processing device 100 will be described with reference to Fig. 2. The image processing device 100 includes a CPU 111, a ROM 112, a RAM 113, an auxiliary storage device 114, a display unit 115, an operation unit 116, a communication I / F 117, and a bus 118.

[0023] The CPU 111 realizes each function of the image processing device 100 shown in Fig. 1 by controlling the entire image processing device 100 using computer programs or data stored in the ROM 112 or the RAM 113. The ROM 112 stores programs that do not require modification. The RAM 113 temporarily stores programs or data supplied from the auxiliary storage device 114, and data supplied from the outside via the communication I / F 117. The auxiliary storage device 114 stores various data such as image data or audio data. The auxiliary storage device 114 is, for example, a hard disk drive.

[0024] The display unit 115 displays various images or information to the user. The display unit 115 can display, for example, a GUI (Graphical User Interface) used by the user to operate the image processing device 100. The display unit 115 is, for example, a liquid crystal display or an LED. The operation unit 116 inputs various instructions to the CPU 111 according to operations by the user. The operation unit 116 is, for example, a keyboard, a mouse, a joystick, or a touch panel. In this example, the CPU 111 operates as a display control unit that controls the display unit 115 and an operation control unit that controls the operation unit 116.

[0025] The communication I / F 117 is an interface for communication with an external device of the image processing device 100. For example, when the image processing device 100 is connected to the external device by wire, a communication cable is connected to the communication I / F 117. Furthermore, when the image processing device 100 has a function of wirelessly communicating with the external device, the communication I / F 117 includes an antenna. The bus 118 connects each part of the image processing device 100 and transmits information between each part.

[0026] 2, the image processing device 100 has a display unit 115 and an operation unit 116. However, at least one of the display unit 115 and the operation unit 116 may exist as a separate device outside the image processing device 100.

[0027] [Embodiment 1] The configuration of an image processing device 100 according to an embodiment will be described with reference to Fig. 1. The image processing device 100 includes a foreground separation unit 101, a division unit 102, a rectangle calculation unit 103, a transmission unit 104, and a control unit 105.

[0028] The foreground separation unit 101 detects a foreground region from an input image. In this way, the foreground separation unit 101 separates a background region in the input image from a foreground region corresponding to a predetermined subject. The method of detecting the foreground region is not particularly limited. For example, the foreground separation unit 101 can generate a background image by a sequential background update method. Specifically, the foreground separation unit 101 can determine, for a plurality of input images obtained by the same imaging device 150, a region in which an image has a change as a foreground region and a region in which an image has no change for a certain period of time as a background region. In addition, the foreground separation unit 101 can extract only the background region of the input image. Then, the foreground separation unit 101 can generate a background image using the image extracted in this way. For example, the foreground separation unit 101 can generate a background image using an image extracted from each of the background regions of a plurality of input images.

[0029] Then, the foreground separation unit 101 can detect a foreground region from the input image by using the background image. For example, the foreground separation unit 101 can detect a foreground region by using a background subtraction method. Furthermore, the foreground separation unit 101 can generate information indicating the detected foreground region. For example, the foreground separation unit 101 can generate a binary image (foreground mask image) indicating the foreground region. The foreground separation unit 101 can generate a foreground mask image by using an input image and a background image. Specifically, the foreground separation unit 101 can generate a foreground mask image by binarizing a difference image obtained by subtracting the background image from the input image by using a predetermined threshold value. In this embodiment, a pixel value of "1" in the foreground mask image indicates that this pixel is in the foreground region, and a pixel value of "0" indicates that this pixel is in the background region. The binary values ​​may be reversed.

[0030] The method of detecting the foreground region and the method of generating the background image and the foreground mask image are not limited to the above methods. For example, an image captured in advance by the imaging device 150 when no subject is present can be stored in a frame memory. The foreground separation unit 101 can read out this image and use it as a background image. Also, instead of using the background difference method, the foreground region and the background region can be discriminated using AI. In this case, the binary image output by AI can be used as the foreground mask image.

[0031] The division unit 102 and the rectangle calculation unit 103 calculate circumscribing rectangles for each of a plurality of divided regions obtained by dividing one foreground region. In this specification, this one foreground region is a group of foreground regions represented by a group of adjacent pixels, and can be called a continuous foreground region. According to the method of this embodiment, when two or more foreground regions are detected from an input image, each foreground region can be divided. For example, when a person is in the foreground, if three people are included in the captured image without overlapping, there will be three foreground regions, and as a result, a division process to be described later is performed on each foreground region.

[0032] In this embodiment, the division unit 102 divides a foreground mask image to divide one foreground region. The division unit 102 can divide the foreground mask image into regions of a predetermined size. Furthermore, the rectangle calculation unit 103 calculates a circumscribing rectangle for each of the multiple divided regions according to each partial image of the foreground mask image obtained by the division. Specifically, the rectangle calculation unit 103 can obtain a circumscribing rectangle for the foreground region in each of the divided foreground mask images. For example, the rectangle calculation unit 103 can calculate the vertex coordinates of the circumscribing rectangle. In this way, the rectangle calculation unit 103 can calculate a circumscribing rectangle for each of the divided regions obtained by dividing the foreground region. These processes will be described later with reference to FIG. 3 and FIG. 4.

[0033] In this embodiment, the multiple divided regions are obtained by dividing the foreground region in the horizontal direction, the vertical direction, or a grid pattern. For this purpose, the dividing unit 102 can divide the foreground mask image in a grid pattern, as described with reference to Figs. 3(C) and (D). The dividing unit 102 may also divide the foreground mask image in the horizontal direction or the vertical direction. In this embodiment, the multiple divided regions are obtained by dividing the foreground region according to a predetermined interval. For example, the dividing unit 102 can divide the foreground mask image according to a predetermined interval, as described with reference to Figs. 3(C) and (D).

[0034] The transmitting unit 104 transmits data of the input image within the circumscribing rectangle to the outside. As described above, the rectangle calculation unit 103 can calculate a plurality of circumscribing rectangles. The transmitting unit 104 can transmit data of partial images (called foreground images, which represent textures) of the input image defined by each of the circumscribing rectangles to the generating device 200. The transmitting unit 104 can also transmit data of a foreground mask image to the outside. Here, the transmitting unit 104 can transmit data of the foreground mask image within the circumscribing rectangle to the outside, similar to the data of the input image. For example, the transmitting unit 104 can transmit data of partial images of the foreground mask image defined by each of the circumscribing rectangles to the generating device 200.

[0035] Thus, in this embodiment, the transmission target is limited to information within the circumscribing rectangle. That is, the transmission unit 104 can exclude the data of the input image outside the circumscribing rectangle and the data of the foreground mask image from the target of transmission to the outside. As described above, the input image can be an image of one frame of a moving image. At this time, the transmission unit 104 transmits the data of the input image within the circumscribing rectangle for each frame to the outside. On the other hand, the transmission unit 104 can exclude the data of the input image outside the circumscribing rectangle for each frame from the target of transmission to the outside. Note that the transmission unit 104 may further transmit background data (for example, the above-mentioned background image) to the outside. Such background data may be generated from the input image. On the other hand, such background data may be common to a plurality of frames. That is, it is not necessary to transmit separate background data corresponding to each frame to the outside.

[0036] The transmitting unit 104 may compress and transmit these image data. In addition, when the area of ​​a circumscribing rectangle is less than a threshold, the transmitting unit 104 may exclude the data of the input image within the circumscribing rectangle from being transmitted to the outside. That is, when the area of ​​a certain circumscribing rectangle is equal to or less than a threshold, the transmitting unit 104 may not transmit the data of the input image and the foreground mask image within this circumscribing rectangle. For example, when using the background subtraction method, noise may be detected as a foreground region. Since the foreground region detected due to noise is often small, such a configuration can determine that a small circumscribing rectangle is due to noise and suppress the transmission of the image data within this circumscribing rectangle. Therefore, the amount of data transmitted from the image processing device 100 to the generating device 200 can be reduced. In addition, the occurrence of artifacts that do not correspond to the subject in the foreground model generated by the generating device 200 can be suppressed.

[0037] Alternatively, the transmission unit 104 may transmit image data that is ranked high in the order of the area of ​​the circumscribing rectangle, and may not transmit the rest. The transmission unit 104 may also specify image data up to the rank in the order of the area of ​​the circumscribing rectangle, compress the specified image data, and control the transmission of the image data according to the total data amount of the compressed data. That is, if the total data amount after compression exceeds a threshold, the transmission unit 104 may not transmit the image data, and if it is below the threshold, the transmission unit 104 may transmit the image data. This makes it possible to suppress congestion of the transmission band. In this way, in one embodiment, the transmission unit 104 can transmit data of the input image within a circumscribing rectangle selected from the multiple circumscribing rectangles. At this time, the selection of the circumscribing rectangle from the multiple circumscribing rectangles can be performed according to the area of ​​the circumscribing rectangle.

[0038] The control unit 105 can control the operation of the image processing device 100 .

[0039] Next, the process performed by the image processing device 100 will be described with reference to Fig. 3 and Fig. 4. Fig. 4 is a flowchart of the process performed by the image processing device 100 in this embodiment. This process can be started when the operation unit 116 accepts a user operation to start the process.

[0040] In S401, the image processing device 100 acquires an input image from the imaging device 150. In S402, the foreground separation unit 101 generates a foreground mask image as described above. Fig. 3(A) shows an example of the foreground mask image. Fig. 3(A) shows a foreground region 301.

[0041] In S403 and S404, the division unit 102 divides the foreground region. In this embodiment, the division unit 102 divides the foreground mask image to divide the foreground region. The method of dividing the foreground region and the foreground mask image is not particularly limited. For example, the division unit 102 can divide the foreground mask image so that the size of each divided region is equal to or smaller than a predetermined size. Furthermore, the division unit 102 can divide the foreground mask image into a lattice shape. In this case, the division unit 102 can divide the foreground mask image into a lattice shape according to a certain interval. In this embodiment, the foreground mask image is divided using a pattern image as shown below.

[0042] In S403, the division unit 102 generates a pattern used to divide the foreground mask image. The division unit 102 can generate, for example, a lattice-shaped pattern image as shown in FIG. 3(C). This pattern image is a binary image having the same resolution as the foreground mask image. The lattice is represented by lines with a width of 1 dot. The lines are drawn at arbitrary intervals or at constant intervals in the horizontal and / or vertical directions. The intervals may be determined by a user who operates the image processing system. In addition, the intervals may be determined according to the communication speed between the image processing device 100 and the generation device 200. In this pattern image, the pixel values ​​of the pixels constituting the lines are "0", and the pixel values ​​of the other pixels are "1". However, the binary values ​​may be reversed.

[0043] The division unit 102 can generate such a pattern image according to the values ​​of the horizontal counter and the vertical counter. The horizontal counter counts a value so that the horizontal interval of the lattice becomes a maximum value. The vertical counter counts a value so that the vertical interval of the lattice becomes a maximum value. In this example, the effective image area of ​​one frame of the input image is raster scanned. Hereinafter, the pixel being scanned is called a pixel of interest. These counters are incremented at the clock edge of this effective image area. For example, in raster scanning, the horizontal counter is incremented in response to the pixel of interest position moving in the horizontal direction, and the vertical counter is incremented in response to the pixel of interest position moving in the vertical direction. These counters are reset when either of the following two reset conditions is satisfied. The first reset condition is that the pixel of interest position is at the start position of the effective image area in the horizontal direction (or vertical direction). The second reset condition is that the value of each counter reaches the maximum value -1 (the interval of the lattice). The reset value of the counter is 0. According to these reset and increment conditions, these counters repeatedly increment within the effective image area from 0 to the maximum value minus 1. If either or both of the counter values ​​are 0, the pixel value of the pixel of interest is set to "0" to draw the grid lines. In all other cases, the pixel value of the pixel of interest is set to "1".

[0044] The method of acquiring the pattern image is not limited to the above method. For example, the pattern image may be generated in advance. For example, the pattern image may be stored in a frame memory. The division unit 102 can read out such a pattern image.

[0045] In S404, division unit 102 divides the foreground mask image using the pattern image obtained in S403. In this embodiment, division unit 102 divides the foreground mask image by calculating the logical product of pixel values ​​of pixels at the same coordinates between the foreground mask image and the pattern image. Equation (1) shows an equation for the logical product. F(x, y) represents the pixel value of the foreground mask image. L(x, y) represents the pixel value of the pattern image. D(x, y) represents the pixel value of the foreground mask image after division. x represents the X coordinate. y represents the Y coordinate. D(x,y)=F(x,y)&L(x,y) ··· Equation (1)

[0046] Fig. 3(D) shows the result of dividing the foreground mask image shown in Fig. 3(A) according to the pattern image shown in Fig. 3(C). Fig. 3(D) shows divided regions 303 obtained by dividing the foreground region. In this way, by dividing the foreground mask image, divided regions that are the division results of the foreground region can be obtained.

[0047] In S405, the rectangle calculation unit 103 calculates a circumscribing rectangle for each of the divided regions obtained by the division. In this embodiment, the rectangle calculation unit 103 calculates the coordinates of the circumscribing rectangle of the foreground region indicated by the divided foreground mask image obtained in S404. In the divided foreground mask image, the foreground region (region with pixel value "1") is divided by a line with pixel value "0". Therefore, by using a method of determining a circumscribing rectangle for each of a plurality of regions present in an image, the circumscribing rectangle of the foreground region in each of the divided mask images can be calculated. FIG. 3(E) shows a result of superimposing a circumscribing rectangle calculated from the divided foreground mask image shown in FIG. 3(D) on the foreground mask image shown in FIG. 3(A). FIG. 3(E) shows a circumscribing rectangle 304 of the divided region. Note that in this embodiment, the rectangle calculation unit 103 performs a process of expanding the circumscribing rectangle of the divided region thus calculated by one dot. That is, the X coordinate and the Y coordinate of each vertex of the circumscribing rectangle are moved outward by one dot. By performing this process, it is possible to fill in the one-dot gaps between the divided regions.

[0048] Here, the advantage of dividing the foreground region before calculating the circumscribing rectangle will be described. FIG. 3B shows the result of superimposing the circumscribing rectangle of the foreground region calculated from the foreground mask image shown in FIG. 3A on FIG. 3A. FIG. 3B shows the circumscribing rectangle 302 of the (undivided) foreground region. Comparing the area of ​​the circumscribing rectangle 302 shown in FIG. 3B with the total area of ​​the circumscribing rectangle 304 shown in FIG. 3E, the latter is smaller. Thus, according to the method of this embodiment, the amount of data transmitted in S406 is reduced. In particular, the larger the ratio of the foreground to the angle of view, the smaller the wasted area (area other than the subject) in the circumscribing rectangle. Therefore, the transfer load of the foreground image in S406 is further reduced.

[0049] In S406, the transmission unit 104 transmits the data of the input image within the circumscribing rectangle calculated in S405 to the generation device 200 as described above. The transmission unit 104 may further transmit a background image to the generation device 200.

[0050] In S407, the control unit 105 determines whether or not to continue the process. For example, if the user has not performed an operation to end the image processing shown in FIG. 4 using the operation unit 116, it is determined that the process is to be continued, and the process returns to S401. In this case, it is possible to calculate a circumscribing rectangle for the input image of the next frame. Also, if the user has performed an operation to end the image processing, the image processing shown in FIG. 4 ends.

[0051] As described above, according to this embodiment, a circumscribing rectangle is calculated for each of a plurality of divided regions obtained by dividing one foreground region. Then, data of the input image within the circumscribing rectangle is transmitted to the outside. With this configuration, it is possible to reduce the transmission load, as shown in Figs. 3(B) and (E).

[0052] (Modification) As described above, the interval of the grid used to divide the foreground region may be determined by the user. On the other hand, the interval of the grid may be determined automatically. In the following modified example, the control unit 105 determines the division method of the foreground region. For example, as described above, a plurality of divided regions can be obtained by dividing the foreground region according to a predetermined interval. Here, the control unit 105 can control this predetermined interval. For example, the control unit 105 can determine the interval of the grid (for example, the horizontal interval and / or the vertical interval). Also, the control unit 105 can determine the area of ​​each rectangle constituting the grid.

[0053] Specifically, the control unit 105 can control the predetermined interval based on the total area of ​​the circumscribing rectangle calculated according to the predetermined interval. In an example described later, the control unit 105 can generate a grid-like pattern image according to a transmission rate of the foreground image defined below. The control unit 105 can also control the predetermined interval according to the communication speed of the image processing device 100. For example, the control unit 105 can generate a grid-like pattern image according to the communication speed of the image processing device 100.

[0054] Hereinafter, the method of controlling the grid spacing will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the processing of the image processing device 100 in this modified example. Note that the processing of S401 to S407 is as described above, and the description will be omitted.

[0055] In S501, the control unit 105 initializes the interval of the grid. For example, the control unit 105 can set the interval of the grid to the same value as the resolution (i.e., the number of pixels) of the foreground mask image. Specifically, the control unit 105 can set the horizontal interval of the grid (Hperiod) to the horizontal resolution (Htotal) of the foreground mask image. In addition, the control unit 105 can set the vertical interval of the grid (Vperiod) to the vertical resolution (Vtotal) of the foreground mask image. In addition, the control unit 105 can set the number of divisions (DIV) of the grid to 1. In this case, the foreground mask image is not divided. However, the initialized interval of the grid is not limited to the above. For example, the interval of the grid may be initialized so that division is performed.

[0056] In S502, control unit 105 calculates the transmission rate of the foreground image. The transmission rate indicates the ratio of the sum of the areas of the circumscribing rectangles of the foreground region to the area of ​​the input image of one frame. In this example, the area can be expressed by the number of pixels. The transmission rate can be calculated according to formula (2). In formula (2), T is the transmission rate. Htotal is the horizontal resolution. Vtotal is the vertical resolution. Hwidth(n) is the width of the nth circumscribing rectangle. Vhigh(n) is the height of the nth circumscribing rectangle. N is the total number of circumscribing rectangles in the input image of one frame.

number

[0057] If the transmission rate is equal to or lower than the upper limit, control unit 105 determines that the foreground image can be transmitted, and the process proceeds to S505. If the transmission rate is greater than the upper limit, control unit 105 determines that the foreground image cannot be transmitted, and the process proceeds to S503.

[0058] In S503, the control unit 105 updates the grid interval so that the grid interval becomes narrower. For example, the control unit 105 can first increment the number of divisions (DIV). Then, the control unit 105 can calculate the grid interval based on the formulas (3) and (4). Hperiod = Htotal / DIV...Equation (3) Vperiod = Vtotal / DIV...Equation (4)

[0059] According to the interval thus updated, a grid-like pattern image used to divide the foreground region of the input image of the next frame is generated. The method of calculating the grid interval is not limited to the above method. For example, it is not necessary to use the same number of divisions (DIV) in equation (3) and equation (4). That is, the number of divisions (DIV) used in equation (3) and the number of divisions (DIV) used in equation (4) may be different. For example, the number of divisions (DIV) used in equation (3) and the number of divisions (DIV) used in equation (4) may be controlled according to the ratio of the horizontal resolution (Htotal) and the vertical resolution (Vtotal) of the foreground mask image. As an example, when Htotal is about twice Vtotal, the number of divisions (DIV) used in equation (3) may be twice the number of divisions (DIV) used in equation (4). With such a configuration, the shape of the grid becomes approximately square.

[0060] Also, an upper limit may be set for the number of divisions (DIV). With this configuration, the area of ​​the circumscribing rectangle becomes extremely small due to an increase in the number of divisions, and the transmitting unit 104 is prevented from determining that the circumscribing rectangle is derived from noise and excluding it from the transmission target. Also, with this configuration, an increase in the transmission load caused by an extremely large number of circumscribing rectangles is prevented.

[0061] In S504, transmission unit 104 transmits null data to generation device 200. This configuration prevents a foreground image from being transmitted even if the upper limit of the transmission rate is exceeded. This also prevents generation device 200 from performing processing based on a foreground image that has not been completely transmitted due to a limit in the transfer speed.

[0062] In S505, control unit 105 calculates the difference between the transmission rate calculated in S502 and the upper limit of the transmission rate. Then, control unit 105 compares this difference value with a margin value. This margin value may be determined in advance in the image processing system. If the difference value is equal to or greater than the margin value, control unit 105 determines that there is a margin in the transmission rate, and the process proceeds to S506. If the difference value is smaller than the margin value, control unit 105 determines that there is no margin in the transmission rate, but the foreground image can be transmitted, and the process proceeds to S406.

[0063] In S506, the control unit 105 updates the lattice interval so that the lattice interval becomes wider. For example, the control unit 105 can first decrement the number of divisions (DIV). Then, the control unit 105 can calculate the lattice interval based on the above-mentioned formulas (3) and (4).

[0064] According to the above-mentioned modified example, it is possible to automatically control the grid spacing to be narrowed or widened. The transmission rate changes according to the installation position and angle of view of the imaging device 150, and the position of the subject. Therefore, the optimal grid spacing also differs for each imaging device 150. Therefore, the burden on the user to sequentially set the grid spacing for each of the multiple imaging devices 150 is large. According to this modified example, it is possible to automatically determine the grid spacing for each of the multiple imaging devices 150.

[0065] The method by which the control unit 105 determines how to divide the foreground region or the foreground mask region is not limited to the method of controlling the interval of the grid as described above. For example, the control unit 105 may shift the position of the grid. Specifically, the control unit 105 may control the start position of the grid. For example, the control unit 105 may control the condition under which the division unit 102 draws the grid lines as described above. In the first embodiment, the grid lines are drawn on the pixel of interest when the counter value is "0". However, the grid lines may be drawn on the pixel of interest when the counter value is a specific value other than "0". The start position of the grid can be controlled by changing this specific value. For example, the specific value may be incremented for each frame so as to scroll the grid. Also, the specific value may be randomly determined for each frame. When the grid lines are located near the edge of the foreground region, the area of ​​the circumscribing rectangle becomes extremely small, and the transmission unit 104 may determine that the circumscribing rectangle is derived from noise and may not be transmitted. However, by shifting the starting position of the grid in this way, it is possible to prevent image data for a part of the foreground region from being continuously excluded from transmission. As described in the second embodiment, if the area of ​​a circumscribing rectangle for a divided region adjacent to another divided region is less than a threshold, the rectangle calculation unit 103 may enlarge the circumscribing rectangle.

[0066] [Embodiment 2] In the first embodiment, the foreground mask image is divided into a grid, and a circumscribing rectangle is calculated from each region. However, the method of calculating the circumscribing rectangle for each divided region obtained by dividing the foreground region is not limited to this method. For example, the foreground region can be divided by any method, and a circumscribing rectangle can be calculated for each divided region. In addition, when the area of ​​the circumscribing rectangle is small, the circumscribing rectangles adjacent to each other can be integrated. In the second embodiment, an example of such a method will be described.

[0067] The configuration of an image processing device 100 according to this embodiment will be described with reference to Fig. 7. The image processing device 100 has a rectangle integration unit 106 in addition to a foreground separation unit 101, a rectangle calculation unit 103, a transmission unit 104, and a control unit 105. The differences from the first embodiment will be described below.

[0068] In this embodiment, the rectangle calculation unit 103 divides the foreground region into a plurality of regions. For example, the rectangle calculation unit 103 can calculate a plurality of rectangles that are part of the circumscribing rectangle and are adjacent to each other as a preprocessing for calculating the circumscribing rectangle of the foreground region. A set of these multiple rectangles can include one foreground region. In this embodiment, the multiple rectangles calculated by the rectangle calculation unit 103 are rectangles that circumscribing each of the regions obtained by dividing the foreground region. Each rectangle can be searched and calculated according to a raster scan.

[0069] The rectangle integration unit 106 sets one divided region by integrating a group of adjacent regions that are part of the multiple regions. In this embodiment, the rectangle integration unit 106 integrates adjacent rectangles one by one. Integration refers to calculating a circumscribing rectangle for two adjacent rectangles. That is, the rectangle integration unit 106 can calculate a circumscribing rectangle for a group of rectangles as a result of integrating the group of rectangles. In this way, the rectangle integration unit 106 can calculate a circumscribing rectangle for one divided region by integrating a group of adjacent rectangles that are part of the rectangles circumscribing each of the multiple regions.

[0070] Here, the rectangle integration unit 106 can set the divided regions such that the area of ​​the circumscribing rectangle for one divided region is equal to or larger than a threshold value. For example, the rectangle integration unit 106 can determine whether or not to integrate an additional rectangle into the rectangle group, depending on the area of ​​the circumscribing rectangle calculated as the integration result of the rectangle group. Specifically, the rectangle integration unit 106 can calculate the area of ​​the circumscribing rectangle calculated as the integration result, and compare this area with a threshold value. Then, based on the comparison result, the rectangle integration unit 106 can determine whether or not to integrate an additional rectangle into the circumscribing rectangle calculated as the integration result.

[0071] When calculating a bounding rectangle for a foreground region of a complex shape using hardware according to the raster scan method, it is not easy to calculate the bounding rectangle at one time (within the period of one frame) due to restrictions on circuit size and processing time. For this reason, in this embodiment, after the rectangle calculation unit 103 calculates multiple rectangles as preprocessing, the rectangle integration unit 106 integrates the rectangles to calculate a bounding rectangle for each divided region.

[0072] Control unit 105 controls the area threshold value referenced by rectangle integration unit 106. Control unit 105 can control the threshold value according to the total area of ​​circumscribing rectangles calculated according to the threshold value. In this embodiment, control unit 105 controls this threshold value according to the transmission rate of the foreground image. Control unit 105 can also control this threshold value according to the communication speed of image processing device 100.

[0073] The process performed by the image processing device 100 in this embodiment will be described below with reference to Fig. 6 and Fig. 8. Fig. 6 is a flowchart of the process performed by the image processing device 100 in this embodiment.

[0074] In S601, the control unit 105 initializes the area threshold and the number of divisions (DIV) to be described later, which are referred to in S607. In this example, the area threshold is initialized to the "approximate area of ​​the circumscribing rectangle of the foreground area." Also, the number of divisions is initialized to "1." The approximate area of ​​the circumscribing rectangle can be determined by the user depending on the angle of view of the camera and the size of the subject to be photographed. However, the initial value is not limited to the above. For example, the number of divisions may be initialized so that division is performed.

[0075] Steps S602 and S603 are performed in the same manner as steps S401 and S402. Fig. 8A shows an example of the foreground mask image generated in step S603. A foreground region 801 is shown in Fig. 8A.

[0076] In S604, the rectangle calculation unit 103 calculates information on a plurality of rectangles that are part of the circumscribing rectangle and are adjacent to each other as a preprocessing for calculating the circumscribing rectangle of the foreground region. As described above, the rectangle calculation unit 103 can search and calculate each rectangle according to raster scan. FIG. 8B shows a plurality of rectangles 802 calculated in this way. Each rectangle is labeled with a number for identifying the rectangle in the order of calculation. Also, as information indicating the adjacency relationship between the rectangles, a number indicating the adjacent rectangle is recorded for each rectangle. The rectangle calculation unit 103 can calculate the vertex coordinates, number, and information indicating the adjacency relationship of each of these rectangles as information for each rectangle. In this example, a plurality of rectangles adjacent in the vertical direction (or horizontal direction) are calculated for one foreground region.

[0077] The rectangle calculation unit 103 can detect rectangles consisting of pixels of the continuous foreground region for each row of the foreground mask image. In addition, when a rectangle located in an upper row is adjacent to a rectangle located in a row adjacent below, and the rectangle located in the upper row is expanded downward, if the rectangle located in the row adjacent below is included in the expanded rectangle, the rectangle located in the upper row and the rectangle located in the row adjacent below are integrated. In this way, multiple rectangles 802 shown in FIG. 8(B) can be calculated.

[0078] In S605, the rectangle integration unit 106 integrates the rectangles calculated in S604. The rectangle integration unit 106 can integrate the two selected rectangles. The rectangle integration unit 106 can generate, as information on the rectangle obtained by integration, the vertex coordinates of the circumscribing rectangles of the two rectangles, the numbers of the two rectangles, and information indicating the adjacency relationship between the two rectangles.

[0079] In S605, the rectangle integration unit 106 can integrate an integrated rectangle (N) with a rectangle adjacent to this rectangle in accordance with the adjacency information. The integrated rectangle refers to a rectangle obtained by the integration process in the loop of S605 to S608. The integrated rectangle (N) refers to a rectangle into which a rectangle is integrated in the (N+1)th loop of S605 to S611. In the loop of S605 to S608, rectangles adjacent to the integrated rectangle (N) are integrated one by one. However, in the first loop of S605 to S608, the integrated rectangle (N) does not yet exist. Therefore, in the first loop, the integrated rectangle (N) is treated as a NULL value.

[0080] For example, when N=0, in S605 in the first loop of S605 to S608, the NULL value is integrated with the rectangle 803 with the smallest number shown in Fig. 8(B) to turn the integrated rectangle (0) into rectangle 803. Then, from the second loop onwards, adjacent rectangles directly below rectangle 803 are integrated one after another. Finally, rectangle 805 shown in Fig. 8(C) is obtained as the completed integrated rectangle (0).

[0081] Also, when N=1, in S605 in the first loop of S605 to S608, a NULL value and an uncombined rectangle 804 with the smallest number are combined, and combined rectangle (1) becomes rectangle 804. Then, from the second loop onwards, adjacent rectangles directly below rectangle 804 are combined one after another. Finally, rectangle 806 shown in FIG. 8C is obtained as the completed combined rectangle (1).

[0082] In S606, the rectangle integration unit 106 calculates the area of ​​the integrated rectangle (N) obtained by the integration in S605. The rectangle integration unit 106 can calculate the area from the vertex coordinates.

[0083] In S607, the rectangle integration unit 106 compares the area calculated in S606 with the area threshold set by the control unit 105. If this area is smaller than the threshold, the process proceeds to S608. If this area is equal to or larger than the threshold, the generation of the integrated rectangle (N) is completed. That is, the vertex coordinates of the integrated rectangle (N) are determined. Then, the process proceeds to S610.

[0084] In S608, the rectangle integration unit 106 refers to the adjacent information for the integrated rectangle (N) obtained in S605 and determines whether or not there is a rectangle adjacent to the integrated rectangle (N) to be integrated next. If there is an adjacent rectangle, the process returns to S605. Then, the rectangle adjacent to the integrated rectangle (N) is integrated into the integrated rectangle (N). If there is no adjacent rectangle, it means that the last rectangle among multiple adjacent rectangles is integrated and the area of ​​the integrated rectangle (N) does not meet the threshold value. In this case, the integrated rectangle (N) is determined to be a fractional rectangle. FIG. 8(C) shows such a fractional rectangle 807. Then, the process proceeds to S609.

[0085] In S609, the rectangle integration unit 106 expands the integrated rectangle (N). In this example, the area of ​​the integrated rectangle (N) is less than the threshold. In this way, the rectangle integration unit 106 can expand the circumscribing rectangle when the area of ​​the circumscribing rectangle for a divided area adjacent to another divided area is less than the threshold. For example, the rectangle integration unit 106 can move the vertex coordinates of the integrated rectangle (N). Specifically, the rectangle integration unit 106 can increase the Y coordinates of the lower left and lower right vertex coordinates of the integrated rectangle (N) so that the area of ​​the integrated rectangle (N) becomes equal to the area threshold set by the control unit 105. FIG. 8(D) shows a rectangle 808 expanded in this way. The rectangle integration unit 106 may also move the X coordinates of the vertices. In addition, when the coordinates after expansion exceed the coordinates of the maximum value of the vertical or horizontal coordinates, the coordinates after expansion can be clipped to the coordinates of the maximum value.

[0086] In S610, the rectangle integration unit 106 increments N. In S611, the rectangle integration unit 106 determines whether or not integration processing has been completed for all rectangles. For example, the rectangle integration unit 106 determines whether the number of the rectangle last integrated in S605 is equal to the number of the rectangle last labeled in S604. If these numbers are equal, the rectangle integration unit 106 determines that integration processing has been completed for all rectangles. Then, the process proceeds to S612. If these numbers are not equal, the rectangle integration unit 106 determines that there remains a rectangle that has not been the target of integration processing. Then, the process returns to S605. Then, in the next loop of S605 to S611, the next integrated rectangle (N) is generated.

[0087] Each integrated rectangle shown in FIG. 8(C) thus obtained represents a circumscribing rectangle for each divided region obtained by dividing one foreground region. Also, in FIG. 8(D), each integrated rectangle other than at least the last obtained rectangle 808 represents a circumscribing rectangle for each divided region obtained by dividing one foreground region. In this example, since rectangle 807 is a circumscribing rectangle for a part of one foreground region, it is unlikely that it is caused by noise. In order to prevent image data for this rectangle 807 from being excluded from the transmission target, rectangle 807 is enlarged to rectangle 808. However, it is not essential to enlarge a circumscribing rectangle smaller than the threshold value in this manner.

[0088] In S612, control unit 105 compares the transmission rate of the foreground image with the upper limit, as in S502. In this embodiment, the transmission rate indicates the ratio of the total area of ​​integrated rectangles to the area of ​​the input image of one frame. If the transmission rate is equal to or less than the upper limit, the process proceeds to S615. If the transmission rate is greater than the upper limit, the process proceeds to S613.

[0089] In S613, the control unit 105 updates the area threshold to be smaller. For example, the control unit 105 can increment the number of divisions (DIV). Then, the control unit 105 calculates the threshold according to formula (5). In formula (5), TH indicates the area threshold. Also, THini indicates the initial value of the area set in S601. Note that, as in the modified example of the first embodiment, an upper limit may be set on the number of divisions (DIV). TH = THini / DIV ··· Equation (5)

[0090] S614 is performed in the same manner as S504.

[0091] In S615, the control unit 105 compares the difference between the transmission rate calculated in S612 and the upper limit with the margin value, as in S505. If the difference is equal to or greater than the margin value, the process proceeds to S616. If the difference is smaller than the margin value, the process proceeds to S617.

[0092] In S616, the control unit 105 updates the area threshold to a larger value. For example, the control unit 105 can first decrement the number of divisions (DIV). Then, the control unit 105 can calculate the area threshold based on the above-mentioned formula (5).

[0093] S617 to S618 are performed in the same manner as S406 to S407. In this embodiment, the transmission target of the foreground image and the foreground mask image in S406 is limited to within each integrated rectangle. Also, if it is determined in S407 that the process is to be continued, the process returns to S601.

[0094] As described above, according to this embodiment, a circumscribing rectangle is calculated for each of a plurality of divided regions obtained by dividing one foreground region. Then, data of the input image in the circumscribing rectangle is transmitted to the outside. This can reduce the transmission load. In particular, according to this embodiment, a circumscribing rectangle is calculated to have a certain area. This can prevent the calculation of an extremely small circumscribing rectangle that may be determined to be derived from noise.

[0095] (Other embodiments) The contents of the present disclosure can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. Also, the contents of the present disclosure can be realized by a circuit (e.g., ASIC) for realizing one or more functions.

[0096] Also, each of the image processing devices 100 at the rear stage of the multiple imaging devices 150 may be connected to the generating device 200, or the multiple image processing devices 100 may be cascade-connected. FIG. 9 shows a configuration example of an image processing system in which multiple image processing devices 100 are cascade-connected. The image processing system may further include multiple imaging devices 150 and a generating device 200. That is, in FIG. 9, data output from the transmission unit 104 of the upstream image processing device 100 is input to the transmission unit 104 of the downstream image processing device 100, and is further transmitted to the subsequent image processing device 100. That is, data processed by the upstream image processing device 100 is received by the downstream image processing device 100. Also, the downstream image processing device 100 transmits data received from the upstream image processing device 100 and data processed by itself to the further downstream image processing device 100, and transmits data to the generating device 200 via the image processing device 100. Thus, according to this cascade connection, data processed by each image processing device 100 is transmitted to the generating device 200. In such a cascade connection configuration, the transmission band for the amount of data becomes tighter, so by applying the above-mentioned embodiment, the tightness of the transmission band is suppressed. Note that all of the multiple image processing devices 100 used in this system may be connected in one cascade connection, or the multiple image processing devices 100 may be classified into several groups, and each group may be cascade-connected.

[0097] Thus, the image processing system according to an embodiment may include a plurality of image processing devices 100 and a generating device 200. The plurality of image processing devices 100 may include a first image processing device and a second image processing device. Here, the second image processing device may receive data processed by the first image processing device. For example, the first image processing device may calculate a circumscribing rectangle of the foreground region from an input image captured by an imaging device 150 connected to the first image processing device as described above. Then, the first image processing device may transmit data of the input image within the calculated circumscribing rectangle to the second image processing device according to the above-mentioned method. Also, the second image processing device may transmit data received from the first image processing device and data processed by the second image processing device to the generating device 200. For example, the second image processing device may also calculate a circumscribing rectangle of the foreground region from an input image captured by an imaging device 150 connected to the second image processing device as described above. Then, the second image processing device may transmit data of the input image within the calculated circumscribing rectangle to the generating device 200 together with data received from the first image processing device according to the above-mentioned method. At this time, the second image processing device can transmit data to the generating device 200 via another image processing device 100.

[0098] The disclosure of this specification includes the following image processing device, image processing method, program, and image processing system. (Item 1) A detection means for detecting a foreground region from an input image; A calculation means for calculating a circumscribing rectangle for each of a plurality of divided regions obtained by dividing one foreground region; a transmitting means for transmitting data of the input image within the circumscribing rectangle to an external device; 13. An image processing device comprising: (Item 2) 2. The image processing device according to item 1, wherein the detection means generates a foreground mask image indicating the foreground region. (Item 3) 3. The image processing device according to item 2, wherein the calculation means divides the foreground mask image and calculates a circumscribing rectangle for each of the plurality of divided regions according to each partial image of the foreground mask image obtained by the division. (Item 4) 4. The image processing device according to any one of items 1 to 3, wherein the plurality of divided regions are obtained by dividing the foreground region in a horizontal direction, a vertical direction, or a grid pattern. (Item 5) 5. The image processing device according to any one of items 1 to 4, wherein the plurality of divided regions are obtained by dividing the foreground region according to a predetermined interval. (Item 6) 6. The image processing device according to item 5, further comprising a control means for controlling the predetermined interval. (Item 7) 7. The image processing device according to item 6, wherein the control means controls the predetermined interval based on a total area of ​​the circumscribing rectangle calculated according to the predetermined interval. (Item 8) The calculation means is means for dividing the foreground region into a plurality of regions; a means for setting one of the divided regions by processing for integrating a group of adjacent regions that are a part of the plurality of regions; 8. The image processing device according to any one of items 1 to 7, comprising: (Item 9) 9. The image processing device according to item 8, wherein the calculation means sets the divided regions such that the area of ​​a circumscribing rectangle for one of the divided regions is equal to or larger than a threshold value. (Item 10) 10. The image processing device according to item 9, further comprising a control means for controlling the threshold value. (Item 11) Item 11. The image processing device according to item 10, wherein the control means controls the threshold value in accordance with a total area of ​​the circumscribing rectangle calculated according to the threshold value. (Item 12) The calculation means is means for calculating a rectangle circumscribing each of a plurality of regions obtained by dividing the foreground region; a means for calculating the circumscribing rectangle for one of the divided regions by integrating a group of adjacent rectangles that are part of the rectangles circumscribing each of the plurality of regions; 12. The image processing device according to any one of items 1 to 11, comprising: (Item 13) Item 13. The image processing device according to item 12, characterized in that the calculation means calculates a circumscribing rectangle of the group of rectangles as a result of integrating the group of rectangles, and determines whether or not to integrate further rectangles into the group of rectangles depending on the area of ​​the calculated circumscribing rectangle. (Item 14) 14. The image processing device according to any one of items 1 to 13, wherein the calculation means enlarges the circumscribing rectangle for a divided area adjacent to another divided area when the area of ​​the circumscribing rectangle for the divided area adjacent to another divided area is less than a threshold value. (Item 15) The image processing device according to any one of items 2 to 14, which cites item 2, is characterized in that the transmission means further transmits data of the foreground mask image within the circumscribing rectangle to an outside. (Item 16) the input image is an image of one frame of a video sequence, 16. The image processing device according to any one of items 1 to 15, wherein the transmission means transmits data of the input image within the circumscribing rectangle for each frame to the outside, and excludes data of the input image outside the circumscribing rectangle for each frame from data to be transmitted to the outside. (Item 17) 17. The image processing device according to any one of items 1 to 16, wherein the transmission means further transmits background data to the outside. (Item 18) 18. The image processing device according to any one of items 1 to 17, wherein the transmission means excludes data of the input image within the circumscribing rectangle from data to be transmitted to the outside if the area of ​​the circumscribing rectangle is less than a threshold value. (Item 19) An image processing method performed by an image processing device, comprising: detecting foreground regions from an input image; calculating a circumscribing rectangle for each of a plurality of divided regions obtained by dividing one foreground region; transmitting data of the input image within the circumscribing rectangle to an external device; 13. An image processing method comprising: (Item 20) 19. A program for causing a computer to function as the image processing device according to any one of items 1 to 18. (Item 21) A plurality of image processing devices and a generating device that generates a virtual viewpoint image, the plurality of image processing devices include a first image processing device and a second image processing device; The second image processing device receives data to be processed by the first image processing device, and transmits the data received from the first image processing device and the data to be processed by the second image processing device to the generation device; The first image processing device, A detection means for detecting a foreground region from an input image; A calculation means for calculating a circumscribing rectangle for each of a plurality of divided regions obtained by dividing one foreground region; a transmitting unit for transmitting data of the input image within the circumscribing rectangle to an outside,

[0099] The present disclosure is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the present disclosure. Accordingly, the following claims are appended to disclose the scope of the present disclosure. [Explanation of symbols]

[0100] 100: image processing device, 101: foreground separation unit, 102: division unit, 103: rectangle calculation unit, 104: transmission unit, 105: control unit, 106: rectangle integration unit, 150: imaging device, 200: generation device

Claims

1. A detection means for detecting a foreground region from an input image; A calculation means for calculating a circumscribing rectangle for each of a plurality of divided regions obtained by dividing one foreground region; a transmitting means for transmitting data of the input image within the circumscribing rectangle to an external device; 13. An image processing device comprising:

2. The image processing apparatus according to claim 1 , wherein the detection means generates a foreground mask image indicating the foreground region.

3. The image processing device according to claim 2 , wherein the calculation means divides the foreground mask image, and calculates a circumscribing rectangle for each of the plurality of divided regions in accordance with each partial image of the foreground mask image obtained by the division.

4. The image processing device according to claim 1 , wherein the plurality of divided regions are obtained by dividing the foreground region in a horizontal direction, a vertical direction, or a lattice shape.

5. The image processing device according to claim 1 , wherein the plurality of divided regions are obtained by dividing the foreground region according to a predetermined interval.

6. 6. The image processing apparatus according to claim 5, further comprising a control means for controlling the predetermined interval.

7. 7. The image processing apparatus according to claim 6, wherein the control means controls the predetermined interval based on a total area of ​​the circumscribing rectangle calculated in accordance with the predetermined interval.

8. The calculation means is means for dividing the foreground region into a plurality of regions; a means for setting one of the divided regions by processing for integrating a group of adjacent regions that are a part of the plurality of regions; The image processing device according to claim 1 , further comprising:

9. The image processing device according to claim 8 , wherein the calculation means sets the divided regions such that an area of ​​a circumscribing rectangle for one of the divided regions is equal to or larger than a threshold value.

10. 10. The image processing device according to claim 9, further comprising a control means for controlling the threshold value.

11. 11. The image processing device according to claim 10, wherein the control means controls the threshold value in accordance with a total area of ​​the circumscribing rectangle calculated in accordance with the threshold value.

12. The calculation means is means for calculating a rectangle circumscribing each of a plurality of regions obtained by dividing the foreground region; a means for calculating the circumscribing rectangle for one of the divided regions by integrating a group of adjacent rectangles that are part of the rectangles circumscribing each of the plurality of regions; The image processing device according to claim 1 , further comprising:

13. 13. The image processing device according to claim 12, wherein the calculation means calculates a circumscribing rectangle of the group of rectangles as a result of integrating the group of rectangles, and determines whether or not to integrate further rectangles into the group of rectangles depending on the area of ​​the calculated circumscribing rectangle.

14. The image processing device according to claim 1 , wherein the calculation means enlarges the circumscribing rectangle when the area of ​​the circumscribing rectangle for a divided area adjacent to another divided area is less than a threshold value.

15. 3. The image processing apparatus according to claim 2, wherein said transmission means further transmits data of the foreground mask image within said circumscribing rectangle to an external device.

16. the input image is an image of one frame of a video sequence, 2. The image processing device according to claim 1, wherein the transmission means transmits data of the input image within the circumscribing rectangle for each frame to the outside, and excludes data of the input image outside the circumscribing rectangle for each frame from data to be transmitted to the outside.

17. 17. The image processing apparatus according to claim 16, wherein said transmission means further transmits background data to said external device.

18. 2 . The image processing apparatus according to claim 1 , wherein said transmission means excludes data of said input image within said circumscribing rectangle from data to be transmitted to the outside when the area of ​​said circumscribing rectangle is less than a threshold value.

19. An image processing method performed by an image processing device, comprising: detecting foreground regions from an input image; A step of calculating a circumscribing rectangle for each of a plurality of divided regions obtained by dividing one foreground region; transmitting data of the input image within the circumscribing rectangle to an external device; 13. An image processing method comprising:

20. A program for causing a computer to function as the image processing device according to any one of claims 1 to 18.

21. A plurality of image processing devices and a generating device that generates a virtual viewpoint image, the plurality of image processing devices include a first image processing device and a second image processing device; The second image processing device receives data to be processed by the first image processing device, and transmits the data received from the first image processing device and the data to be processed by the second image processing device to the generation device; The first image processing device includes: A detection means for detecting a foreground region from an input image; A calculation means for calculating a circumscribing rectangle for each of a plurality of divided regions obtained by dividing one foreground region; a transmitting unit for transmitting data of the input image within the circumscribing rectangle to an outside,

Images