Information processing apparatus, information processing method, and program

Abstract

To improve extraction accuracy when using a trained model to extract a foreground.SOLUTION: An information processing apparatus 100 specifies a first area in an image plane corresponding to a first space in which a target object may exist within a field of view of an imaging apparatus, and a second area in the image plane corresponding to a second space in which accuracy in separation between a foreground area and a background area may be degraded within the field of view in foreground / background separation processing, and the information processing apparatus 100 determines a plurality of divided areas for dividing an image being a processing object in the foreground / background separation processing on the basis of the specified first area and second area.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to the art of foreground-background separation. [Background technology]

[0002] There is a technique called foreground / background separation, which separates a foreground region from a background region by extracting a region corresponding to an image of a target object (hereinafter simply referred to as "object") from an image as the foreground region. Patent Document 1 discloses a method of foreground / background separation, i.e., extraction of a foreground region (hereinafter referred to as "foreground extraction"), using a trained model obtained as a result of learning by machine learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-182320 Summary of the Invention [Problem to be solved by the invention]

[0004] When performing foreground extraction processing using the above-described trained model, a conceivable method is to perform foreground extraction processing in parallel on each of multiple equally divided images obtained by simply dividing the image to be processed equally. However, when simply dividing the image to be processed equally, areas in the image to be processed that contain images of objects are not taken into consideration. Therefore, the equally divided images may not be sufficient for extracting foreground areas, which may result in reduced accuracy of foreground extraction in the image to be processed. Therefore, an object of the present disclosure is to provide a technology for improving extraction accuracy when performing foreground extraction using a trained model obtained as a result of learning through machine learning. [Means for solving the problem]

[0005] The information processing device according to the present disclosure includes an identification means for identifying a first region in an image plane corresponding to a first space in which a target object may exist within an angle of view of an imaging device, and a second region in the image plane corresponding to a second space within the angle of view in which the accuracy of separation between a foreground region and a background region by foreground-background separation processing may deteriorate, and a determination means step for determining a plurality of divided regions for dividing an image to be processed in the foreground-background separation processing based on the identified first and second regions. [Effects of the Invention]

[0006] According to the present disclosure, it is possible to improve extraction accuracy when performing foreground extraction using a trained model. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a block diagram illustrating an example of a configuration of an information processing system according to a first embodiment. [Figure 2] 1 is a block diagram showing an example of a functional configuration of an information processing device according to a first embodiment. [Figure 3] 3 is a block diagram showing an example of the internal configuration of a rectangle generation unit according to the first embodiment. FIG. [Figure 4] 3A and 3B are diagrams for explaining an example of an angle of view and a rectangular region according to the first embodiment. [Figure 5] 1 is a block diagram showing an example of a hardware configuration of an information processing device according to a first embodiment. [Figure 6] 4 is a flowchart showing an example of a processing flow of the information processing device according to the first embodiment. [Figure 7] FIG. 10 is a block diagram showing an example of a functional configuration of an information processing device according to a second embodiment. [Figure 8] 10 is a flowchart showing an example of a processing flow of an information processing device according to a second embodiment. [Figure 9] FIG. 11 is a block diagram showing an example of a functional configuration of an information processing device according to a third embodiment. [Figure 10]11 is a flowchart showing an example of a processing flow of an information processing device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the means for solving the problems according to the present disclosure are not limited to the configurations shown in the following embodiments.

[0009] [Embodiment 1] <Configuration of information processing system> 1 is a block diagram showing an example of the configuration of an information processing system 1. The information processing system 1 includes a first input device 110, a second input device 120, a first image processing device 130, an information processing device 100, a second image processing device 140, and a third image processing device 150.

[0010] The first input device 110 is connected to the first image processing device 130 so that they can communicate with each other. The first input device 110 inputs data (hereinafter referred to as "captured image data") of images obtained by capturing images using a plurality of image capturing devices (hereinafter referred to as "captured images") to the first image processing device 130 and the third image processing device 150. The second input device 120 is connected to the information processing device 100 so that they can communicate with each other. The second input device 120 inputs data of a background model (hereinafter referred to as "background data") used for volumetric video and information indicating the position of each image capturing device (hereinafter referred to as "captured information") to the information processing device 100 and the third image processing device 150.

[0011] The information processing device 100 determines a rectangular area for performing foreground / background separation processing based on the background data and each piece of imaging information input by the second input device 120. Information indicating the rectangular area determined by the information processing device 100 (hereinafter referred to as "rectangle information") is output to the first image processing device 130 and the second image processing device 140.

[0012] The first image processing device 130 generates a plurality of rectangular images by dividing the captured image input by the first input device 110 into a plurality of rectangular regions based on the rectangular information acquired from the information processing device 100. Next, the first image processing device 130 performs foreground / background separation processing on each of the generated rectangular images to extract a foreground region from each rectangular image. The first image processing device 130 generates a mask image (hereinafter referred to as a "rectangular foreground mask image") corresponding to the foreground region extracted from the rectangular image for each rectangular image. The first image processing device 130 is connected to the second image processing device 140 so that they can communicate with each other. The first image processing device 130 outputs data of the rectangular foreground mask image (hereinafter referred to as "rectangular foreground mask image data") generated for each rectangular image to the second image processing device 140. Note that the extraction of the foreground region corresponding to the object in the first image processing device 130 uses a trained model obtained as a result of machine learning of a predefined learning model. In the following description, the trained model used to extract the foreground region is assumed to be constructed using a convolutional neural network (CNN). However, the trained model is not limited to a CNN and may be constructed using a device other than a CNN.

[0013] The second image processing device 140 generates a foreground mask image corresponding to the image region of each captured image input to the first image processing device 130, based on the multiple rectangular foreground mask image data acquired from the first image processing device 130 and the rectangle information acquired from the information processing device 100. Specifically, the second image processing device 140 generates a foreground mask image corresponding to each captured image by combining the multiple rectangular foreground mask images divided into rectangles, based on the rectangle information. The second image processing device 140 outputs data of the multiple generated foreground mask images (hereinafter referred to as "foreground mask image data") to the third image processing device 150.

[0014] The third image processing device 150 generates shape data indicating the three-dimensional shape of an object based on foreground mask image data corresponding to each captured image acquired from the second image processing device 140 and each piece of imaging information input by the second input device 120. Specifically, for example, the third image processing device 150 acquires shape data based on a plurality of pieces of foreground mask image data and a plurality of pieces of imaging information using a volume intersection method. Note that the method for generating shape data in the third image processing device 150 is not limited to the volume intersection method, as long as the shape data is based on foreground mask image data.

[0015] Furthermore, the third image processing device 150 generates a virtual viewpoint image corresponding to the view from an arbitrary viewpoint (hereinafter referred to as a "virtual viewpoint") set by a user or the like, based on the acquired shape data and background data input by the second input device 120. The method for generating the virtual viewpoint image in the third image processing device 150 may be a known volumetric video generation method. The third image processing device 150 outputs the generated virtual viewpoint image. Specifically, for example, the third image processing device 150 may output a display image including the virtual viewpoint image as a display control signal to display the image on a display device not shown in FIG. 1, or may output data of the virtual viewpoint image to store the data in a storage device not shown in FIG. 1.

[0016] <Functional configuration of information processing device> FIG. 2 is a block diagram showing an example of the functional configuration of the information processing device 100 according to the first embodiment. The information processing device 100 includes a background model acquisition unit 201, an information acquisition unit 202, a field of view generation unit 203, a rectangle generation unit 204, and an output unit 205. The background model acquisition unit 201 acquires background model data (background data) input via the second input device 120. The background data acquired by the background model acquisition unit 201 is accompanied by complex space information indicating a complex space in the background and imaging space information indicating an imaging space. Here, the imaging space specifically refers to the range of imaging space in which a volumetric video can be generated or the range of space in which an object can exist within the field of view of an imaging device. Furthermore, the complex space refers to the range of space in which it is considered difficult to separate a foreground region from a background region by foreground-background separation processing, such as a space corresponding to spectator seats in a venue to be imaged. The background data acquired by the background model acquisition unit 201 is transmitted to the field of view generation unit 203. The information acquisition unit 202 acquires imaging information indicating the position of the imaging device input by the second input device 120. The imaging information acquired by the information acquisition unit 202 is transmitted to the field of view generation unit 203.

[0017] The angle-of-view generation unit 203 generates angle-of-view information based on the background data and imaging information acquired by the background model acquisition unit 201 or the information acquisition unit 202. Specifically, the angle-of-view generation unit 203 identifies the imaging space and the complex space in the background within the angle of view of each imaging device, and generates angle-of-view information indicating an image region corresponding to the imaging space and an image region corresponding to the complex space in each captured image. Hereinafter, the image region corresponding to the complex space will be referred to as a "complex region," and the image region corresponding to the imaging space will be referred to as an imaging region. The angle-of-view information generated by the angle-of-view generation unit 203 is transmitted to the rectangle generation unit 204. The rectangle generation unit 204 receives the angle-of-view information from the angle-of-view generation unit 203, and generates rectangle information indicating a plurality of rectangular regions for dividing the captured image when the first image processing device 130 performs foreground / background separation processing, based on the angle-of-view information corresponding to each imaging device. The rectangle information generated by the rectangle generation unit 204 is transmitted to the output unit 205. The output unit 205 outputs the rectangle information received from the rectangle generation unit 204 to the first image processing device 130, the second image processing device 140, etc. The output unit 205 may output the rectangle information to a storage device not shown in FIGS. 1 and 2 and store the rectangle information in the storage device.

[0018] FIG. 3 is a block diagram showing an example of the internal configuration of the rectangle generation unit 204 according to the first embodiment. The rectangle generation unit 204 includes a field of view acquisition unit 301, a rectangle setting unit 302, a proportion acquisition unit 303, a direction acquisition unit 304, and a rectangle determination unit 305. The field of view acquisition unit 301 receives field of view information from the field of view generation unit 203 and transmits the received field of view information to the proportion acquisition unit 303 and the direction acquisition unit 304. FIG. 4 is a diagram for explaining an example of field of view information and rectangular regions for dividing a captured image according to the first embodiment. Specifically, FIG. 4(a) shows an example of an image region 401 of a captured image corresponding to the field of view of the imaging device indicated by the field of view information received by the field of view acquisition unit 301. FIG. 4(a) also shows an example of a complex region 402 corresponding to a complex space and an imaging region 403 corresponding to an imaging space in the image region 401. FIGS. 4(b) and 4(c) will be described later.

[0019] The rectangle setting unit 302 uses the angle of view information received from the angle of view generation unit 203 to set a reference rectangular area (hereinafter referred to as a "reference rectangular area") in the image area of ​​the captured image corresponding to the angle of view of the imaging device indicated by the angle of view information. Here, setting a reference rectangular area means setting a specified number of rectangular areas evenly in the image area of ​​the captured image. FIG. 4B shows an example of multiple reference rectangular areas 411 to 416 set in the image area 401 shown in FIG. 4A. Reference rectangular information indicating each reference rectangular area set by the rectangle setting unit 302 is transmitted to the proportion acquisition unit 303. The proportion acquisition unit 303 calculates and acquires the proportion of the complex area and the proportion of the captured area included in each reference rectangular area using the angle of view information received from the angle of view acquisition unit 301 and the reference rectangular information received from the rectangle setting unit 302. The proportion value acquired by the proportion acquisition unit 303, indicating the proportion of the complex area and the proportion of the captured area in each reference rectangular area, is transmitted to the direction acquisition unit 304 and the rectangle determination unit 305.

[0020] The direction acquisition unit 304 determines, from among the multiple reference rectangular areas, a reference rectangular area to be regenerated at a new position different from the position of the reference rectangular area, based on the proportion value of the complex area received from the proportion acquisition unit 303. The direction acquisition unit 304 calculates a direction vector indicating the movement direction when the reference rectangular area determined to be regenerated is regenerated by moving it from the original reference rectangular area toward the imaging area, thereby acquiring vector information indicating the direction vector. The vector information indicating the direction vector acquired by the direction acquisition unit 304 is transmitted to the rectangle determination unit 305. Note that the point used as a reference in the imaging area when calculating the movement direction of the original reference rectangular area may be any point within the imaging area, such as the center point or center of gravity of the imaging area, or a point that is closest to the center of the original reference rectangular area.

[0021] The rectangle determination unit 305 determines the position of a new reference rectangular area to be regenerated based on the proportion value of the complex area received from the proportion acquisition unit 303 and the vector information received from the direction acquisition unit 304. Specifically, the rectangle determination unit 305 determines the position of the new reference rectangular area to be regenerated by determining the amount of movement when regenerating a new reference rectangular area by moving the reference rectangular area in the direction indicated by the vector information, depending on the proportion of the complex area in the original reference rectangular area. The rectangle determination unit 305 generates rectangle information including information indicating the determined position of the new reference rectangular area to be regenerated and information indicating the positions of reference rectangular areas that were not selected for regeneration of a new reference rectangular area from among the multiple reference rectangular areas set by the rectangle setting unit 302. The rectangle information generated by the rectangle determination unit 305 is transmitted to the output unit 205.

[0022] Fig. 4(c) shows an example of multiple rectangular areas 421 to 426 used to extract a foreground area after a series of processes have been performed in the rectangle generation unit 204. Rectangular areas 421 to 423 shown in Fig. 4(c) correspond to new regenerated reference rectangular areas corresponding to the reference rectangular areas 411 to 413 shown in Fig. 4(b). Rectangular areas 424 to 426 shown in Fig. 4(c) correspond to reference rectangular areas 414 to 416 shown in Fig. 4(b) that were not determined to be regenerated.

[0023] <Hardware configuration of information processing device> FIG. 5 is a block diagram showing an example of the hardware configuration of the information processing device 100 according to the first embodiment. The information processing device 100 includes a CPU 501, a ROM 502, a RAM 503, an auxiliary storage device 504, a communication I / F 505, and a bus 506. The CPU 501 controls the entire information processing device 100 using computer programs and data stored in the ROM 502, the RAM 503, or the auxiliary storage device 504, thereby realizing the functions of each unit of the information processing device 100 shown in FIG. 2. The information processing device 100 may include one or more dedicated processing hardware components different from the CPU 501. In this case, at least a portion of the processing performed by the CPU 501 may be executed by the processing hardware components. Examples of the processing hardware include an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), and a DSP (digital signal processor).

[0024] The ROM 502 is a memory for storing programs and the like that do not require modification. The RAM 503 is a memory for temporarily storing programs and data supplied from the ROM 502 or the auxiliary storage device 504, as well as data and the like supplied from the outside via the communication I / F 505. The RAM 503 is used as a work area for the CPU 501. The auxiliary storage device 504 is configured with a storage device such as a hard disk drive, and stores various data and the like, such as image data and audio data. The communication I / F 505 is used for communication with devices external to the information processing device 100. For example, when the information processing device 100 is connected to an external device via a wired connection, a communication cable is connected to the communication I / F 505. When the information processing device 100 communicates wirelessly with an external device, the communication I / F 505 is equipped with an antenna. The bus 506 connects each unit included in the hardware configuration of the information processing device 100 to each other so that they can communicate with each other and transmit information.

[0025] <Operation of information processing device> Fig. 6 is a flowchart showing an example of a processing flow of the information processing device 100 according to the first embodiment. The information processing device 100 determines a plurality of rectangular regions for performing foreground / background separation processing on a captured image by executing the processing of the flowchart shown in Fig. 6. The processing of the flowchart shown in Fig. 6 is realized by the CPU 501 reading a control program stored in the ROM 502 or the like into the RAM 503 and executing it. Note that, hereinafter, each processing step (process) in the flowchart will be represented by adding "S" to the beginning of the reference numeral.

[0026] First, in S601, the background model acquisition unit 201 acquires background data input from the second input device 120. Next, in S602, the information acquisition unit 202 acquires imaging information corresponding to each imaging device input from the second input device 120. Next, in S603, the field of view generation unit 203 generates a virtual field of view corresponding to the field of view of each imaging device based on the background data and imaging information acquired in S601 or S602. Next, in S604, the field of view generation unit 203 identifies the imaging space and complex space included in the virtual field of view based on the virtual field of view generated in S603 and the background data acquired in S601. Next, in S605, the rectangle setting unit 302 sets multiple reference rectangular areas on the image plane corresponding to the virtual field of view generated in S603.

[0027] Next, in S606, for example, the rectangle determination unit 305 selects an arbitrary reference rectangular area from the multiple reference rectangular areas set in S605. Hereinafter, the reference rectangular area selected in S606 will be referred to as the "selected area." Next, in S607, the proportion acquisition unit 303 sets a complex area and an image area corresponding to the complex space and image space identified in S604 on the image plane, and acquires the proportion of the complex area and the proportion of the image area included in the selected area selected in S606. Next, in S608, the rectangle determination unit 305 determines whether the proportion of the complex area acquired in S607 exceeds a predetermined threshold. Note that the accuracy of foreground area extraction depends on the processing performance of a trained model configured by CNN or the like, and therefore the threshold is determined in consideration of the processing performance.

[0028] If it is determined in S608 that the proportion of complex regions exceeds the threshold, in S609, the direction acquisition unit 304 acquires a direction vector indicating the direction in which the selected region should be moved. After S609, in S610, the rectangle determination unit 305 determines the position of the rectangular region to be used in the foreground / background separation process by moving the selected region in the direction indicated by the direction vector acquired in S609. Specifically, for example, the rectangle determination unit 305 determines the position of the rectangular region to be used in the foreground / background separation process by moving the selected region to a position where the proportion of complex regions acquired by a method similar to the method for acquiring the proportion of complex regions in S606 is equal to or less than the threshold. The conditions for moving the selected region are not limited to the above-mentioned conditions. For example, the rectangle determination unit 305 may move the selected region until the proportion of complex regions after the movement of the selected region reaches a predetermined proportion, such as a proportion corresponding to the proportion of the captured image area. After S610, in S611, the rectangle determination unit 305 generates (updates) rectangle information by including information indicating the position of the rectangular area determined in S610 in the rectangle information.

[0029] If it is determined in S608 that the proportion of complex regions does not exceed the threshold, then in S612, the rectangle determination unit 305 generates (updates) the rectangle information by including, in the rectangle information, information indicating the position of the selected region selected in S606. After S611 or S612, in S613, for example, the rectangle determination unit 305 determines whether all of the reference rectangular regions set in S605 have been selected in S606. If it is determined in S613 that at least some of the reference rectangular regions have not been selected, the information processing device 100 returns to the process of S606 and executes the processes from S606 to S613 as appropriate until it is determined in S613 that all of the reference rectangular regions have been selected. In this case, for example, in S606, an arbitrary reference rectangular region that has not yet been selected is selected from among the multiple reference rectangular regions. If it is determined in S613 that all of the reference rectangular regions have been selected, then in S614, the output unit 205 outputs the rectangle information. After S614, the information processing device 100 ends the process of the flowchart shown in FIG. 6.

[0030] The information processing device 100 configured as described above can determine rectangular regions for dividing an image into regions suitable for performing foreground-background separation processing. As a result, by using the rectangular information generated by the information processing device 100, it is possible to improve the accuracy of foreground extraction using a trained model.

[0031] [Modification 1 of Embodiment 1] In the first embodiment, a method has been described in which a reference rectangular area is first set, and then a rectangular area suitable for performing foreground / background separation processing is determined by moving the set reference rectangular area. However, the method for determining a rectangular area is not limited to this. For example, a rectangular area may be determined directly without setting a reference rectangular area. In this case, for example, the number of rectangular areas is determined in advance, and an area that can cover the entire captured image and in which the proportion of complex areas in the rectangular area is equal to or less than a predetermined threshold is set as the rectangular area.

[0032] [Modification 2 of Embodiment 1] In the first embodiment, a method for determining a static rectangular area by moving a reference rectangular area in the direction of a static captured image area determined based on a background model has been described. However, the method for determining the rectangular area is not limited to this, and the rectangular area may be determined dynamically. In this case, for example, an object detection function for detecting an image of an object in a captured image may be added to the information processing device 100, and an area including the image of the detected object may be determined as the rectangular area. If the target object is a moving object, such a method for determining a rectangular area can determine a dynamic rectangular area. Here, the object detection function may be a function that performs detection using a background subtraction method or the like, or may be a function that uses any object recognition method that uses a trained model obtained as a result of learning by machine learning.

[0033] [Embodiment 2] In the first embodiment, a method for determining rectangular regions for dividing a captured image so that the proportion of complex regions is equal to or less than a predetermined threshold has been described. In the second embodiment, a method for determining rectangular regions so as not to include regions in the captured image corresponding to images of predetermined obstacles or the like that do not require foreground / background separation processing will be described. Note that the device configuration of the information processing system according to the second embodiment is the same as that of the information processing system 1 according to the first embodiment.

[0034] FIG. 7 is a block diagram showing an example of the functional configuration of an information processing device 100 according to the second embodiment (hereinafter simply referred to as "information processing device 100"). The information processing device 100 differs from the information processing device 100 according to the first embodiment in that the angle of view generation unit 203 is replaced with an angle of view generation unit 703 and a mask acquisition unit 706 is added. That is, the information processing device 100 includes a background model acquisition unit 201, an information acquisition unit 202, a mask acquisition unit 706, an angle of view generation unit 703, a rectangle generation unit 204, and an output unit 205. Hereinafter, functional components having the same functions as those shown in FIG. 2 are denoted by the same reference numerals, and description thereof will be omitted. Note that each unit included in the functional configuration of the information processing device 100 is realized by the CPU 501 controlling the entire information processing device 100 using computer programs and data stored in the ROM 502, the RAM 503, or the auxiliary storage device 504.

[0035] The mask acquisition unit 706 acquires data of a mask image (hereinafter referred to as a "non-target mask image") for masking an image region corresponding to an image of a non-target object from which a foreground region does not need to be extracted when performing foreground / background separation processing on a captured image. For example, the mask acquisition unit 706 acquires data of an asymmetric mask image (hereinafter referred to as "non-target mask image data") input by the second input device 120. The source of the non-target mask image data is not limited to the second input device 120; for example, the mask acquisition unit 706 may acquire the non-target mask image data by reading it from a storage device not shown in FIG. 7.

[0036] The angle of view generation unit 703 generates angle of view information based on various information and data acquired by the background model acquisition unit 201, the information acquisition unit 202, and the mask acquisition unit 706. Specifically, the angle of view generation unit 703 identifies the imaging space and the complex space in the background within the angle of view of each imaging device, and generates angle of view information indicating the imaging region and the complex region corresponding to the imaging space and the complex space in each captured image. More specifically, since it is not necessary to extract a foreground region for the mask region indicated by the asymmetric mask image, the angle of view generation unit 703 performs a removal process to remove an overlapping region between the imaging region and the mask region (hereinafter referred to as an "overlapping region") from the imaging region. The angle of view information generated by the angle of view generation unit 703 is transmitted to the rectangle generation unit 204.

[0037] Fig. 8 is a flowchart showing an example of a processing flow of the information processing device 100 according to the second embodiment. The processing of the flowchart shown in Fig. 8 is realized by the CPU 501 reading a control program stored in the ROM 502 or the like into the RAM 503 and executing it. The processing flow according to the second embodiment shown in Fig. 8 differs from the processing flow according to the first embodiment shown in Fig. 6 in that S603 is changed to S802 and S801 and S803 are added. In the following description of the flowchart shown in Fig. 8, processing steps that perform the same processing as the processing steps shown in Fig. 6 are denoted by the same reference numerals and description thereof will be omitted.

[0038] First, the information processing device 100 executes the processes of S601 and S602. After S602, in S801, the mask acquisition unit 706 acquires non-target mask image data corresponding to each imaging device. After S801, the information processing device 100 executes the processes of S603 and S604. After S604, in S802, the field of view generation unit 703 detects an overlapping area (overlapping area) between the imaging area identified in S604 and the mask area indicated by the non-target mask image acquired in S801, and executes removal processing to remove the detected overlapping area from the imaging area. After S802, the information processing device 100 appropriately executes the processes of S605 to S613. Note that in the processing of S605, the information processing device 100 treats the area in the imaging area after the overlapping area has been removed as the imaging area. After S613, the information processing device 100 ends the processing of the flowchart shown in FIG. 8.

[0039] According to the information processing device 100 configured as described above, it is possible to determine rectangular regions for dividing an image into regions suitable for performing foreground / background separation processing, without including regions in the imaging region that do not require foreground / background separation processing. As a result, by using the rectangular information generated by the information processing device 100, it is possible to extract foreground regions only from imaging regions that require foreground / background separation processing, thereby improving the accuracy of foreground extraction using a trained model.

[0040] [Embodiment 3] In the first embodiment, a method for identifying an imaging region and a complex region in each captured image is described using complex space information indicating a complex space in the background and background model data (background data) accompanied by imaging space information indicating an imaging space. In the third embodiment, a method for identifying an imaging region and a complex region in each captured image is described using captured image data. Note that the device configuration of an information processing system according to the second embodiment is the same as that of the information processing system 1 according to the first embodiment.

[0041] FIG. 9 is a block diagram showing an example of the functional configuration of an information processing device 100 according to the third embodiment (hereinafter simply referred to as "information processing device 100"). The information processing device 100 differs from the information processing device 100 according to the first embodiment in that the background model acquisition unit 201, the information acquisition unit 202, and the angle of view generation unit 203 are eliminated, and an image acquisition unit 901 and an area detection unit 902 are added. That is, the information processing device 100 includes the image acquisition unit 901, the area detection unit 902, the rectangle generation unit 204, and the output unit 205. Hereinafter, functional components having the same functions as those shown in FIG. 2 are denoted by the same reference numerals, and description thereof will be omitted. Note that each unit included in the functional configuration of the information processing device 100 is realized by the CPU 501 controlling the entire information processing device 100 using computer programs and data stored in the ROM 502, the RAM 503, or the auxiliary storage device 504.

[0042] The image acquisition unit 901 acquires captured image data obtained by imaging with each imaging device, which is input via the first input device 110. The captured image data acquired by the image acquisition unit 901 is transmitted to the region detection unit 902. The region detection unit 902 detects an imaging region corresponding to an imaging space from each captured image using each captured image data received from the image acquisition unit 901, and generates imaging region information indicating the imaging region in each captured image. Furthermore, the region detection unit 902 detects a complex region corresponding to a complex space in the background from each captured image using each captured image data received from the image acquisition unit 901, and generates complex region information indicating the complex region in each captured image. Furthermore, the region detection unit 902 generates field of view information including the generated imaging region information and complex region information. The field of view information generated by the region detection unit 902 is transmitted to the rectangle generation unit 204.

[0043] Fig. 10 is a flowchart showing an example of a processing flow of the information processing device 100 according to the third embodiment. The processing of the flowchart shown in Fig. 10 is realized by the CPU 501 reading a control program stored in the ROM 502 or the like into the RAM 503 and executing it. The processing flow according to the third embodiment shown in Fig. 10 differs from the processing flow according to the first embodiment shown in Fig. 6 in that steps S601 to S604 are replaced by steps S1001 and S1002. In the following description of the flowchart shown in Fig. 10, processing steps that perform the same processing as the processing steps shown in Fig. 6 are denoted by the same reference numerals, and description thereof will be omitted.

[0044] First, in S1001, the image acquisition unit 901 acquires multiple captured image data. Next, in S1002, the area detection unit 902 detects captured areas and complex areas in each captured image using the captured image data acquired in S1001. The method for detecting the captured areas and complex areas may be any object recognition method, such as a method using a trained model obtained as a result of learning by machine learning. Information indicating the captured areas and complex areas detected in S1002 is transmitted to the rectangle generation unit 204 as angle of view information. After S1002, the information processing device 100 appropriately executes processes from S605 to S613. After S613, the information processing device 100 ends the process of the flowchart shown in FIG. 10.

[0045] As described above, the information processing device 100 uses acquired captured image data to detect and identify captured regions and complex regions in a captured image, and determines a rectangular region using the identified captured regions and complex regions. The information processing device 100 configured as described above can determine rectangular regions for dividing an image into regions suitable for foreground / background separation processing without using background data or image information. As a result, by using the rectangular information generated by the information processing device 100, the accuracy of foreground extraction using a trained model can be improved.

[0046] [Modification 1 of Embodiment 3] In the third embodiment, a method for identifying an imaging region and a complex region in each captured image using captured image data has been described. However, the method for identifying an imaging region in each captured image is not limited to this. For example, an imaging region may be identified based on the captured image data and the non-target mask image data described in the second embodiment so as to avoid the mask region indicated by the non-target mask image. In this case, the information processing device 100 according to the first modification of the third embodiment includes a mask acquisition unit 706 shown in FIG. 7 in addition to the functional configuration of the information processing device 100 according to the third embodiment.

[0047] For example, the non-target mask image data acquired by the mask acquisition unit 706 is transmitted to the angle of view generation unit 203. The angle of view generation unit 203 removes from the imaging area detected and identified by the area detection unit 902 an area that overlaps with the mask area indicated by the non-target mask image, and generates rectangle information based on the imaging area after removal. Also, for example, the non-target mask image data acquired by the mask acquisition unit 706 may be transmitted to the area detection unit 902. In this case, the area detection unit 902 uses the captured image data and the non-target mask image data to identify an imaging area such that the mask area is not included in the imaging area.

[0048] According to the information processing device 100 configured as described above, it is possible to determine rectangular regions for dividing an image into regions suitable for performing foreground / background separation processing, without including regions in the imaging region that do not require foreground / background separation processing. As a result, by using the rectangular information generated by the information processing device 100, it is possible to extract a foreground region from an imaging region that requires foreground / background separation processing, thereby improving the accuracy of foreground extraction using a trained model.

[0049] [Other embodiments] In the above-described embodiment, data on multiple divided images obtained by dividing an image into multiple rectangular regions is input to a trained model configured using a CNN or the like that performs foreground / background separation processing. However, the shape of the divided images is not limited to a rectangle. For example, the divided images may be any polygonal shape, such as a triangle, trapezoid, or hexagon, or may be any shape that can be specified by a closed curve, such as a circle or ellipse. Furthermore, in the above-described embodiment, an image is divided into multiple rectangular regions, and the information processing device 100 generates and outputs rectangular information indicating the rectangular regions. However, if the divided regions are not rectangular, the terms "rectangular region" and "reference rectangular region" in the above-described embodiment may simply be read as "divided region" and "reference divided region," and the rectangular information may be appropriately read as "division information" indicating the region.

[0050] The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0051] It should be noted that within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0052] [Configuration of the present disclosure] The present disclosure includes the following configurations, methods, and programs.

[0053] <Configuration 1> an identification means for identifying a first region on an image plane corresponding to a first space in which a target object may exist within an angle of view of the imaging device, and a second region on the image plane corresponding to a second space in which separation accuracy between a foreground region and a background region may deteriorate within the angle of view; a determination means for determining a plurality of divided regions for dividing an image to be processed in a foreground / background separation process based on the identified first region and second region; An information processing device comprising:

[0054] <Configuration 2> the specifying means specifies the first area and the second area based on background model data including information indicating the first space and information indicating the second space, and imaging information including information indicating a position of the imaging device; 2. The information processing device according to configuration 1,

[0055] <Configuration 3> the specifying means specifies the second area by detecting an area that satisfies a predetermined condition from the captured image obtained by imaging using the imaging device; 3. The information processing device according to configuration 1 or 2, characterized in that:

[0056] <Configuration 4> the identification means inputs data of the captured image into a trained model obtained as a result of learning by machine learning, and acquires an inference result output by the trained model, thereby detecting an area that satisfies the predetermined condition based on the inference result; 4. The information processing device according to configuration 3,

[0057] <Configuration 5> the specifying means specifies the first area by detecting an area including an image of the object from a captured image obtained by imaging using the imaging device; 5. The information processing device according to any one of configurations 1 to 4, characterized in that:

[0058] <Configuration 6> the identification means inputs data of the captured image into a trained model obtained as a result of learning by machine learning, and obtains an inference result output by the trained model, thereby detecting a region including an image of the object based on the inference result; 6. The information processing device according to configuration 5,

[0059] <Configuration 7> the specifying means specifies the first region using mask information for masking an image region corresponding to an image of a non-target object from which extraction of a foreground region by foreground / background separation processing is not required, so as not to include a mask region indicated by the mask information; 7. The information processing device according to any one of configurations 1 to 6,

[0060] <Configuration 8> the determining means determines the plurality of divided regions so that the entirety of the identified first region is covered by the plurality of divided regions; 8. The information processing device according to any one of configurations 1 to 7,

[0061] <Configuration 9> the determining means determines a specified number of divided regions; 9. The information processing device according to any one of configurations 1 to 8,

[0062] <Configuration 10> the determining means determines the plurality of divided regions based on at least one of a ratio of the second region and a ratio of the first region in each of the plurality of divided regions; 10. The information processing device according to any one of configurations 1 to 9, characterized in that:

[0063] <Configuration 11> an output means for outputting area information indicating the determined plurality of divided areas; further comprising: 11. The information processing device according to any one of configurations 1 to 10, characterized in that:

[0064] <Configuration 12> a dividing means for dividing the image to be processed based on the determined plurality of divided regions; an extraction means for extracting, as a foreground region, an image region including an image of the object in each of a plurality of divided images generated by dividing the image to be processed, by performing a foreground / background separation process on each of the plurality of divided images using a trained model obtained as a result of learning by machine learning; further comprising: 12. The information processing device according to any one of configurations 1 to 11,

[0065] <Configuration 13> an acquisition means for generating a foreground region image representing the foreground region in the image to be processed by combining the foreground regions in each of the plurality of extracted divided images; further comprising: 13. The information processing device according to configuration 12,

[0066] <Configuration 14> the specifying means specifies the first region and the second region in the image plane corresponding to each of the plurality of images to be processed; the determining means determines the plurality of divided regions for each of the plurality of images to be processed; 14. The information processing device according to any one of configurations 1 to 13,

[0067] <Configuration 15> each of the plurality of divided regions is a rectangular region; 15. The information processing device according to any one of configurations 1 to 14, wherein:

[0068] <Method> an identifying step of identifying a first region on an image plane corresponding to a first space in which a target object may exist within an angle of view of the imaging device, and a second region on the image plane corresponding to a second space in which separation accuracy between a foreground region and a background region by foreground / background separation processing may deteriorate within the angle of view; a determining step of determining a plurality of divided regions for dividing an image to be processed in a foreground / background separation process based on the identified first region and second region; An information processing method comprising:

[0069] <Program> 16. A program for causing a computer to function as the information processing device according to any one of configurations 1 to 15. [Explanation of symbols]

[0070] 100 Information processing device 203 Angle of view generation unit 204 Rectangle generator

Claims

1. an identification means for identifying a first region on an image plane corresponding to a first space in which a target object may exist within an angle of view of the imaging device, and a second region on the image plane corresponding to a second space in which separation accuracy between a foreground region and a background region may be degraded within the angle of view of the imaging device; a determination means for determining a plurality of divided regions for dividing an image to be processed in a foreground / background separation process based on the identified first region and second region; An information processing device comprising:

2. the specifying means specifies the first area and the second area based on background model data including information indicating the first space and information indicating the second space, and imaging information including information indicating a position of the imaging device; 2. The information processing device according to claim 1,

3. the specifying means specifies the second area by detecting an area that satisfies a predetermined condition from the captured image obtained by imaging using the imaging device; 2. The information processing device according to claim 1,

4. the identification means inputs data of the captured image into a trained model obtained as a result of learning by machine learning, and acquires an inference result output by the trained model, thereby detecting an area that satisfies the predetermined condition based on the inference result; 4. The information processing device according to claim 3,

5. the specifying means specifies the first area by detecting an area including an image of the object from a captured image obtained by imaging using the imaging device; 2. The information processing device according to claim 1,

6. the identification means inputs data of the captured image into a trained model obtained as a result of learning by machine learning, and obtains an inference result output by the trained model, thereby detecting a region including an image of the object based on the inference result; 6. The information processing device according to claim 5,

7. the specifying means specifies the first region using mask information for masking an image region corresponding to an image of a non-target object from which extraction of a foreground region by foreground / background separation processing is not required, so as not to include a mask region indicated by the mask information; 2. The information processing device according to claim 1,

8. the determining means determines the plurality of divided regions so that the entirety of the identified first region is covered by the plurality of divided regions; 2. The information processing device according to claim 1,

9. the determining means determines a specified number of divided regions; 2. The information processing device according to claim 1,

10. the determining means determines the plurality of divided regions based on at least one of a ratio of the second region and a ratio of the first region in each of the plurality of divided regions; 2. The information processing device according to claim 1,

11. an output means for outputting area information indicating the determined plurality of divided areas; further comprising:

2. The information processing device according to claim 1,

12. a dividing means for dividing the image to be processed based on the determined plurality of divided regions; an extraction means for extracting, as a foreground region, an image region including an image of the object in each of a plurality of divided images generated by dividing the image to be processed, by performing a foreground / background separation process on each of the plurality of divided images using a trained model obtained as a result of learning by machine learning; further comprising:

2. The information processing device according to claim 1,

13. an acquisition means for generating a foreground region image representing the foreground region in the image to be processed by combining the foreground regions in each of the plurality of extracted divided images; further comprising: The information processing device according to claim 12 .

14. the specifying means specifies the first area and the second area in the image plane corresponding to each of the plurality of images to be processed; the determining means determines the plurality of divided regions for each of the plurality of images to be processed; 2. The information processing device according to claim 1,

15. each of the plurality of divided regions is a rectangular region; The information processing apparatus according to claim 1 , wherein:

16. an identifying step of identifying a first region on an image plane corresponding to a first space in which a target object may exist within an angle of view of the imaging device, and a second region on the image plane corresponding to a second space in which accuracy of separation between a foreground region and a background region by foreground / background separation processing may deteriorate within the angle of view; a determining step of determining a plurality of divided regions for dividing an image to be processed in a foreground / background separation process based on the identified first region and second region; An information processing method comprising:

17. A program for causing a computer to function as the information processing device according to any one of claims 1 to 15.

Images