Information processing device, information processing method, and program

Abstract

This technology provides appropriate processing using captured images. [Solution] The information processing device includes: detection means for detecting the markers contained in each of a plurality of captured images generated by an imaging device capturing markers at a plurality of different distances and calculating detection accuracy; distance calculation means for calculating the distance between each of the detected markers and the imaging device; and setting means for setting an area in the captured image to be used in processing performed using the captured image, based on the detection accuracy, a predetermined reference accuracy, and the distance.

Description

【Technical Field】 【0001】 The present disclosure relates to an information processing apparatus, an information processing method, and a program. 【Background Art】 【0002】 A technique for detecting some images from a plurality of images taken from different directions or the like and generating a virtual viewpoint video or the like is known. 【0003】 Patent Document 1 discloses a method of determining whether a subject is located within the depth of field of a camera when generating a shape model of a subject by outdoor photography, and processing a volume intersection method using only the subject image of the camera in which the subject is located within the depth of field. 【0004】 Here, in the above method, for the depth of field of each camera, instead of the measured value, the result calculated using the camera parameters is used. Specifically, it is calculated from the focal length of the camera, the aperture value, the allowable circle of confusion diameter, and the distance between the camera and the fixation point within the angle of view determined at the time of design. By using the calculated parameters, the depth of field of each camera can be obtained at the stage of performing a camera installation simulation, and the camera arrangement study and the image quality study of the virtual viewpoint video can be efficiently performed. 【Prior Art Documents】 【Patent Documents】 【0005】 【Patent Document 1】 Japanese Patent Application Laid-Open No. 2022-110751 【Summary of the Invention】 【Problems to be Solved by the Invention】 【0006】 However, there may be discrepancies between the camera placement determined in the installation simulation and the actual camera placement. Specifically, this could be due to venue constraints preventing placement in the simulated location, or slight misalignments during installation. If the depth of field calculated from the parameters during the simulation is applied without considering such discrepancies, a difference may occur between this and the actual depth of field of the camera. If the shape model of the subject and the virtual viewpoint image are generated with this discrepancy, a blurred image outside the depth of field will be used. As a result, the accuracy of the shape model of the subject will decrease or be lost, leading to a degradation in the image quality of the virtual viewpoint image and other related content. 【0007】 Therefore, this disclosure provides a technology for performing appropriate processing using captured images. [Means for solving the problem] 【0008】 To solve this problem, for example, the information processing device of this disclosure has the following configuration. That is, A detection means that detects the markers contained in each of the multiple captured images generated by the imaging device capturing markers at multiple different distances and calculates the detection accuracy, Distance calculation means for calculating the distance between each of the detected markers and the imaging device, A setting means for setting an area to be used in processing performed using the captured image, based on the detection accuracy, a predetermined reference accuracy, and the distance, It holds. [Effects of the Invention] 【0009】 According to this disclosure, appropriate processing can be performed using the captured image. [Brief explanation of the drawing] 【0010】 [Figure 1] A schematic diagram showing an example of an image processing system according to an embodiment. [Figure 2] A block diagram illustrating the functions of the image processing apparatus according to the embodiment. [Figure 3] A diagram illustrating the method for calculating the distance between the imaging device and the marker in the embodiment. [Figure 4] A diagram illustrating the method for predicting detection accuracy in the embodiment. [Figure 5] A diagram illustrating how the detection range of the marker is calculated. [Figure 6] A flowchart of the detection range calculation process for calculating the detection range of the embodiment. [Figure 7] A block diagram showing the hardware configuration of an image processing device. [Modes for carrying out the invention] 【0011】 The embodiments will be described in detail below with reference to the attached drawings. Although several features are described in the embodiments, not all of these features are essential to this disclosure, and the features may be combined in any way. Furthermore, in the attached drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted. 【0012】 (Embodiment) This embodiment will be described in detail with reference to the drawings. Figure 1 is a schematic diagram showing an example of an image processing system 101 of the embodiment. 【0013】 The image processing system 101 generates multi-viewpoint video from multiple images generated by capturing a shooting area including a subject from multiple directions. Specifically, the image processing system 101 generates video observed from a virtual viewpoint (hereinafter referred to as virtual viewpoint video) using multiple captured images generated by multiple imaging devices capturing a subject from different directions. The virtual viewpoint is a viewpoint based on information such as the position of the imaging devices and the direction of line of sight set in a virtual space. To generate virtual viewpoint video, a shape model representing the three-dimensional shape of the subject is required. A widely known method for estimating a shape model from captured images of multiple imaging devices is the view volume cross-eyed method. In this method, the shape model of the subject is represented by a collection of cubes (voxels). In order to represent the shape model of the subject using the view volume cross-eyed method, a silhouette image is required in the image of the imaging device to show the image area in which the subject is located. 【0014】 If defects occur in the silhouette image, defects will also occur in the shape model obtained by the viewing volume cross-eyed method, potentially leading to a degradation in the image quality of the virtual viewpoint video. One factor causing defects in the silhouette image is blurring of the captured image due to the depth of field of the imaging device. When there is blurring, the contour shape of the subject cannot be accurately acquired, and defects in the silhouette image are likely to occur relative to the actual shape of the subject. Therefore, in this embodiment, the detection range for detecting images to generate virtual viewpoint video is set to a range at which there are few blurred images of the subject, based on the frequency threshold of markers detected from the captured image. 【0015】 The image processing system 101 includes a plurality of imaging devices 102 and an image processing device 103 connected to the imaging devices 102. In the following description, the term "image" may include still images, videos, images of a single frame of a video, and their data. 【0016】 The imaging device 102 may be a digital camera that images a shooting area including a subject and generates electrical image data. A plurality of imaging devices 102 are arranged so as to surround an arena 104, which is a shooting range, in order to image the arena 104 from a plurality of directions for generating a virtual viewpoint video. Each imaging device 102 images a marker 105, which is a shooting target, a plurality of times in setting a detection range described later, and generates a plurality of captured images including the marker 105. Note that the marker 105 may be printed on a recording medium such as paper, or may be displayed on a liquid crystal display or the like. The marker 105 is, for example, planar. The imaging device 102 images the marker 105 moving along a path indicated by a broken line, for example. Therefore, in the plurality of captured images captured by one imaging device 102, the position of the marker 105 in the captured image and the distance between the marker 105 and the imaging device 102 are different. The marker 105 uses a marker 105 whose coordinates can be specified by calculating the frequency components of an image as shown in FIG. 1. The imaging device 102 transmits the captured image of the marker to an image processing device 103. 【0017】 The image processing device 103 is an example of an information processing device and may be a computer. The image processing device 103 performs any one of parameter setting, for example, processing for generating a virtual viewpoint video, generation of a shape model representing the three-dimensional shape of a subject, and setting of a detection range used for texture generation and region information in a captured image based on the detection range. 【0018】 FIG. 2 is a block diagram showing the functions of the image processing device according to the embodiment. The image processing device 103 includes an image acquisition unit 201, a marker detection unit 202, a position acquisition unit 203, a marker distance calculation unit 204, a marker detection accuracy prediction unit 205, a setting unit 206, and a determination unit 207. 【0019】 A part of the above-described configuration of the image processing system 101 may be provided outside. For example, the position acquisition unit 203 may be provided outside the image processing device 103 and may be connected to the image processing device 103 so as to be able to transmit and receive data. 【0020】 The image acquisition unit 201 acquires multiple images captured by a single imaging device 102 (in this case, imaging device 102a). For example, the image acquisition unit 201 acquires multiple images generated when the imaging device 102 images the marker 105 while moving the marker 105 as shown by the dashed line in Figure 1. In this way, the image acquisition unit 201 acquires multiple images of the marker imaged by the imaging device 102, even when the distance between the marker 105 and the imaging device 102 is different. Alternatively, the image acquisition unit 201 may acquire a video captured while moving the marker 105 and acquire multiple frame images contained in the video as multiple images. 【0021】 The marker detection unit 202 performs a marker detection process to detect markers in each of the multiple captured images acquired by the image acquisition unit 201 and calculate the detection accuracy. The detection accuracy may be, for example, the accuracy of the markers that were detected. Specifically, the marker detection unit 202 acquires the frequency component values ​​of the captured images and detects the marker 105 based on the frequency component values ​​and a preset frequency threshold. Specifically, the marker detection unit 202 may determine that the marker 105 has been detected if a frequency component value equal to or greater than the frequency threshold is observed, or if a frequency component value greater than the frequency threshold is observed. 【0022】 Next, the marker detection unit 202 calculates the coordinate information of the detected marker 105 within the captured image, the size information of the marker 105, and the angle information between the marker 105 and the imaging device 102. 【0023】 Furthermore, if the marker detection unit 202 fails to detect the marker 105 in the image using a preset frequency threshold, it may change the frequency threshold and perform the marker detection process again on the same captured image. For example, the marker detection unit 202 may lower the frequency threshold and perform the marker detection process. If the marker detection unit 202 fails to detect a marker even when the frequency threshold is lowered to its lowest value, it may determine that the marker 105 is not included in the captured image. Next, the marker detection unit 202 calculates or obtains the detection accuracy of the marker 105 based on the frequency threshold at which the marker was detected. For example, the marker detection unit 202 may calculate the detection accuracy based on the following formula, which includes the frequency threshold and a predetermined reference frequency. The reference frequency may be, for example, the frequency component value of the marker image when it is in the best focus. The marker detection unit 202 may normalize the detection accuracy so that its maximum value is 100. The reference frequency may also be set so that the maximum value of the detection accuracy is 100. Detection accuracy = Frequency threshold at which the marker was detected / Reference frequency 【0024】 Furthermore, the marker detection unit 202 may obtain the marker detection accuracy corresponding to the frequency threshold used when detecting the marker 105 from a conversion table. The conversion table may be a table that associates the frequency threshold for comparison with the frequency component value with the marker detection accuracy. 【0025】 The position acquisition unit 203 acquires position information of the imaging device 102. Specifically, the position acquisition unit 203 may acquire position information from the calibration results for the three-dimensional coordinates of the imaging device 102. 【0026】 The marker distance calculation unit 204 calculates the distance between each of the markers 105 detected by the marker detection unit 202 from multiple captured images and the imaging device 102. For example, the marker distance calculation unit 204 may calculate the distance between a marker facing directly and the imaging device 102. Figure 3 is a schematic diagram illustrating the method for calculating the distance between the marker 105 and the imaging device 102. The method for calculating the distance between the marker 105 and the imaging device 102 will be specifically explained with reference to Figure 3. 【0027】 Figures 3(a) and 3(b) show captured images with marker 105. Figure 3(c) is a schematic diagram showing the positional relationship between the imaging device 102 and marker 105, viewed from above in the y-direction of the coordinate system of the captured image. Figures 3(d) and 3(e) are schematic diagrams showing the positional relationship between the imaging device 102 and marker 105, viewed from above in the x-direction of the coordinate system of the captured image. 【0028】 First, for the marker 105 detected in the captured image, the marker detection unit 202 acquires position information, size information, and angle information within the image, as shown in Figure 3(a). In Figure 3(a), the x-axis 301, z-axis 302, and y-axis 303 represent the x, z, and y axes of the marker, respectively, with the center of the marker 105 as the origin. The coordinate system based on x-axis 301, z-axis 302, and y-axis 303 is defined as the marker coordinate system. Additionally, the x-axis 304, z-axis 305, and y-axis 306 represent the x, z, and y axes of the captured image, respectively. The coordinate system based on x-axis 304, z-axis 305, and y-axis 306 is defined as the captured image coordinate system. (x,y) represents the coordinates of the center of the detected marker 105 in the captured image coordinate system, and this is defined as the position information of the marker 105. 【0029】 Furthermore, (width, height) represents the length of the detected marker 105 in the x-axis 301 direction and the length in the y-axis 303 direction on the marker coordinate system, respectively, and this is used as the size information of the marker 105. For example, in Figure 3(a), it is (w1, h1). 【0030】 Next, the marker distance calculation unit 204 calculates a plane perpendicular to the line segment connecting the center of the imaging device 102 and the center of the marker 105, as shown in Figure 3(c), based on the size information and angle information of the marker 105 shown in Figure 3(a). The marker distance calculation unit 204 determines the facing marker 307, which is the marker when the marker 105 is pointed toward this perpendicular plane. In other words, the facing marker 307 is the marker when it is facing the imaging device 102 directly. The marker distance calculation unit 204 determines the size information (w', h') of the facing marker 307, as shown in Figure 3(b). 【0031】 Specifically, as shown in Figure 3(c), the marker distance calculation unit 204 calculates the length w' of the facing marker 307 in the direction of the x-axis 304 in the marker coordinate system, which is directly facing the imaging device 102, using equation (1) based on three angles Φ, θ, and ε on the xz plane of the imaging image coordinate system. Angle Φ is the angle between the z-axis 302 of the marker coordinate system and the z-axis 305 of the imaging image coordinate system. Angle θ is the angle between the line segment connecting the center of the facing marker 307 and the imaging device 102 and the z-axis 305 of the imaging image coordinate system. Angle ε is the angle between the line segment connecting the imaging device 102 and the center of the facing marker 307 and the x-axis 301 of the marker coordinate system. 【number】 【0032】 Similarly, as shown in Figure 3(d), the marker distance calculation unit 204 calculates the length h' in the y-axis direction of the marker coordinate system when it is directly facing the imaging device 102, using equation (2) based on angles Φ', θ', and ε' on the zy-plane of the imaging image coordinate system. Angle Φ' is the angle between the z-axis 302 of the marker coordinate system and the z-axis 305 of the imaging image coordinate system. Angle θ' is the angle between the line segment connecting the center of the directly facing marker 307 and the imaging device 102 and the z-axis 305 of the imaging image coordinate system. Angle ε' is the angle between the line segment connecting the imaging device 102 and the center of the directly facing marker 307 and the y-axis 303 of the marker coordinate system. 【number】 【0033】 As a result, the marker distance calculation unit 204 can determine the size information (w',h') of the facing marker 307 using equations (1) and (2). Finally, the marker distance calculation unit 204 calculates the distance r between the imaging device 102 and the marker 105 from the size information (w',h') of the facing marker 307 rotated to face it, the original size (w1,h1) of the marker 105, the size s of the image sensor of the imaging device 102, and the focal length z of the imaging device 102. Specifically, as shown in Figure 3(e), the marker distance calculation unit 204 calculates the distance r using the following equation (3) based on the size information (w',h'), size s, and focal length z. 【number】 【0034】 The marker detection accuracy prediction unit 205 is an example of a determination means that predicts and determines the detection accuracy of markers at distances where no markers are detected in the captured image. For example, the marker detection accuracy prediction unit 205 predicts the detection accuracy of markers at distances where no markers are detected in the captured image, based on the distance r between the imaging device 102 and the detected marker 105, and the detection accuracy of the detected marker, which are calculated for multiple captured images acquired by the image acquisition unit 201. 【0035】 Figure 4 is a graph illustrating the method for predicting the detection accuracy of a marker in this embodiment. Referring to Figure 4, the method for predicting the detection accuracy of a marker with respect to the distance between the imaging device and the marker will be explained. Figure 4(a) is a graph plotting the results of determining the marker detection accuracy at each distance between the imaging device 102 and the marker 105. Figure 4(b) is a diagram showing an approximation curve of the detection accuracy. We want to use this graph to determine the distance between the imaging device 102 and the marker 105 for the detection accuracy we want to obtain, but there is no data other than the plotted points. Therefore, the marker detection accuracy prediction unit 205 creates an approximation curve as shown in Figure 4(b) based on the marker detection accuracy plotted based on the relationship between the marker detection accuracy and the distance to the imaging device 102. The marker detection accuracy prediction unit 205 refers to the created approximation curve and sets the detection accuracy 402 on the approximation curve corresponding to the camera-marker distance 401 we want to predict as the predicted detection accuracy. In this embodiment, an approximation curve based on points plotted on the graph is used as the method for predicting the detection accuracy, but this is not the only method, and the detection accuracy may be predicted using another method. 【0036】 The setting unit 206 sets an area (also called the detection range) to be used in processing using the captured image. Processing using the captured image may be a process to generate a virtual viewpoint image or a process to generate a shape model representing the three-dimensional shape of a subject. The image (also called the silhouette image) included in the area used here may be, for example, a specific image included in the captured image. Specifically, the image included in the area may be an image of a player's face or a full-body image of a player in a captured image of a sport in progress. The setting unit 206 sets the detection range based on the detection accuracy of the marker 105, a predetermined reference accuracy, and the distance between the marker 105 and the imaging device 102. Specifically, the setting unit 206 compares the detection accuracy associated with the distance with a reference accuracy set for the area used for generating a virtual viewpoint image, and may set the range in which the detection accuracy is equal to or greater than the reference accuracy (or greater than the reference accuracy) as the detection range. Here, the detection accuracy includes the detection accuracy of the marker 105 detected by the marker detection unit 202, and the detection accuracy of the marker 105 predicted by the marker detection accuracy prediction unit 205. Hereinafter, the reference accuracy will also be referred to as the virtual viewpoint detection reference accuracy 502. 【0037】 Figure 5 illustrates the method for calculating the marker detection range. The setting unit 206 determines the distance range (hereinafter referred to as the detection range 503) in which the marker can be detected with a detection accuracy of at least the set virtual viewpoint detection reference accuracy 502 (or a detection accuracy greater than the virtual viewpoint detection reference accuracy 502) by finding the intersection point of the marker detection accuracy approximation curve 501 and the pre-set virtual viewpoint detection reference accuracy 502. The virtual viewpoint detection reference accuracy 502 may be a parameter pre-set from parameters such as the focal length and field of view of the imaging device 102. For example, if the detection accuracy is normalized so that the maximum value is 100, the virtual viewpoint detection reference accuracy 502 may be 25. 【0038】 When the virtual viewpoint detection criterion accuracy 502 is set high, the image processing system 101 can use only images that are in focus on the marker for generating the virtual viewpoint image, although the detection range 503 used for generating the virtual viewpoint image will be narrowed in each imaging device 102. Conversely, when the virtual viewpoint detection criterion accuracy 502 is set low, the image processing system 101 can widen the detection range 503 used for generating the virtual viewpoint image, although slightly out-of-focus images will be used in each imaging device 102, allowing more images to be used for generating the virtual viewpoint image. The virtual viewpoint detection criterion accuracy 502 may be set for each imaging device 102. For example, for an imaging device 102 that wants to use high-resolution textures for generating the virtual viewpoint image, the virtual viewpoint detection criterion accuracy 502 may be set higher. This allows the image processing system 101 to adjust the system so that only images in the detection range 503 that are in focus, even within the original depth of field, are used for generating the virtual viewpoint image. 【0039】 The determination unit 207 determines an image region based on the detection range 503 used for generating virtual viewpoint video, and generates and outputs video generation information including region information, which is information about the said image region. The image region may be, for example, a region in the captured image that can detect images with less blur, as set by the detection range 503. The determination unit 207 may also determine the image region based on the position information of the imaging device 102 along with the detection range 503. The output destination may be directly to the system that generates the virtual viewpoint video, or it may be output as an arbitrary file. 【0040】 Figure 7 is a block diagram showing the hardware configuration of the image processing device. The hardware configuration of the image processing device 103 will be explained using Figure 7. The hardware configuration of the control system of the imaging device 102 is the same as that of the image processing device 103, which will be explained below. The image processing device 103 includes a CPU 211, ROM 212, RAM 213, auxiliary storage device 214, display unit 215, operation unit 216, communication I / F 217, and bus 218. 【0041】 CPU211 stands for Central Processing Unit and is a type of processor. The CPU211 controls the entire image processing device 103 using computer programs and data stored in either ROM212, RAM213, or auxiliary storage device 214. As a result, the CPU211 may implement all or part of the functions of the image processing device 103 shown in Figure 2. The image processing device 103 may have other processors such as an MPU (Micro Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), and QPU (Quantum Processing Unit) in place of, or in addition to, the CPU211. Furthermore, the image processing device 103 may have multiple processors of the same type, each implementing a different function. Additionally, the image processing device 103 may have one or more dedicated hardware components distinct from the CPU211, with the dedicated hardware executing at least some of the processing and functions performed by the CPU211. Examples of dedicated hardware include ASICs (Application-Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and DSPs (Digital Signal Processors). 【0042】 ROM212 stands for Read Only Memory and is a type of non-volatile memory. ROM212 is used to store programs and other data that do not require modification. 【0043】 RAM213 stands for Random Access Memory and is a high-speed read and write memory. RAM213 temporarily stores programs and data supplied from auxiliary storage device 214, as well as data supplied from external sources via communication interface 217. 【0044】 The auxiliary storage device 214 is a non-volatile storage device composed of, for example, a hard disk drive and an SSD (Solid State Drive). The auxiliary storage device 214 stores various types of data, such as image data and audio data. 【0045】 The display unit 215 is composed of, for example, a liquid crystal display or LEDs, and displays a GUI (Graphical User Interface) for the user to operate the image processing device 103. The operation unit 216 is composed of, for example, a keyboard, mouse, joystick, touch panel, etc., and receives various instructions from the user and inputs them to the CPU 211. The CPU 211 operates as a display control unit that controls the display unit 215, and as an operation control unit that controls the operation unit 216. 【0046】 The communication interface 217 is used for communication between the image processing device 103 and external devices. For example, if the image processing device 103 is connected to an external device by a wired connection, a communication cable is connected to the communication interface 217. If the image processing device 103 has a function for wireless communication with an external device, the communication interface 217 is equipped with an antenna. The bus 218 connects the various parts of the image processing device 103 and transmits information. 【0047】 In this embodiment, the display unit 215 and the operation unit 216 are assumed to be located inside the image processing device 103, but at least one of the display unit 215 and the operation unit 216 may be located outside the image processing device 103 as a separate device. 【0048】 Figure 6 is a flowchart of the detection range calculation process for calculating the detection range in the embodiment. In the detection range calculation process, the detection range for detecting the image used to generate the virtual viewpoint image is detected, and region information based on the detection range is output. The process flow will be explained in detail below, following the flowchart in Figure 6. 【0049】 In S600, the image acquisition unit 201 acquires an image of the marker. The marker image may be, for example, an image generated by the imaging device 102 when it images the marker 105. The imaging device 102 may be an imaging device that wants to acquire parameters necessary for generating a virtual viewpoint image. 【0050】 In S601, the position acquisition unit 203 acquires the position information of the imaging device 102. 【0051】 In S602, the marker detection unit 202 performs marker detection processing on the image acquired in S600 based on a frequency threshold for marker detection to be compared with frequency component values, and detects markers. 【0052】 In S603, the marker detection unit 202 determines whether it was able to detect a marker in the captured image as a result of the marker detection process performed in S602. If the marker detection unit 202 determines that no marker was detected, the process proceeds to S605. 【0053】 In S605, the marker detection unit 202 determines whether the detection accuracy is at its lowest value based on the detection accuracy of the markers newly acquired in S604. If the marker detection unit 202 determines that the detection accuracy is at its lowest value, it determines that no markers are visible in the captured image for which the detection process was performed in S602, and skips the processes up to S608, proceeding to S609. 【0054】 In S606, the marker detection unit 202 reduces the detection accuracy of the markers acquired in S604. Then, using the reduced frequency threshold, the marker detection unit 202 performs processing, including marker detection, again on the captured image where markers could not be detected. The marker detection unit 202 repeats steps S602 onwards for the same captured image until markers can be detected or the frequency threshold reaches its lowest value. 【0055】 In S603, if the marker detection unit 202 determines that it has detected a marker, the process proceeds to S604. 【0056】 In S604, the marker detection unit 202 calculates the detection accuracy of the detected marker based on the frequency threshold and reference frequency at which the marker was detected. The marker detection unit 202 may calculate or acquire marker information such as the coordinate information of the detected marker 105 in the captured image, the size information of the marker 105, the angle information between the marker 105 and the imaging device 102, and the detection accuracy of the marker. 【0057】 In S607, the marker distance calculation unit 204 acquires information about the marker detected by the marker detection unit 202. The marker distance calculation unit 204 may also acquire position information of the imaging device 102. 【0058】 In S608, the marker distance calculation unit 204 calculates the distance between the marker 105 and the imaging device 102 based on the position information of the imaging device 102 acquired in S601, the size information of the marker 105 acquired in S607, and the angle information. 【0059】 In S609, the marker detection unit 202 determines whether there are any images among the multiple images acquired in S600 for which the marker detection process from S602 onwards has not been completed. If the marker detection unit 202 determines that there are images for which the process from S602 onwards has not been completed, it executes the process from S602 onwards for those images. If the marker detection unit 202 determines that the process from S602 onwards has been completed for all images, it proceeds to S610. 【0060】 In S610, the marker detection accuracy prediction unit 205 generates an approximate curve of detection accuracy based on the detection accuracy of the detected markers and the calculated distance, and predicts the detection accuracy at distances where no markers have been detected. 【0061】 In S611, the setting unit 206 calculates and sets the detection range based on the detection accuracy. For example, the setting unit 206 may set the detection range to a distance at which a marker can be detected with a detection accuracy equal to or greater than a pre-set virtual viewpoint detection threshold, based on the detection accuracy calculated from the detected marker and the approximation formula of the approximation curve of the detection accuracy generated for prediction in S610. 【0062】 In S612, the determination unit 207 generates and outputs video generation information, which includes region information indicating the image region of the captured image used for generating virtual viewpoint video in the imaging device 102, based on the detection range information obtained in S611 that enables image detection with an accuracy equal to or greater than the virtual viewpoint detection reference threshold. 【0063】 As described above, this embodiment sets the area (detection range) used in processing using the captured image based on the detection accuracy of the detected marker and the virtual viewpoint detection reference accuracy. As a result, this embodiment can set an appropriate detection range that can detect images with a detection accuracy greater than the virtual viewpoint detection reference accuracy, thereby suppressing the detection of blurred images. Consequently, this embodiment can improve the image quality of virtual viewpoint videos and other images generated from the detected images, and can perform appropriate processing using the captured image. 【0064】 For example, this embodiment calculates the range of distances at which markers can be detected with a detection accuracy equal to or greater than the virtual viewpoint detection threshold from the actual image acquired from the imaging device 102, and can acquire regions to be used for generating virtual viewpoint images (e.g., shape models representing 3D shapes) and textures based on the actual captured image. Furthermore, by changing the virtual viewpoint detection threshold accuracy used for determination, it is possible to acquire not only regions where markers can be detected based on the actual captured image, but also regions that can be used for high-precision model generation and texture generation processing, such as using only the regions where marker detection was achieved with higher detection accuracy among the detectable regions. 【0065】 In this embodiment, the detection range is set by predicting the detection accuracy at distances where no marker has been detected, thus allowing for a more appropriate setting of the detection range. 【0066】 In this embodiment, markers are detected and the detection range is set based on frequency component values ​​related to blurring, thus enabling more appropriate setting of the detection range while suppressing an increase in processing load. 【0067】 In this embodiment, if a marker is not detected, the frequency threshold used for comparison with the frequency component value is changed to detect the marker. This allows for the detection of markers that are out of focus, and enables the calculation of the detection accuracy of such markers with high precision. 【0068】 In this embodiment, the detection accuracy is predicted by an approximation curve generated based on the relationship between the detection accuracy and distance of the detected marker, thereby improving the prediction accuracy. 【0069】 (Other embodiments) In the embodiment described above, a detection range was set for detecting an image to generate a virtual viewpoint image. However, the technology of this embodiment may also be applied to other techniques that detect and use an image within an appropriate depth of field. 【0070】 In the above-described embodiment, whether or not a marker was detected was determined by comparing the frequency value of the captured image with a frequency threshold, but the detection of a marker is not limited to this method. For example, whether or not a marker was detected may be determined by comparing the frequency characteristics, such as the frequency range of the captured image, with a predetermined frequency comparison characteristic. 【0071】 In the above-described embodiment, a detection marker was used to detect the marker, but the marker may be a subject present in the shooting area. Alternatively, a moving subject present in the shooting area may be used as the marker. 【0072】 This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, this disclosure can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions. 【0073】 The disclosures herein include the following information processing devices, information processing methods, and programs. (Item 1) A detection means that detects the markers contained in each of the multiple captured images generated by the imaging device capturing markers at multiple different distances and calculates the detection accuracy, Distance calculation means for calculating the distance between each of the detected markers and the imaging device, A setting means for setting an area to be used in processing performed using the captured image, based on the detection accuracy, a predetermined reference accuracy, and the distance, An information processing device characterized by having the following features. (Item 2) The system includes a determination means for determining the detection accuracy of the marker at a distance where the marker has not been detected. The information processing device described in item 1, characterized by the features described herein. (Item 3) The detection means detects the marker based on the frequency component values ​​of the captured image and a predetermined frequency threshold. The information processing device described in item 1, characterized by the features described herein. (Item 4) If the detection means fails to detect the marker, it changes the frequency threshold and tries to detect the marker again on the same captured image to calculate the detection accuracy. The information processing device described in item 3, characterized by the features described herein. (Item 5) The determination means determines the detection accuracy at distances where no markers are detected, based on the detection accuracy of the detected markers and the distance. The setting means sets the area based on the calculated and determined detection accuracy. An information processing device according to any one of items 2 to 4, characterized in that it is an information processing device. (Item 6) The detection means calculates the detection accuracy based on the frequency threshold at which the marker was detected and a predetermined reference frequency. An information processing device according to item 4 or item 5, characterized in that it is an information processing device. (Item 7) The setting means compares the detection accuracy with a predetermined reference accuracy to set the range. An information processing device according to any one of items 1 to 6, characterized by the features described in item 1 to 6. (Item 8) The distance calculation means calculates the distance when the detected marker is directly facing the imaging device. An information processing device according to any one of items 1 to 7, characterized by the features described in item 1 to 7. (Item 9) The determination means determines the detection accuracy using an approximation curve generated based on the relationship between the detection accuracy of the detected marker and the distance. An information processing device according to any one of items 2 to 8, characterized by the features described in item 2 to 8. (Item 10) The processing performed using the aforementioned captured image is either a process to generate a virtual viewpoint image or a process to generate a shape model representing the three-dimensional shape of the subject. An information processing device according to any one of items 1 to 9, characterized in that it is an information processing device. (Item 11) A step of detecting the markers contained in each of the multiple captured images generated by the imaging device capturing markers at multiple different distances and calculating the detection accuracy, A step of calculating the distance between each of the detected markers and the imaging device, A step of setting an area in the captured image to be used in processing performed using the captured image, based on the detection accuracy, a predetermined reference accuracy, and the distance. An information processing method characterized by having the following features. (Item 12) A program that, when read and executed by a computer, causes the computer to perform each step of the information processing method described in item 11. 【0074】 This disclosure is not limited to the embodiments described above, and various modifications and variations are possible. [Explanation of symbols] 【0075】 101...Image processing system, 102...Imaging device, 103...Image processing device, 201...Image acquisition unit, 202...Marker detection unit, 203...Position acquisition unit, 204...Marker distance calculation unit, 205...Marker detection accuracy prediction unit, 206...Setting unit, 207...Decision unit.

Claims

[Claim 1] A detection means that detects the markers contained in each of the multiple captured images generated by the imaging device capturing markers at multiple different distances and calculates the detection accuracy, Distance calculation means for calculating the distance between each of the detected markers and the imaging device, A setting means for setting an area to be used in processing performed using the captured image, based on the detection accuracy, a predetermined reference accuracy, and the distance, An information processing device characterized by having the following features. [Claim 2] The system includes a determination means for determining the detection accuracy of the marker at a distance where the marker has not been detected. The information processing apparatus according to feature 1. [Claim 3] The detection means detects the marker based on the frequency component values ​​of the captured image and a predetermined frequency threshold. The information processing apparatus according to feature 1. [Claim 4] If the detection means fails to detect the marker, it changes the frequency threshold and tries to detect the marker again on the same captured image to calculate the detection accuracy. The information processing apparatus according to feature 3. [Claim 5] The determination means determines the detection accuracy at distances where no markers are detected, based on the detection accuracy of the detected markers and the distance. The setting means sets the area based on the calculated and determined detection accuracy. The information processing apparatus according to feature 2. [Claim 6] The detection means calculates the detection accuracy based on the frequency threshold at which the marker was detected and a predetermined reference frequency. The information processing apparatus according to feature 4. [Claim 7] The setting means compares the detection accuracy with a predetermined reference accuracy to set the range. The information processing apparatus according to feature 1. [Claim 8] The distance calculation means calculates the distance when the detected marker is directly facing the imaging device. The information processing apparatus according to feature 1. [Claim 9] The determination means determines the detection accuracy using an approximation curve generated based on the relationship between the detection accuracy of the detected marker and the distance. The information processing apparatus according to feature 2. [Claim 10] The processing performed using the aforementioned captured image is either a process to generate a virtual viewpoint image or a process to generate a shape model representing the three-dimensional shape of the subject. The information processing apparatus according to feature 1. [Claim 11] A step of detecting the markers contained in each of the multiple captured images generated by the imaging device capturing markers at multiple different distances and calculating the detection accuracy, A step of calculating the distance between each of the detected markers and the imaging device, A step of setting an area in the captured image to be used in processing performed using the captured image, based on the detection accuracy, a predetermined reference accuracy, and the distance. An information processing method characterized by having the following features. [Claim 12] A program that, when read and executed by a computer, causes the computer to perform each step of the information processing method described in claim 11.

Images