Information processing device, information processing system, information processing method, and program

Abstract

The present invention provides an information processing device capable of appropriately determining additional feature regions in an environment where objects are moving. [Solution] The information processing device (120) includes an acquisition means (121) that acquires the distribution of features used to measure the position and orientation of the viewpoint of an imaging device from an image (201) captured by an imaging device (110); a calculation means (122) that calculates a blind spot area (207) based on the difference between the image and a past image captured by the imaging device at the viewpoint; a determination means (121) that determines a feature addition area using the distribution of features and the ratio of the blind spot area; and an output means (126) that superimposes the feature addition area onto the image and outputs it.

Description

Technical Field

[0006] , ,

[0005] , , ,

[0001] The present invention relates to an information processing apparatus, an information processing system, an information processing method, and a program.

Background Art

[0002] Conventionally, a self-position and orientation technology using an image captured by an imaging device has been known. In this technology, the calculation accuracy of the position and orientation of the imaging device varies according to the arrangement of features in the usage environment. In order to calculate the position and orientation of the measurement viewpoint with a desired accuracy, it is necessary to arrange features in advance so that the calculation accuracy of the position and orientation of the imaging device becomes high. However, it is difficult for the user to determine the arrangement of features that can increase the calculation accuracy of the position and orientation of the imaging device.

[0003] Therefore, a technology for assisting the work of arranging features in order to increase the calculation accuracy of the position and orientation of the imaging device has been developed. Patent Document 1 discloses a method of obtaining features used for calculating the position and orientation of an imaging device from an image, presenting to the user the additional position and orientation of the features such that the distribution of the features becomes uniform, and assisting in the arrangement of the features.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the method disclosed in Patent Document 1, due to the movement of objects such as in a logistics warehouse, the feature addition area may become a blind spot and become invisible. Also, since the viewpoint of the user and the viewpoint of the imaging device are different, the user cannot recognize the blind spot area from the viewpoint of the imaging device, and it is difficult for the user to determine the position of the feature addition area considering the blind spot.

[0006] Therefore, the present invention aims to provide an information processing device that can appropriately determine additional feature regions in an environment in which objects are moving. [Means for solving the problem]

[0007] An information processing device as one aspect of the present invention is characterized by comprising: acquisition means for acquiring the distribution of features used to measure the position and orientation of the viewpoint of an imaging device from an image captured by an imaging device; calculation means for calculating a blind spot region based on the difference between the image and a past image captured by the imaging device at the viewpoint; determination means for determining a feature addition region using the distribution of features and the ratio of the blind spot region; and output means for superimposing the feature addition region onto the image and outputting it.

[0008] Other objects and features of the present invention are described in the following embodiments. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an information processing device that can appropriately determine additional feature regions in an environment in which an object is moving. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram showing an example of the system configuration in the first embodiment. [Figure 2] This figure shows an example of a display screen in the first embodiment. [Figure 3] This is a flowchart showing the process in the first embodiment. [Figure 4] This is a flowchart showing the process for calculating the blind spot area in the first embodiment. [Figure 5] This flowchart shows the process for calculating the blind spot area in the second embodiment. [Figure 6] This flowchart shows the process for calculating the blind spot area in the second embodiment. [Figure 7]This flowchart shows the process for calculating the blind spot area in the second embodiment. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0012] (First Embodiment) <System Configuration> First, with reference to Figure 1, the system (feature addition region presentation system, information processing system) 100 in the first embodiment will be described. Figure 1 is a diagram showing an example of the configuration of system 100. The system 100 in this embodiment is composed of an imaging device 110, a feature addition region presentation device (information processing device) 120, a server device 130, and a network 140. Note that the configuration of system 100 shown in Figure 1 is merely an example, and other components may be added, or some components may be deleted.

[0013] The imaging device 110 is a camera with a built-in processor. The imaging device 110 captures images of the surrounding environment and estimates the camera's own position and orientation from the captured images. The obtained data is stored in the memory of the imaging device 110 and transmitted externally. The imaging device 110 is mounted on transport robots and forklifts, and the estimated position and orientation is used for autonomous driving and position management. The imaging device 110 comprises an imaging unit 111, a position and orientation estimation unit 112, a storage unit 113, and a communication unit 114.

[0014] The imaging unit 111 is configured to include an image sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) sensor. The imaging unit 111 is a functional unit that captures the surrounding environment as an image.

[0015] The position and orientation estimation unit 112 is a functional unit composed of a processor such as a CPU, which reads a control program and executes position and orientation estimation processing. The position and orientation estimation processing estimates the position and orientation of the camera based on natural features detected from an image, for example, by using the technology of Visual SLAM (Visual Simultaneous Localization and Mapping). The position and orientation estimation unit 112 receives an image from the imaging unit 111, performs position and orientation estimation processing, and transmits and stores the image, natural features, and estimated position and orientation used in the processing to the storage unit 113. The position and orientation estimation unit 112 may add sensors other than the camera and use the information obtained from the sensors in combination with the image for position and orientation estimation. For example, inertial information that can be obtained from an IMU sensor may be used for position and orientation estimation.

[0016] The storage unit 113 is composed of a memory device and stores data from the imaging unit 111 and the position and orientation estimation unit 112. The storage unit 113 stores, as a set, the image captured by the imaging unit 111, the natural features calculated from the image, and the estimated position and orientation.

[0017] The communication unit 114 includes a communication module for communicating with external devices and networks. The communication unit 114 transmits, via the network 140, to the server device 130, as a set, the image captured by the imaging unit 111, the natural features calculated from the image, and the estimated position and orientation stored in the storage unit 113.

[0018] The feature addition area presentation device 120 is at least one computer such as a PC including a CPU, a memory, etc. The imaging device 110 and the feature addition area presentation device 120 are communicably connected via the network 140. The feature addition area presentation device 120 includes a feature addition area determination unit (acquisition means, determination means) 121, a blind spot area calculation unit (calculation means) 122, a storage unit 123, a communication unit 124, an operation unit 125, and a display unit (output means) 126.

[0019] The feature addition area determination unit 121 is a functional unit composed of a processor such as a CPU, which reads a control program and executes feature addition area determination processing. The feature addition area determination processing will be described later using a figure.

[0020] The dead angle area calculation unit 122 is a functional unit composed of a processor such as a CPU, which reads a control program and executes dead angle area estimation processing. The dead angle area estimation processing will be described later by referring to a figure.

[0021] The storage unit 123 is composed of a memory device, and stores a set of a captured image received from the communication unit 124, natural features calculated from the captured image, and a position and orientation estimated from the captured image.

[0022] The communication unit 124 includes a communication module for communicating with an external device or network. When the communication unit 124 receives an instruction for feature addition and a feature addition viewpoint from the operation unit 125, it transmits an instruction to transmit an image of the feature addition viewpoint (the viewpoint of the imaging device 110) to the server device 130. Thereafter, it receives a set of a captured image, natural features calculated from the captured image, and a position and orientation estimated from the captured image from the server device 130.

[0023] The operation unit 125 receives operations from the user. The operation unit 125 is, for example, a mouse or a keyboard, and it is assumed that the user operates these to input a user operation to the feature addition area presentation device 120. However, it is not limited thereto, and the operation unit 125 may be any device capable of detecting the intention of other users, such as a touch panel or a microphone. The operation unit 125 receives an operation for changing the display on the display unit 126. When a feature addition viewpoint is input from the user, the operation unit 125 outputs it to the feature addition area determination unit 121. Only an instruction for feature addition may be input from the user. When only an instruction for feature addition is input, the position and orientation with the lowest estimation accuracy when the imaging device 110 for feature addition estimates the position and orientation is used as the feature addition viewpoint.

[0024] The display unit 126 presents (outputs) information to the user, for example, via a display. Information may also be presented (output) by methods other than screen display, such as sound or vibration. The display unit 126 overlays the blind spot area estimated by the blind spot area calculation unit 122 and the feature addition area determined by the feature addition area determination unit 121 onto the captured image acquired from the storage unit 123. A more specific display method will be described later with reference to Figure 2.

[0025] The server device 130 is composed of multiple processors, memory modules, and storage devices, and is a device that stores data received via the network and transmits the stored data. The server device 130 stores a set of captured images received from the imaging device 110, natural features calculated from the captured images, and position and orientation estimated from the captured images, and stores them as past captured images. When the server device 130 receives an instruction from the feature addition region presentation device 120 to transmit an image of the feature addition viewpoint, it searches for a past image of the feature addition viewpoint (a past image captured by the imaging device 110 at the viewpoint of the imaging device 110) using the estimated position and orientation, and transmits it to the feature addition region presentation device 120.

[0026] Network 140 is comprised of multiple routers, switches, cables, etc., that meet communication standards such as Ethernet®. Network 140 may be any network that enables communication between the imaging device 110 and the feature addition area presentation device 120, and can be constructed with any size, configuration, and communication standards. For example, Network 140 may be the Internet, a wired LAN (Local Area Network), a wireless LAN, a WAN (Wide Area Network), etc.

[0027] Next, with reference to Figures 2(a) and 2(b), the display screen 200 that the display unit 126 displays to the user will be described. Figure 2(a) is an example of the display screen for the feature addition area. The display screen in Figure 2(a) shows the feature addition viewpoint image 201, the feature addition area 202, the feature addition area list 203, the estimated blind spot area display button 204, the estimated blind spot area hide button 205, and the estimated blind spot area edit button 206. Figure 2(b) is an example of the display screen for the estimated blind spot area (blind spot area) 207.

[0028] The feature addition viewpoint image 201 is the image of the viewpoint from which features will be added. The feature addition region 202 shows the region that is a candidate for feature addition superimposed on the feature addition viewpoint image 201. The feature addition candidate selected by the user from the feature addition region list 203 is displayed.

[0029] The feature addition area list 203 is a list of candidate features for addition. When the user selects an item from the list, the corresponding feature addition area 202 is displayed. The estimated blind spot area display button 204 is a button that displays the estimated blind spot area 207.

[0030] The estimated blind spot area hide button 205 hides the estimated blind spot area 207. The estimated blind spot area edit button 206 edits the estimated blind spot area 207. Editing the estimated blind spot area 207 allows the user to move the vertices of the area to any position or orientation in the feature addition viewpoint image 201 by pressing the estimated blind spot area edit button 206 once. The editing is completed by pressing the estimated blind spot area edit button 206 again.

[0031] The estimated blind spot area 207 indicates the estimated blind spot area superimposed on the feature-added viewpoint image 201.

[0032] <Processing flow> Next, the processing (information processing method) in this embodiment will be described with reference to Figures 3 and 4. Figures 3 and 4 are flowcharts illustrating the processing performed by the feature addition area presentation device 120. However, the feature addition area presentation device 120 does not necessarily have to perform all the steps described in each flowchart.

[0033] Figure 3 is a flowchart illustrating the process in which a user requests the display of a feature addition location, and the feature addition area display device 120 determines and displays the feature addition location.

[0034] First, in step S301, the feature addition region determination unit 121 receives the feature addition viewpoint specified by the user from the operation unit 125. Next, in step S302, the feature addition region determination unit (acquisition means) 121 acquires all pairs of past captured images and feature distributions from the storage unit 123 whose estimated position and orientation match the feature addition viewpoint. The feature distribution is used to measure the position and orientation of the viewpoint of the imaging device 110 from the images captured by the imaging device 110.

[0035] Next, in step S303, the feature addition area determination unit 121 outputs the captured image of the feature addition viewpoint to the blind spot area calculation unit 122 in order to calculate the blind spot area. The specific processing (flow for calculating the blind spot area) will be described later with reference to Figure 4.

[0036] Next, in step S304, the feature addition region determination unit 121 divides the captured image into multiple rectangular regions by dividing it into equal vertical and horizontal intervals. The number of vertical and horizontal divisions is determined by setting the minimum number of pixels to be included based on, for example, the size of the feature to be added on the image. Specifically, for an image with dimensions of 640 horizontally and 480 vertically, if the goal is to include at least 10,000 pixels, the number of divisions can be, for example, 5 horizontal divisions and 4 horizontal divisions. The division method is not limited to this; the image may also be divided into polygons such as triangles, or the upper part of the image may be divided into smaller shapes and the lower part into larger shapes. Next, in step S305, the feature addition region determination unit 121 calculates the amount of natural features in the divided regions.

[0037] The following steps S306 to S308 loop through and process each divided region. In step S306, the feature addition region determination unit 121 determines whether the amount of features in the divided region is less than or equal to a first threshold. At this time, the first threshold may be determined based on the amount of features required by the position and orientation estimation unit 112 of the imaging device 110. If the amount of features in the divided region is less than or equal to the first threshold, the process proceeds to step S307. On the other hand, if the amount of features in the divided region is greater than the first threshold, the processing of the next divided region begins.

[0038] In step S307, the feature addition region determination unit 121 determines whether the area ratio (percentage) of the blind spot region to the divided region is less than or equal to the second threshold. At this time, the second threshold may be determined based on the size of the feature added by the user on the image. If the area ratio is less than or equal to the second threshold, the process proceeds to step S308. On the other hand, if the area ratio is greater than the second threshold, the processing of the next divided region begins.

[0039] In step S308, the feature addition region determination unit 121 adds the divided region being processed to the feature addition region list.

[0040] Next, in step S309, the display unit 126 displays the feature addition location based on the captured image, the blind spot area, and the feature addition area list.

[0041] In this embodiment, the first threshold and the second threshold used in the determination process in steps S306 and S307 are not limited to fixed values, but may be changed according to other conditions.

[0042] Figure 4 is a flowchart illustrating the process of calculating the blind spot area performed by the blind spot area calculation unit 122. First, in step S401, the blind spot area calculation unit 122 receives the captured image of the feature addition viewpoint. Next, in step S402, the blind spot area calculation unit 122 calculates the image difference between captured images (the difference between multiple captured images taken at different times from the same viewpoint). For example, the image difference can be calculated by simply comparing the brightness of the images and defining the area where the brightness difference is above a certain threshold as the difference area. Alternatively, the image may be converted to grayscale before calculating the difference. Next, in step S403, the blind spot area calculation unit 122 outputs the union of the image differences between captured images as the blind spot area.

[0043] <Modified form of the first embodiment> Next, a modification of this embodiment will be described. In this embodiment, in step S307, the area ratio of the blind spot region was used to determine whether to add it to the feature addition region list. However, in addition to the area ratio of the blind spot region, the frequency of the blind spot region may also be used to determine whether to adopt it as a feature addition location. For example, if the frequency of the blind spot region is less than or equal to the third threshold, the decision of whether to add it to the feature addition region list is made regardless of the area ratio of the blind spot region. The frequency of the blind spot region can be calculated by dividing the sum of the differences in the captured images calculated by the blind spot region calculation unit 122 by the number of differences. Also, in step S309 in Figure 3, if the feature addition region list is empty, a region with a small amount of features and a low frequency of being a blind spot region may be added.

[0044] As described above, in this modified example, the feature addition region determination unit 121 calculates the frequency of blind spots from past images and determines regions with a frequency lower than the third threshold as feature addition regions. This makes it possible to determine regions where adding features is highly effective, even in environments where there are many blind spots when an object is placed and there are no regions to add features to due to the blind spots.

[0045] (Second Embodiment) Next, a second embodiment of the present invention will be described. In the first embodiment described above, the blind spot area is calculated from the difference of past images, but in this embodiment, a case is described in which the vanishing point of the image is considered in addition to the difference of past images. In the description of this embodiment, the same points as in the first embodiment will be omitted from the explanation.

[0046] The processing of this embodiment can be carried out by a system 100 similar to that of the first embodiment described with reference to Figure 1. In this embodiment, the process for calculating the blind spot area described using Figure 4 of the first embodiment is replaced. Figure 5 is a flowchart illustrating the process for calculating the blind spot area performed by the blind spot area calculation unit 122 in this embodiment. The processes in steps S501 to S503 are the same as the processes in steps S401 to S403, respectively.

[0047] In step S504, the blind spot area calculation unit 122 calculates the vanishing point from the captured image. The vanishing point is calculated by extracting line segments in the image and determining the intersection of these line segments as the vanishing point. Line segments in the image can be extracted using, for example, the Hough transform or LSD (Line Segment Detector). Next, in step S505, the blind spot area calculation unit 122 connects each vertex of the union region to the vanishing point with a line segment. The blind spot area calculation unit 122 then outputs the region enclosed by the union region and the line segments as the blind spot area. In other words, the blind spot area calculation unit 122 calculates the region enclosed by lines connecting points in the image difference region to the vanishing point as the blind spot area.

[0048] <Modification 1 of the second embodiment> Next, a modification 1 of this embodiment will be described. In this modification, a process to equalize the height of the blind spots in the viewpoint (shooting viewpoint) of the imaging device 110 is added after step S503 in Figure 5. Figure 6 is a flowchart of the process to equalize the height of the blind spots in the shooting viewpoint. The processes of steps S604 to S608 added to the flowchart in Figure 5 will be described.

[0049] In step S604, the blind spot area calculation unit 122 calculates the length of the vertical line segment at each vertex of the union region. Subsequently, in step S605, the blind spot area calculation unit 122 groups together nearby line segments based on the vertical position of the lower end of the calculated line segments.

[0050] Steps S606 and S607 are performed in a loop for each group. In step S606, the blind spot area calculation unit 122 calculates the length of the longest line segment among the line segments in the group. Subsequently, in step S607, the blind spot area calculation unit 122 extends the position of the upper end of the line segment in the group so that the length of the line segment matches the length of the longest line segment.

[0051] Next, in step S608, the blind spot area calculation unit 122 updates the polygon with the extended line segment as its vertex, as a new union area.

[0052] As described above, in this modified example, the blind spot area calculation unit 122 calculates the length of the vertical line segment at the vertex of the blind spot area. The blind spot area calculation unit 122 then adjusts the position of the upper end of the line segment to match the length of the line segment in a set of line segments, where the difference in the vertical position of the lower end of the line segment is less than or equal to a predetermined difference, to the longest line segment in that set, and updates the adjusted position to the vertex of the blind spot area. This makes it possible to assume a blind spot area and perform calculations in environments where the upper limit of the height of objects stacked in the shooting viewpoint is fixed, even if images of stacked objects have not been recorded.

[0053] <Modification 2 of the second embodiment> Next, a modified example 2 of this embodiment will be described. In this modified example, a process to limit the depth extending the blind spot area is added after step S505 in Figure 5. Figure 7 is a flowchart showing the process to limit the depth extending the blind spot area. The processes of steps S706 and S707 added to the flowchart in Figure 5 will be described.

[0054] In step S706, the blind spot area calculation unit 122 calculates the lateral distance between the union area and the vanishing point to find the shortest distance. Subsequently, in step S707, the blind spot area calculation unit 122 removes from the blind spot area any areas within the blind spot area where the lateral distance is less than the shortest distance.

[0055] As described above, in this modified example, the blind spot area calculation unit 122 calculates the lateral distance between the vertex of the blind spot area and the vanishing point to determine the shortest lateral distance, and excludes areas within the blind spot area where the lateral distance from the vanishing point is shorter than the shortest distance. This allows the depth to be limited to the position recorded in the image when the environment in which objects are stacked on the left and right is separated by a wall or the like.

[0056] (Other embodiments) Although embodiments have been described in detail above, the present invention can take the form of, for example, a system, apparatus, method, program, or recording medium (storage medium). Specifically, it may be applied to a system consisting of multiple devices (for example, a host computer, interface devices, imaging devices, web applications, etc.), or to an apparatus consisting of a single device.

[0057] Furthermore, the present invention can also be realized by the following process: supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium; and having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0058] According to each embodiment, it is possible to provide an information processing device, an information processing system, an information processing method, and a program that can appropriately determine additional feature regions in an environment in which objects are moving.

[0059] Each embodiment of the disclosure includes the following configuration and method. (Composition 1) An acquisition means for acquiring the distribution of features used to measure the position and orientation of the viewpoint of the imaging device from an image captured by the imaging device, A calculation means for calculating a blind spot area based on the difference between the aforementioned image and a past image captured by the imaging device at the aforementioned viewpoint, A determination means for determining a feature addition region using the distribution of the features and the ratio of the blind spot region, Output means for superimposing the aforementioned feature-added region onto the image and outputting it, An information processing device characterized by comprising: (Configuration 2) The determination means determines a region where the distribution of the features is smaller than a first threshold and the proportion of the blind spot region is smaller than a second threshold as the feature addition region. The information processing device according to configuration 1, characterized by the above. (Composition 3) The information processing apparatus according to configuration 1 or 2, wherein the determination means determines the feature addition area using the distribution of the features in the divided region of the image and the ratio of the blind spot area. (Composition 4) The calculation means calculates the blind spot region based on the difference and the vanishing point in the image. An information processing device according to any one of configurations 1 to 3, characterized by the above. (Composition 5) The calculation means calculates the area enclosed by the line connecting the point in the difference area and the vanishing point as the blind spot area. The information processing apparatus according to configuration 4, characterized by the features described above. (Composition 6) The aforementioned calculation means is The length of the vertical line segment at the vertex of the blind spot region is calculated. The position of the upper end is adjusted so that the length of the line segments in the set matches the length of the longest line segment in the set where the difference in the vertical position of the lower end of the line segment is less than or equal to a predetermined difference. The adjusted position is updated to the vertex of the blind spot area. An information processing device according to any one of configurations 1 to 5, characterized by the above. (Composition 7) The aforementioned calculation means is The lateral distance between the vertex of the blind spot region and the vanishing point is calculated to determine the shortest lateral distance. The area within the blind spot region where the lateral distance from the vanishing point is shorter than the shortest distance is excluded from the blind spot region. The information processing apparatus according to configuration 4 or 5, characterized by the above. (Composition 8) The aforementioned determination means is The frequency of blind spots is calculated from the aforementioned past images. The region where the frequency is lower than the third threshold is determined to be the feature addition region. An information processing device according to any one of configurations 1 to 7, characterized by the above. (Composition 9) The output means displays the blind spot area superimposed on the image. An information processing device according to any one of configurations 1 to 8, characterized by the above. (Composition 10) An information processing system having an imaging device and an information processing device, The aforementioned information processing device is An acquisition means for acquiring the distribution of features used to measure the position and orientation of the viewpoint of the imaging device from an image captured by the imaging device, A calculation means for calculating a blind spot area based on the difference between the aforementioned image and a past image captured by the imaging device at the aforementioned viewpoint, A determination means for determining a feature addition region using the distribution of the features and the ratio of the blind spot region, Output means for superimposing the aforementioned feature-added region onto the image and outputting it, An information processing system characterized by comprising the following features. (Method 1) An acquisition step of obtaining the distribution of features used to measure the position and orientation of the viewpoint of the imaging device from an image captured by the imaging device, A calculation step of calculating a blind spot area based on the difference between the aforementioned image and a past image captured by the imaging device at the aforementioned viewpoint, A determination step of determining a feature addition region using the distribution of the features and the ratio of the blind spot region, Output process includes superimposing the aforementioned feature-added region onto the image and outputting it, An information processing method characterized by comprising: (Composition 11) A program characterized by causing a computer to execute each step of the information processing method described in Method 1.

[0060] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. [Explanation of Symbols]

[0061] 110 Imaging device 120 Feature Addition Area Presentation Device (Information Processing Device) 121 Feature Addition Region Determination Unit (Acquisition Means, Determination Means) 122 Blind spot area calculation unit (calculation means) 201 Feature-added viewpoint image (image) 202 Feature Addition Area 207 Estimated blind spot area (blind spot area)

Claims

1. An acquisition means for acquiring the distribution of features used to measure the position and orientation of the viewpoint of the imaging device from an image captured by the imaging device, A calculation means for calculating a blind spot area based on the difference between the aforementioned image and a past image captured by the imaging device at the aforementioned viewpoint, A determination means for determining a feature addition region using the distribution of the features and the ratio of the blind spot region, Output means for superimposing the aforementioned feature-added region onto the image and outputting it, An information processing device characterized by comprising:

2. The determination means determines a region where the distribution of the features is smaller than a first threshold and the proportion of the blind spot region is smaller than a second threshold as the feature addition region. The information processing apparatus according to feature 1.

3. The information processing apparatus according to claim 1 or 2, wherein the determination means determines the feature addition area using the distribution of the features in the divided area of ​​the image and the ratio of the blind spot area.

4. The calculation means calculates the blind spot region based on the difference and the vanishing point in the image. The information processing apparatus according to claim 1 or 2.

5. The calculation means calculates the area enclosed by the line connecting the point in the difference area and the vanishing point as the blind spot area. The information processing apparatus according to feature 4.

6. The aforementioned calculation means is The length of the vertical line segment at the vertex of the blind spot region is calculated. The position of the upper end is adjusted so that the length of the line segments in the set matches the length of the longest line segment in the set where the difference in the vertical position of the lower end of the line segment is less than or equal to a predetermined difference. The adjusted position is updated to the vertex of the blind spot area. The information processing apparatus according to claim 1 or 2.

7. The aforementioned calculation means is The lateral distance between the vertex of the blind spot region and the vanishing point is calculated to determine the shortest lateral distance. The area within the blind spot region where the lateral distance from the vanishing point is shorter than the shortest distance is excluded from the blind spot region. The information processing apparatus according to feature 4.

8. The aforementioned determination means is The frequency of blind spots is calculated from the aforementioned past images. The region where the frequency is lower than the third threshold is determined to be the feature addition region. The information processing apparatus according to claim 1 or 2.

9. The output means displays the blind spot area superimposed on the image. An information processing apparatus according to claim 1 or 2, characterized by the above.

10. An information processing system having an imaging device and an information processing device, The aforementioned information processing device is An acquisition means for acquiring the distribution of features used to measure the position and orientation of the viewpoint of the imaging device from an image captured by the imaging device, A calculation means for calculating a blind spot area based on the difference between the aforementioned image and a past image captured by the imaging device at the aforementioned viewpoint, A determination means for determining a feature addition region using the distribution of the features and the ratio of the blind spot region, Output means for superimposing the aforementioned feature-added region onto the image and outputting it, An information processing system characterized by comprising the following features.

11. An acquisition step of obtaining the distribution of features used to measure the position and orientation of the viewpoint of the imaging device from an image captured by the imaging device, A calculation step of calculating a blind spot area based on the difference between the aforementioned image and a past image captured by the imaging device at the aforementioned viewpoint, A determination step of determining a feature addition region using the distribution of the features and the ratio of the blind spot region, Output process includes superimposing the aforementioned feature-added region onto the image and outputting it, An information processing method characterized by comprising:

12. A program characterized by causing a computer to execute each step of the information processing method described in claim 11.

Images