Information processing device, information processing method, and program

Abstract

This information processing device is configured to comprise a control unit that performs processing for grouping a plurality of feature points on captured images used for motion detection of an imaging device and displays each group in display modes that differ from each other, and performs processing for collectively deleting a plurality of feature points belonging to the same group according to a user operation.

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program, and in particular to setting feature points on an image.

[0002] Match move is a well-known technique for combining computer graphics (CG) materials with live-action images. This technique involves detecting the movement of the imaging device (hereinafter also referred to as the camera) or subject based on the live-action image, and using this information to combine with the CG material, resulting in a seamless image composition. Detecting the movement of the camera from the movement of a specific subject or area within the image is called camera tracking.

[0003] The following Patent Document 1 discloses that it is possible to automatically obtain placement information of geometric features existing in real space based on captured images input from a camera, and to provide an interface for selecting geometric features sufficient for estimating the position and orientation of the camera.

[0004] JP 2009-110304 A

[0005] To improve the accuracy of camera tracking, it is important to properly set feature points in captured images. While feature points in an image can be set automatically to a certain extent, to improve accuracy, the operator must adjust them by adding or deleting individual feature points. In particular, the accuracy of match move depends on which feature points are added or deleted, so the operator must be aware of this and operate the system accordingly, which is extremely time-consuming. For example, selecting only feature points on the same plane in three-dimensional object space reduces accuracy, so operators must be conscious of dispersing feature points three-dimensionally so that they do not lie on the same plane. Furthermore, when removing unreliable feature points on a low-contrast object, for example, it is necessary to delete feature points on the same object one by one. Furthermore, because moving objects move, feature points on moving objects should not be used for camera tracking. These factors make the operation of setting feature points by the user extremely tedious.

[0006] Therefore, an object of the present technology is to provide a user interface (UI) that realizes efficient setting operations of feature points used for camera tracking.

[0007] An information processing device according to the present technology includes a control unit that performs a process of grouping multiple feature points on a captured image used for motion detection of an imaging device and displaying each group in a different display mode, and a process of collectively deleting multiple feature points in the same group in response to a user operation. When detecting the motion of the imaging device at the time of image capture from a captured image, feature points are set in the captured image, and the motion of the imaging device at the time of image capture is estimated from the movement of the feature points. Depending on the setting locations and number of feature points, estimation accuracy can be improved and calculation efficiency can be increased. In this case, a user interface for setting feature points is provided.

[0008] 1 is an explanatory diagram of a match move. FIG. 1 is a block diagram of an information processing device according to an embodiment. FIG. 2 is an explanatory diagram of an example of a feature point setting screen in object mode according to an embodiment. FIG. 3 is a flowchart of UI processing in object mode according to an embodiment. FIG. 4 is a flowchart of designation operation response processing according to an embodiment. FIG. 5 is an explanatory diagram of a dialog display in response to a designation operation according to an embodiment. FIG. 6 is an explanatory diagram of a display after batch deletion according to an embodiment. FIG. 7 is a flowchart of designation operation response processing according to an embodiment. FIG. 8 is an explanatory diagram of a dialog display in response to a designation operation according to an embodiment. FIG. 9 is an explanatory diagram of a display after thinning-out deletion according to an embodiment. FIG. 10 is an explanatory diagram of an example of a feature point setting screen in spatial position mode according to an embodiment. FIG. 11 is a flowchart of UI processing in spatial position mode according to an embodiment. FIG. 12 is a flowchart of a first grouping processing example according to an embodiment. FIG. 13 is a flowchart of a second grouping processing example according to an embodiment. FIG. 14 is an explanatory diagram of a second grouping processing example according to an embodiment. FIG. 15 is an explanatory diagram of display of planar information in spatial position mode according to an embodiment. FIG. 16 is a flowchart of UI processing when planar information is displayed in spatial position mode according to an embodiment. FIG. 17 is a flowchart of mode switching according to an embodiment.

[0009] Hereinafter, the embodiments will be described in the following order: <1. Configuration of information processing device> <2. UI in object mode> <3. UI in spatial position mode> <4. Mode switching> <5. Summary and modified examples>

[0010] In this disclosure, "image" refers to both a moving image and a still image. In the embodiment, an example of performing match move on a moving image captured by a camera will be described, and the image on the feature point setting screen is an image of one frame that constitutes the moving image. Furthermore, "image" refers to the image that is actually displayed on the screen, but "image" in the signal processing process and transmission path leading up to display on the screen refers to image data.

[0011] 1 shows an example of match move. Assume that this is an image in which a CG image 2 is composited with a background image 1, which is a live-action image captured by a camera. By analyzing the live-action image and detecting the movement of the camera at the time of capture, it is possible to composite the CG image 2 seamlessly with the background image 1. Therefore, when performing match move, camera tracking is performed to detect the amount of camera movement from the movement of subjects and areas in the captured live-action image.

[0012] To improve the accuracy of camera tracking, it is necessary to set appropriate feature points in the images. Feature points are set to estimate the camera movement, and there are various points to keep in mind for setting them appropriately, such as the following:

[0013] - The more feature points there are in one frame, the higher the estimation accuracy, but the more there are, the greater the processing load, so a moderate number of feature points is desirable. - It is not desirable for feature points to be concentrated on the same plane in three-dimensional subject space; to improve camera tracking accuracy, it is better for feature points to be scattered three-dimensionally in space. - Because moving subjects move unrelated to the movement of the camera in video, feature points on moving subjects will reduce camera tracking accuracy. Therefore, it is desirable not to set feature points on moving subjects. - Feature points in low-contrast areas (blurred areas) have low reliability, so it is desirable not to set feature points in low-contrast areas.

[0014] For example, when performing work for match move, the user (worker) sets and adjusts feature points for camera tracking while paying attention to the above points. This embodiment provides a UI that is suitable for such work.

[0015] An example of the configuration of an information processing device 70 that provides such a UI is described with reference to Fig. 2. The information processing device 70 can be configured as, for example, a dedicated workstation, a general-purpose personal computer, a mobile terminal device, a computer device serving as a cloud server, or the like.

[0016] 2 is configured with a processor such as a CPU. The control unit 71 executes various processes in accordance with programs stored in a ROM 72 or a nonvolatile memory unit 74 such as an EEPROM (Electrically Erasable Programmable Read-Only Memory), or programs loaded from a storage unit 79 to a RAM 73. The RAM 73 also stores data necessary for the control unit 71 to execute various processes, as appropriate.

[0017] The image processing unit 85 is configured as a processor that performs various types of image processing, such as image generation processing for video clips, image analysis processing for captured images, generation processing for animation images and CG images, and match move processing.

[0018] The image processing unit 85 can be realized by, for example, a CPU, a graphics processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), an AI (artificial intelligence) processor, or the like, which is separate from the control unit 71. The image processing unit 85 may be provided as a function within the control unit 71. In other words, the control unit 71 and the image processing unit 85 may be configured as an integrated unit.

[0019] The control unit 71, ROM 72, RAM 73, nonvolatile memory unit 74, and image processing unit 85 are interconnected via a bus 83. To this bus 83, an input / output interface 75 is also connected.

[0020] An input unit 76 including operators and operation devices is connected to the input / output interface 75. For example, the input unit 76 may be various operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. The input unit 76 detects a user operation, and a signal corresponding to the input operation is interpreted by the control unit 71.

[0021] The input / output interface 75 is also connected, either integrally or separately, to a display unit 77 such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 78 such as a speaker.

[0022] The display unit 77 performs various displays as a user interface. The display unit 77 is configured, for example, by a display device provided in the housing of the information processing device 70 or a separate display device connected to the information processing device 70. The display unit 77 displays various images on the display screen based on instructions from the control unit 71. The display unit 77 also displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the control unit 71.

[0023] A storage unit 79 configured with an SSD (Solid State Drive) or an HDD (Hard Disk Drive) or a communication unit 80 configured with a modem or the like may be connected to the input / output interface 75. The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, or the like.

[0024] A drive 82 is also connected to the input / output interface 75 as needed, and a removable recording medium 81 such as a flash memory, a memory card, a magnetic disk, an optical disk, or a magneto-optical disk is appropriately attached. The drive 82 allows data files such as image files and various computer programs to be read from the removable recording medium 81. The read data files are stored in the storage unit 79, and images and sounds contained in the data files are output on the display unit 77 and the audio output unit 78. In addition, the computer programs and the like read from the removable recording medium 81 are installed in the storage unit 79 as needed.

[0025] In this information processing device 70, software can be installed via network communication by the communication unit 80 or via a removable recording medium 81. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.

[0026] By using this information processing device 70 for image editing processing, it becomes a device that executes the processing of the present disclosure. For example, the information processing device 70 can perform image editing as a match move using live-action video and CG material. In this case, frames within the video are identified for camera tracking and feature point setting is performed, and a UI processing unit 71a is implemented in the control unit 71 as one of the processing functions for setting feature points. Specifically, the control unit 71 performs processing to provide a UI, which will be described below, using a software program.

[0027] The UI processing unit 71a controls the screen display of the feature point setting operation on the display unit 77, accepts user operations via the input unit 76, and performs processing according to the operations.

[0028] <2. UI in Object Mode> Examples of UIs described in this embodiment include a UI in object mode and a UI in spatial position mode. First, the UI in object mode will be described. Each processing example described below is an example of processing executed by the control unit 71 using the functions of the UI processing unit 71a. Note that the control unit 71 performs UI processing while sequentially referring to the image analysis results of the image processing unit 85 during the processing process.

[0029] 3 shows an example of a feature point setting screen in object mode. The displayed image 10 is a single frame of a live-action video that is the target of match move. After displaying the single frame of image 10 in this manner, the information processing device 70 accepts user operations to set and adjust feature points.

[0030] In image 10, feature points 3 are shown superimposed with circular marks. Feature points 3 are set as motion detection points for camera tracking, and appropriate positions and numbers are determined as described above. Note that in FIG. 3 , feature points 3 are indicated by symbols "3a" to "3e," which are circular marks with different colors. In the figure, the color differences are expressed by differences in the patterns within the circular marks. For the sake of explanation, when feature points 3a to 3f are not particularly distinguished, they are referred to as "feature points 3." Since labeling all feature points would make the figure cumbersome, some feature points within each group are labeled with symbols.

[0031] The feature points 3 may be set by the user specifying positions within the image 10, starting from the first one, or an appropriate number may be set automatically. For example, the information processing device 70 may perform image analysis of the image 10 using the image processing unit 85, extract points that can be evaluated as corners within the image, and set them as feature points 3. However, whether the feature points are set automatically initially or the user sets them starting from the first one, manual adjustment by the user is required to achieve a more appropriate feature point arrangement.

[0032] The UI in the object mode described here can be said to be a convenient UI for when the user performs adjustment work after the feature points 3 have been set automatically to a certain extent.

[0033] The object mode UI performs object detection on the image 10, color-codes feature points 3 for each object, and enables the collective deletion of feature points on the same object. Specifically, when feature points are automatically detected using match move software, feature points corresponding to different objects are displayed in different colors. For example, by using an analysis method such as semantic segmentation, objects in the image 10 are distinguished and feature points 3 are automatically color-coded, such as "feature point 3b of a television is blue," "feature point 3d of a chair is green," and "feature point 3e of a person is yellow," as shown in the example. Feature points 3 of the same color are set as one group, allowing for collective deletion.

[0034] For example, if a user thinks, "I don't want to use the feature points of the moving subject when performing match move, so I want to delete all the feature points of people," all the yellow feature points 3e can be deleted with one click, reducing the effort required to delete feature points.

[0035] Specific processing examples will be described with reference to Figures 4 and 5. In step S101 of Figure 4, the control unit 71 sets a frame to be used as a feature point setting screen. That is, the control unit 71 selects a frame from an image of a live-action video, and sets it as image 10 to be displayed as shown in Figure 3.

[0036] In step S102, the control unit 71 performs feature point setting, which is a process of automatically setting several feature points 3 in the image 10 based on the image analysis results of the image processing unit 85.

[0037] In step S103, the control unit 71 performs grouping of feature points 3 based on object detection. The control unit 71 acquires the results of semantic segmentation by the image processing unit 85 and determines various objects that are present in the image 10 as subjects. The control unit 71 then groups each of the feature points 3 set in step S102 for each determined object. For example, in FIG. 3 , the control unit 71 sets a group for curtain feature point 3a, a group for television feature point 3b, a group for person feature point 3e, and so on. In other words, the control unit 71 groups multiple feature points 3 on one object into the same group.

[0038] In step S104, the control unit 71 assigns a color to each group, for example, purple for the group of feature points 3a, blue for the group of feature points 3b, red for the group of feature points 3c, green for the group of feature points 3d, and yellow for the group of feature points 3e.

[0039] In step S105, the control unit 71 performs display control to display the above-described feature point 3 on the display unit 77. As a result, a display such as that shown in FIG.

[0040] In this state, the user can see a large number of feature points 3, each color-coded for each object in the image 10. The control unit 71 then provides a UI that enables the user to add, delete, or delete feature points 3 as desired, or to delete them all at once in groups. That is, the control unit 71 monitors the user's operations in steps S106 and S107 until it determines in step S108 that the feature point setting operation for the current frame has ended.

[0041] There may also be an example where the process proceeds from step S101 to step S105 without automatically setting feature points in step S102 as described above. In that case, feature points 3 are not displayed initially, and the user can add feature points 3 arbitrarily through subsequent operations, with grouping being performed sequentially as feature points are added.

[0042] In step S106, the control unit 71 monitors whether or not the user has performed an operation to add a feature point 3. For example, when the user performs a predetermined operation such as clicking a location in the image 10 where a feature point 3 is not set, the control unit 71 recognizes this as an operation to add a feature point 3, and proceeds to step S110 to perform processing for the addition operation.

[0043] As processing for an addition operation, the control unit 71 sets a new feature point at a position specified by, for example, clicking. Furthermore, it determines the object at the specified position and sets a group for the new feature point 3. If the specified position is on an object on which a feature point 3 has already been set, the new feature point 3 is added to the group of feature points 3 on that object. For example, if a position on the chair in FIG. 3 where no feature point 3 has been set is specified, the new feature point 3 is incorporated into the chair group as feature point 3d. On the other hand, if the specified position is on an object on which no feature point 3 has been set, the new feature point 3 does not belong to any existing group. Therefore, a new group including only the new feature point 3 is set, and a color not used in any existing group is set as the display color for the feature point 3.

[0044] As described above, in step S110, the control unit 71 adds new feature points 3 in response to the designation operation by adding them to an existing group or by setting a new group, and displays them on the image 10 in a color corresponding to the new group.

[0045] In step S107, the control unit 71 monitors whether or not the user has performed an operation to designate a certain feature point 3. For example, when the user performs an operation such as right-clicking on a certain feature point 3 in the image 10, the control unit 71 recognizes this as an operation to designate the feature point 3, and proceeds to step S111 to perform processing corresponding to the designation operation.

[0046] An example of the process for responding to a specifying operation is shown in Fig. 5. In response to the specifying operation, the control unit 71 controls to display a dialog box prompting the user for the next operation in step S120. For example, as shown in Fig. 6, when the user places the cursor 5 on a certain feature point 3e and then right-clicks, a dialog box 6 is displayed.

[0047] For example, "Delete one item," "Delete all," and "Cancel" can be selected in the dialog box 6. The control unit 71 waits for a user operation on the dialog box 6 in a loop of steps S121, S122, and S123.

[0048] If the user selects "Cancel" in the dialog 6, the control unit 71 proceeds from step S121 to step S125, terminates the display of the dialog 6, ends the process of Fig. 5 (step S111 in Fig. 4), and returns to the monitoring loop of steps S106, S107, and S108 in Fig. 4. In other words, if the user clicks on feature point 3 but cancels the operation, the state before the click operation is restored.

[0049] If the user selects "Delete one point" in the dialog 6, the control unit 71 proceeds from step S122 to step S140 in Fig. 5, where it deletes the specified feature point 3 from the settings. Then, in step S141, the control unit 71 updates the display so that the specified feature point 3 is deleted from the display, and also ends the display of the dialog 6. This completes the processing in Fig. 5, and the process returns to the monitoring loop of steps S106, S107, and S108 in Fig. 4. This allows the user to delete only the specified feature point 3. On the display, for example, from the state in Fig. 6, only the feature point 3e where the cursor 5 is positioned is deleted.

[0050] If the user selects "Delete All" in the dialog 6, the control unit 71 proceeds from step S123 to step S130 in FIG. 5, where it deletes the specified feature point 3 and all feature points 3 in the same group from the settings. Then, in step S131, the control unit 71 updates the display so that all feature points 3 in the specified group are deleted from the display, and it also terminates the display of the dialog 6. This completes the processing in FIG. 5, and the process returns to the monitoring loop of steps S106, S107, and S108 in FIG. 4. For example, if "Delete All" is selected from the state in FIG. 6, all feature points 3e set for people are deleted from the settings, and all feature points 3e are deleted from the display, as shown in FIG. 7. This allows the user to delete all of the feature points 3e set for people by operating only one feature point 3e.

[0051] In step S108 of FIG. 4, it is monitored whether the feature point setting operation for the current frame is terminated by a predetermined user operation or the like. If it is determined that the operation has been terminated, the control unit 71 determines in step S109 whether the feature point setting process for another frame in the video is to be continued. If the process is to be continued, the process returns to the frame setting in step S101. This allows the user to select another frame and perform a similar feature point setting operation. For example, when performing match move on a single video, it is desirable to set feature points in several frames in response to changes in the subject caused by camera movement, etc. Therefore, the user will perform the same feature point setting operation on several frames.

[0052] If the user performs an operation to end the feature point setting operation for the video, the control unit 71 ends the processing of Fig. 4 from step S109. According to the processing described above with reference to Figs. 4 and 5, the user can collectively delete multiple feature points 3 grouped by object as a subject. For example, a person is a moving subject, so it is not desirable to use them as feature points 3 for camera tracking. In such a case, instead of deleting many feature points 3e one by one, it is possible to delete them all at once with a single operation. This makes the operation extremely efficient.

[0053] In the above example, "Delete one point" and "Delete all" can be selected in the dialog 6, but an example where "Delete one point", "Delete by thinning out", or "Delete all" can be selected is also conceivable, as shown in Fig. 9. Thinning out deletion is an operation of deleting feature points 3 in the same group by thinning out the number of feature points, rather than deleting all of them. For example, it is an operation of halving the number of feature points in a group.

[0054] An example of processing that enables thinning and deletion in step S111 in Fig. 4 is shown in Fig. 8. In Fig. 8, the same steps as those in Fig. 5 are assigned the same step numbers to avoid redundant explanation.

[0055] In the case of Fig. 8, the control unit 71 displays the dialog 6A as shown in Fig. 9 in step S120, allowing the user to select "Delete one item," "Delete thinned items," "Delete all," or "Cancel." Then, in steps S121, S122, S150, and S123, the control unit 71 waits for a user operation on the dialog 6A. When the user selects "Delete one item," "Delete all," or "Cancel," the processing of the control unit 71 is the same as that in Fig. 5.

[0056] If the user selects "thin out and delete" in the dialog 6A, the control unit 71 proceeds from step S150 to step S151, where it selects a feature point 3 to be deleted in order to thin out some of all feature points 3 in the same group as the specified feature point 3. Then, in step S152, the control unit 71 deletes the selected feature point 3 from the settings. In step S153, the control unit 71 updates the display so that some feature points 3 in the specified group are erased from the display, and also ends the display of the dialog 6A. This completes the processing in FIG. 8, and the process returns to the monitoring loop of steps S106, S107, and S108 in FIG. 4.

[0057] For example, Fig. 9 shows a state in which the cursor 5 is placed on one of the feature points 3b on the television, a designation operation is performed, and a dialog box 6A is displayed. Suppose the user selects "thin-out deletion." In response, the control unit 71 selects some of the feature points 3b on the television and deletes the selected feature points 3b. The deleted feature points 3b are then erased from the display, as shown in Fig. 10.

[0058] There are various methods for selecting feature points 3 to be deleted within a group. First, there is an example in which a fixed ratio is set for the number of feature points to be thinned out. For example, 50% deletion, 40% deletion, etc. Alternatively, the user may be allowed to select the thinning ratio. Figures 9 and 10 show an example in which five of ten feature points 3b on a television are thinned out, i.e., an example in which 50% deletion is performed.

[0059] Alternatively, the number of feature points 3 to be thinned out may be set to a fixed number, such as 3 or 4. Alternatively, the user may be allowed to select the number of feature points to be thinned out. If the number of feature points to be thinned out is greater than the number of feature points in the group, processing may be added to subtract the number of feature points to be thinned out so that some feature points remain untouched.

[0060] Examples of which feature points 3 to select as thinning targets include the following. First, it is conceivable to select feature points 3 at random. It is also conceivable to preferentially select feature points 3 with low image contrast as deletion targets. Even if feature points 3 are on the same subject, if there is a difference in image contrast, feature points 3 in the lower contrast area are deleted first. It is also conceivable to preferentially select one of the feature points 3 that are close to each other. If multiple feature points 3 are arranged closely together, one of them is left and the others are selected as deletion targets.

[0061] For example, by automatically selecting and deleting the feature points to be thinned out as described above, the number of feature points 3 in one group can be efficiently reduced. For example, when there are too many feature points 3 set on a certain object, the operation of reducing the feature points 3 can be made more efficient.

[0062] Up to this point, the operations of "delete all" and "delete thinned out" have been explained for the feature points 3 grouped in the object mode, but these are also effective in the UI in the spatial position mode, which will be explained next.

[0063] 3. UI in Spatial Position Mode Next, the UI in the spatial position mode will be described.

[0064] In the UI in the spatial position mode, feature points 3 that are closely spaced in the subject space in the image 10 are grouped. Then, the feature points 3 are color-coded for each group, and it is possible to delete all feature points 3 in the same group at once.

[0065] Specifically, when match move software automatically detects feature points 3, points that are close to each other in three-dimensional space are displayed in the same color. For example, the three-dimensional coordinate values ​​of each feature point 3 are used to color-code points, such as "points within a certain distance are displayed in the same color." For camera tracking to perform match move, it is desirable for feature points 3 to be widely dispersed in space to improve accuracy. Therefore, by using color coding in this way, users can prevent performance degradation by being careful not to delete too many feature points of the same color when deleting feature points. Conversely, if a user wants to add feature points, they can improve performance by adding them in areas with few feature points of the same color. Note that while distance information is used in spatial position mode, semantic segmentation is not used.

[0066] 11 shows an example of a feature point setting screen in the spatial position mode. As in FIG. 3, the displayed image 10 is a single frame of a live-action video that is the target of match move. After displaying the single frame of image 10 in this manner, the information processing device 70 accepts user operations to set and adjust feature points.

[0067] In image 10, feature points 3 are shown superimposed with circular marks. In FIG. 11, feature points 3 are shown as "3g," "3h," "3i," "3j," "3k," "3m," "3n," "3o," "3p," and "3q," which are circular marks but of different colors. As before, when feature points 3g to 3q are not particularly distinguished, they are written as "feature points 3." Furthermore, since assigning reference numerals to all feature points would make the diagram complicated, reference numerals are assigned to some of the feature points in each group.

[0068] As can be seen from the figure, in the spatial position mode, the feature points 3 are grouped (color-coded) not by object, but by proximity.

[0069] A specific example of the process will be described with reference to Fig. 12. In step S201 of Fig. 12, the control unit 71 sets a frame to be used as a feature point setting screen. That is, the control unit 71 selects a frame from an image as a live-action video, and sets it as image 10 to be displayed as shown in Fig. 11.

[0070] In step S202, the control unit 71 performs feature point setting, which is a process of automatically setting several feature points 3 in the image 10 based on the image analysis results of the image processing unit 85.

[0071] In step S203, the control unit 71 groups the feature points 3 according to the spatial position determination. It also sets a display color for each group. That is, the control unit 71 groups the feature points 3 so that objects that are close to each other in the three-dimensional space as the subject space are grouped together, and selects a different color for each group. A specific example of the processing in step S203 will be described later.

[0072] In step S105, control unit 71 performs display control to display grouped feature points 3 on display unit 77. As a result, a display such as that shown in Fig. 11 is executed. Since the subsequent steps S106 to S111 are generally similar to those in Fig. 4, detailed description will not be repeated, but differences from the above-described object mode will be described.

[0073] 11, the user can see a large number of feature points 3 that are color-coded according to their spatial positions within the image 10. The control unit 71 then provides a UI that allows the user to add or delete feature points 3 at will, or delete them all at once in groups.

[0074] If the control unit 71 detects an operation to add a feature point 3 by the user, the process proceeds from step S106 to step S110, and the control unit 71 performs processing to handle the addition operation.

[0075] As processing for an addition operation, the control unit 71 sets a new feature point at a position specified by clicking, for example. Furthermore, the control unit 71 determines the distance between the specified position and a nearby feature point 3, and determines whether or not to include the new feature point 3 in the same group as the nearby feature point 3. If the new feature point 3 is to be included in a group, the new feature point 3 is added to the nearby group. On the other hand, if it is determined that the specified position is not close to feature points 3 in other groups, the new feature point 3 is not considered to belong to any existing group, and a new group including only the new feature point 3 is set. A color that is not used in any existing group is set as the display color of the feature point 3.

[0076] As described above, new feature points 3 are added to an existing group or a new group is set depending on the position determination, and are displayed on the image 10 in a color corresponding to the group.

[0077] If the control unit 71 detects a user operation to specify feature point 3, the process proceeds from step S107 to step S111, where the control unit 71 performs processing for the adding operation. In this case, the control unit 71 performs the processing shown in FIG. 5 or 8 described above. For example, the control unit 71 displays a dialog box 6 or 6A, and allows the user to select an operation. Depending on the user's selection, the control unit 71 executes one feature point deletion, all feature points in a group deletion, or thinning deletion within a group.

[0078] Therefore, even in the spatial position mode, the user can arbitrarily select to delete only one feature point 3, delete all feature points, or thin out and delete feature points 3 within the same group, i.e., feature points displayed in the same color. This is convenient for, for example, collectively deleting multiple feature points 3 set in a blurred area in the image, i.e., multiple feature points 3 grouped together because they are close to each other. The display in spatial position mode also makes it easier to see how closely together feature points 3 are grouped, and thinning out and deleting is a convenient operation when you want to reduce the number of multiple feature points 3 that are closely grouped.

[0079] In this spatial position mode, in step S203 of FIG. 12, feature points 3 that are close to each other in three-dimensional space are grouped together, and two specific examples of this will be described.

[0080] An example of the first grouping process is shown in Fig. 13. In step S301, the control unit 71 calculates the value d used as a reference value for the distance for determining proximity within a space.

[0081] For example, the three-dimensional coordinates of the n feature points set in step S202 in Fig. 2 are (x0, y0, z0), (x1, y1, z1), (x2, y2, z2), ... (xn-1, yn-1, zn-1). In this case, the maximum value from x0 to xn-1 is xmax, the minimum value is xmin, the maximum value from y0 to yn-1 is ymax, the minimum value is ymin, and the maximum value from z0 to zn-1 is zmax, the minimum value is zmin.

[0082] Here, the largest value among the three values ​​xmax-xmin, ymax-ymin, and zmax-zmin is defined as diff_max. In other words, diff_max means the maximum value of the variation of the feature point 3 in the x, y, and z directions in the three-dimensional space.

[0083] Then, using an arbitrary value m, the d value, which is the reference value described above, is calculated as follows: d=diff_max / m

[0084] The value m can be a parameter determined by the user, but a default value can also be determined in software.

[0085] For example, after calculating the d value as described above, the control unit 71 sets all n feature points 3 to white in step S302. Then, the processes of steps S303 and S304 are repeated until it is determined in step S305 that there are no more white feature points 3.

[0086] In step S303, the control unit 71 initially randomly selects one feature point 3 from among n feature points 3 (white feature points 3). In step S304, the control unit 71 sets all white feature points 3 that exist within a space within a radius d in the subject space from the selected feature point 3 to the same color. In this case, the color selected is a color other than white that has not yet been used for other feature points 3.

[0087] In step S305, the control unit 71 checks whether any white feature points 3 remain, and if so, repeats steps S303 and S304. In step S303, one of the remaining white feature points 3 is randomly selected. In step S304, the control unit 71 sets a white feature point 3 that exists within a radius d from the selected feature point 3 to a color that has not yet been used. Note that there may be feature points 3 within the radius d that are no longer white, but such feature points 3 are excluded from the target, and only feature points that are still white at the present time are targeted to set a new color.

[0088] By repeating the above process, at some point there will be no more white feature points 3. When there are no more white feature points 3, the control unit 71 proceeds from step S305 to step S306, and sets feature points 3 of the same color as a group. This makes it possible to group adjacent feature points 3 based on the spherical range of radius = d.

[0089] An example of the second grouping process is shown in Fig. 14. In step S310, the control unit 71 calculates the value d used as a reference value for the distance for determining proximity within a space. This is the same as step S301 in Fig. 13, and for example, d = diff_max / m.

[0090] Once the d value has been calculated, in step S311 the control unit 71 generates a lattice point space using the d value. FIG. 15 shows the lattice point space. Consider setting a point (id, jd, kd) in three-dimensional space using three integers i, j, and k. Here, if the three integer values ​​i, j, and k are changed to various values, the collection of points (id, jd, kd) becomes a lattice point in three-dimensional space. The dots in FIG. 15 are lattice points. Then, if adjacent lattice points are connected with lines, many cubes with sides of d value are created, as shown in FIG. 15.

[0091] This lattice point space is fitted to the three-dimensional space of the subject, and the feature points 3 present within each cube are grouped together, with a different color assigned to each group. In other words, the feature points 3 contained within each cube are grouped. This makes it possible to group adjacent feature points 3 based on the cubic range of side = d. By performing the above first or second grouping process, a display such as that shown in FIG. 11 becomes possible in step S105 of FIG. 12.

[0092] Next, an example will be described in which feature points 3 that exist on the same plane in three-dimensional space are presented, also in the spatial position mode. Fig. 16 shows an example in which feature points 3 are displayed in the spatial position mode, as in Fig. 11, and plane information 4 is also displayed. This plane information 4 is displayed so as to surround multiple feature points 3 that form the same plane in three-dimensional space, thereby presenting to the user that these multiple feature points 3 exist on the same plane.

[0093] Note that the term "coplanar" is not limited to a plane directly facing the camera, but may refer to any plane of any orientation as long as it constitutes a plane. Furthermore, when two or three feature points 3 are considered, a plane that includes those two or three feature points 3 is necessarily constituted. Such planes are excluded, and when there is a plane that includes a predetermined number of feature points 3 (e.g., five or more, ten or more), which is greater than three, plane information 4 is displayed for those feature points 3.

[0094] A specific example of the process is shown in Fig. 17. Note that the same processes as those in Fig. 12 are given the same step numbers and the explanation thereof will be omitted.

[0095] 17, step S204 is added after step S203. In step S204, the control unit 71 performs a coplanar determination for the set feature points 3. For example, the control unit 71 determines whether a predetermined number or more of feature points 3, such as five or more, ten or more, are on the same plane, based on the three-dimensional coordinate values ​​of each feature point 3.

[0096] When the control unit 71 performs display control in step S105, if there are multiple feature points 3 determined to be on the same plane, the control unit 71 displays plane information 4 as shown in Fig. 16. This allows the user to know when multiple feature points 3 are on the same plane.

[0097] If feature points 3 are concentrated on the same plane when performing a match move, performance will decrease. Therefore, plane information 4 is displayed so that the user can add or delete feature points 3 so that they are not concentrated on the same plane. This display function can improve the efficiency of adding and deleting feature points 3.

[0098] The process for handling the addition operation of the feature point 3 in step S110 of FIG. 17 is as follows. As the process for handling the addition operation, the control unit 71 sets a new feature point at a position specified by clicking or the like, for example. As in the case of FIG. 12 , the control unit 71 determines the distance between the position specified by clicking or the like and a nearby feature point 3, and determines whether to include the new feature point 3 in the same group as the nearby feature point 3. If the new feature point 3 is to be included in a group, the new feature point 3 is added to the nearby group. On the other hand, if it is determined that the specified position is not close to feature points 3 in other groups, the new feature point 3 is not considered to belong to any existing group, and a new group including only the new feature point 3 is set. A color that is not used for any existing group is set as the display color of the feature point 3. In addition to the above, the control unit 71 performs the same co-plane determination as in step S204, including the new feature point 3, and determines whether or not a predetermined number or more of feature points 3 are on the same plane as a result of the addition.

[0099] The control unit 71 then adds the new feature points 3 to an existing group or sets a new group, and displays them in a color corresponding to the group on the image 10. If the addition results in more than a predetermined number of feature points 3 constituting the same plane, plane information 4 is displayed for those feature points 3. This allows the user to recognize that the added feature points 3 have caused feature points to be concentrated on the same plane.

[0100] In step S111 of Fig. 17, the control unit 71 performs the same process as step S111 of Fig. 12, and also performs a co-plane determination. That is, if the state in which a predetermined number or more of feature points 3 are on the same plane is resolved by deleting one or more feature points 3, the control unit 71 also performs a process of terminating the display of the plane information 4. Terminating the display of the plane information 4 allows the user to recognize that the state in which feature points 3 are concentrated on the same plane has been resolved.

[0101] 4. Mode Switching Up to this point, we have explained the UI for the object mode and the spatial position mode, but each mode has its own advantages, so it is desirable to allow the user to switch modes at will. For example, an operation button for switching modes may be displayed, and the control unit 71 may perform the processing of FIG. 18 during the processing of FIG. 4 or FIG. 12 (or FIG. 17).

[0102] For example, if a mode switching operation is detected while feature point 3 is being displayed in the spatial position mode by the processing in Fig. 12 (or Fig. 17), the control unit 71 proceeds from step S401 to step S402 in Fig. 18, and since the current mode is the spatial position mode, it determines that this is an instruction to switch to the object mode, and proceeds to step S103 in Fig. 4. As a result, the screen display is switched to feature point display in the object mode.

[0103] Also, for example, if a mode switching operation is detected while feature point 3 is being displayed in object mode by the processing in Fig. 4, the control unit 71 determines that the current mode is object mode and therefore that the instruction to switch to spatial position mode has been issued. That is, the control unit 71 proceeds from step S401 to step S402 in Fig. 18, passes through step S402, and proceeds to step S403, determines that the instruction to switch to spatial position mode has been issued, and proceeds to step S203 in Fig. 12 (or Fig. 17). This switches the screen display to feature point display in spatial position mode.

[0104] It is best to use Object Mode and Spatial Position Mode as follows: Object Mode is useful for deleting feature points on a moving subject all at once. If you select feature points on a moving subject and perform match move, the accuracy will decrease. Therefore, if feature points on a moving subject such as a person or animal are selected, Object Mode makes it possible to delete the feature points of that moving subject all at once.

[0105] On the other hand, the spatial position mode is a UI that aims to improve accuracy by using distance information. If the feature points 3 are clustered in one place or on the same plane, the accuracy of the match move will decrease. Therefore, this mode is suitable for avoiding the selection of such feature points 3.

[0106] From the above, for example, it is possible to first use the object mode to delete all feature points 3 set for a moving subject at once, and then use the spatial position mode to add or delete feature points 3 so as to avoid positional concentration of feature points 3 or concentration on the same plane.

[0107] 5. Summary and Modifications The above embodiment provides the following effects.

[0108] An information processing device 70 according to an embodiment includes a control unit 71 that performs processing for grouping multiple feature points 3 on a captured image used for motion detection of the imaging device and displaying each group in a different display mode, and processing for collectively deleting multiple feature points of the same group in response to a user operation. It is desirable to set a large number of feature points on an image, but deleting them can be time-consuming. By providing a UI that groups feature points and enables their collective deletion, the user's selection of feature points can be made significantly more efficient.

[0109] In the embodiment, the feature points 3 displayed by the control unit 71 are used to detect the movement of the imaging device for match move, which combines another image with an image captured by the imaging device. In the process of working on match move, the user's operation for setting the feature points 3 can be made more efficient, which can contribute to improving the accuracy of match move.

[0110] In the embodiment, the control unit 71 displays each feature point 3 in a different color for each group. By unifying multiple feature points in the same group with a certain color and setting different colors for each group, the grouped state can be presented to the user in an extremely easy-to-understand manner. Note that, as a different aspect, other than color, the feature points for each group can be displayed with different brightness, different shapes, different patterns, or different sizes.

[0111] In the embodiment, an example has been given in which the control unit 71 performs object detection on a subject appearing in a captured image and performs processing to group multiple feature points set on the detected object. That is, in object mode, image analysis processing, such as semantic segmentation, is performed to detect objects in the image, and it is determined which object each feature point is set on. Then, feature points on the same object are grouped together. This enables extremely efficient operations when it is desired to delete all feature points on a certain object.

[0112] In the embodiment, an example has been given in which the control unit 71 determines the position in the subject space of feature points set in a captured image and performs processing to group the feature points based on the position of each feature point. That is, in the spatial position mode, by grouping each feature point 3 according to its position in the subject space, it becomes possible to group, for example, feature points 3 that are close to each other. This makes it easier to check where feature points 3 are concentrated, and also provides a useful UI when you want to delete them all at once or reduce their number. As a result, it becomes easier to resolve a situation in which feature points 3 are concentrated in physically close positions.

[0113] In the spatial position mode of the embodiment, an example has been given in which the control unit 71 calculates the value d as a reference value for grouping, and performs grouping of feature points 3 using a space with the value d as its radius. For example, as in the example of Fig. 13, the value d serving as a reference value is calculated based on the distance between each feature point 3, and grouping is performed considering a space with the value d as its radius. This makes it possible to group feature points that are at an appropriate distance apart into one group according to the variation of the feature points 3. In particular, by calculating the value d based on the value of the variation of the feature points 3 in the three-dimensional subject space, it is possible to perform grouping according to the current variation of the feature points.

[0114] In the spatial position mode of the embodiment, an example has been given in which the control unit 71 calculates a d value as a reference value for grouping, and groups the feature points 3 using a space divided using the d value. For example, as in the examples of Figures 14 and 15, a value d serving as a reference value is calculated based on the distance between each feature point 3, and a cubic space with one side equal to the value d is set, and grouping is performed. Even with this method, it is possible to group feature points 3 that are at an appropriate distance from each other into one group, taking into account variations in the feature points 3.

[0115] In the embodiment, an example has been given in which the control unit 71 performs processing to present a plurality of feature points 3 located on the same plane in the subject space. For example, as shown in Fig. 16 , a plurality of feature points located on the same plane are presented by the plane information 4. Since it is better not to have feature points concentrated on the same plane, displaying the plane information 4 serves as a suitable guide for the user to reset and set more appropriate feature points.

[0116] In the embodiment, an example has been given in which the control unit 71 is capable of switching between an object mode in which it performs object detection on a subject appearing in a captured image and groups a plurality of feature points 3 set on the detected object, and a spatial position mode in which it determines the positions in the subject space of feature points 3 set in the captured image and groups the feature points 3 based on the positions of each feature point 3. For example, as shown in Fig. 18 , the object mode and spatial position mode can be switched in response to a user operation. This allows the user to, for example, collectively delete unnecessary feature points set on a moving subject in the object mode, and then check and adjust spatial variations in the feature points in the spatial position mode.

[0117] In the embodiment, an example has been given in which the control unit 71, for a feature point 3 designated by a user operation, displays a display that allows the user to select between deleting the feature point 3 and deleting all feature points 3 in the same group as the designated feature point 3, and performs the process of deleting the feature point 3 in accordance with the user operation corresponding to the display. When the feature point 3 is designated, for example, the dialog 6 in Fig. 6 is displayed. This allows the user to perform the operation while arbitrarily selecting between deleting one point and deleting all feature points in the group, and an easy-to-understand UI can be provided.

[0118] In the embodiment, an example has been given in which the control unit 71 displays a display that allows the user to select thinning-out deletion from all feature points 3 in the same group as the feature point 3, for the feature point 3 specified by the user operation, and performs deletion processing for the feature point 3 in response to the user operation corresponding to the display. When the feature point 3 is specified, a display that allows the user to select thinning-out deletion is displayed, for example, as in the dialog 6A in Fig. 9. This makes it possible to provide the user with a UI that is convenient for when they want to reduce the number of feature points in a group.

[0119] In the embodiment, an example has been given in which the control unit 71 performs processing to add and set feature points 3 at locations on an image designated by a user operation. The user is allowed to perform an operation to add feature points 3. This enables an operation to optimize feature points while adding, deleting, or deleting feature points 3 all at once.

[0120] In the embodiment, it has been described that the control unit 71 performs processing to incorporate the additionally set feature points 3 into an existing group or to set a new group in response to a user operation. That is, in the processing for handling the addition operation in step S110 in Figures 4, 12, and 17, if the new feature points 3 correspond to an existing group, they are incorporated, and if they do not correspond, a new group is set. This makes it possible to apply operations such as batch deletion and thinning deletion to the added feature points 3.

[0121] The technology disclosed herein is not limited to feature point setting operations for match move, but can also be applied to feature point setting operations when performing camera tracking from feature points in an image, even for purposes other than match move.

[0122] The program according to the embodiment is a program that causes, for example, a CPU, a DSP (digital signal processor), an AI processor, or the like, or an information processing device 70 that includes these, to execute processes such as those shown in Figures 4, 5, 8, 12, 13, 14, 17, and 18. That is, the program according to the embodiment is a program that causes the information processing device to execute a process of grouping a plurality of feature points 3 on a captured image used for motion detection of the imaging device and displaying each group in a different display mode, and a process of collectively deleting a plurality of feature points 3 in the same group in response to a user operation.

[0123] By using such a program, the information processing device 70 according to the embodiment can be realized in, for example, a computer device, a mobile terminal device, or other device capable of executing information processing.

[0124] Such a program can be pre-recorded on a hard disk drive (HDD) as a recording medium built into a computer or other device, or on a ROM within a microcomputer having a CPU. Alternatively, the program can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disc, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called packaged software. Furthermore, such a program can be installed on a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0125] Furthermore, such a program is suitable for widely providing the information processing device 70 of the embodiment. For example, by downloading the program to a mobile terminal device such as a smartphone or tablet, an imaging device, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), or the like, these devices can function as the information processing device 70 of the present disclosure.

[0126] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0127] The present technology may also be configured as follows. (1) An information processing device including a control unit that performs the following processes: grouping a plurality of feature points on a captured image used for motion detection of an imaging device and displaying each group in a different display mode; and collectively deleting a plurality of feature points in the same group in response to a user operation. (2) The information processing device according to (1), wherein the feature points displayed by the control unit are feature points used for motion detection of the imaging device for match move, which combines another image with an actual image captured by the imaging device. (3) The information processing device according to (1) or (2), wherein the control unit displays each feature point in a different color for each group. (4) The information processing device according to any of (1) to (3), wherein the control unit performs object detection for a subject appearing in the captured image and performs a process of grouping a plurality of feature points set on the detected object. (5) The information processing device according to any of (1) to (4), wherein the control unit determines the position in the subject space of feature points set in the captured image and performs a process of grouping the feature points based on the position of each feature point. (6) The information processing device according to (5) above, wherein the control unit calculates a reference value for grouping and groups feature points using a space having a radius equal to the reference value. (7) The information processing device according to (5) above, wherein the control unit calculates a reference value for grouping and groups feature points using a space divided using the reference value. (8) The information processing device according to any of (5) to (7) above, wherein the control unit performs processing to present multiple feature points located on the same plane in a subject space. (9) The information processing device according to any of (1) to (3) above, wherein the control unit is capable of switching between a mode in which object detection is performed on a subject appearing in a captured image and processing to group multiple feature points set on the detected object, and a mode in which positions in the subject space of feature points set in the captured image are determined and processing to group the feature points based on the positions of each feature point.(10) The information processing device according to any one of (1) to (9) above, wherein the control unit executes a display that allows a user to select between deleting a feature point specified by a user operation and deleting all feature points in the same group as the feature point, and performs a feature point deletion process in accordance with the user operation corresponding to the display. (11) The information processing device according to any one of (1) to (10) above, wherein the control unit executes a display that allows a user to select between thinning out and deleting all feature points in the same group as the feature point specified by the user operation, and performs a feature point deletion process in accordance with the user operation corresponding to the display. (12) The information processing device according to any one of (1) to (10) above, wherein the control unit executes a process to add and set a feature point at a location on the image specified by the user operation. (13) The information processing device according to (12), wherein the control unit executes a process to incorporate the added feature point into an existing group or to set a new group in accordance with the user operation. (14) An information processing method in which an information processing device performs a process of grouping multiple feature points on a captured image used for motion detection of an imaging device and displaying each group in a different display mode, and a process of deleting multiple feature points in the same group all at once in response to a user operation. (15) A program that causes an information processing device to perform a process of grouping multiple feature points on a captured image used for motion detection of an imaging device and displaying each group in a different display mode, and a process of deleting multiple feature points in the same group all at once in response to a user operation.

[0128] REFERENCE SIGNS LIST 1 Background image 2 CG image 3 Feature points 4 Plane information 5 Cursor 6, 6A Dialog 10 Image 70 Information processing device 71 Control unit 71a UI processing unit

Claims

1. An information processing device having a control unit that performs the following processes: grouping multiple feature points on a captured image used for motion detection of an imaging device and displaying each group in a different display mode; and deleting multiple feature points in the same group all at once in response to a user operation.

2. The information processing device according to claim 1, wherein the feature points displayed by the control unit are feature points used to detect the movement of the imaging device for match move, which combines another image with an image captured by the imaging device.

3. The information processing device according to claim 1, wherein the control unit displays each feature point in a different color for each group.

4. The information processing device according to claim 1, wherein the control unit performs object detection on a subject appearing in a captured image and performs processing to group multiple feature points set on the detected object.

5. The information processing device according to claim 1, wherein the control unit determines the positions in the subject space of feature points set in the captured image and performs processing to group the feature points based on the positions of each feature point.

6. The information processing device according to claim 5, wherein the control unit calculates a reference value for grouping, and performs grouping of feature points using a space having a radius equal to the reference value.

7. The information processing device according to claim 5, wherein the control unit calculates a reference value for grouping, and performs grouping of feature points using a space divided using the reference value.

8. The information processing device according to claim 5, wherein the control unit performs processing to present a plurality of feature points located on the same plane within the subject space.

9. The information processing device according to claim 1, wherein the control unit is capable of switching between a mode in which object detection is performed on a subject appearing in a captured image and a process of grouping multiple feature points set on the detected object, and a mode in which the position in the subject space of feature points set in the captured image is determined and a process of grouping the feature points based on the position of each feature point.

10. The information processing device according to claim 1, wherein the control unit executes a display that allows the user to select between deleting a feature point designated by a user operation and deleting all feature points in the same group as the designated feature point, and performs a feature point deletion process in accordance with the user operation corresponding to the display.

11. The information processing device according to claim 1, wherein the control unit executes a display that allows the user to select which feature points to thin out and delete from among all feature points in the same group as the feature point specified by user operation, and performs a feature point deletion process in accordance with the user operation corresponding to the display.

12. An information processing device according to claim 1, wherein the control unit performs processing to add and set feature points at locations on an image designated by a user operation.

13. The information processing device according to claim 12, wherein the control unit performs processing to incorporate the feature points added in response to a user operation into an existing group or to set a new group.

14. An information processing method in which an information processing device performs the following process: grouping multiple feature points on a captured image used for motion detection by an imaging device and displaying each group in a different display mode; and collectively deleting multiple feature points in the same group in response to a user operation.

15. A program that causes an information processing device to perform the following processes: grouping multiple feature points on a captured image used for motion detection by an imaging device and displaying each group in a different display mode; and deleting multiple feature points in the same group all at once in response to a user operation.

Images