Input Support System, Input Support Method, and Program

Abstract

An input assistance system (100) comprises a calculation means (110) that calculates a position relating to a feature point from an input image, a display control means (140) that displays the input image and a pointer operated by a user on a screen, a reception means (120) that receives first input from the user, and an identification means (130) that identifies a first position on the basis of the position relating to the feature point calculated by the calculation means (110). The display control means (140) moves the pointer to the first position when the first input is received.

Description

Input support system, input support method, and non-transitory computer-readable medium

[0001] The present disclosure relates to an input support system, an input support method, and a program, and more particularly to an input support system, an input support method, and a program that reduce the burden of input work on a user.

[0002] A commonly known method is to detect feature points from an image of a target object by image processing, then allow the user to check the detection results and make corrections as necessary (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2019-57111

[0004] However, the above-mentioned background art has a problem in that the user has to manually move the pointer, which places a heavy burden on the user. Therefore, one of the objectives to be achieved by the embodiments disclosed in this specification is to provide an input support system, an input support method, and a program that can reduce the burden of pointer movement operations.

[0005] An input support system according to a first aspect of the present disclosure includes: a calculation means for calculating a position related to a feature point from an input image; a display control means for displaying the input image and a pointer operated by a user on a screen; a reception means for receiving a first input from the user; and an identification means for identifying a first position based on the position related to the feature point calculated by the calculation means, wherein the display control means moves the pointer to the first position when the first input is received.

[0006] In an input assistance method according to a second aspect of this disclosure, a computer calculates positions related to feature points from an input image, displays the input image and a pointer operated by a user on a screen, accepts a first input from the user, identifies a first position based on the calculated positions related to the feature points, and moves the pointer to the first position when the first input is accepted.

[0007] A program according to a third aspect of this disclosure causes a computer to execute the following steps: a calculation step of calculating a position related to a feature point from an input image; a display control step of displaying the input image and a pointer operated by a user on a screen; a reception step of receiving a first input from the user; and an identification step of identifying a first position based on the position related to the feature point calculated in the calculation step; and in the display control step, when the first input is received, the pointer is moved to the first position.

[0008] 1 is a block diagram showing an example of the configuration of an input support system of a first embodiment. FIG. 2 is a flowchart showing an example of the operation of the input support system of the first embodiment. FIG. 3 is a block diagram showing an example of the configuration of an input support system of a second embodiment. FIG. 4 is a diagram showing an example of a screen display by the input support system of the second embodiment, in particular a diagram showing an example of an input image and calculated feature points. FIG. 5 is a flowchart showing an example of the operation of the input support system of the second embodiment. FIG. 6 is a diagram showing the hardware configuration of the input support system of the second embodiment. FIG. 7 is a block diagram showing an example of the configuration of an input support system of a third embodiment. FIG. 8 is a diagram showing an example of a screen display by the input support system of the third embodiment, in particular a diagram showing an example of pointer movement. FIG. 9 is a flowchart showing an example of the operation of the input support system of the third embodiment. FIG. 10 is a block diagram showing an example of the configuration of an input support system of a fourth embodiment. FIG. 11 is a diagram showing an example of a screen display by the input support system of the fourth embodiment, in particular a diagram showing an example of an input image and a calculated bounding box. FIG. 12 is a diagram showing an example of a screen display by the input support system of the fourth embodiment, in particular a diagram showing an example of a bounding box display. FIG. 13 is a diagram showing an example of a screen display by the input support system of the fourth embodiment, in particular a diagram showing an example of Fig. 10 is a flowchart showing an example of operation of the input support system of the sixth embodiment. Fig. 11 is a block diagram showing an example of the configuration of the input support system of the seventh embodiment. Fig. 12 is a diagram showing an example of display on a screen by the input support system of the seventh embodiment, and in particular a diagram showing an example of a visual area. Fig. 13 is a flowchart showing an example of operation of the input support system of the seventh embodiment.

[0009] The input support system according to the embodiment described below is a system that supports a user (human) in viewing an input image and determining the positions of feature points and bounding boxes in the input image. Therefore, the positions calculated by the calculation unit of the input support system are used to assist the user in inputting, and are not immediately adopted as the positions of feature points and bounding boxes in the input image.

[0010] [First embodiment] First, a first embodiment will be described. [Configuration description] Fig. 1 is a block diagram showing an example configuration of an input support system according to a first embodiment. As shown in Fig. 1, the input support system 100 includes a calculation unit 110, a reception unit 120, an identification unit 130, and a display control unit 140.

[0011] The calculation unit 110 calculates positions related to feature points from the input image. Specifically, the positions related to feature points refer to the positions of the feature points or the positions of the bounding boxes surrounding the feature points of the target object. The reception unit 120 receives a first input from the user. The first input is a predetermined input that signals the start of a user operation for newly specifying a feature point. This first input may be referred to as a specification start input. Note that the first input also refers to the input described here in other embodiments described below. Specifically, the first input is, for example, an operation for specifying the position of a (previous) feature point or an operation for moving a pointer to the feature point to be specified. The pointer is a mark displayed on the screen for the user to specify a feature point. The identification unit 130 identifies a first position based on the positions related to the feature points calculated by the calculation unit 110. The first position is the calculated position related to the feature point or a position near the calculated position related to the feature point (in other words, a position a predetermined distance away from the calculated position related to the feature point). Note that in other embodiments described below, the "first position" refers to the position described here. Specifically, the first position is, for example, the position of a feature point or a vertex of a bounding box, or a position in the vicinity thereof. The display control unit 140 displays a pointer operated by the user and an input image on the screen. Then, when the receiving unit 120 receives a first input, the display control unit 140 moves the pointer to the first position identified by the identifying unit 130.

[0012] 2 is a flowchart showing the operation of the input support system 100 according to the first embodiment. A series of processes of the input support system 100 will be described with reference to FIG.

[0013] The calculation unit 110 acquires an input image (step S110). The calculation unit 110 calculates positions related to feature points from the acquired input image (step S120). The reception unit 120 determines whether a first input has been received from the user (step S130). If the first input has been received, the process proceeds to step S140; otherwise, the process of step S130 is repeated. If the reception unit 120 has received the first input, the identification unit 130 identifies a first position based on the positions related to the feature points calculated in step S120 (step S140). Then, the display control unit 140 controls the display so that the pointer position is at the first position identified by the identification unit 130 (step S150).

[0014] Note that the timing for performing step S140 is not limited to this, and may be any time from after the position of the feature point is calculated in step S120 until the process of moving the pointer to the first position in step S150 is performed. For example, by specifying the first position in step S140 before the first input is accepted in step S130, the process of specifying the first position can be performed in advance. Therefore, step S150 is performed after step S130, allowing the pointer to be moved more intuitively.

[0015] [Explanation of Effects] In the input support system 100 according to the first embodiment, the calculation unit 110 calculates the position of a feature point in an input image, and when the user provides a first input, the display control unit 140 moves the pointer to the first position identified by the identification unit 130. A user of this input support system 100 can move the pointer to the first position, such as near the feature point to be specified, simply by providing a predetermined first input. This allows the input support system 100 to move the pointer to the first position, such as near the feature point, without the user having to perform a pointer movement operation. As a result, the input support system 100 can reduce the burden of the user's task of inputting feature points, particularly the burden of the movement operation during the input task.

[0016] Second Embodiment Next, a second embodiment will be described.

[0017] [Configuration Description] Fig. 3 is a block diagram showing an example of the configuration of an input support system according to Embodiment 2. As shown in Fig. 3, the input support system 200 includes a calculation unit 210, a reception unit 220, an identification unit 230, a display control unit 240, and a database 250.

[0018] The calculation unit 210 calculates feature points from an input image. For example, when a face image is input to the input support system 200 as an input image so that a user can specify the positions of facial feature points in the image, the calculation unit 210 calculates features present on the surface of a person's face as feature points. Note that features present on the surface of a person's face include features visible on the skin surface, such as eyes, nose, mouth, ears, eyebrows, contours, moles, freckles, tattoos, birthmarks, wrinkles, dimples, scars, warts, bumps, uneven skin, and discolored areas. The method for calculating feature points is not particularly limited. For example, feature extraction techniques such as SIFT (Scale Invariant Feature Transform) and KAZE may be used, or feature points may be extracted using a pre-trained machine learning model. Here, a face image has been described as an example of an input image, but other images may also be used. For example, for object detection, an image acquired by a camera may be input and feature points of a car appearing in the image may be calculated, or cracks or manholes in road conditions may be calculated as feature points. Furthermore, the feature points and input image that the user wishes to specify can be changed as appropriate. In this embodiment, an embodiment using a face image as the input image will be described.

[0019] The receiving unit 220 receives a first input (designation start input) from the user via an input interface such as the input device 202 (described later), for example.

[0020] The identification unit 230 identifies the first position based on the feature point calculated by the calculation unit 210. In this embodiment, specifically, the identification unit 230 identifies the position of the feature point or a position nearby the feature point as the first position. For example, as shown in FIG. 4 , if a feature point 21 is calculated from the input image 20 by the calculation unit 210, the first position is the position of the feature point 21. The first position may also be a position that is a predetermined distance away from the position of the feature point. This prevents the moved pointer 22 from being displayed overlapping the feature point when the pointer 22 is moved to the first position under the control of the display control unit 240 (described later). The predetermined distance between the first position related to a certain feature point a and the position of the feature point a may be set to be shorter than the distance between the first position related to the feature point a and the positions of other feature points other than the feature point a. Preferably, the predetermined distance, i.e., the distance between the first position and the feature point, is set to several pixels. As a result, the pointer 22 that has moved to the first position does not overlap with the feature point, and therefore, operation can be performed with higher visibility.

[0021] The identification unit 230 may determine a feature point for which the user is about to specify a position from among the plurality of feature points 21 calculated by the calculation unit 210, and identify a first position for the determined feature point. This allows the pointer to be automatically moved to a position preferred by the user, further improving user convenience. For example, the identification unit 230 may determine the feature point closest to the position of the pointer at the time the first input is received as the feature point for which the user is about to specify a position. That is, the identification unit 230 may identify the first position based on the position of the pointer and the position of the feature point. The identification unit 230 may also compare data recording the positions of feature points already input by the user with the positions of the feature points calculated by the calculation unit 210, and determine, among the plurality of calculated feature points 21, a feature point whose position differs from the position of the feature point already input by the user by a predetermined value or more as the feature point for which the user is about to specify a position. That is, the identification unit 230 may determine, among the feature points whose positions have been calculated by the calculation unit 210, a feature point within a predetermined distance from the feature point for which the user has not yet specified a position, as a feature point for which the user is about to specify a position. In this regard, the identification unit 230 may search for a point that has been specified by the user that is closest to the feature point, and determine that the user has specified that feature point if the distance between the two points is within a predetermined distance. The above-described process can identify feature points for which the user has not yet specified a position, and determine the first position of such feature points. As another example, the identification unit 230 may calculate an estimated value (described later) for each feature point calculated for the input image currently being processed, based on history data of processing on previously input images, and determine only feature points for which the estimated value is equal to or greater than a predetermined value as feature points for which the user is about to specify a position. Furthermore, for feature points whose estimated values ​​are less than a predetermined value, the position of the feature point calculated by the calculation unit 210 may be used as the position of the feature point in the input image without the user having to specify the position.The above-mentioned history data is data recording, for each feature point in an input image input in the past, the distance between the calculated position of the feature point and the position specified by the user for that feature point (i.e., the difference between the two positions). That is, the history data is data recording, for each feature point, the magnitude of deviation between the position of the feature point calculated by the calculation unit 210 and the position of the feature point specified by the user. The estimated value is an estimated value of the distance between the position of the feature point calculated for the input image currently being processed and the position specified by the user for that feature point (i.e., the difference between the two positions). That is, the estimated value is an estimated value of the magnitude of deviation between the position of the feature point calculated by the calculation unit 210 and the position of the feature point specified by the user. The calculation of the estimated value may be performed, for example, as follows: Using history data for multiple input images input in the past, the average value of the distances between the calculated position of the feature point and the position of the feature point specified by the user may be calculated for each feature point, and this average value may be used as the estimated value. Alternatively, the estimated value may be obtained by machine learning using the history data as training data. For example, if there have been many cases in the past where the position of the feature point of the outer corner of the eye, among the positions of the facial feature points calculated by calculation unit 210, has been significantly modified by the user, the above-mentioned estimated value for the feature point of the outer corner of the eye will exceed the above-mentioned predetermined value (i.e., threshold value). In this way, for each feature point whose position is calculated by calculation unit 210, identification unit 230 may calculate an estimated value of the difference between the position calculated by calculation unit 210 and the position specified by the user, and determine, among the feature points whose positions are calculated by calculation unit 210, a feature point whose estimated value is equal to or greater than the predetermined value as the feature point whose position is about to be specified by the user.

[0022] The display control unit 240 displays a pointer operated by the user and an input image on the screen. The display control unit 240 may also display feature points (the positions of the feature points). All detected feature points may be displayed, or only a certain portion of the feature points may be displayed. Displaying all detected feature points makes it easier for the user to determine where to specify the feature points. Setting which feature points to display and which not to display, such as displaying only a certain portion of the feature points, improves user visibility. For example, the display control unit 240 can inform the user of the feature points to focus on by displaying only the feature points desired by the user, or by repeatedly flashing only the feature points desired by the user between display and non-display while not displaying other feature points. Furthermore, the display control unit 240 moves the pointer to the first position when the receiving unit 220 receives a first input.

[0023] The database 250 stores input images for the calculation unit 210 to calculate feature points. In this embodiment, the stored input images are images including faces, but are not limited to such images. They may also be images from which feature points can be extracted, such as images including automobiles, road conditions, or people. For example, to detect lesions from images of the inside of a body acquired by an endoscopic camera, the input support system 200 may provide the following input support. That is, the calculation unit 210 may calculate the lesions as feature points, and the display control unit 240 may support input by moving a pointer so that a specialist can specify the correct position of the feature points. This makes it possible to create learning data based on expert knowledge and improve detection accuracy. This embodiment can be used for input support when manually assigning meaning to feature points or improving the accuracy of feature points after feature points are detected by image processing, and can also be used for input support in situations where expert confirmation is required for the specification of feature points.

[0024] [Explanation of Operation] Fig. 5 is a flowchart showing the operation of the input support system 200 in the second embodiment. A series of processes of the input support system 200 will be described with reference to Fig. 5 .

[0025] The calculation unit 210 acquires a face image as an input image from the database 250 (step S210). Note that the calculation unit 210 may acquire an image from an external device such as a camera.

[0026] 4, the display control unit 240 displays on the screen the input image 20 acquired by the calculation unit 210 and the pointer 22 operated by the user (step S220). However, the timing for executing step S220 is not limited to this, and step S220 may be executed between the time when the image is acquired by the calculation unit 210 in step S210 and the time when the pointer is moved to the first position in step S270, which will be described later.

[0027] The calculation unit 210 calculates facial feature points from the acquired face image (step S230).

[0028] The display control unit 240 displays the facial feature points calculated by the calculation unit 210 on the screen (step S240). However, the timing at which the display control unit 240 executes step S240 is not limited to this, and it may be executed between step S230 and the end of the flow. Alternatively, step S240 may not be executed, and the calculated feature points may not be displayed on the screen. By not displaying the feature points, operability is ensured while allowing the user to input feature points based solely on their own knowledge. In other words, in this case, the user inputs feature points based solely on their own knowledge, without relying on the positions of feature points suggested by the calculation unit 210.

[0029] The reception unit 220 determines whether a first input has been received from the user (step S250). If the first input has not been received, the reception unit 220 waits until the first input is received from the user. If the reception unit 220 has received the first input, the process proceeds to the next step S260.

[0030] When the receiving unit 220 receives the first input, the identifying unit 230 identifies the first position based on the feature points calculated in step S230 (step S260). Multiple feature points may be calculated. Therefore, for such cases, it may be predetermined which feature points to use to identify the first position. For example, if the input image is a face image and it is predetermined that the first position is to be identified using the feature point of the outer corner of the left eye, the identifying unit 230 identifies the position of the feature point of the outer corner of the left eye as the first position from among the feature points calculated in step S230. Furthermore, step S260 may be executed before step S250, i.e., before the first input is received.

[0031] The display control unit 240 controls the display so as to move the position of the pointer to the first position identified by the identification unit 230 (step S270).

[0032] [Explanation of Hardware Configuration] Fig. 6 is a block diagram showing an example of the hardware configuration of the input support system 200. As shown in Fig. 6, the input support system 200 includes an output device 201, an input device 202, a storage device 203, a memory 204, and a processor 205.

[0033] The output device 201 is an output device such as a display that outputs information to the outside. The display may be, for example, a flat panel display such as a liquid crystal display, a plasma display, or an organic electroluminescence (EL) display. The output device 201 displays the input image, feature points, and pointer output by the display control unit 240. The output device 201 receives an instruction from the display control unit 240, which has acquired the first position from the identification unit 230, to move the pointer from the currently displayed position to a first position and output the pointer, and displays the pointer at the first position.

[0034] The input device 202 is a device for a user to input, and is, for example, an input device such as a pointing device or a keyboard. Examples of pointing devices include a mouse and a trackball. An operation input by the input device 202 is, for example, an operation of clicking a mouse button or an operation of moving the mouse, but is not limited to these. For example, this operation may be input of a predetermined key on a keyboard, or may be an operation that is predetermined as an operation for specifying the position of a feature point or an operation for moving a pointer to a feature point to be specified. When this operation for specifying the position of a feature point or an operation for moving a pointer to a feature point to be specified is input by the input device 202, the receiving unit 220 determines that a first input has been received.

[0035] The storage device 203 is a non-volatile storage device such as a hard disk, flash memory, etc. The above-mentioned database 250 and the databases of the embodiments described below are realized by the storage device 203, but these may also be realized by other storage devices such as a server or cloud. Furthermore, the storage device may not only be connected to other hardware via an electric circuit, but may also be capable of communicating information via wireless signals or the Internet, and may be configured as a single device or multiple devices.

[0036] The memory 204 is configured, for example, by a combination of volatile memory and non-volatile memory. The memory 204 is used to store software (computer programs) including one or more instructions to be executed by the processor 205, and data used for various processes of the input support system 200. The memory 204 may include multiple memories.

[0037] The processor 205 reads software (computer programs) from the memory 204 and executes them to perform the processes of the calculation unit 210, the reception unit 220, the identification unit 230, and the display control unit 240. Similarly, in other embodiments, the processor 205 reads software (computer programs) from the memory 204 and executes them to perform the processes of the input support system of each embodiment. The processor 205 may be, for example, a microprocessor, an MPU (Micro Processor Unit), or a CPU (Central Processing Unit). The processor 205 may include multiple processors. In this way, the input support system 200 has the functionality of a computer.

[0038] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0039] The hardware configuration of the input support system 200 described above is the same as that of the input support systems of the other embodiments, unless otherwise specified.

[0040] [Explanation of Effect] The user of the input support system 200 can move the pointer to the vicinity of the feature point to be specified simply by performing a predetermined first input. This reduces the amount of operation required for the user to move the pointer to the position of the feature point, thereby reducing the burden of input work on the user.

[0041] [Third Embodiment] Next, a third embodiment will be described. The third embodiment differs from the second embodiment in that the order of feature points designated by the user is acquired and a second input is accepted. The second input is a predetermined input that instructs returning the position of a moved pointer to a previous position. This second input may also be referred to as a return input. Note that in other embodiments described later, the second input refers to the input described here.

[0042] The following will specifically describe the differences from the second embodiment, and will omit redundant descriptions as appropriate. Note that the first embodiment and the second embodiment described above can be applied to the third embodiment.

[0043] [Configuration Description] Fig. 7 is a block diagram showing an example of the configuration of an input support system according to the third embodiment. As shown in Fig. 7, the input support system 300 includes a calculation unit 310, a reception unit 320, an identification unit 330, a display control unit 340, and a database 350.

[0044] The database 350 stores input images for the calculation unit 310 to calculate feature points. The database 350 may also store a machine learning model for the calculation unit 310 to calculate feature points corresponding to the input image through machine learning. In addition to the input images, the database 350 may also store information on feature points corresponding to the input images. The information on feature points may include the positions of the feature points, the labels of the feature points, the order of the feature points, etc.

[0045] The machine learning model is trained in advance by machine learning such as deep learning using pairs of images and the positions of feature points of objects depicted in the images as training data. In this training data, the positions of the feature points are specified in advance, for example, by an expert in the task of specifying the positions of feature points. Therefore, this machine learning model can calculate the positions of the feature points from the input image. The machine learning model may also be trained by associating the order of the feature points with the training data. In this case, by inputting an input image into the machine learning model, the order of the feature points as well as the positions of the feature points can be calculated.

[0046] The positions of the feature points included in the information on the feature points are the positions of the feature points in the input image. For example, the database 350 stores the positions on a two-dimensional plane of the feature points corresponding to the input image as the information on the feature points.

[0047] A feature point label indicates what the feature point in an image refers to. For example, labels attached to feature points in a face image indicate facial features such as the nose, mouth, and eyes. If the labels are further subdivided, the labels indicate finer facial features such as the left corner of the eye, the right corner of the eye, the left nostril, and the right nostril.

[0048] The order of feature points refers to the order in which the user specifies the positions of multiple feature points. When generating data about a group of feature points included in a single input image based on the user's specification of the positions of the feature points, it may be necessary to generate data in which the positions of the feature points are specified in a predetermined order. In this case, the user needs to specify the positions of the feature points in the predetermined order. For example, in the case of a group of feature points in a facial image, it may be necessary to generate data in which the positions of the feature points are specified in the order of a predetermined feature, for example, the outer corner of the left eye, the left pupil, the inner corner of the left eye, and the outer corner of the right eye. In such a case, the feature point corresponding to the outer corner of the left eye is the first feature point, the feature point corresponding to the left pupil is the second feature point, the feature point corresponding to the inner corner of the left eye is the third feature point, and the feature point corresponding to the outer corner of the right eye is the fourth feature point, etc.

[0049] The calculation unit 310 may calculate feature points and the order of the feature points, which is the order in which the user inputs the feature points, from the input image. For example, when a face image is input as the input image, the calculation unit 310 calculates features present on the surface of a person's face as feature points, detects which facial features the feature points correspond to, and assigns an order to the detected feature points based on a predetermined order of the features. In this way, the order of the feature points is calculated. Here, a face image has been described as an example of an input image, but as described in the second embodiment, the calculation can also be applied to other images.

[0050] Furthermore, the calculation unit 310 may calculate the feature points and the order of the feature points as follows: The calculation unit 310 reads out, from the database 350, a machine learning model that outputs the feature points and the order of the feature points, and inputs the input image to the machine learning model, thereby calculating the feature points as well as the order in which the user should input the feature points.

[0051] Furthermore, the calculation unit 310 may calculate the feature points and the order of the feature points as follows. When the calculation unit 310 reads out information about the feature points from the database 350, the calculation unit 310 can calculate the feature points and the order of the feature points by referring to the information about the feature points. For example, if the information about the feature points includes a correspondence between the positions of the feature points and the labels of the feature points, the calculation unit 310 can calculate the order in which the user should input the feature points by comparing the parts indicated by the labels of the feature points at each position with a predetermined order of the parts.

[0052] The identification unit 330 identifies first positions related to the feature points. In this embodiment, the first positions related to the feature points are identified based on the positions and order of the feature points. The identification unit 330 acquires the positions and order of the feature points from the calculation unit 310 or the database 350. For example, if the order of feature points calculated by the calculation unit 310 from the input image 30 as shown in FIG. 8 is from the feature point 31a at the outer corner of the right eye to the feature point 31b at the inner corner of the right eye, the first first position is the position of the feature point 31a at the outer corner of the right eye. Thereafter, when the reception unit 320 further receives a first input (a designation start input), the next first position is the position of the feature point 31b at the inner corner of the right eye. The identification unit 330 may identify one first position each time the reception unit 320 receives a first input, or may identify multiple first positions when one first input is received. The timing at which the identification unit 330 identifies the first position is not limited to this, and may be any time after the feature points and the order of the feature points are calculated by the calculation unit 310. Furthermore, when the positions and order of the feature points are stored in the database 350 in association with the input image as information on the feature points, the identification unit 330 may acquire the positions and order of the feature points from the database 350 and identify the first position related to the feature points.

[0053] The display control unit 340 moves the pointer to the first position when the receiving unit 320 receives the first input. Therefore, when the first input is received, the display control unit 340 moves the pointer 32 to the position of feature point 31a, which is the first position, as shown in FIG. 8 . When the receiving unit 320 receives the first input again, the identification unit 330 updates the first position to the position of feature point 31b so that the user designates the next feature point, the feature point at the inner corner of the right eye. The display control unit 340 then moves the pointer to the position of feature point 31b so that the pointer moves to the updated first position. Note that the first input does not necessarily have to be a uniform operation. For example, first, an operation to move the pointer may be received as the first input, and the pointer may be moved to the position of the first feature point. Next, an operation to designate that feature point may be received as the first input, and the pointer may be moved to the second feature point. 8, an operation to move the pointer is received as the first input, so that the pointer 32 moves to the position of the feature point 31a. Then, an operation to specify the position of the feature point 31a is received as the first input, so that the pointer 32 moves to the position of the feature point 31b.

[0054] Furthermore, the identification unit 330 may identify a second position related to the feature point. The first position is identified so as to transition in the order in which the user inputs the feature points, whereas the second position related to the feature points is identified so as to transition in the reverse order to the order in which the user inputs the feature points. Note that in other embodiments described later as well, the second position refers to the position described here.

[0055] The receiving unit 320 may receive a second input in addition to the functions of the receiving unit 220 of the second embodiment. While the first input (designation start input) is an operation for moving the pointer in the order in which the feature points are input, the second input (return input) is an operation for moving the pointer in the reverse order of the order in which the feature points are input. By performing the second input, the pointer can be returned to the feature point to which it has previously moved. The second input is an input different from the operation set as the first input. The second input may be, for example, an operation of clicking a mouse button or an operation of moving the mouse, but is not limited thereto. For example, this operation may be the input of a predetermined key on the keyboard, or any operation predetermined as an operation for moving to the position of a feature point. For example, if the first input is a left click, the pointer position can be reversed to the position of a predetermined feature point by inputting a right click as the second input.

[0056] In addition to the functions of the display control unit 240 of the second embodiment, the display control unit 340 moves the pointer to a second position when the receiving unit 320 receives a second input. Therefore, when the first input is received and the pointer 32 is moved from the feature point 31a to the feature point 31b as shown in FIG. 8 , the identification unit 330 identifies the second position as follows. That is, the identification unit 330 identifies the feature point 31a as the second position so that the pointer returns to the feature point 31a at the outer corner of the right eye, which is in the opposite direction to the order of the features. In this way, the identification unit 330 identifies the second position by tracing back the transition of the identified first position. For example, each time the receiving unit 320 receives a second input, the identification unit 330 identifies the second position by tracing back the transition of the identified first position one by one. The display control unit 340 moves the pointer from the feature point 31b to the feature point 31a so that the pointer moves to the second position. That is, the display control unit 340 returns the position of the pointer. The identification unit 330 and the display control unit 340 can also be described as follows: The identification unit 330 identifies a new first position and further identifies a second position. The second position can also be described as a first position identified before the new first position. When the display control unit 340 receives a second input from the user, it moves the pointer to the second position. More specifically, the identification unit 330 identifies the first position in the order in which the user inputs positions related to the feature points, and identifies the first position immediately before the new first position as the second position in the reverse order to the order used to identify the first position.

[0057] [Explanation of Operation] Next, the operation of the input support system 300 according to the third embodiment will be described with reference to a flowchart. FIG. 9 is a flowchart showing an example of the operation of the input support system 300 according to the third embodiment. As shown in FIG. 9, the flowchart shown here differs from the flowchart shown in FIG. 5 in that steps S310 to S370 are newly executed. Furthermore, steps S210, S220, S240, and S270 in FIG. 9 differ only in that the operations of the calculation unit 210 and the display control unit 240 described in FIG. 5 are executed by the calculation unit 310 and the display control unit 340, and therefore description thereof will be omitted. Below, differences from the flowchart shown in FIG. 5 will be described.

[0058] After the calculation unit 310 acquires an image from the database 350 in step S210, the calculation unit 310 calculates feature points and the order of the feature points designated by the user (steps S310 and S320). In step S310, the calculation unit 310 calculates the feature points using a machine learning model read from the database 350. Note that the feature points may not be calculated using a machine learning method, but may be calculated from the input image, or may be calculated based on feature point information as described above. In step S320, the calculation unit 310 calculates the order of the feature points using the machine learning model read from the database 350. Note that instead of such a calculation method, the order of the feature points may be calculated based on the feature points calculated from the image and the order of the parts, or the order of the feature points may be calculated from feature point information associated with the acquired image. Although step S320 is executed after step S310 in FIG. 9 , these steps may be executed simultaneously.

[0059] The reception unit 320 determines whether a first input has been received from the user (step S330). If the first input and the second input have not been received, the reception unit 320 waits until the first input and the second input are received from the user. If the reception unit 320 has received the first input, the process proceeds to the next step S340.

[0060] The identification unit 330 identifies the first position based on the feature points and the order of the feature points calculated by the calculation unit 310 (step S340). In this way, the identification unit 330 identifies the first position based on the order in which the user inputs the positions related to the feature points. After executing step S340, the flow proceeds to step S270, where the display control unit 340 displays the pointer at the first position, and the flow ends. Note that the timing for performing step S340 is not limited to this, and may be any time after the order of the feature points is calculated in step S320 and before the process of moving the pointer to the first position in S270.

[0061] The reception unit 320 determines whether a second input has been received from the user (step S350). If the reception unit 320 receives the second input, the process proceeds to the next step S360. Therefore, when the reception unit 320 receives the first input or the second input, the reception unit 320 performs an operation of transitioning the process to the corresponding process.

[0062] When the receiving unit 320 receives the second input, the identifying unit 330 identifies the position of the previous feature point as the second position based on the order of the feature points specified by the user (step S360). The timing at which step S360 is performed is not limited to this, and step S360 may be performed at any time after the order of the feature points is calculated in step S320 and before the process of moving the pointer to the second position in step S370 is performed.

[0063] In addition to the operation of the display control unit 240 of the second embodiment, when the receiving unit 320 receives a second input, the display control unit 340 displays the pointer at the second position identified by the identification unit 330 (step S370).

[0064] [Explanation of Effects] The input support system 300 according to the third embodiment can use a first input as a trigger to move a pointer to the vicinity of a feature point to be designated by the user, and can use a predetermined second input as a trigger to return the pointer to the position of the previous feature point by tracing back the transition. This allows the input support system 300 to sequentially move the pointer to the vicinity of the feature point while reducing the amount of operation required by the user to move the pointer to the feature point position. Furthermore, because the input support system 300 can return the pointer to the position of the previous feature point, the user can redo the feature point designation. As a result, the input support system 300 can reduce the burden on the user of the task of inputting feature points. Furthermore, in this embodiment, the calculation unit 310 calculates the order in which the user inputs positions related to the feature points from the input image. This automatically calculates the order, providing a highly convenient system.

[0065] Fourth Embodiment Next, a fourth embodiment will be described. The fourth embodiment differs from the third embodiment in that the calculation unit calculates a bounding box instead of calculating feature points.

[0066] The following will specifically describe the differences from the third embodiment, and will omit redundant descriptions as appropriate. Note that the first, second, and third embodiments can also be applied to the fourth embodiment.

[0067] 10 is a block diagram showing an example of the configuration of the fourth embodiment. As shown in FIG. 10, the input support system 400 includes a calculation unit 410, a reception unit 320, an identification unit 430, a display control unit 440, and a database 350.

[0068] The calculation unit 410 calculates a bounding box from an input image. A bounding box is a polygonal boundary line that surrounds an object, such as a person, face, facial features (eyebrows, eyes, nose, mouth, ears, etc.), or a car, that appears in the image. A bounding box can also be considered a boundary line that surrounds feature points that make up the object. For example, if a face image is input as an input image to specify a bounding box for the eyes, etc., the calculation unit 410 calculates a bounding box 41 that surrounds features present on the surface of the person's face from the input image 40, as shown in FIG. 11 . More specifically, the calculation unit 410 calculates the vertices of the bounding box.

[0069] Here, a facial image has been described as an example of an input image, but other images may also be used. For example, the present embodiment may be implemented in an operation in which an image acquired from an onboard camera mounted in a vehicle is sent to a server or to a device connected to the onboard camera, and then the user specifies road surface defects, cars, people, etc. using bounding boxes. This can assist the user in inputting the bounding boxes. This can also assist in the creation of training data for machine learning. Using a machine learning model trained using this training data, it becomes possible to accurately identify objects such as people and cars using bounding boxes from driving footage and accident footage. This identification result can also be used to assess driving skills and calculate the percentage of fault in a traffic accident. Additionally, images acquired from cameras installed in various environments, such as cameras pointed at roads or inside facilities, may be input into the input support system 400 to assist the user in creating training data used to learn the facial and body movements of cars and passersby appearing in the images. This embodiment supports the operation of specifying a whole person or parts such as the face or body using a bounding box from an image acquired from a camera, making it easier to create training data for machine learning. A model trained using this training data may be used to track people in video captured by a camera using a bounding box. Based on feature points within the bounding box, it becomes possible to identify people and detect suspicious behavior.

[0070] The bounding box and input image that the user wants to specify can be changed as needed. Furthermore, the calculation unit 410 may calculate the order of bounding boxes specified by the user, similar to the calculation of the order of feature points in the third embodiment. This embodiment uses a facial image as the input image, and will be described as an embodiment that supports the specification of rectangular bounding boxes for each facial feature.

[0071] The bounding box can be calculated using the same calculation method as that used to calculate feature points using the machine learning model described in the third embodiment. However, while pairs of an image and the positions of feature points related to an object are used as training data for learning the machine learning model described in the third embodiment, the bounding box can be calculated using, for example, pairs of an image and a bounding box related to the object as training data. Furthermore, the bounding box may be calculated based on the positional relationship between feature points identified from the feature points calculated using the method described in the third embodiment.

[0072] The identification unit 430 identifies a vertex of the bounding box calculated by the calculation unit 410 as the first position. In this embodiment, a rectangular bounding box is specified, so at least two vertices are required to specify one bounding box. Therefore, for example, to specify the bounding box of the left eye, the user can specify the upper left vertex of the bounding box and then the lower right vertex of the bounding box. That is, in this case, the identification unit 430 identifies the upper left vertex of the bounding box calculated by the calculation unit 410 as the first position, and then identifies the lower right vertex of this bounding box as the next first position. Alternatively, the identification unit 430 may identify the first and second positions by having the user specify each vertex according to the shape of the bounding box to be specified. Furthermore, if the order of the bounding boxes has been calculated by the calculation unit 410, the first and second positions can be identified based on the vertices and order of the bounding boxes. In this way, the identification unit 430 may identify the first position or the second position based on the order in which the user inputs the positions related to the feature points.

[0073] The display control unit 440 moves the pointer to the first position when the receiving unit 320 receives a first input (designation start input). Therefore, each time the display control unit 440 receives a first input, it moves the pointer 42 sequentially to the vertices 41a, 41b, 41c, and 41d of the bounding boxes calculated by the calculation unit 410, as shown in FIG. 12 . This allows the bounding boxes of each facial feature to be specified in order, such as the bounding box for the right eye and the bounding box for the left eye. Specifically, when the first input is first received, the pointer 42 moves to the position of the upper left vertex 41a of the bounding box, which is the first position. When the receiving unit 320 receives the first input again, the identification unit 430 updates the first position to the lower right vertex 41b of the bounding box to allow the user to specify the position of the next vertex of the bounding box to be specified. Accordingly, the display control unit 440 moves the pointer to the position of the vertex 41b so that the pointer moves to the updated first position. Here, if a second input (return input) is made after the pointer has been moved to the vertex 41b of the bounding box, the identification unit 430 identifies the vertex 41a of the bounding box as the second position, and the display control unit 440 can move the pointer back to the second position.

[0074] [Explanation of Operation] Next, the operation of the input support system 400 according to the fourth embodiment will be described with reference to a flowchart. FIG. 13 is a flowchart showing an example of the operation of the input support system 400 according to the fourth embodiment. As shown in FIG. 13, the flowchart shown here differs from the flowchart shown in FIG. 9 in that steps S410 to S470 are newly executed. Furthermore, steps S210 and S220 in FIG. 13 differ only in that the operations of the calculation unit 310 and the display control unit 340 described in FIG. 9 are executed by the calculation unit 410 and the display control unit 440, respectively, and therefore their description will be omitted. Furthermore, steps S330 and S350 are the same as those described above, and therefore their description will be omitted. Differences from the flowchart shown in FIG. 9 will be described below.

[0075] After acquiring an image from the database 350 in step S210, the calculation unit 410 calculates the order of bounding boxes and bounding boxes designated by the user (steps S410 and S420). The order of bounding boxes is determined based on the predetermined order of each object for which a bounding box is designated and the predetermined order for designating the vertices of the bounding box for each object. For example, the order of bounding boxes for designating rectangular bounding boxes for each facial feature is calculated as follows: first, the upper left vertex of the bounding box for the right eye, then the lower right vertex of the bounding box for the right eye, then the upper left vertex of the bounding box for the left eye, and then the lower right vertex of the bounding box for the left eye. In this way, the order may be calculated so that the upper left and lower right vertices of the bounding boxes corresponding to each feature can be selected in order.

[0076] The display control unit 440 displays the bounding box calculated by the calculation unit 410 on the screen (step S430). However, the timing at which the display control unit 440 executes step S430 is not limited to this, and it may be executed between step S410 and the end of the flow. Displaying the bounding box assists the user's operation. The bounding box may be displayed with lines of a predetermined density or less, as in bounding box 43a in FIG. 14, or with dashed lines, as in bounding box 43b, so that it is clear that the bounding box is displayed for reference purposes. Alternatively, step S430 may not be executed, and the calculated bounding box may not be displayed on the screen. Not displaying the bounding box ensures operability while allowing the user to input the bounding box based solely on their own knowledge. In other words, in this case, the user inputs the bounding box based solely on their own knowledge, without relying on the position of the bounding box suggested by the calculation unit 410.

[0077] When the receiving unit 320 receives a first input from the user in step S330, the identifying unit 430 identifies the first position based on the bounding boxes and the order of the bounding boxes calculated by the calculating unit 410 (step S440). Note that the timing for performing step S440 is not limited to this and may be any time after the order of the bounding boxes is calculated in step S420 and before the process of moving the pointer to the first position in S450.

[0078] When the receiving unit 320 receives the first input, the display control unit 440 displays the pointer at the first position identified by the identifying unit 430 (step S450).

[0079] When a second input is received from the user in step S350, the identification unit 430 identifies a second position based on the bounding boxes and the order of the bounding boxes calculated by the calculation unit 410 (step S460). The identification unit 430 identifies the second position by tracing back the transition of the identified first position. In this way, the second position related to the bounding box is identified so as to transition in the reverse order of the order in which the user inputs the vertices of the bounding box. The timing for performing step S460 is not limited to this and may be any time after the order of the bounding boxes is calculated in step S420 and before the process of moving the pointer to the second position in S470.

[0080] When the receiving unit 320 receives the second input, the display control unit 440 displays the pointer at the second position identified by the identifying unit 430 (step S470).

[0081] [Explanation of Effects] In the input support system 400 according to the fourth embodiment, the calculation unit 410 calculates the bounding boxes and their order. Then, triggered by a first input or a second input from the user, the display control unit 440 moves the pointer to an appropriate position. This allows the input support system 400 to allow the user to specify bounding boxes in order while reducing the amount of pointer movement required when the user specifies bounding box areas. As a result, the input support system 400 can reduce the burden of input work on the user.

[0082] Fifth Embodiment Next, a fifth embodiment will be described. The fifth embodiment differs from the other embodiments in that it newly includes a warning unit that issues a warning based on a comparison between calculated feature points and feature points specified by a user. Below, differences from the other embodiments will be specifically described, and overlapping descriptions will be omitted as appropriate.

[0083] 15 is a block diagram showing an example of the configuration of an input support system according to the fifth embodiment. As shown in Fig. 15, the input support system 500 includes a calculation unit 310, a reception unit 520, an identification unit 330, a display control unit 540, a database 350, and a warning unit 560.

[0084] The receiving unit 520 receives an operation by the user to specify feature points. As described in the above embodiment, this operation may be received as a first input (specifying start input). Alternatively, the receiving unit 520 may receive an operation to specify a bounding box instead of the operation to specify feature points. Alternatively, the receiving unit 520 may receive the first input separately from the operation by the user to specify feature points. Alternatively, the receiving unit 520 may receive a second input (return input).

[0085] The display control unit 540 performs the same processing as the display control unit of any of the above-described embodiments, but may also superimpose the specified feature point on the image when the receiving unit 520 receives an operation from the user to specify the feature point. This allows the user to confirm where the feature point has been specified.

[0086] The warning unit 560 outputs a warning based on the distance between the position of the feature point calculated by the calculation unit 310 and the position of the feature point specified by the user and accepted by the acceptance unit 520. Specifically, the warning unit 560 determines whether the distance between the position of the feature point calculated by the calculation unit 310 and the position of the feature point accepted by the acceptance unit 520 is greater than or equal to a predetermined distance, and if it determines that the distance is greater than or equal to the predetermined distance, displays a warning to the user on the output device 201. For example, the warning unit 560 outputs a warning when the position of the feature point specified by the user after the pointer is moved to a first position is greater than or equal to a predetermined distance from the position calculated by the calculation unit 310 for the feature point corresponding to the first position. The warning can also be audible from a speaker built into or connected to the input support system. This allows the user to know that their input differs from the position of the feature point calculated by the calculation unit 310. This can indicate the possibility of an erroneous input or that the feature point was incorrectly specified based on the user's own knowledge. A distance greater than or equal to the predetermined distance refers to, for example, a case where the distance between the two points is several tens of pixels. Therefore, the warning allows the user to notice erroneous input, enabling more accurate feature point specification. The user can set this predetermined distance as appropriate; shortening the predetermined distance results in stricter judgment that is more in line with the position of the feature point calculated by the calculation unit 310. Even in a configuration in which a bounding box is specified instead of a feature point, the system can determine whether to issue a warning by comparing the positions of specific vertices of the bounding box calculated by the calculation unit 410 with the positions of specific vertices of the bounding box specified by the user. In this manner, the warning unit 560 issues a warning when the position of the feature point calculated by the calculation unit 310 (the position of the feature point or the bounding box) and the position of the feature point input by the user (the position of the feature point or the bounding box) are separated by more than a predetermined distance. This embodiment describes an embodiment in which an operation to specify a feature point is performed. The input support system according to the embodiment is a system that supports a user (human) in determining the positions of feature points and bounding boxes in an input image by viewing the input image.Therefore, by using the input support system according to the embodiment, it is expected that the user will be able to specify the positions of feature points and bounding boxes more accurately than the positions calculated by the calculation unit. Therefore, the positions specified by the user are generally more appropriate than the positions calculated by the calculation unit. Therefore, this warning does not necessarily indicate an error in the user's specification.

[0087] [Explanation of Operation] Next, the operation of the input support system 500 according to the fifth embodiment will be described with reference to a flowchart. FIG. 16 is a flowchart showing an example of the operation of the input support system 500 according to the fifth embodiment. As shown in FIG. 16, the flowchart shown here differs from the flowchart shown in FIG. 9 in that steps S510 to S540 are newly executed. Also, FIG. 16 omits the processing from steps S210 to S240. The processing of these steps differs only in that the processing previously performed by the reception unit 320 and the display control unit 340 is now performed by the reception unit 520 and the display control unit 540, and therefore illustration and description of these steps are omitted. Below, the differences from the flowchart shown in FIG. 9 will be described.

[0088] After the display control unit 540 moves the pointer to the first position or the second position in step S270 or step S370, the accepting unit 520 accepts an operation to specify a feature point from the user (step S510). If an operation to specify a feature point from the user is accepted, the process proceeds to the next step S520. If an operation to specify a feature point from the user is not accepted, the process waits until an operation is accepted.

[0089] The display control unit 540 is triggered by the reception unit 520 receiving an operation to specify a feature point from the user, and displays the feature point specified by the user superimposed on the image (step S520). The timing for executing step S520 is not limited to this, and step S520 may be executed at any time after step S510.

[0090] The warning unit 560 compares the positions of the feature points calculated by the calculation unit 310 with the positions of the feature points specified by the user and accepted by the acceptance unit 520, and determines whether the distance between the two points is within a predetermined distance (step S530). If the distance is not within the predetermined distance, the process proceeds to step S540. If the distance is within the predetermined distance, the process terminates. If the distance is within the predetermined distance, the warning unit 560 may output a message indicating that the user's input is correct, or may execute step S520 so that the display control unit 540 displays the feature points entered by the user to notify the user that the input is correct. Specifically, if the warning unit 560 determines that the distance between the two points is within the predetermined distance, the warning unit 560 displays a notification to the user on the output device. Alternatively, the warning unit 560 may output a sound indicating that the input is correct from a speaker built into or connected to the input assistance system.

[0091] If the distance between the two points is not within a predetermined distance, the warning unit 560 outputs a warning to the display control unit 540 indicating that the feature points calculated by the calculation unit 310 do not match the feature points input by the user (step S540). By the display control unit 540 outputting the warning on the screen of the output device, the user can be informed that their input may be incorrect. Furthermore, by outputting a message prompting re-input together with the warning in step S540, the user can be prompted to re-input the feature points. After outputting the warning in step S540, the process can transition to S510 and accept re-input. Note that the accepting unit 520 may accept input from the user to ignore the warning, and if such input is accepted, the process may end without transitioning to step S510.

[0092] [Explanation of Hardware Configuration] The hardware configuration of the fifth embodiment is the same as the hardware configuration of the input support system 200 described in the second embodiment, and only the differences in operation will be described below.

[0093] The processor 205 reads out and executes software (computer programs) from the memory 204 to further perform the processing of the warning unit 560 described above.

[0094] [Explanation of Effects] In addition to the embodiments described so far, the fifth embodiment further compares the calculated feature points with the feature points input by the user. This makes it possible to determine erroneous inputs where there is a large difference between the two or operations that indicate an error in the user's knowledge, and to output a warning. This allows the user to know whether their input may be incorrect, enabling more accurate specification of feature points and bounding boxes.

[0095] [Sixth Embodiment] Next, a sixth embodiment will be described. In the sixth embodiment, a configuration in which an identification unit identifies a first position and a second position based on the movement direction of a pointer operated by a user will be specifically described. Below, differences from other embodiments will be specifically described, and overlapping descriptions will be omitted as appropriate.

[0096] 17 is a block diagram showing an example of the configuration of an input support system according to the sixth embodiment. As shown in FIG. 17 , the input support system 600 includes a calculation unit 310, a reception unit 620, an identification unit 630, a display control unit 340, and a database 350.

[0097] The receiving unit 620 detects a user's operation of operating the input device to move a pointer to the position of a feature point to be designated, and receives the direction of the pointer movement as a first input (designation start input). At this time, the conditions for determining whether the user's pointer operation is the first input can be set to one of the following, or a combination of two or more of the following: 1) the pointer is away from the calculated feature point by a predetermined distance or more; 2) another operation indicating that the pointer is the first input is simultaneously performed along with the pointer movement operation; or 3) the acceleration of the pointer movement caused by the user's operation is equal to or greater than a threshold. By employing these conditions, it is possible to determine whether the pointer operation is an operation by the user to move the pointer to the next feature point. The receiving unit 620 may also receive the direction of the pointer movement as a second input (return input) along with a predetermined input instructing the user to return the moved pointer to a previous position.

[0098] The identification unit 630 identifies the first position based on the position of the pointer and the moving direction of the pointer at the time of input of the first input accepted by the accepting unit 620. Specifically, the identification unit 630 searches for a feature point along the moving direction of the pointer accepted as the first input, using the position of the pointer at the time of accepting the first input as the starting point of the search. The identification unit 630 then identifies the position of the searched feature point that is closest to the position of the pointer at the time of accepting the first input as the first position. In this way, the identification unit 630 identifies the first position based on the position of the pointer at the time of the first input, the moving direction of the pointer, and the position of the feature point. By using the first input to identify the first position, the first position can be identified based on an intuitive operation by the user, thereby supporting the user's operation of specifying a feature point with a more intuitive operation.

[0099] [Explanation of Operation] Next, the operation of the input support system 600 according to the sixth embodiment will be described with reference to a flowchart. FIG. 18 is a flowchart showing an example of the operation of the input support system 600 according to the sixth embodiment. As shown in FIG. 18, the flowchart shown here differs from the flowchart shown in FIG. 9 in that steps S610 to S630 are newly executed. Furthermore, the processing from steps S210 to S240 differs only in that the processing previously performed by the receiving unit 320 is now performed by the receiving unit 620, and therefore a description thereof will be omitted. Similarly, the processing from steps S310 to S370 differs only in that the processing previously performed by the receiving unit 320 and the identifying unit 330 is now performed by the receiving unit 620 and the identifying unit 630, and therefore a description thereof will be omitted. Below, the differences from the flowchart shown in FIG. 9 will be described.

[0100] The reception unit 620 determines whether the user has operated the pointer as the first input (step S610), and if it is determined that the first input has been made, the process proceeds to the next step S620.

[0101] The identification unit 630 acquires the position of the pointer at the time when the receiving unit 620 receives the first input (Step S620).

[0102] The identification unit 630 identifies, as a first position, the position of the feature point that is on an extension of the movement direction and that is closest to the current pointer position (the position acquired in step S620) from among the multiple feature points calculated by the calculation unit 310 (step S630). That is, the identification unit 630 uses the current pointer position (the position acquired in step S620) as the search start point and searches for the feature points calculated by the calculation unit 310 along the pointer movement direction. Then, the identification unit 630 identifies, as the first position, the position of the feature point that is closest to the current pointer position among the searched feature points.

[0103] After step S630, the display control unit 340 moves the pointer to the first position identified in step S630 (step S270), and the flow ends.

[0104] [Explanation of Effects] In the sixth embodiment, the movement direction of the pointer is accepted as a first input, and the first input is used to identify the first position, thereby enabling the user to identify the feature point they want to specify through a more intuitive operation, thereby improving operability.

[0105] [Seventh Embodiment] Next, a seventh embodiment will be described. The seventh embodiment differs from the above-described embodiments in that the display control unit further displays a visual area, which is a window in which the pointer and an image around the pointer are enlarged. Below, differences from the other embodiments will be specifically described, and overlapping descriptions will be omitted as appropriate.

[0106] 19 is a block diagram showing an example of the configuration of an input support system according to the seventh embodiment. As shown in FIG. 19 , the input support system 700 includes a calculation unit 310, a reception unit 320, an identification unit 330, a display control unit 740, and a database 350.

[0107] The display control unit 740 performs display control similar to that of the display control unit of the other embodiments described above. However, the display control unit 740 differs from the display control unit of the other embodiments described above in that it displays a visual area that enlarges and displays the pointer and the image around the pointer after the pointer is moved to the first position or the second position. The visual area is, for example, a window displayed around the pointer, such as the upper right of the pointer 72, as shown in FIG. 20 , and is a rectangular-bordered area 73. The area 73 is displayed superimposed on the input image 70. The user can more accurately specify a feature point by operating the pointer while viewing the enlarged image displayed in the area 73. The display control unit 740 may also display the visual area when the pointer position is within a predetermined range from the position of the feature point calculated by the calculation unit 310. For example, the display control unit 740 may be configured to display the visual area when the pointer is within several pixels of the calculated position of the feature point. In this way, the visual area can be displayed even if the user moves the pointer to the feature point themselves without performing the first or second input.

[0108] Furthermore, when the pointer is located within a predetermined range from the feature point calculated by the calculation unit 310, the display control unit 740 may lower the sensitivity of the pointer compared to when the pointer is located outside the predetermined range from the feature point. Pointer sensitivity refers to the ratio of the amount of pointer movement to the amount of user operation. In other words, lowering the pointer sensitivity means reducing the ratio of the amount of pointer movement to the amount of user operation. Lowering the pointer sensitivity improves the accuracy of the user's operation to move the pointer, enabling accurate designation of feature points.

[0109] [Explanation of Operation] Next, the operation of the input support system 700 according to the seventh embodiment will be described with reference to Fig. 21. Fig. 21 is a flowchart showing an example of the operation of the input support system 700 according to the seventh embodiment. As shown in Fig. 21, the difference is that the display control unit 740 processes step S710 of displaying the visual area and step S720 of changing the sensitivity of the pointer in addition to the operation of the display control unit 340, but the other processing is the same as in Fig. 9. Below, the differences from the flowchart shown in Fig. 9 will be described.

[0110] After moving the pointer to the first position in step S270, the display control unit 740 displays a visual area to the upper right of the pointer, which displays an enlarged image of the pointer and the area surrounding the pointer (step S710). The timing for the display control unit 740 to display the visual area may be any time when the position of the pointer and the position of the feature point calculated by the calculation unit 310 are within a predetermined distance. Therefore, for example, the display of the visual area may be executed after displaying the pointer at the second position in step S370. In this way, when the position of the pointer and the position related to the calculated feature point are within a predetermined distance, the display control unit 740 superimposes on the input image an enlarged partial image of the area surrounding the pointer that is included in the input image.

[0111] After moving the pointer to the first position or the second position, if the pointer is present within a predetermined range from the feature point calculated by the calculation unit 310, the display control unit 740 reduces the sensitivity of the pointer to a value lower than the sensitivity when the pointer is present outside the predetermined range from the feature point (step S720).

[0112] In the seventh embodiment, when the position of the pointer and the position of the calculated feature point are within a predetermined distance, the display control unit displays the visual area, which allows the user to specify the feature point more accurately while checking an enlarged image of the pointer and its surroundings.

[0113] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the invention. Furthermore, each embodiment can be combined.

[0114] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes. [Supplementary Note 1] An input assistance system comprising: a calculation means for calculating a position related to a feature point from an input image; a display control means for displaying the input image and a pointer operated by a user on a screen; a reception means for receiving a first input from the user; and an identification means for identifying a first position based on the position related to the feature point calculated by the calculation means, wherein the display control means moves the pointer to the first position when the first input is received. [Supplementary Note 2] The input assistance system according to Supplementary Note 1, wherein the identification means further identifies the first position based on an order in which the user inputs the positions related to the feature points. [Supplementary Note 3] The input assistance system according to Supplementary Note 2, wherein the calculation means further calculates the order from the input image. [Supplementary Note 4] The input assistance system according to Supplementary Note 1, wherein the identification means identifies the first position based on the position of the pointer and the positions related to the feature points. [Supplementary Note 5] The input support system according to Supplementary Note 1, wherein the specifying means specifies the first position based on the position of the pointer at the time of the first input, the moving direction of the pointer, and a position related to the feature point. [Supplementary Note 6] The input support system according to Supplementary Note 1, wherein the specifying means specifies a new first position and further specifies a second position, the second position being the first position specified before the new first position, and the display control means, upon receiving a second input from the user, moves the pointer to the second position. [Supplementary Note 7] The input support system according to Supplementary Note 6, wherein the specifying means specifies the first position in accordance with an order in which the user inputs positions related to feature points, and specifies the first position immediately before the new first position as the second position in accordance with the reverse order used to specify the first position. [Supplementary Note 8] The input support system according to Supplementary Note 1, wherein the specifying means specifies the first position based on feature points whose positions have been calculated by the calculating means and which are within a predetermined distance from the feature points and for which the user has not yet specified a position.[Supplementary Note 9] The input support system according to Supplementary Note 1, wherein the identification means calculates, for each of the feature points whose positions are calculated by the calculation means, an estimated value of a difference between the position calculated by the calculation means and a position specified by the user, and identifies the first position based on feature points, among the feature points whose positions are calculated by the calculation means, for which the estimated value is equal to or greater than a predetermined value. [Supplementary Note 10] The input support system according to any one of Supplements 1 to 9, further comprising: a warning means for issuing a warning when the position related to the feature point calculated by the calculation means and the position related to the feature point input by the user are separated by a predetermined distance or more. [Supplementary Note 11] The input support system according to any one of Supplements 1 to 10, wherein the position related to the feature point is the position of the feature point. [Supplementary Note 12] The input support system according to any one of Supplements 1 to 10, wherein the position related to the feature point is the position of a bounding box that surrounds the feature points that constitute an object. [Supplementary Note 13] The input support system according to Supplementary Note 11, wherein the display control means further displays the positions of the feature points. [Supplementary Note 14] The input support system according to Supplementary Note 12, wherein the display control means further displays the bounding box. [Supplementary Note 15] The input support system according to any one of Supplements 1 to 14, wherein, when the position of the pointer and a position related to the feature point are within a predetermined distance, the display control means displays an enlarged partial image included in the input image around the pointer, superimposed on the input image. [Supplementary Note 16] An input support method, wherein a computer calculates a position related to the feature point from the input image, displays the input image and a pointer operated by a user on a screen, accepts a first input from the user, identifies a first position based on the calculated position related to the feature point, and moves the pointer to the first position when the first input is accepted.[Supplementary Note 17] A non-transitory computer-readable medium storing a program that causes a computer to execute the following steps: a calculation step of calculating positions related to feature points from an input image; a display control step of displaying the input image and a pointer operated by a user on a screen; a reception step of receiving a first input from the user; and an identification step of identifying a first position based on the positions related to the feature points calculated in the calculation step, wherein in the display control step, when the first input is received, the pointer is moved to the first position.

[0115] 20, 30, 40, 70 Input image 21, 31a, 31b Feature points 22, 32, 42, 72 Pointer 41, 43a, 43b Bounding box 41a, 41b Vertex 73 Area 100, 200, 300, 400, 500, 600, 700 Input support system 110, 210, 310, 410 Calculation unit 120, 220, 320, 520, 620 Reception unit 130, 230, 330, 430, 630 Identification unit 140, 240, 340, 440, 540, 740 Display control unit 201 Output device 202 Input device 203 Storage device 204 Memory 205 Processor 250, 350 Database 560 Warning unit

Claims

1. A calculation means for calculating a position related to a feature point from an input image; a display control means for displaying the input image and a pointer operated by a user on a screen; A receiving means for receiving a first input from the user; a specifying means for specifying a first position based on a position related to the feature point calculated by the calculating means; having The display control means moves the pointer to the first position when the first input is received. Input support system.

2. The specifying means further specifies the first position based on the order in which the user inputs positions related to the feature points. The input support system according to claim 1 .

3. The calculation means further calculates the order from an input image. The input support system according to claim 2 .

4. The specifying means specifies the first position based on the position of the pointer and a position related to the feature point. The input support system according to claim 1 .

5. The specifying means specifies the first position based on a position of the pointer at the time of the first input, a moving direction of the pointer, and a position related to the feature point. The input support system according to claim 1 .

6. The identifying means identifies a new first position and further identifies a second position; the second location is the first location identified prior to the new first location; When the display control means receives a second input from the user, the display control means moves the pointer to the second position. The input support system according to claim 1 .

7. The specifying means specifies the first position according to an order in which a user inputs positions related to feature points, and specifies the first position immediately preceding the new first position as the second position in a reverse order to the order used to specify the first position. The input support system according to claim 6 .

8. The specifying means specifies the first position based on a feature point whose position has not yet been specified by the user within a region within a predetermined distance from the feature point, among the feature points whose positions have been calculated by the calculation means. The input support system according to claim 1 .

9. The computer Calculate the positions of feature points from the input image; Displaying the input image and a pointer operated by a user on a screen; Accepting a first input from the user; determining a first position based on the calculated position of the feature point; When the first input is received, the pointer is moved to the first position. Input assistance method.

10. On the computer, A calculation step of calculating a position of a feature point from an input image; a display control step of displaying the input image and a pointer operated by a user on a screen; a receiving step of receiving a first input from the user; a specifying step of specifying a first position based on the position related to the feature point calculated in the calculating step; Run the command, In the display control step, when the first input is received, the pointer is moved to the first position. program.