Input support system, input support method, and program

Abstract

An input assistance system (100) comprises a calculation means (110) that calculates a position relating to a feature point from an input image, a display control means (140) that displays the input image and a pointer operated by a user on a screen, a reception means (120) that receives first input from the user, and an identification means (130) that identifies a first position on the basis of the position relating to the feature point calculated by the calculation means (110). The display control means (140) moves the pointer to the first position when the first input is received.

Description

Technical Field

[0004] , ,

[0005] , , , , , ,

[0001] The present disclosure relates to an input support system, an input support method, and a program, and particularly to an input support system, an input support method, and a program for reducing the burden of a user's input work.

Background Art

[0002] <​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​A means for identifying a first position based on the position of the feature point calculated by the calculation means, It has, When the display control means receives the first input, it moves the pointer to the first position.

[0006] In the input support method relating to the second aspect of this disclosure, Computers The position of feature points is calculated from the input image. The input image and the user's pointer are displayed on the screen. The first input is received from the aforementioned user. Based on the calculated position of the feature point, a first position is identified. When the first input is received, the pointer is moved to the first position.

[0007] The program relating to the third aspect of this disclosure is On the computer, A calculation step that calculates the position of feature points from the input image, A display control step that displays the input image and a pointer operated by the user on the screen, A reception step in which a first input is received from the user, A determination step to identify a first position based on the position of the feature point calculated in the calculation step, Make it run, In the display control step, when the first input is received, the pointer is moved to the first position. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example configuration of the input support system according to the first embodiment. [Figure 2] This flowchart shows an example of the operation of the input support system according to the first embodiment. [Figure 3]It is a block diagram showing a configuration example of an input support system according to a second embodiment. [Figure 4] It is a diagram showing a display example of a screen by the input support system according to the second embodiment, and particularly, it is a diagram showing an example of an input image and calculated feature points. [Figure 5] It is a flowchart showing an operation example of the input support system according to the second embodiment. [Figure 6] It is a diagram showing the hardware configuration of the input support system according to the second embodiment. [Figure 7] It is a block diagram showing a configuration example of an input support system according to a third embodiment. [Figure 8] It is a diagram showing a display example of a screen by the input support system according to the third embodiment, and particularly, it is a diagram showing an example of pointer movement. [Figure 9] It is a flowchart showing an operation example of the input support system according to the third embodiment. [Figure 10] It is a block diagram showing a configuration example of an input support system according to a fourth embodiment. [Figure 11] It is a diagram showing a display example of a screen by the input support system according to the fourth embodiment, and particularly, it is a diagram showing an example of an input image and a calculated bounding box. [Figure 12] It is a diagram showing a display example of a screen by the input support system according to the fourth embodiment, and particularly, it is a diagram showing an example of pointer movement. [Figure 13] It is a flowchart showing an operation example of the input support system according to the fourth embodiment. [Figure 14] It is a diagram showing a display example of a screen by the input support system according to the fourth embodiment, and particularly, it is a diagram showing an example of the display of a bounding box. [Figure 15] It is a block diagram showing a configuration example of an input support system according to a fifth embodiment. [Figure 16] It is a flowchart showing an operation example of the input support system according to the fifth embodiment. [Figure 17] It is a block diagram showing a configuration example of an input support system according to a sixth embodiment. [Figure 18] It is a flowchart showing an operation example of the input support system according to the sixth embodiment. [Figure 19] It is a block diagram showing a configuration example of the input support system according to the seventh embodiment. [Figure 20] It is a diagram showing a display example of a screen by the input support system according to the seventh embodiment, and particularly, it is a diagram showing an example of a visual area. [Figure 21] It is a flowchart showing an operation example of the input support system according to the seventh embodiment.

Embodiments for Carrying Out the Invention

[0009] The input support system according to the embodiments described below is a system that assists a user (human) in looking at an input image and determining the positions of feature points and the position of a bounding box in this input image. Therefore, the positions calculated by the calculation unit of the input support system are used to assist the user's input, and the positions calculated by the calculation unit are not immediately adopted as the positions of the feature points and the position of the bounding box in the input image.

[0010] [First Embodiment] First, the first embodiment will be described. [Description of Configuration] FIG. 1 is a block diagram showing a configuration example of the input support system according to the first embodiment. As shown in FIG. 1, the input support system 100 includes a calculation unit 110, a reception unit 120, a specification unit 130, and a display control unit 140.

[0011] The calculation unit 110 calculates the position of feature points from the input image. Specifically, the position of a feature point refers to the position of the feature point, or the position of the bounding box surrounding the feature points that constitute the object. The reception unit 120 receives a first input from the user. The first input is a predetermined input that signals the start of the user's operation for specifying a new feature point. This first input may also be called a specification start input. In other embodiments described later, the first input refers to the input described here. Specifically, the first input is, for example, an operation to specify the position of the (previous) feature point, or an operation to move the pointer to the feature point to be specified. The pointer is a marker displayed on the screen for the user to specify a feature point. The identification unit 130 identifies a first position based on the position of the feature point calculated by the calculation unit 110. The first position is the calculated position of the feature point, or a position near the calculated position of the feature point (in other words, a position a predetermined distance away from the calculated position of the feature point). In addition, in other embodiments described later, the first position refers to the position described here. Specifically, the first position is, for example, the position of a feature point or the vertex of a bounding box, or a position in its vicinity. The display control unit 140 displays the pointer operated by the user and the input image on the screen. When the reception unit 120 receives the first input, the display control unit 140 moves the pointer to the first position identified by the identification unit 130.

[0012] [Explanation of operation] Figure 2 is a flowchart showing the operation of the input support system 100 in the first embodiment. A series of processes of the input support system 100 will be described with reference to Figure 2.

[0013] The calculation unit 110 acquires the input image (step S110). The calculation unit 110 calculates the position of feature points from the acquired input image (step S120). The reception unit 120 determines whether or not it has received a first input from the user (step S130). If a first input is received, the process proceeds to step S140; otherwise, the process in step S130 is repeated. If the reception unit 120 has received a first input, the identification unit 130 identifies a first position based on the position of feature points calculated in step S120 (step S140). The display control unit 140 then controls the display so that the pointer's position is at the first position identified by the identification unit 130 (step S150).

[0014] Note that the timing of step S140 is not limited to this; it can be performed anytime between the time the position of the feature point is calculated in step S120 and the time the pointer is moved to the first position in step S150. For example, by determining the first position in step S140 before receiving the first input in step S130, the process of determining the first position can be performed in advance. As a result, step S150 is executed after step S130, allowing the pointer to be moved more intuitively.

[0015] [Explanation of effects] In the first embodiment, the input support system 100 calculates the position of feature points in the input image using a calculation unit 110, and when a first input is made by the user, the display control unit 140 moves the pointer to the first position identified by the identification unit 130. With this input support system 100, the user can move the pointer to the first position, such as the vicinity of the feature point to be specified, simply by making a predetermined first input. As a result, the input support system 100 allows the user to move the pointer to the first position, such as the vicinity of the feature point, without having to perform any pointer movement operations. Consequently, the input support system 100 can reduce the burden on the user in the task of inputting feature points, especially the burden of movement operations during the input process.

[0016] [Second Embodiment] Next, a second embodiment will be described.

[0017] [Explanation of the structure] Figure 3 is a block diagram showing an example configuration of the input support system according to the second embodiment. As shown in Figure 3, the input support system 200 includes a calculation unit 210, a reception unit 220, a specification unit 230, a display control unit 240, and a database 250.

[0018] The calculation unit 210 calculates feature points from the input image. For example, if a face image is input to the input support system 200 as an input image so that the user can specify the location of facial feature points in the image, the calculation unit 210 calculates features present on the surface of the person's face as feature points. Features present on the surface of a person's face include, for example, eyes, nose, mouth, ears, eyebrows, contours, moles, freckles, tattoos, birthmarks, wrinkles, dimples, scars, warts, bumps, skin irregularities, and areas of skin discoloration. The method for calculating feature points is not particularly limited. For example, feature extraction techniques such as SIFT (Scale Invariant Feature Transform) or KAZE may be used, or feature points may be extracted using a pre-trained machine learning model. Here, a face image was described as an example of an input image, but other images may also be used. For example, for object detection, an image acquired from a camera may be input, and feature points of cars in the image may be calculated, or cracks in the road surface or manholes may be calculated as feature points. Furthermore, the feature points and input image specified by the user can be changed as appropriate. In this embodiment, an embodiment in which a face image is used as the input image will be described.

[0019] The reception unit 220 receives a first input (specified start input) from the user. The reception unit 220 receives the first input from the user via an input interface, such as the input device 202 described later.

[0020] The identification unit 230 identifies a first position based on the feature points calculated by the calculation unit 210. In this embodiment, specifically, the identification unit 230 identifies the position of the feature point or a position near it as the first position. For example, as shown in Figure 4, if a feature point 21 is calculated by the calculation unit 210 from the input image 20, the first position is the position of feature point 21. Alternatively, the first position may be a position located at a predetermined distance from the feature point. This prevents the pointer 22 from overlapping the feature point when it moves to the first position under the control of the display control unit 240, which will be described later. The predetermined distance between a first position and the position of a feature point a may be set to be shorter than the distance between the first position and the positions of other feature points. Preferably, the predetermined distance, i.e., the distance between the first position and the feature point, is set to a few pixels. As a result, the pointer 22, which has moved to the first position, does not overlap with the feature point, allowing for operation with improved visibility.

[0021] Furthermore, the identification unit 230 may determine from among the multiple feature points 21 calculated by the calculation unit 210 which feature point the user is about to specify, and identify a first position for the determined feature point. This will automatically move the pointer to a position favorable to the user, further improving user convenience. For example, the identification unit 230 may determine the feature point closest to the pointer's position at the time the first input is received as the feature point the user is about to specify. In other words, the identification unit 230 may identify the first position based on the pointer's position and the position of the feature point. Alternatively, the identification unit 230 may compare data recording the positions of feature points already entered by the user with the positions of feature points calculated by the calculation unit 210, and determine among the multiple calculated feature points 21 which feature points are located at a position where the difference between the position of the feature point already entered by the user is greater than or equal to a predetermined value as the feature point the user is about to specify. In other words, the identification unit 230 may determine that feature points whose positions have been calculated by the calculation unit 210, and which are within a predetermined distance from the feature point and for which the user has not yet specified a position, are feature points for which the user is about to specify a position. In this regard, the identification unit 230 may search for the user-specified point closest to the feature point, and if the distance between those two points is within a predetermined distance, it may determine that the user has specified a position for that feature point. Through the above process, feature points for which the user has not yet specified a position can be identified, and a first position can be identified for such feature points. As another example, the identification unit 230 may calculate an estimated value for each feature point calculated for the input image that is currently being processed, based on the processing history data for previously input images, and determine that only feature points for which the estimated value is greater than or equal to a predetermined value are feature points for which the user is about to specify a position. Furthermore, for feature points whose estimated value is less than a predetermined value, the position of the feature point calculated by the calculation unit 210 may be used directly as the position of the feature point in the input image without requiring user specification of the position.The historical data mentioned above is data that records the distance (i.e., the difference between the two positions) between the calculated position of each feature point and the position specified by the user relative to that feature point, for each feature point. In other words, the historical data is data that records the magnitude of the discrepancy between the position of the feature point calculated by the calculation unit 210 and the position of the feature point specified by the user, for each feature point. The estimated value is an estimated value of the distance (i.e., the difference between the two positions) between the position of the calculated feature point for the input image currently being processed and the position specified by the user relative to that feature point. In other words, the estimated value is an estimated value of the magnitude of the discrepancy between the position of the feature point calculated by the calculation unit 210 and the position of the feature point specified by the user. The calculation of the estimated value may be performed, for example, as follows: Using historical data from multiple input images that have been input in the past, the average value of the distance between the calculated position of the feature point and the position of the feature point specified by the user may be calculated for each feature point, and this average value may be used as the estimated value. Alternatively, the estimated value may be obtained by performing machine learning using the historical data as training data. For example, if there have been many cases in the past where the position of the feature point at the outer corner of the eye has been significantly modified by the user among the facial feature points calculated by the calculation unit 210, the above-mentioned estimated value for the feature point at the outer corner of the eye will exceed the predetermined value (i.e., threshold) mentioned above. In this way, the identification unit 230 may calculate an estimated value of the difference between the position calculated by the calculation unit 210 and the position specified by the user for each feature point whose position has been calculated by the calculation unit 210, and may determine that the feature points whose estimated value is greater than or equal to the predetermined value among the feature points whose position has been calculated by the calculation unit 210 are the feature points whose position the user is about to specify.

[0022] The display control unit 240 displays the pointer operated by the user and the input image on the screen. In addition, the display control unit 240 may also display feature points (the location of the feature points). Furthermore, all detected feature points may be displayed, or only some of the feature points may be displayed. Displaying all detected feature points makes it easier for the user to confirm where to specify the feature points. Also, by setting which feature points to display and which not to display, such as displaying only some of the feature points, the user's visibility can be improved. For example, by displaying only the feature points that the user should specify, or by repeatedly flashing the display and hiding only the feature points that the user should specify, while not displaying other feature points, the user can be notified of the feature points that should be focused on. Furthermore, triggered by the reception unit 220 receiving the first input, the display control unit 240 moves the pointer to the first position.

[0023] The database 250 stores input images for the calculation unit 210 to calculate feature points. In this embodiment, the stored input images are images that include faces, but are not limited to faces; they may be images that can extract certain feature points, such as images that include cars, road conditions, or people. For example, in order to detect lesions from images of the inside of the body acquired by an endoscope camera, the input support system 200 may be used to provide the following input support. That is, after the calculation unit 210 calculates the lesions as feature points, the display control unit 240 may move a pointer to assist a specialist in specifying the correct location of the feature points. This makes it possible to create training data that aligns with expert knowledge and to improve detection accuracy. This embodiment can be used to provide input support when manually assigning meaning to feature points or improving the accuracy of feature points after feature point detection by image processing, and it can also be used to provide input support in situations where expert confirmation is required for the specification of feature points.

[0024] [Explanation of operation] Figure 5 is a flowchart showing the operation of the input support system 200 in the second embodiment. A series of processes of the input support system 200 will be described with reference to Figure 5.

[0025] The calculation unit 210 acquires a face image as an input image from the database 250 (step S210). The calculation unit 210 may also acquire images from an external device such as a camera.

[0026] As shown in Figure 4, the display control unit 240 displays the input image 20 acquired by the calculation unit 210 and the pointer 22 operated by the user on the screen (step S220). However, the timing of executing step S220 is not limited to this; it can be executed anytime between the time the calculation unit 210 acquires the image in step S210 and the time the pointer is moved to the first position in step S270, which will be described later.

[0027] The calculation unit 210 calculates facial feature points from the acquired facial image (step S230).

[0028] The display control unit 240 displays the facial feature points calculated by the calculation unit 210 on the screen (step S240). However, the timing at which the display control unit 240 executes step S240 is not limited to this; it can be executed anytime from step S230 until the end of the flow. Alternatively, step S240 may be omitted, and the calculated feature points may not be displayed on the screen. By not displaying the feature points, operability can be ensured while allowing the user to input feature points based solely on their own knowledge. In other words, in this case, the user will input feature points based solely on their own knowledge, without relying on the location of the feature points suggested by the calculation unit 210.

[0029] The reception unit 220 determines whether or not it has received the first input from the user (step S250). If there is no first input, the reception unit 220 waits until it receives the first input from the user. If the reception unit 220 has received the first input, the process transitions to the next step S260.

[0030] When the receiving unit 220 receives the first input, the identification unit 230 identifies the first position based on the feature points calculated in step S230 (step S260). Multiple feature points may be calculated. For this reason, it may be predetermined which feature points to use to identify the first position in such cases. For example, if the input image is a face image and it is predetermined that the feature point at the outer corner of the left eye will be used to identify the first position, the identification unit 230 will identify the position of the feature point at the outer corner of the left eye as the first position from among the feature points calculated in step S230. Step S260 may also be executed before step S250, which is before the first input is received.

[0031] The display control unit 240 controls the display to move the pointer position to the first position identified by the identification unit 230 (step S270).

[0032] [Description of hardware configuration] Figure 6 is a block diagram showing an example of the hardware configuration of the input support system 200. As shown in Figure 6, the input support system 200 includes an output device 201, an input device 202, a storage device 203, memory 204, and a processor 205.

[0033] The output device 201 is an output device such as a display that outputs information to the outside. The display may be a flat panel display such as a liquid crystal display, plasma display, or organic EL (Electro-Luminescence) display. The output device 201 displays the input image, feature points, and pointer output by the display control unit 240. The output device 201 receives a command from the display control unit 240, which has obtained the first position from the identification unit 230, to move the pointer from the currently displayed position to the first position and output it, and displays the pointer at the first position.

[0034] The input device 202 is a device for the user to input data, such as a pointing device or a keyboard. Examples of pointing devices include a mouse or a trackball. The operation input by the input device 202 may be, for example, clicking a mouse button or moving the mouse, but is not limited to these. For example, this operation may be the input of a specific key on the keyboard, or any operation predetermined to specify the position of a feature point, or to move the pointer to a feature point to be specified. When this operation to specify the position of a feature point, or to move the pointer to a feature point to be specified, is input by the input device 202, the reception unit 220 determines that it has received the first input.

[0035] The storage device 203 is a non-volatile storage device such as a hard disk or flash memory. The storage device 203 realizes the database 250 described above and the database of the embodiment described later, but these may also be realized by other storage devices such as servers or clouds. Furthermore, the storage device may not only be connected to other hardware by electrical circuits, but may also communicate information via wireless signals or the internet, and the storage device may be composed of a single device or multiple devices.

[0036] Memory 204 is composed of, for example, a combination of volatile memory and non-volatile memory. Memory 204 is used to store software (computer programs) containing one or more instructions executed by the processor 205, and data used for various processes of the input assistance system 200. Memory 204 may include multiple memory locations.

[0037] The processor 205 reads and executes software (computer programs) from the memory 204 to perform the processing of the calculation unit 210, reception unit 220, identification unit 230, and display control unit 240 described above. Similarly, in other embodiments, the processor 205 reads and executes software (computer programs) from the memory 204 to perform the processing of the input support system in each embodiment. The processor 205 may be, for example, a microprocessor, an MPU (Micro Processor Unit), or a CPU (Central Processing Unit). The processor 205 may include multiple processors. Thus, the input support system 200 possesses the functionality of a computer.

[0038] The program, when loaded into a computer, includes a set of instructions (or software code) for causing the computer to perform one or more functions as described in the embodiments. The program may be stored on a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically or otherwise propagating signals.

[0039] Furthermore, the hardware configuration of the input support system 200 described above is the same in input support systems of other embodiments unless otherwise specified.

[0040] [Explanation of effects] The user of the input support system 200 can move the pointer to the vicinity of the feature point they wish to specify simply by making a predetermined first input. This reduces the amount of operation required for the user to move the pointer to the location of the feature point, thereby reducing the burden on the user's input work.

[0041] [Third Embodiment] Next, a third embodiment will be described. The third embodiment differs from the second embodiment in that it obtains the order of feature points specified by the user and accepts a second input. The second input is a predetermined input that instructs the system to return the position of the moved pointer to its previous position. This second input may also be called a return input. In other embodiments described later, the second input refers to the input described here.

[0042] The differences from the second embodiment will be explained in detail below, with redundant explanations omitted where appropriate. It should be noted that the first embodiment described above can be applied to the third embodiment, as can the second embodiment described above.

[0043] [Explanation of the structure] Figure 7 is a block diagram showing an example configuration of the input support system according to the third embodiment. As shown in Figure 7, the input support system 300 includes a calculation unit 310, a reception unit 320, a specification unit 330, a display control unit 340, and a database 350.

[0044] The database 350 stores input images for the calculation unit 310 to calculate feature points. It may also store a machine learning model for the calculation unit 310 to calculate feature points corresponding to the input images using machine learning. In addition to the input images, it may also store information about the feature points corresponding to the input images. The feature point information may include the position of the feature point, the label of the feature point, the order of the feature points, etc.

[0045] The machine learning model is pre-trained using machine learning methods such as deep learning, with training data consisting of pairs of images and the locations of feature points of objects depicted in those images. In this training data, the locations of the feature points are pre-specified, for example, by an expert in the task of specifying the locations of feature points. Therefore, this machine learning model can calculate the locations of feature points from the input image. Alternatively, the machine learning model may be trained to associate the order of the feature points with the training data. In this case, by inputting the input image into the machine learning model, both the locations and the order of the feature points can be calculated.

[0046] The position of a feature point included in the feature point information refers to the position of the feature point in the input image. For example, database 350 stores the position of the feature point corresponding to the input image on a two-dimensional plane as feature point information.

[0047] Feature point labels indicate what the feature point in an image represents. For example, labels attached to feature points in a face image would indicate facial features such as the nose, mouth, and eyes. If the labels are further subdivided, they would indicate more specific facial features, such as the outer corner of the left eye, the outer corner of the right eye, the left nostril, and the right nostril.

[0048] The order of feature points refers to the order in which feature points are specified when a user specifies the locations of multiple feature points. When generating data for a set of feature points contained in a single input image based on the user's specification of feature point locations, it may be necessary to generate data in which the feature point locations are specified in a predetermined order. In this case, the user must specify the feature point locations in a predetermined order. For example, in the case of a set of feature points in a face image, it may be necessary to generate data in which the feature point locations are specified in a predetermined order of parts, such as the outer corner of the left eye, the left pupil, the inner corner of the left eye, and the outer corner of the right eye. In such a case, the feature point corresponding to the outer corner of the left eye would be the first feature point, the feature point corresponding to the left pupil would be the second feature point, the feature point corresponding to the inner corner of the left eye would be the third feature point, and the feature point corresponding to the outer corner of the right eye would be the fourth feature point, and so on.

[0049] The calculation unit 310 may calculate feature points and the order of the feature points, which is the order in which the user inputs the feature points, from the input image. For example, if a face image is input as the input image, the calculation unit 310 calculates features present on the surface of a person's face as feature points, detects which part of the face each feature point represents, and assigns an order to the detected feature points based on a predetermined order of parts. This calculates the order of the feature points. Here, a face image was described as an example of an input image, but as described in the second embodiment, it can also be applied to other images.

[0050] Furthermore, the calculation unit 310 may calculate feature points and the order of feature points as follows: The calculation unit 310 reads a machine learning model that outputs feature points and the order of feature points from the database 350, and inputs the input image into the machine learning model, thereby calculating not only the feature points but also the order in which the user should input the feature points.

[0051] Furthermore, the calculation unit 310 may calculate feature points and their order as follows. When the calculation unit 310 reads feature point information from the database 350, it can calculate feature points and their order by referring to the feature point information. For example, if the feature point information includes a correspondence between the location of the feature point and the label of the feature point, the calculation unit 310 can calculate the order in which the user should input the feature points by comparing the parts indicated by the labels of the feature points at each location with a predetermined order of parts.

[0052] The identification unit 330 identifies a first position relating to a feature point. In this embodiment, the first position relating to a feature point is identified based on the position of the feature point and the order of the feature points. The identification unit 330 obtains the position of the feature point and the order of the feature points from the calculation unit 310 or the database 350. For example, as shown in Figure 8, if the order of feature points calculated by the calculation unit 310 from the input image 30 is from the feature point 31a at the outer corner of the right eye to the feature point 31b at the inner corner of the right eye, the first position is initially the position of the feature point 31a at the outer corner of the right eye. Then, when the receiving unit 320 receives another first input (specified start input), the next first position is the position of the feature point 31b at the inner corner of the right eye. The identification unit 330 may identify one first position each time the receiving unit 320 receives a first input, or it may identify multiple first positions when it receives one first input. The timing at which the identification unit 330 identifies the first position is not limited to this; it can be done at any time after the calculation unit 310 has calculated the feature points and their order. Furthermore, if the database 350 stores the position and order of the feature points as feature point information linked to the input image, the identification unit 330 may retrieve the position and order of the feature points from the database 350 and then identify the first position related to the feature points.

[0053] The display control unit 340 moves the pointer to the first position when the reception unit 320 receives the first input. Therefore, upon receiving the first input, the pointer 32 is moved to the position of feature point 31a, which is the first position, as shown in Figure 8. When the reception unit 320 receives the first input again, the identification unit 330 updates the first position to the position of feature point 31b so that the user can specify the next feature point, which is the feature point at the inner corner of the right eye. The display control unit 340 then moves the pointer to the position of feature point 31b so that the pointer moves to the updated first position. Note that the first inputs do not all have to be the same operation. For example, first, the operation to move the pointer may be received as the first input, and the pointer may be moved to the position of the first feature point. Next, the operation to specify that feature point may be received as the first input, and the pointer may be moved to the second feature point. To explain using Figure 8 as an example, by receiving an operation to move the pointer as the first input, pointer 32 moves to the position of feature point 31a. Then, by receiving an operation to specify the position of feature point 31a as the first input, pointer 32 moves to the position of feature point 31b.

[0054] Furthermore, the identification unit 330 may identify a second position relating to the feature point. While the first position is identified so as to transition in the order in which the user inputs the feature points, the second position relating to the feature point is identified so as to transition in the reverse order of the order in which the user inputs the feature points. In other embodiments described later, the second position refers to the position described here.

[0055] The reception unit 320 may accept a second input in addition to the functions of the reception unit 220 in the second embodiment. While the first input (specified start input) is an operation to move the pointer in the order in which feature points are entered, the second input (return input) is an operation to move the pointer in the reverse order of the order in which feature points are entered. By performing the second input, the pointer can be returned to the feature points to which it has been moved so far. The second input is a different input from the operation set as the first input. The second input is, for example, an operation to click a mouse button or an operation to move the mouse, but is not limited to these. For example, this operation may be the input of a predetermined key on the keyboard, or any operation predetermined as an operation to move to the position of a feature point. For example, if the first input is a left click, the pointer position can be reversed to the position of a predetermined feature point by inputting a right click as the second input.

[0056] In addition to the functions of the display control unit 240 in the second embodiment, the display control unit 340 moves the pointer to the second position when the reception unit 320 receives the second input, triggered by this. Therefore, if the first input is received and the pointer 32 is moved from feature point 31a to feature point 31b as shown in Figure 8, the identification unit 330 identifies the second position as follows. That is, the identification unit 330 identifies feature point 31a as the second position so that the pointer returns to feature point 31a, which is at the outer corner of the right eye, in the opposite direction to the order of the parts. In this way, the identification unit 330 identifies the second position by tracing back the transitions of the identified first position. For example, each time the reception unit 320 receives the second input, the identification unit 330 identifies the second position by tracing back one step at a time the transitions of the identified first position. The display control unit 340 moves the pointer from feature point 31b to feature point 31a so that the pointer moves to the second position. In other words, the display control unit 340 returns the pointer to its original position. The identification unit 330 and the display control unit 340 can also be described as follows: The identification unit 330 identifies a new first position and further identifies a second position. The second position can also be described as the first position identified before the new first position. When the display control unit 340 receives a second input from the user, it moves the pointer to the second position. More specifically, the identification unit 330 identifies the first position in the order in which the user inputs the positions of the feature points, and identifies the first position that preceded the new first position as the second position in the reverse order of the order in which the first position was identified.

[0057] [Explanation of operation] Next, the operation of the input support system 300 according to the third embodiment will be described with reference to a flowchart. Figure 9 is a flowchart showing an example of the operation of the input support system 300 according to the third embodiment. As shown in Figure 9, the flowchart shown here differs from the flowchart shown in Figure 5 in that steps S310 to S370 are newly executed. Also, steps S210, S220, S240, and S270 in Figure 9 differ only in that the calculation unit 310 and display control unit 340 execute the operations of the calculation unit 210 and display control unit 240 described in Figure 5, so their explanation will be omitted. The differences from the flowchart shown in Figure 5 will be explained below.

[0058] In step S210, the calculation unit 310 acquires an image from the database 350, and then calculates the order of feature points and the feature points specified by the user (steps S310 and S320). In step S310, the calculation unit 310 calculates feature points using a machine learning model read from the database 350. Note that feature points may be calculated from the input image, or based on the feature point information as described above, rather than using a machine learning method. In step S320, the calculation unit 310 calculates the order of feature points using a machine learning model read from the database 350. Alternatively, the order of feature points may be calculated based on the feature points calculated from the image and the order of the parts, or the order of feature points may be calculated from the feature point information associated with the acquired image. In Figure 9, step S320 is executed after step S310, but they may be executed simultaneously.

[0059] The reception unit 320 determines whether or not it has received a first input from the user (step S330). If there is no first or second input, the reception unit 320 waits until it receives a first and second input from the user. If the reception unit 320 has received a first input, the process transitions to the next step S340.

[0060] The identification unit 330 identifies a first position based on the feature points and their order calculated by the calculation unit 310 (step S340). In this way, the identification unit 330 identifies a first position based on the order in which the user inputs the positions of the feature points. After executing step S340, the process transitions to step S270, and the flow ends when the display control unit 340 displays the pointer at the first position. Note that the timing of step S340 is not limited to this; it can be performed at any time between the calculation of the order of the feature points in step S320 and the processing of moving the pointer to the first position in S270.

[0061] The reception unit 320 determines whether or not it has received a second input from the user (step S350). If the reception unit 320 has received a second input, the process transitions to the next step S360. Therefore, the reception unit 320 performs an operation to transition to the corresponding process when it receives either the first or second input.

[0062] When the receiving unit 320 receives the second input, the identification unit 330 identifies the position of the previous feature point as the second position based on the order of feature points specified by the user (step S360). The timing of step S360 is not limited to this; it can be performed at any time after the order of feature points is calculated in step S320 and before the process of moving the pointer to the second position in S370.

[0063] In addition to the operation of the display control unit 240 in the second embodiment, the display control unit 340 also causes the pointer to be displayed at the second position identified by the identification unit 330 when the reception unit 320 receives the second input (step S370).

[0064] [Explanation of effects] In the third embodiment, the input support system 300 can move the pointer to the vicinity of the feature point to be specified by the user, triggered by a first input, and then, triggered by a predetermined second input, reverse the transition and return the pointer to the position of the previous feature point. This allows the input support system 300 to move the pointer sequentially to the vicinity of feature points while reducing the amount of operation required for the user to move the pointer to the location of the feature point. Furthermore, because the input support system 300 can return the pointer to the position of the previous feature point, the user can redo the specification of the feature point. As a result, the input support system 300 can reduce the burden on the user in the task of inputting feature points. In addition, in this embodiment, the calculation unit 310 calculates the order in which the user inputs the positions of feature points from the input image. Therefore, the order is calculated automatically, providing a highly convenient system.

[0065] [Fourth Embodiment] Next, a fourth embodiment will be described. The fourth embodiment differs from the third embodiment in that the calculation unit calculates bounding boxes instead of calculating feature points.

[0066] The differences from the third embodiment will be explained in detail below, with redundant explanations omitted where appropriate. It should be noted that the first, second, and third embodiments described above can also be applied to the fourth embodiment.

[0067] [Explanation of the structure] Figure 10 is a block diagram showing an example configuration of the fourth embodiment. As shown in Figure 10, the input support system 400 includes a calculation unit 410, a reception unit 320, a specification unit 430, a display control unit 440, and a database 350.

[0068] The calculation unit 410 calculates a bounding box from the input image. A bounding box is a polygonal boundary that surrounds an object in an image, such as a person, face, various facial features (eyebrows, eyes, nose, mouth, ears, etc.), or a car. A bounding box can also be described as a boundary that surrounds the feature points that make up the object. For example, if a face image is input as the input image in order to specify the bounding box for the eyes of a face, the calculation unit 410 calculates a bounding box 41 that surrounds the features present on the surface of the person's face from the input image 40, as shown in Figure 11. More specifically, the calculation unit 410 calculates the vertices of the bounding box.

[0069] Here, a facial image was described as an example of an input image, but this embodiment can be applied to other images as well. For example, this embodiment may be implemented in an operation where an image acquired from an in-vehicle camera mounted on a car is sent to a server or to a device connected to the in-vehicle camera, and then the user specifies road scratches, cars, people, etc., using bounding boxes. This can assist the user in inputting bounding boxes. It also becomes possible to assist in the creation of training data for machine learning. By using a machine learning model trained with this training data, it becomes possible to accurately identify objects such as people and cars from driving footage or accident footage using bounding boxes. The results of this identification can also be used to determine driving skills or calculate fault ratios in traffic accidents. In addition, by inputting images acquired from cameras installed in various environments, such as cameras pointed at roads or inside facilities, into the input support system 400, the system may assist the user in creating training data used to learn the movements of cars, faces, and bodies of pedestrians in the images. This embodiment facilitates the creation of training data for machine learning by supporting the operation of specifying the entire person or parts such as the face and body using bounding boxes from images acquired from a camera. Using a model trained with this training data, a person appearing in the video captured by the camera can be tracked using bounding boxes. Based on the feature points within the bounding boxes, it becomes possible to identify the person or detect suspicious behavior.

[0070] Furthermore, the bounding box and input image specified by the user can be changed as appropriate. In addition, the calculation unit 410 may calculate the order of the bounding boxes specified by the user, similar to how the order of feature points was calculated in the third embodiment. In this embodiment, an embodiment in which a face image is used as the input image and which assists in specifying rectangular bounding boxes for each part of the face will be described.

[0071] The method for calculating the bounding box is the same as the method used to calculate feature points by the machine learning model described in the third embodiment. However, while the training of the machine learning model described in the third embodiment used pairs of images and the positions of feature points related to the object as training data, the calculation of the bounding box uses, for example, pairs consisting of an image and the bounding box related to the object as training data. Alternatively, the bounding box may be calculated based on the positional relationships between feature points identified from the feature points calculated by the method described in the third embodiment.

[0072] The identification unit 430 identifies the vertices of the bounding box calculated by the calculation unit 410 as the first position. In this embodiment, since a rectangular bounding box is to be specified, at least two vertices need to be specified to specify one bounding box. Therefore, for example, to specify the bounding box of the left eye, the user can specify the upper left vertex of the bounding box, and then the lower right vertex. In other words, in this case, the identification unit 430 identifies the upper left vertex of the bounding box calculated by the calculation unit 410 as the first position, and then identifies the lower right vertex of this bounding box as the next first position. However, the identification unit 430 is not limited to this, and may also identify the first and second positions in such a way that the user specifies each vertex according to the shape of the bounding box to be specified. Furthermore, if the order of the bounding boxes has been calculated by the calculation unit 410, the first and second positions can be identified based on the vertices and order of the bounding boxes. Thus, the identification unit 430 may identify a first position or a second position based on the order in which the user inputs the positions of the feature points.

[0073] The display control unit 440 moves the pointer to the first position when the reception unit 320 receives the first input (specification start input). Therefore, each time the display control unit 440 receives the first input, it moves the pointer 42 in order to the vertices 41a, 41b, 41c, and 41d of the bounding boxes calculated by the calculation unit 410, as shown in Figure 12. This allows the bounding boxes of each face part to be specified in order, such as the bounding box of the right eye, the bounding box of the left eye, and so on. Specifically, when the first input is received, the pointer 42 is moved to the first position, which is the upper left vertex 41a of the bounding box. When the reception unit 320 receives the first input again, the identification unit 430 updates the first position to the lower right vertex 41b of the bounding box in order to allow the user to specify the position of the next bounding box vertex to be specified. Accordingly, the display control unit 440 moves the pointer to the position of vertex 41b so that the pointer moves to the updated first position. If a second input (return input) is received after the pointer has been moved to vertex 41b of the bounding box, the identification unit 430 identifies vertex 41a of the bounding box as the second position, and the display control unit 440 can move the pointer back to the second position.

[0074] [Explanation of operation] Next, the operation of the input support system 400 according to the fourth embodiment will be described with reference to a flowchart. Figure 13 is a flowchart showing an example of the operation of the input support system 400 according to the fourth embodiment. As shown in Figure 13, the flowchart shown here differs from the flowchart shown in Figure 9 in that steps S410 to S470 are newly executed. Also, steps S210 and S220 in Figure 13 differ only in that the calculation unit 410 and display control unit 440 perform the operations of the calculation unit 310 and display control unit 340 described in Figure 9, so their explanation will be omitted. Furthermore, steps S330 and S350 are the same operations as described above, so their explanation will be omitted. The differences from the flowchart shown in Figure 9 will be explained below.

[0075] After acquiring an image from the database 350 in step S210, the calculation unit 410 calculates the order of the bounding boxes and the bounding boxes specified by the user (steps S410 and S420). The order of the bounding boxes is determined based on a predetermined order for each object to which a bounding box is specified and a predetermined order for specifying the vertices of the bounding box for a single object. For example, the order of bounding boxes for specifying rectangular bounding boxes for each part of a face is calculated as follows: first, the top-left vertex of the bounding box for the right eye; then, the bottom-right vertex of the bounding box for the right eye; then, the top-left vertex of the bounding box for the left eye; and finally, the bottom-right vertex of the bounding box for the left eye. In this way, the order should be calculated so that the top-left and bottom-right vertices of the bounding boxes corresponding to each part can be selected in order.

[0076] The display control unit 440 displays the bounding box calculated by the calculation unit 410 on the screen (step S430). However, the timing at which the display control unit 440 executes step S430 is not limited to this; it may be executed anytime between step S410 and the end of the flow. User operation is supported by displaying the bounding box. The displayed bounding box may be displayed as a line of a predetermined density or less, as shown in bounding box 43a in Figure 14, or as a dashed line, as shown in bounding box 43b, so that it is clear that it is a reference display. Alternatively, step S430 may be omitted, and the calculated bounding box may not be displayed on the screen. By not displaying the bounding box, it is possible to input the bounding box based solely on the user's knowledge while ensuring operability. In other words, in this case, the user inputs the bounding box based solely on their own knowledge, without relying on the position of the bounding box proposed by the calculation unit 410.

[0077] In step S330, when the reception unit 320 receives a first input from the user, the identification unit 430 identifies the first position based on the bounding box and the order of the bounding boxes calculated by the calculation unit 410 (step S440). Note that step S440 is not limited to this timing, and can be performed at any time after the order of the bounding boxes is calculated in step S420 and before the process of moving the pointer to the first position in S450.

[0078] When the reception unit 320 receives the first input, the display control unit 440 causes the pointer to be displayed at the first position identified by the identification unit 430 (step S450).

[0079] In step S350, when the user provides a second input, the identification unit 430 identifies a second position based on the bounding box and the order of the bounding boxes calculated by the calculation unit 410 (step S460). The identification unit 430 identifies the second position by tracing back the transitions of the identified first position. In this way, the second position with respect to the bounding box is identified so that it transitions in the reverse order of the order in which the user inputs the vertices of the bounding box. The timing of step S460 is not limited to this; it can be performed anytime between the calculation of the order of the bounding boxes in step S420 and the processing of moving the pointer to the second position in S470.

[0080] When the reception unit 320 receives the second input, the display control unit 440 causes the pointer to be displayed at the second position identified by the identification unit 430 (step S470).

[0081] [Explanation of effects] In the input support system 400 of the fourth embodiment, the calculation unit 410 calculates the bounding boxes and their order. Then, triggered by the first or second input from the user, the display control unit 440 moves the pointer to the appropriate position. As a result, the input support system 400 allows the user to specify the bounding boxes sequentially while reducing the amount of operation required for pointer movement during the user's task of specifying the bounding box area. Consequently, the input support system 400 can reduce the burden on the user's input work.

[0082] [Fifth Embodiment] Next, a fifth embodiment will be described. The fifth embodiment differs from the other embodiments in that it newly includes a warning unit that issues a warning based on a comparison between the calculated feature points and the feature points specified by the user. The differences from the other embodiments will be explained in detail below, and redundant explanations will be omitted as appropriate.

[0083] [Explanation of the structure] Figure 15 is a block diagram showing an example configuration of the input support system according to the fifth embodiment. As shown in Figure 15, the input support system 500 includes a calculation unit 310, a reception unit 520, a specification unit 330, a display control unit 540, a database 350, and a warning unit 560.

[0084] The reception unit 520 accepts an operation by the user to specify feature points. This operation may be accepted as a first input (specification start input) as described in the embodiment described above. The reception unit 520 may also accept an operation to specify a bounding box instead of an operation to specify feature points. The reception unit 520 may also accept a first input separate from the operation by the user to specify feature points. The reception unit 520 may also accept a second input (return input).

[0085] The display control unit 540 performs the same processing as the display control unit in any of the embodiments described above, but may also overlay the specified feature point on the image when the reception unit 520 receives an operation from the user to specify a feature point. This allows the user to confirm where they specified the feature point.

[0086] The warning unit 560 outputs a warning based on the distance between the position of the feature point calculated by the calculation unit 310 and the position of the feature point specified by the user, which is received by the reception unit 520. Specifically, the warning unit 560 determines whether the distance between the position of the feature point calculated by the calculation unit 310 and the position of the feature point received by the reception unit 520 is greater than a predetermined distance, and if it determines that the distance is greater than the predetermined distance, it displays a warning to the user on the output device 201. For example, the warning unit 560 outputs a warning if the position of the feature point specified by the user after the pointer has moved to a first position is greater than a predetermined distance from the position calculated by the calculation unit 310 for the feature point corresponding to this first position. In addition, a warning sound can be output from a speaker built into or connected to the input support system as a warning. This allows the user to know that their input differs from the position of the feature point calculated by the calculation unit 310. This can suggest the possibility of incorrect input or the possibility that the user's specification of the feature point based on their own knowledge is incorrect. Note that a distance greater than the predetermined distance refers to, for example, when the distance between two points is several tens of pixels. Therefore, the user can notice input errors through the warning, enabling more accurate specification of feature points. This predetermined distance can be set by the user as appropriate, and shortening the predetermined distance will result in stricter judgment to align with the position of the feature point calculated by the calculation unit 310. Furthermore, even in a configuration where a bounding box is specified instead of a feature point, the system can determine whether or not to issue a warning by comparing the position of a specific vertex of the bounding box calculated by the calculation unit 410 with the position of a specific vertex of the bounding box specified by the user. In this way, the warning unit 560 issues a warning if the position of the feature point calculated by the calculation unit 310 (the position of the feature point or the bounding box) and the position of the feature point entered by the user (the position of the feature point or the bounding box) are separated by a predetermined distance or more. In this embodiment, an embodiment in which the operation of specifying feature points is performed will be described. The input support system according to this embodiment is a system that assists a user (human) in determining the position of feature points and the position of the bounding box in the input image by looking at the input image.Therefore, by using the input support system according to this embodiment, it is expected that the user will be able to specify the position of feature points and bounding boxes more accurately than the position calculated by the calculation unit. For this reason, the position specified by the user is generally more appropriate than the position calculated by the calculation unit. Therefore, this warning does not necessarily indicate an error in the user's specification.

[0087] [Explanation of operation] Next, the operation of the input support system 500 according to the fifth embodiment will be described with reference to a flowchart. Figure 16 is a flowchart showing an example of the operation of the input support system 500 according to the fifth embodiment. As shown in Figure 16, the flowchart shown here differs from the flowchart shown in Figure 9 in that steps S510 to S540 are newly executed. Also, in Figure 16, the processing from steps S210 to S240 is omitted from the illustration. The processing of these steps differs only in that the processing previously performed by the reception unit 320 and the display control unit 340 is now performed by the reception unit 520 and the display control unit 540, so the illustration and explanation are omitted. The differences from the flowchart shown in Figure 9 will be explained below.

[0088] After the display control unit 540 moves the pointer to the first or second position in step S270 or step S370, the reception unit 520 accepts an operation from the user to specify a feature point (step S510). If an operation to specify a feature point is accepted from the user, the process transitions to the next step S520. If an operation to specify a feature point is not accepted from the user, the unit waits until an operation is accepted.

[0089] The display control unit 540, triggered by the reception unit 520 receiving an operation from the user to specify a feature point, overlays the feature point specified by the user onto the image and displays it (step S520). The timing of step S520 is not limited to this; it can be executed at any time after step S510.

[0090] The warning unit 560 compares the position of the feature point calculated by the calculation unit 310 with the position of the feature point specified by the user received by the reception unit 520, and determines whether the distance between the two points is within a predetermined distance (step S530). If it is not within the predetermined distance, the process proceeds to step S540. If it is within the predetermined distance, the process ends. If it is within the predetermined distance, the warning unit 560 may output that the user's input is correct, or by executing step S520, the display control unit 540 may display the feature point entered by the user to inform the user that the input is correct. Specifically, if the warning unit 560 determines that the distance between the two points is within a predetermined distance, it displays a notification to the user on the output device. Alternatively, the system may output a sound indicating that the input is correct from a speaker built into or connected to the input support system to inform the user that the input is correct.

[0091] If the distance between two points is not within a predetermined distance, the warning unit 560 outputs a warning to the display control unit 540 indicating that the feature points calculated by the calculation unit 310 do not match the feature points input from the user (step S540). The display control unit 540 outputs the warning to the screen of the output device, allowing the user to know that their input may be incorrect. Furthermore, by outputting a prompt for re-input along with the warning in step S540, the user can be prompted to re-enter the feature points. After outputting the warning in step S540, the process transitions to S510, where re-input can be accepted. The reception unit 520 may also accept input from the user that ignores the warning, and if such input is received, the process may terminate without transitioning to step S510.

[0092] [Description of hardware configuration] The hardware configuration of the fifth embodiment is the same as the hardware configuration of the input support system 200 described in the second embodiment; only the differences in operation are described below.

[0093] The processor 205 reads and executes software (computer programs) from memory 204, thereby further performing the processing of the warning unit 560 described above.

[0094] [Explanation of effects] In the fifth embodiment, in addition to the embodiments described so far, the calculated feature points are compared with the feature points entered by the user. This makes it possible to identify erroneous inputs or operations that indicate errors in the user's knowledge if the difference between the two is too large, and to output a warning. Therefore, the user can know if their input may be incorrect and can specify feature points and bounding boxes more accurately.

[0095] [Sixth Embodiment] Next, a sixth embodiment will be described. The sixth embodiment will specifically describe a configuration in which a specific unit determines a first position and a second position based on the direction of movement of a pointer operated by the user. The differences from the other embodiments will be described in detail below, and redundant explanations will be omitted as appropriate.

[0096] [Explanation of the structure] Figure 17 is a block diagram showing an example configuration of the input support system according to the sixth embodiment. As shown in Figure 17, the input support system 600 comprises a calculation unit 310, a reception unit 620, a specification unit 630, a display control unit 340, and a database 350.

[0097] The reception unit 620 detects when the user operates the input device to move the pointer to the location of the feature point to be specified, and accepts the direction of the pointer movement as the first input (specification start input). At this time, the conditions for determining whether the user's pointer operation is the first input are: 1) The pointer is located more than a predetermined distance away from the calculated feature point. 2) A pointer movement operation is performed simultaneously with another operation that indicates it is the first input. 3) The acceleration of the pointer movement caused by user operation is above a threshold. Any one or a combination of any two or more of the above conditions can be set. By adopting the above conditions, it is possible to determine whether the pointer operation is an operation in which the user moves the pointer to the next feature point. The receiving unit 620 may also accept the direction of pointer movement as a second input (return input), along with a predetermined input instructing to return the moved pointer position to its previous position.

[0098] The identification unit 630 identifies the first position based on the pointer position and the direction of movement of the pointer at the time of input and the first input received by the reception unit 620. Specifically, the identification unit 630 uses the pointer position at the time of input reception as the starting point for the search and searches for feature points along the direction of movement of the pointer received as the first input. The identification unit 630 then identifies the position of the feature point that is closest to the pointer position at the time of input reception as the first position. In this way, the identification unit 630 identifies the first position based on the pointer position at the time of input, the direction of movement of the pointer, and the position of the feature point. By using the first input to identify the first position, the first position can be identified from the user's intuitive operation, and the operation to specify feature points can be supported with a more intuitive operation.

[0099] [Explanation of operation] Next, the operation of the input support system 600 according to the sixth embodiment will be explained with reference to a flowchart. Figure 18 is a flowchart showing an example of the operation of the input support system 600 according to the sixth embodiment. As shown in Figure 18, the flowchart shown here differs from the flowchart shown in Figure 9 in that steps S610 to S630 are newly executed. Also, the processing from steps S210 to S240 is omitted from the explanation because the only difference is that the processing previously performed by the reception unit 320 is now performed by the reception unit 620. Similarly, the processing from steps S310 to S370 is omitted from the explanation because the only difference is that the processing previously performed by the reception unit 320 and the identification unit 330 is now performed by the reception unit 620 and the identification unit 630. The differences from the flowchart shown in Figure 9 will be explained below.

[0100] The reception unit 620 determines whether the user has operated the pointer as the first input (step S610). If it determines that the first input has been made, the process proceeds to the next step S620.

[0101] The identification unit 630 obtains the pointer position at the time the receiving unit 620 receives the first input (step S620).

[0102] The identification unit 630 identifies the position of the feature point closest to the current pointer position (the position obtained in step S620) from among the multiple feature points calculated by the calculation unit 310 as the first position (step S630). That is, the identification unit 630 uses the current pointer position (the position obtained in step S620) as the starting point of the search and searches for the feature points calculated by the calculation unit 310 along the direction of the pointer's movement. Then, the identification unit 630 identifies the position of the feature point closest to the current pointer position among the searched feature points as the first position.

[0103] After step S630, the display control unit 340 moves the pointer to the first position identified in step S630 (step S270), and the flow ends.

[0104] [Explanation of effects] In the sixth embodiment, the direction of pointer movement is accepted as a first input, and the first input is used to identify a first position, making it possible for the user to identify the feature point they wish to specify through more intuitive operation, thereby improving usability.

[0105] [Seventh Embodiment] Next, a seventh embodiment will be described. The seventh embodiment differs from the embodiments described above in that the display control unit further displays a visual area, which is a window in which the pointer and the image around the pointer are displayed in an enlarged state. The differences from the other embodiments will be described in detail below, and redundant explanations will be omitted as appropriate.

[0106] [Explanation of the structure] Figure 19 is a block diagram showing an example configuration of the input support system according to the seventh embodiment. As shown in Figure 19, the input support system 700 includes a calculation unit 310, a reception unit 320, a specification unit 330, a display control unit 740, and a database 350.

[0107] The display control unit 740 performs display control similar to that of the display control units of the other embodiments described above. However, the display control unit 740 differs from the display control units of the other embodiments described above in that it displays a visual area that magnifies and displays the pointer and the image around the pointer after the pointer has been moved to a first or second position. The visual area is a window displayed around the pointer, such as to the upper right of the pointer 72 as shown in Figure 20, and is a rectangular-bordered area 73. The area 73 is displayed superimposed on the input image 70. The user can specify feature points more accurately by operating the pointer while viewing the magnified image displayed in the area 73. The display control unit 740 may also display the visual area when the position of the pointer is within a predetermined range from the position of the feature point calculated by the calculation unit 310. For example, the display control unit 740 may be set to display the visual area when the pointer is within a few pixels from the position of the calculated feature point. In this way, the visual area can be displayed even if the user moves the pointer to the feature point themselves without making a first or second input.

[0108] Furthermore, the display control unit 740 may lower the sensitivity of the pointer when the pointer is within a predetermined range from the feature point calculated by the calculation unit 310 compared to when the pointer is outside the predetermined range from the feature point. Pointer sensitivity refers to the ratio of the amount of pointer movement to the amount of user operation. In other words, lowering the pointer sensitivity means reducing the ratio of the amount of pointer movement to the amount of user operation. Lowering the pointer sensitivity improves the accuracy of user operation to move the pointer, enabling accurate specification of feature points.

[0109] [Explanation of operation] Next, the operation of the input support system 700 according to the seventh embodiment will be described using Figure 21. Figure 21 is a flowchart showing an example of the operation of the input support system 700 according to the seventh embodiment. As shown in Figure 21, the display control unit 740 differs from the display control unit 340 in that, in addition to the operation of the display control unit 340, it processes the step S710 of displaying the visual area and the step S720 of changing the sensitivity of the pointer, while the other processes are the same as in Figure 9. The differences from the flowchart shown in Figure 9 will be explained below.

[0110] After moving the pointer to the first position in step S270, the display control unit 740 displays a magnified visual area above the pointer to the upper right of the pointer (step S710). The timing of when the display control unit 740 displays the visual area is anytime as long as the position of the pointer and the position of the feature point calculated by the calculation unit 310 are within a predetermined distance. Therefore, for example, the display of the visual area may be performed after the pointer is displayed at the second position in step S370. In this way, when the position of the pointer and the position of the calculated feature point are within a predetermined distance, the display control unit 740 displays a magnified image of a partial image included in the input image, which is a partial image of the area around the pointer, superimposed on the input image.

[0111] After moving the pointer to the first or second position, the display control unit 740, if the pointer is within a predetermined range from the feature point calculated by the calculation unit 310, lowers the sensitivity of the pointer to that of when the pointer is outside the predetermined range from the feature point (step S720).

[0112] [Explanation of effects] In the seventh embodiment, the display control unit displays a visual area when the pointer position and the calculated feature point position are within a predetermined distance. This allows the user to specify the feature point more accurately while viewing a magnified image of the pointer and its surroundings.

[0113] Although the present invention has been described above with reference to embodiments, the present invention is not limited thereto. Various modifications to the structure and details of the present invention can be made within the scope of the invention as can be understood by those skilled in the art. Furthermore, each embodiment can be combined.

[0114] Some or all of the above embodiments may also be described as follows, but are not limited to the following: [Note 1] A calculation means for calculating the position of feature points from an input image, A display control means that displays the input image and a pointer operated by the user on the screen, A receiving means for receiving a first input from the user, A means for identifying a first position based on the position of the feature point calculated by the calculation means, It has, When the display control means receives the first input, it moves the pointer to the first position. Input assistance system. [Note 2] The identification means further identifies the first position based on the order in which the user inputs the positions of the feature points. The input support system described in Appendix 1. [Note 3] The calculation means further calculates the order from the input image. The input support system described in Appendix 2. [Note 4] The identifying means identifies the first position based on the position of the pointer and the position of the feature point. The input support system described in Appendix 1. [Note 5] The identifying means identifies the first position based on the position of the pointer at the time of the first input, the direction of movement of the pointer, and the position of the feature point. The input support system described in Appendix 1. [Note 6] The identifying means identifies a new first position and further identifies a second position. The second position is the first position identified before the new first position, When the display control means receives a second input from the user, it moves the pointer to the second position. The input support system described in Appendix 1. [Note 7] The identification means identifies the first position in the order in which the user inputs the positions of the feature points, and identifies the first position that precedes the new first position as the second position in the reverse order of the order used to identify the first position. The input support system described in Appendix 6. [Note 8] The identification means identifies the first position based on feature points whose positions have been calculated by the calculation means, and which are within a predetermined distance from the feature point and whose position has not yet been specified by the user. The input support system described in Appendix 1. [Note 9] The identification means calculates an estimated difference between the position calculated by the calculation means and a position specified by the user for each of the feature points whose position has been calculated by the calculation means, and identifies the first position based on the feature points whose position has been calculated by the calculation means and whose estimated value is equal to or greater than a predetermined value. The input support system described in Appendix 1. [Note 10] The system further includes a warning means that issues a warning if the position of the feature point calculated by the calculation means and the position of the feature point entered by the user are separated by a predetermined distance or more. An input support system described in any one of the appendices 1 through 9. [Note 11] The position of the aforementioned feature point is the position of the feature point. An input support system described in any one of the appendices 1 through 10. [Note 12] The position of the aforementioned feature point is the position of the bounding box that surrounds the feature point constituting the object. An input support system described in any one of the appendices 1 through 10. [Note 13] The display control means further displays the position of the feature point. The input support system described in Appendix 11. [Note 14] The display control means further displays the bounding box. The input support system described in Appendix 12. [Note 15] The display control means, when the position of the pointer and the position of the feature point are within a predetermined distance, displays an enlarged image of a portion of the input image, specifically a portion of the input image surrounding the pointer, superimposed on the input image. An input support system described in any one of the appendices 1 through 14. [Note 16] Computers The position of feature points is calculated from the input image. The input image and the user's pointer are displayed on the screen. The first input is received from the aforementioned user. Based on the calculated position of the feature point, a first position is identified. When the first input is received, the pointer is moved to the first position. Input assistance methods. [Note 17] On the computer, A calculation step that calculates the position of feature points from the input image, A display control step that displays the input image and a pointer operated by the user on the screen, A reception step in which a first input is received from the user, A determination step to identify a first position based on the position of the feature point calculated in the calculation step, Make it run, In the display control step, when the first input is received, the pointer is moved to the first position. A non-temporary, computer-readable medium on which a program is stored. [Explanation of symbols]

[0115] 20, 30, 40, 70 Input images 21, 31a, 31b Feature points 22, 32, 42, 72 pointers 41, 43a, 43b Bounding boxes Vertices 41a and 41b 73 areas 100, 200, 300, 400, 500, 600, 700 Input Assistance System 110, 210, 310, 410 Calculation Unit 120, 220, 320, 520, 620 Reception Desk 130, 230, 330, 430, 630 Specific part 140, 240, 340, 440, 540, 740 Display Control Unit 201 Output device 202 Input device 203 Storage device 204 memory 205 Processor 250, 350 databases 560 Warning section

Claims

1. A calculation means that calculates the position of feature points from an input image and calculates the order in which the user inputs the positions of feature points from the input image, A display control means that displays the input image and a pointer operated by the user on the screen, A receiving means for receiving a first input from the user, A means for identifying a first position based on the position of the feature point calculated by the calculation means and the order thereof, It has, When the display control means receives the first input, it moves the pointer to the first position. Input assistance system.

2. A calculation means for calculating the position of feature points from an input image, A display control means that displays the input image and a pointer operated by the user on the screen, A receiving means for receiving a first input from the user, A first position identification means that identifies a feature point whose position has been calculated by the calculation means, based on a feature point within a predetermined distance from the feature point for which the user has not yet specified a position, It has, When the display control means receives the first input, it moves the pointer to the first position. Input assistance system.

3. The identifying means identifies a new first position and further identifies a second position. The second position is the first position which was identified before the new first position. When the display control means receives a second input from the user, it moves the pointer to the second position. The input support system according to claim 1 or 2.

4. The identification means identifies the first position in the order in which the user inputs the positions of the feature points, and identifies the first position that precedes the new first position as the second position in the reverse order of the order used to identify the first position. The input support system according to claim 3.

5. Computers The system calculates the position of feature points from the input image, and calculates the order in which the user inputs the positions of feature points from the input image. The input image and the user's pointer are displayed on the screen. The first input is received from the aforementioned user. Based on the calculated position of the feature points and the order, the first position is identified. When the first input is received, the pointer is moved to the first position. Input assistance methods.

6. A computer, The position of feature points is calculated from the input image. The input image and the user's pointer are displayed on the screen. The first input is received from the aforementioned user. Based on the calculated feature points, a first position is identified based on feature points within a predetermined distance from the feature point that have not yet been assigned a location by the user. When the first input is received, the pointer is moved to the first position. Input assistance methods.

7. On the computer, A calculation step that calculates the position of feature points from the input image and calculates the order in which the user inputs the positions of feature points from the input image, A display control step that displays the input image and a pointer operated by the user on the screen, A reception step in which a first input is received from the user, A determination step to identify a first position based on the position of the feature point calculated in the calculation step and the order thereof, Make it run, In the display control step, when the first input is received, the pointer is moved to the first position. program.

8. A computer, A calculation step that calculates the position of feature points from the input image, A display control step that displays the input image and a pointer operated by the user on the screen, A reception step in which a first input is received from the user, A first position identification step is performed based on feature points whose positions have been calculated in the above calculation step, and which are located within a predetermined distance from the feature point and whose position has not yet been specified by the user. Make it run, In the display control step, when the first input is received, the pointer is moved to the first position. program.

Images