Information processing device and program

Abstract

To provide an information processing device and a program which urge a user to complete a series of operations, even when a position at which a procedure of work is displayed in accordance with a video cannot be properly detected.SOLUTION: An operation navigation system comprises an image forming device which is an object for which a user works. A user terminal and a server are connected through a network. The server 10 which is an information processing device comprises a processor. The processor: acquires an object image which is an image obtained by capturing a work object in which a series of operations is defined; recognizes whether the object image is in a specific state associated with each procedure in advance, for each procedure of the series of operations; when it is in the specific state, overlaps specific content associated with the specific state in advance to display the object image; and when it is not the specific state, displays non-specific content associated with each procedure in advance.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an information processing device and a program. [Background technology]

[0002] Patent document 1 describes an information processing device that has an imaging means for imaging an object, a determination means for determining the object as a superimposition object based on the imaging results by the imaging means, an identification means for identifying the type of content associated with the superimposition object, and a control means for controlling the display of the superimposition object and content displayed on a display screen, and the control means determines the display mode of the content to be a tracking display in which the content is superimposed by following the image of the superimposition object, or a non-tracking display in which the content is displayed at a predetermined position on the display screen, depending on the type of content identified. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-3550 A Summary of the Invention [Problem to be solved by the invention]

[0004] To inform users of work procedures, some technologies capture images of the object to be worked on and overlay the work procedures on the captured image. For example, when an abnormality occurs in an electrical appliance, a technology is available that captures an image of the appliance where the abnormality occurred and displays repair procedures on the captured image. To detect the position where the repair procedures should be displayed on the captured image, for example, AR markers or feature points of the electrical appliance are used for positioning. However, even if the AR markers and feature points of the electrical appliances are captured in the video, the tilt of the camera and the influence of the light source may prevent the AR markers and feature points from being recognized, making it impossible to correctly detect the position where the work procedures should be displayed in accordance with the video. An object of the present invention is to encourage a user to complete a series of operations even when it is not possible to correctly detect the position where the work procedure should be displayed in accordance with the video. [Means for solving the problem]

[0005] The invention of claim 1 includes a processor, and the processor acquires an object image, which is an image of an object to be worked on for which a series of operations is defined, and executes the series of operations. Hand In each order, the target image is That hand If the specific situation is present, the target image is displayed with specific content pre-associated with the specific situation superimposed thereon. If the specific situation is not present, the target image is displayed with specific content pre-associated with the specific situation superimposed thereon. note-taker Display non-specific content that is pre-associated in order The recognition of whether the target image is in the specific situation is performed based on feature points of the target, and the specific situation is when a position where the specific content is to be superimposed and displayed can be recognized from the feature points of the target, and when the target image is in the specific situation, the specific content is superimposed on the display position and the target image is displayed, and when the display position cannot be recognized, image recognition of the target image is further performed, and when the result of the image recognition satisfies a predetermined condition, the non-specific content associated with the predetermined condition is superimposed and the target image is displayed. The information processing device is characterized by 。 request request 2 The invention described in the above provides that the image recognition is carried out by using two or more different image recognition methods, and the series of operations Hand The method of image recognition is characterized in that a priority order is assigned to each image recognition method. 1 The information processing device according to 。 request request 3 The described invention is a processor, the processor acquiring a target image which is an image of a target performing work for which a series of operations is defined, recognizing for each step of the series of operations whether the target image is in a specific situation associated with the step in advance, and if the specific situation is met, displaying the target image with a specific content associated with the specific situation in advance superimposed thereon, and if the specific situation is not met, displaying a non-specific content associated with the step in advance, and performing image recognition of the target image to determine one of the steps as an execution step to be performed in the work; a first similarity indicating a degree of similarity between an image previously associated with the execution procedure and the target image; a second similarity indicating a degree of similarity between an image previously associated with a subsequent step of the execution procedure and the target image; and updating the execution procedure based on the calculated first similarity and second similarity. Feelings It is an information processing device. Claim 4 The invention described in the document provides a computer with a function for acquiring an image of an object that is an image of an object that is to be subjected to a task for which a series of operations are defined, and a function for acquiring an image of an object that is to be subjected to a task for which a series of operations are defined, Hand In each order, the target image is That handa function of recognizing whether the target image is in a specific situation associated in advance, a function of superimposing specific content associated in advance with the specific situation on the target image when the target image is in the specific situation, and a function of superimposing specific content associated in advance with the specific situation on the target image when the target image is not in the specific situation. note-taker It also provides a function to display non-specific content that is pre-associated in order. The recognition of whether the target image is in the specific situation is performed based on feature points of the target, and the specific situation is when a position where the specific content is to be superimposed and displayed can be recognized from the feature points of the target, and when the target image is in the specific situation, the specific content is superimposed on the display position and the target image is displayed, and when the display position cannot be recognized, image recognition of the target image is further performed, and when the result of the image recognition satisfies a predetermined condition, the non-specific content associated with the predetermined condition is superimposed and the target image is displayed. This is a program for The invention described in claim 5 is a program for making a computer realize the following functions: acquiring a target image, which is an image of a target performing a task for which a series of operations are defined; recognizing, for each step in the series of operations, whether the target image is in a specific situation that is pre-associated with the step; if the specific situation is determined, displaying the target image by superimposing specific content that is pre-associated with the specific situation; and if the specific situation is not determined, displaying non-specific content that is pre-associated with the step; and for recognizing the target image as an execution procedure to be performed in the task, by image-recognizing the target image, calculating a first similarity that indicates the degree of similarity between the image pre-associated with the execution procedure and the target image, calculating a second similarity that indicates the degree of similarity between an image pre-associated with the next step in the execution procedure and the target image, and updating the execution procedure based on the calculated first similarity and second similarity. [Effects of the Invention]

[0006] According to the invention of claim 1, even if it is not possible to correctly detect the position where the work procedure should be displayed in accordance with the video, it is possible to prompt the user to complete the series of operations. 。 request request 2 According to the invention, the accuracy of image recognition can be improved compared to when the priority of image recognition means is not determined for each work procedure. 。 request request 3 According to the invention, the procedure of the work can be displayed according to the actual situation of the work object. Claim 4 and 5 According to the invention, even if it is not possible to correctly detect the position where the work procedure is to be displayed in accordance with the video, it is possible to prompt the user to complete the series of operations. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram showing an example of the overall configuration of an operation navigation system according to an embodiment of the present invention; [Figure 2] 10A and 10B are diagrams illustrating an example of content displayed on a screen of a user terminal when feature points of an image forming apparatus are successfully extracted. [Figure 3] 10A and 10B are diagrams illustrating an example of content displayed on the screen of a user terminal when feature points of an image forming apparatus cannot be extracted. [Figure 4]FIG. 2 is a diagram illustrating an example of a hardware configuration of a server of the operation navigation system. [Figure 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of a user terminal of the operation navigation system. [Figure 6] FIG. 2 is a diagram illustrating an example of a hardware configuration of an image forming apparatus in the operation navigation system. [Figure 7] FIG. 2 is a diagram showing the functional configuration of a user terminal and a server. [Figure 8] 10 is a table for explaining an example of information related to a work procedure. [Figure 9] 10 is a flowchart illustrating an example of a process in which a server outputs content to be displayed on a user terminal based on an image captured by an image forming apparatus. [Figure 10] 10 is a flowchart showing an example of a process in which a server performs image recognition and determines content to be output when no feature points are extracted. [Figure 11] 10 is a flowchart illustrating an example of a process in which the server determines the execution procedure of a task based on a captured image. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. <Overall configuration of the operation navigation system> 1 is a diagram showing an example of the overall configuration of an operation navigation system 1 according to this embodiment. The operation navigation system 1 according to this embodiment includes a server 10 that provides information on the procedures of a task to be performed by the user to the user terminal 30 to assist the user, and the user terminal 30 that displays the information on the procedures of the task provided by the server 10. The server 10 and the user terminal 30 are connected via a network 70.

[0009] The image forming apparatus 50 is also provided as an object on which a user performs work. In this embodiment, the image forming apparatus 50 is used as an example of an object on which a user performs work, but the object on which a user performs work is not particularly limited. The work performed by the user is also not particularly limited as long as the work procedure is predetermined. Examples of work performed by the user include clearing a paper jam that has occurred in the image forming apparatus 50, replacing toner, changing settings of the image forming apparatus 50, and maintenance and inspection work. The image forming apparatus 50 may or may not be connected to one or both of the server 10 and the user terminal 30.

[0010] In the operation navigation system 1, a server 10, which is an example of an information processing device, sequentially transmits information indicating the procedure of the work to be currently performed to a user terminal 30 according to the progress of the work performed by the user. More specifically, the server 10 performs a process of extracting feature points from an image captured by the image forming device 50 and acquired by the user using the user terminal 30. If the extracted feature points match predetermined feature points, the server 10 transmits to the user terminal 30 display position information based on the feature points and information indicating the work to be currently performed. Hereinafter, the "procedure of the work to be currently performed" may be referred to as an "execution procedure," and the "information indicating the procedure of the work to be currently performed" may be referred to as "execution procedure information." Furthermore, if the server 10 cannot extract predetermined feature points from the captured image, it transmits to the user terminal 30 display position information based on the display area displaying the captured image and execution procedure information. Here, the "display area displaying the captured image" refers to the area in which the captured image is displayed on the display screen of the user terminal 30. More specifically, the window displaying the captured image on the user terminal 30 is an example of the "display area displaying the captured image." The display position information based on the display area displaying the captured image is display position information based on the window. For example, it is specified by a portion of the window, and examples include the upper right area or the lower half area within the window. In other words, the display position information based on the display area displaying the captured image is position information that is determined regardless of the positions of the feature points within the captured image. Furthermore, execution procedure information is predetermined for each procedure, and is predetermined separately for cases where feature points can be extracted and cases where they cannot. A case where feature points can be extracted is an example of a "specific situation," and execution procedure information that is predetermined when feature points can be extracted is an example of "specific content." Furthermore, execution procedure information that is predetermined when feature points cannot be extracted is an example of "non-specific content."

[0011] The user terminal 30 captures an image of the image forming device 50 on which the user is to perform work, transmits the captured image to the server 10, and displays the captured image on the user terminal 30. The user terminal 30 further acquires execution procedure information and display position information from the server 10, and displays the acquired execution procedure information on the user terminal 30 based on the acquired display position information. When the display position information is based on a feature point of the captured image, the execution procedure information is displayed superimposed on the position of the image forming device 50 displayed on the screen of the user terminal 30. Then, while the predetermined feature point is being extracted, the execution procedure information is displayed following the movement of the image forming device 50 displayed on the screen. When the display position information is a display position based on the display screen, the execution procedure information is displayed in a predetermined area regardless of the movement of the image forming device 50 displayed on the screen of the user terminal 30.

[0012] <Display screen example> FIG. 2 is a diagram for explaining an example of the content displayed on the screen of the user terminal 30 when the feature points of the image forming device 50 have been extracted. In the example of FIG. 2, a paper jam has occurred in the folding unit of the image forming apparatus 50, and the user is about to perform an operation to clear the jam. The execution procedure corresponds to a procedure for opening the door of the folding unit. The user photographs the image forming apparatus 50 using the user terminal 30, and the captured image is displayed on the entire display screen of the user terminal 30. Furthermore, execution procedure information 801a and 801b are superimposed on the captured image based on the positions of feature points in the captured image. Execution procedure information 801a is superimposed on the position of the door to be opened and is a graphic arrow pointing in the direction of opening the door. Execution procedure information 801b is placed above execution procedure information 801a and is composed of the text "Please open the door." The user proceeds with the work according to the instructions in the execution procedure information 801a and 801b.

[0013] FIG. 3 is a diagram for explaining an example of the content displayed on the screen of the user terminal 30 when the feature points of the image forming device 50 cannot be extracted. In the example of FIG. 3 , similar to the example of FIG. 2 , the image forming apparatus 50 is in the process of opening the folding unit door. In the example of FIG. 3 , light from a fluorescent lamp above the image forming apparatus 50 shines on the image forming apparatus 50, generating reflected light 921. Although the image forming apparatus 50 is within the angle of view captured by the user terminal 30, the reflected light 921 makes it difficult to detect feature points of the image forming apparatus 50, making it impossible to extract feature points of the image forming apparatus 50. In this case, when feature points cannot be extracted, predetermined execution procedure information 901 is displayed in the upper right area of ​​the entire display screen, which is the area displaying the captured image, in addition to the execution procedure information 801 a and 801 b. The execution procedure information 901 is an image that combines a diagram showing the image forming apparatus 50 opening the folding unit door with text that reads, "First, open the folding unit door." The user proceeds with the work by referring to the execution procedure information 901 displayed on the user terminal 30.

[0014] <Server hardware configuration> FIG. 4 is a diagram showing an example of the hardware configuration of the server 10 of the operation navigation system 1. As shown in FIG. The server 10 may be, for example, an electronic device such as a server machine or a personal computer. The server 10 includes a CPU (Central Processing Unit) 101, which is a computing means; a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, and a storage device 104, which are storage means; and a communication I / F (Communication Interface) 105 for communicating with the outside. The RAM 102 is used as a working memory when the CPU 101 performs computational processing. The ROM 103 holds programs and data such as pre-prepared setting values. The storage device 104 is a means for storing programs and data. For example, a magnetic disk device or an SSD (Solid State Drive) is used as the storage device 104. The CPU 101 is an example of a processor.

[0015] In this embodiment, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPUs) and dedicated processors (e.g., GPUs: Graphics Processing Units, ASICs: Application Specific Integrated Circuits, FPGAs: Field Programmable Gate Arrays, programmable logic devices, etc.). Furthermore, the operations of the processors may not only be performed by a single processor, but may also be performed by multiple processors located at physically separate locations working together. The order of the operations of the processors is not limited to the order described in this embodiment, and may be changed.

[0016] <Hardware configuration of user terminal> FIG. 5 is a diagram showing an example of the hardware configuration of the user terminal 30 of the operation navigation system 1. As shown in FIG. The user terminal 30 includes a CPU 301 as a computing means, RAM 302, ROM 303, and storage device 304 as storage means, a communication I / F 305 for communicating with the outside world, a display device 306 for presenting information to the user, and an input device 307 for the user to input information to the user terminal. The user terminal 30 also includes an imaging device 308 for capturing images of the surroundings. Examples of the user terminal 30 include electronic devices such as a smartphone, a tablet personal computer, or a head-mounted display carried by the user.

[0017] The display device 306 is a device that has the function of displaying images and presenting information to the user, and is, for example, a liquid crystal display or an organic EL display. The input device 307 is a device that has the function of allowing the user to input information to the user terminal 30, and is, for example, a touch panel or physical buttons. The imaging device 308 captures an image of the image forming apparatus 50 in real space and generates image information. The imaging device 308 includes, for example, an optical system that converges incident light and an image sensor that is an imaging means that detects the light converged by the optical system.

[0018] <Hardware Configuration of Image Forming Apparatus 50> FIG. 6 is a diagram showing an example of the hardware configuration of the image forming device 50 of the operation navigation system 1. As shown in FIG. The image forming apparatus 50 includes a CPU 501 as a calculation means, and a RAM 502 and a ROM 503 as storage means.

[0019] The image forming device 50 also has a storage device 504 that stores image data, management data, etc., an image processing unit 505 that performs image processing such as color correction and gradation correction on the image data, an image forming unit 506 that forms an image on paper using, for example, an electrophotographic method or an inkjet method, and a post-processing unit 507 that performs, for example, tri-folding or binding processing on the image-formed paper. The image forming apparatus 50 is also provided with a sensor unit 508 that monitors the operating state of the image forming apparatus 50. Examples of the sensor unit 508 include a sensor that recognizes the open / close state of a door of the image forming apparatus 50, and a sensor that recognizes whether a print medium is jammed in a conveyance path that is an internal structure of the image forming apparatus 50. The image forming apparatus 50 is also provided with a display unit 509 that displays a user interface screen, an operation reception unit 510 that receives user operations, and a communication I / F 511 that is used for communication with the outside.

[0020] FIG. 7 is a diagram showing the functional configuration of the user terminal 30 and the server 10. As shown in FIG. <User terminal functional configuration> The user terminal 30 includes a reception unit 31 that receives operations from a user, an imaging unit 32 that captures an image of the image forming device 50 (see FIG. 1), and a content display unit 33 that displays the captured image captured by the imaging unit 32 together with an image received from the server 10. The reception unit 31 receives information input from the user. For example, the reception unit 31 receives information from the user that identifies the work that the user will perform on the image forming apparatus 50. The reception unit 31 also receives from the user that the user has performed the work and proceeded to the next step. The reception unit 31 transmits the information received from the user to the server 10 via the communication I / F 305 (see FIG. 5). The imaging unit 32 captures an image of the image forming apparatus 50 and transmits the captured image to the server 10 via the communication I / F 305. The content display unit 33 displays the image captured by the imaging unit 32 on the display device 306 (see FIG. 5). Furthermore, the content display unit 33 acquires information about the work procedure from the server 10 and displays the information about the work procedure on the display device 306.

[0021] <Server functional configuration> The server 10 comprises a procedure control unit 11 that determines which step the user is performing the work at; a feature point extraction unit 12 that extracts feature points of the object the user is performing the work at from the captured image acquired from the user terminal 30; a feature point comparison unit 13 that compares the feature points extracted by the feature point extraction unit 12 with predetermined feature points and determines whether the predetermined conditions are met; an image recognition unit 14 that performs image recognition processing on the acquired captured image and determines whether the predetermined conditions are met; a content selection unit 15 that selects content to be displayed on the user terminal 30 based on the determinations of the feature point comparison unit 13 and the image recognition unit 14; and a memory unit 16 that stores information regarding the work procedures.

[0022] The procedure control unit 11 acquires information identifying the work to be performed by the user on the image forming device 50, and acquires the work procedure from information about the work procedure stored in the storage unit 16. The procedure control unit 11 then identifies the execution procedure. The execution procedure can be identified in various ways, and for example, the execution procedure may be identified based on whether or not the image of the image forming device 50 received from the user terminal 30 is similar to an image predetermined for each procedure. Alternatively, the user may determine the progress of the work procedure and grasp the execution procedure based on the result of the user's determination. Furthermore, information detected by a sensor included in the image forming apparatus 50 may be acquired, and the execution procedure may be determined based on the information acquired from the sensor. For example, the execution procedure may be determined based on information detected by a sensor that recognizes the open / close state of a door included in the image forming apparatus 50, or information detected by a sensor that recognizes that a print medium is caught in a conveyance path, which is one of the internal structures of the image forming apparatus 50.

[0023] The feature point extraction unit 12 extracts feature points from the captured image acquired from the user terminal 30, which serve as references for the position at which execution procedure information is to be displayed. These feature points are feature quantities based on the shape of the object on which the user is working. Feature points are calculated from local regions of the captured image, and, for example, edges and corners within the image are extracted. Methods for extracting feature points include, for example, SIFT (Scale Invariant Feature Transform) and SURF (Speeded Up Robust Features). Here, the feature points only need to determine the position at which the AR content is to be displayed, and the method for extracting feature quantities is not particularly limited. AR markers may be used as feature points. The AR markers are, for example, two-dimensional images attached to the image forming device 50.

[0024] The feature point comparison unit 13 compares the feature points extracted by the feature point extraction unit 12 with feature point data stored in the storage unit 16, which will be described later. Depending on the comparison result, it is determined whether the extracted feature points match those of the feature point data. Here, "matching" does not necessarily mean a perfect match; for example, it may be determined that the feature points match when the number or percentage of matching feature points is greater than a predetermined number.

[0025] The image recognition unit 14 performs image recognition on the acquired captured image and calculates the similarity with an image stored in the storage unit 16 (described later). The image recognition unit 14 also performs character recognition processing on the acquired captured image and calculates the similarity between the recognized characters and the characters stored in the storage unit 16 (described later).

[0026] The content selection unit 15 selects content to be displayed on the user terminal 30 based on the determination by the feature point comparison unit 13 as to whether the feature points match and the similarity calculated by the image recognition unit 14. Then, the content selection unit 15 outputs the selected content from the server 10 to the user terminal 30. The storage unit 16 stores work procedure information in which procedures are predetermined for each work. The work procedure information includes image and text information to be presented to the user for each procedure to inform the user of the work procedure.

[0027] <Configuration of work procedure information> FIG. 8 is a table for explaining an example of information related to work procedures. Information on work procedures is prepared for each type of work. A work type is a work that has been predetermined as a set of procedures. Specifically, for example, information on work procedures is stored for each work to clear a paper jam that has occurred in the image forming device 50, to replace toner in the image forming device 50, and inspection work for the image forming device 50. Then, a procedure for the work is predetermined for each type of work. The example in Figure 8 is information about the work procedure for a work type whose "work name" is "paper jam." The "paper jam" work is divided into five steps, and the step numbers are set to 1 to 5. The user performs the work of the step with step number 1, then the work of the step with step number 2. Then, the user performs the work in order of the step numbers until all the steps are completed.

[0028] The information regarding the work procedure is stored in such a manner that, for each procedure number, an existing manual, AR feature point data, AR content, image recognition means, non-AR comparison data, and non-AR content are associated with each other. An existing manual is a manual exchanged over a computer communication network, and is prepared, for example, in HTML format, to display work procedures on the user terminal 30. It is composed of text, images, etc. to inform the user of the work procedures. The existing manual is displayed in an area separate from the area in which the captured image is displayed. The existing manual is an example of non-specific content. The AR feature point data is data to be compared with feature points of an object on which a user is working, which are extracted from a captured image. The AR content has display content and a display position based on the feature points of the object on which the user is working. The display content is execution procedure information displayed on the user terminal 30. For example, it may be a graphic such as an arrow, an image, text, or a combination thereof. The display content is not limited to a still image, but may also be a video. The display position based on the feature point is a position corresponding to the feature point of the object on which the user is working, and is a position based on the position of the object in the captured image. That is, when feature points are extracted from the video received from the user terminal 30, the execution procedure information is displayed at a position corresponding to the feature point, and when the feature point of the image forming device 50 in the video moves, the execution procedure information is displayed to follow the movement.

[0029] The image recognition priority order indicates the order in which image recognition methods are used to determine whether to display non-AR content regardless of the position of feature points in the captured image when feature points cannot be identified from the captured image. In the example of Figure 8, image recognition methods 1 and 2 are assigned as priorities for each task. "Similar" is a method for calculating the degree of similarity between a captured image and a predetermined image. "Character" is a method for calculating the degree of similarity between extracted characters obtained by performing character recognition processing on a captured image and predetermined characters. "None" indicates a state in which no image recognition method is defined. When image recognition methods 1 and 2 are defined, this means that the image recognition method for method 1 is used first, followed by the image recognition method for method 2. This allows, for example, character recognition to be prioritized as the image recognition method when characters are displayed on a display device attached to the target of the task in a certain task.

[0030] The non-AR comparison data is data to be compared when using image recognition means, and is images and characters that are predetermined for each work procedure. The non-AR content has display content and a display position relative to the window. The display content is content that is displayed on the user terminal 30 to indicate the task that is currently being performed. For example, it is text information or image information. The display content is not limited to a still image and may be a video. The display position relative to the window is, for example, specified as "top right" with respect to the upper right area of ​​the window relative to the window. In this embodiment, non-AR content also includes content that is displayed superimposed without being aligned with a feature point in a captured image. For example, this is a mode in which an image is displayed superimposed according to the position of the window, rather than based on the position of a feature point of image forming device 50 in a captured image. Specifically, this is a mode in which a semi-transparent image is displayed superimposed on the "full surface" of the window, or a mode in which a semi-transparent image is displayed superimposed in the upper right area of ​​the window.

[0031] In FIG. 8, step number 1 is associated with "Text 1" and "Existing Manual Image 1" as existing manuals. Also, step number 1 is associated with "Feature Point Data 1" as AR feature point data, "AR Content 1" as the display content of the AR content, and "Display Position 1" as the display position based on the feature points of the AR content. Furthermore, step number 1 is associated with "Similar" as the first image recognition priority and "None" as the second image recognition priority. Furthermore, step number 1 is associated with "Comparative Image 1" as non-AR comparison data, "Non-AR Content 1" as the display content of the non-AR content, and "Top Right" as the display position based on the window.

[0032] The process performed by the server 10 will be described below. <Server device processing> FIG. 9 is a flowchart showing an example of a process in which the server 10 outputs content to be displayed on the user terminal 30 based on an image captured by the image forming device 50. Hereinafter, an example of a process in which the server 10 outputs content to be displayed on the user terminal 30 based on an image captured by the image forming device 50 will be described with reference to FIGS.

[0033] The operation navigation system 1 is started, for example, when the user operates the user terminal 30 to execute an application for using the operation navigation system 1. First, the procedure control unit 11 acquires information for identifying the work to be performed by the user on the image forming apparatus 50 from the user terminal 30, and refers to the work procedure information stored in the storage unit 16 (step 401). Next, the feature point extraction unit 12 acquires an image captured by the image forming apparatus 50 from the user terminal 30, and extracts feature points from the acquired captured image (step 402). Then, the feature point comparison unit 13 determines whether the feature points extracted in step 402 match the feature point data linked to the current procedure (step 403). If the feature point comparison unit 13 determines that the feature points extracted in step 402 match the feature point data (YES in step 403), the content selection unit 15 outputs the AR content to the user terminal 30 (step 404). On the other hand, if the feature point comparison unit 13 determines that the feature points do not match the feature point data in step 403 (NO in step 403), the content selection unit 15 outputs the existing manual linked to the current procedure to the user terminal 30 (step 405). When the AR content or the existing manual is output to the user terminal 30 in step 404 or step 405, the procedure control unit 11 determines whether or not the procedure has proceeded to the next step (step 406). If it determines that the procedure has not proceeded to the next step (NO in step 406), the procedure control unit 11 returns to step 402, acquires a captured image, and extracts feature points (step 402). If it determines that the procedure has proceeded to the next step in step 406 (YES in step 406), the procedure control unit 11 determines whether or not a next step exists (step 407). If it determines that a next step exists (YES in step 407), the procedure control unit 11 returns to step 402, acquires a captured image, and extracts feature points (step 402). If the procedure control unit 11 determines in step 407 that a next step does not exist (NO in step 407), all steps of the work being performed have been completed, and the process ends.

[0034] According to the process of FIG. 9, if the feature points extracted from the captured image do not match the feature point data linked to the current procedure, the existing manual linked to the current procedure is output. As described above, in this embodiment, if the extracted feature points do not match, the AR content is not output, thereby preventing the AR content from being displayed in an unexpected position. If the feature points cannot be extracted, the existing manual linked to the current procedure is output. The user continues working by referring to the output existing manual.

[0035] In the process of Figure 9, an existing manual is output if the extracted feature points do not match, but if the extracted feature points do not match, image recognition can be further performed to select the content to be output. FIG. 10 is a flowchart showing an example of a process in which the server 10 performs image recognition and selects content to be output when the extracted feature points do not match. An example of a process in which the server 10 outputs display information to the user terminal 30 based on a captured image will be described below with reference to FIGS.

[0036] First, the procedure control unit 11 acquires information for identifying the work that the user is to perform on the image forming apparatus 50 from the user terminal 30 (step 601). Next, the feature point extraction unit 12 acquires from the user terminal 30 an image of the image forming apparatus 50 captured by the user terminal 30, and performs processing to extract feature points (step 602).

[0037] Then, the feature point comparison unit 13 determines whether the feature points extracted in step 602 match the feature point data linked to the current procedure (step 603). If the feature point comparison unit 13 determines that the feature points extracted in step 602 match the feature point data (YES in step 603), the content selection unit 15 outputs the AR content linked to the current procedure to the user terminal 30 (step 604). On the other hand, if the feature point comparison unit 13 determines that the feature points do not match the feature point data in step 603 (NO in step 603), the image recognition unit 14 determines whether the captured image is similar to the non-AR comparison data linked to the current procedure (step 605). Here, the determination of similarity is made, for example, based on whether the similarity calculated by the image recognition unit 14 is greater than a predetermined value. The condition used to determine similarity is an example of a "predetermined condition." If the image recognition unit 14 determines that the images are similar (YES in step 605), the content selection unit 15 outputs the non-AR content to the user terminal 30 (step 606). If the image recognition unit 14 determines that the images are not similar (NO in step 605), the content selection unit 15 outputs the existing manual (step 607).

[0038] Then, after outputting display information to the user terminal 30 in step 604, step 606, or step 607, the procedure control unit 11 determines whether or not the procedure has proceeded to the next step (step 608). If it determines that the procedure has not proceeded to the next step (NO in step 608), the procedure control unit 11 returns to step 602, acquires a captured image, and extracts feature points (step 602). If it determines in step 608 that the procedure has proceeded to the next step (YES in step 608), the procedure control unit 11 determines whether or not a next step exists (step 609). If it determines that a next step exists (YES in step 609), the procedure control unit 11 returns to step 602, acquires a captured image, and extracts feature points (step 602). On the other hand, if the procedure control unit 11 determines in step 609 that a next step does not exist (NO in step 609), all steps in the work being performed have been completed, and the process ends.

[0039] As described above, in the process of FIG. 10, image recognition is performed, and if the image is similar to a predetermined image, predetermined non-AR content is output. To superimpose AR content at a specific position within the video, information is required to identify the position at which the AR content is to be displayed, such as by using feature points within the video. However, when viewed in units of work steps, there may be multiple pieces of content that do not need to be superimposed at a specific position within the video but are useful for informing the user of the work content. In the process of FIG. 10, one of the multiple pieces of useful content is selected based on the results of image recognition and output to the user terminal 30.

[0040] 11 is a flowchart showing an example of a process in which the server 10 determines the execution procedure of a task based on a captured image. Hereinafter, an example of a process in which the server 10 determines the execution procedure of a task will be described with reference to FIGS. 7 and 11. First, the procedure control unit 11 acquires, from the user terminal 30, information for identifying the work that the user will perform on the image forming device 50 (step 701). Next, the feature point extraction unit 12 acquires, from the user terminal 30, an image of the image forming device 50 captured by the user terminal 30, and performs processing to extract feature points (step 702).

[0041] Then, the feature point comparison unit 13 determines whether the feature points extracted in step 702 match the feature point data linked to the current procedure (step 703). If the feature point comparison unit 13 determines that the feature points extracted in step 702 match the feature point data (YES in step 703), the content selection unit 15 outputs the AR content linked to the current procedure to the user terminal 30 (step 704). Then, the procedure control unit 11 determines whether the procedure has progressed to the next procedure (step 709), and if it determines that the procedure has not progressed to the next procedure (NO in step 709), the process returns to step 702, and the feature point extraction unit 12 extracts feature points from the captured image. On the other hand, if it is determined in step 709 that the process has proceeded to the next step (YES in step 709), the procedure control unit 11 determines whether or not the next step exists (step 710). If it is determined that the next step exists (YES in step 710), the process returns to step 702, where the feature point extraction unit 12 extracts feature points from the captured image. If it is determined in step 710 that there is no next step (NO in step 710), all steps in the work being performed have been completed, and the process ends.

[0042] In step 703, if the feature point comparison unit 13 determines that the feature points extracted in step 702 do not match the feature point data (NO in step 703), the image recognition unit 14 calculates similarity 1 with the non-AR comparison data (see FIG. 8) associated with the current procedure and similarity 2 with the non-AR comparison data associated with the next procedure (step 705). Then, the procedure control unit 11 compares similarity 1 and similarity 2 calculated by the image recognition unit 14 to determine whether similarity 1 is greater (step 706). If it is determined in step 706 that similarity 1 is greater (YES in step 706), the procedure control unit 11 outputs the non-AR content (step 707). Then, the procedure control unit 11 performs processing in step 709. On the other hand, if it is determined in step 706 that similarity 2 is greater (NO in step 706), the procedure control unit 11 advances the current procedure to the next procedure (step 708). Then, the procedure control unit 11 performs processing in step 710.

[0043] Note that the processing in step 706 in the flowchart of FIG. 11 is determined based only on a comparison between similarity 1 and similarity 2. Therefore, if both similarity 1 and similarity 2 are small, the actual procedure may not be the execution procedure or the next procedure. Therefore, even if similarity 1 is greater than similarity 2, if similarity 1 is smaller than a predetermined size, non-AR content may not be output. Furthermore, even if similarity 2 is greater than similarity 1, if similarity 2 is smaller than a predetermined size, the processing in step 709 may be performed without advancing the current procedure to the next procedure. Furthermore, when calculating similarity 1 and similarity 2, similarity 1 and similarity 2 were calculated using non-AR comparison data, but similarity may also be calculated using images from existing manuals or images included in non-specific content, as long as images for calculating similarity for each procedure are predetermined.

[0044] <Other variations> Although the image forming device 50 has been given as an example of an object on which a series of operations are to be performed, various other devices can be handled by the operation navigation system 1 of this embodiment as long as the object is an object on which the procedure of the operation performed by the user is predetermined. For example, a refrigerator, a personal computer (PC), a car, etc. can be taken as examples of objects on which the user performs operations. The user terminal 30 may be a smart glass or a head-mounted display, rather than a smartphone. A user wearing the glasses-type terminal can view images in which virtual images and text are superimposed on real objects, allowing them to check information about work procedures alongside the real work object.

[0045] In the present embodiment, the work procedure information has been described using a relational database format as shown in Fig. 8, but the work procedure information may be stored in other formats. For example, the work procedure information may be stored in XML (Extensible Markup Language). If an XML format manual is already available, AR feature point data and AR content can be stored in the existing XML format manual by tagging each procedure, for example. [Explanation of symbols]

[0046] 1...operation navigation system, 10...server, 11...procedure control unit, 12...feature point extraction unit, 13...feature point comparison unit, 14...image recognition unit, 15...content selection unit, 16...storage unit, 30...user terminal, 31...reception unit, 32...imaging unit, 33...content display unit, 50...image forming device, 70...network, 101...CPU, 801a, 801b, 901...execution procedure information

Claims

1. a processor; The processor: Acquire a target image, which is an image of a target performing a task for which a series of operations is defined; For each step of the series of operations, recognize whether the target image is in a specific situation that is pre-associated with the step; When the specific situation occurs, the target image is displayed with specific content that is pre-associated with the specific situation superimposed thereon; When the specific situation does not exist, displaying non-specific content that is pre-associated with the procedure; The recognition of whether the target image is the specific situation is performed based on feature points of the target; the specific situation is a situation where a position where the specific content is to be superimposed and displayed can be recognized from the feature points of the object; When the target image is in the specific situation, the specific content is superimposed at the display position and the target image is displayed; If the display position cannot be recognized, image recognition of the target image is further performed; When the result of the image recognition satisfies a predetermined condition, the information processing device superimposes the non-specific content associated with the predetermined condition and displays the target image.

2. 2. The information processing apparatus according to claim 1, wherein two or more different image recognition methods are prepared for the image recognition, and a priority order for using the image recognition method is assigned to each step of the series of operations.

3. A processor is provided, The processor: Acquire a target image, which is an image of a target performing a task for which a series of operations is defined; For each step of the series of operations, recognize whether the target image is in a specific situation that is pre-associated with the step; When the specific situation occurs, the target image is displayed with specific content that is pre-associated with the specific situation superimposed thereon; When the specific situation does not exist, displaying non-specific content that is pre-associated with the procedure; By performing image recognition on the target image, one of the procedures is determined as an execution procedure to be executed in the work; calculating a first similarity indicating a degree of similarity between an image previously associated with the execution procedure and the target image; calculating a second similarity indicating a degree of similarity between an image previously associated with a subsequent step in the execution procedure and the target image; The information processing apparatus is characterized in that the execution procedure is updated based on the calculated first similarity and second similarity.

4. On the computer, A function of acquiring a target image, which is an image of a target performing a task for which a series of operations is defined; a function of recognizing, for each step of the series of operations, whether the target image is in a specific situation that is pre-associated with the step; a function of displaying the target image with specific content that is pre-associated with the specific situation superimposed thereon when the specific situation occurs; a function of displaying non-specific content associated with the procedure in advance when the specific situation does not exist; Realize this, The recognition of whether the target image is the specific situation is performed based on feature points of the target; the specific situation is a situation where a position where the specific content is to be superimposed and displayed can be recognized from the feature points of the object; When the target image is in the specific situation, the specific content is superimposed at the display position and the target image is displayed; If the display position cannot be recognized, image recognition of the target image is further performed; A program for displaying the target image by superimposing the non-specific content associated with the predetermined condition when the result of the image recognition satisfies the predetermined condition.

5. A computer, A function of acquiring a target image, which is an image of a target performing a task for which a series of operations is defined; a function of recognizing, for each step of the series of operations, whether the target image is in a specific situation that is pre-associated with the step; a function of displaying the target image with specific content that is pre-associated with the specific situation superimposed thereon when the specific situation occurs; a function of displaying non-specific content associated with the procedure in advance when the specific situation does not exist; Realize this, By performing image recognition on the target image, one of the procedures is determined as an execution procedure to be executed in the work; calculating a first similarity indicating a degree of similarity between an image previously associated with the execution procedure and the target image; calculating a second similarity indicating a degree of similarity between an image previously associated with a subsequent step in the execution procedure and the target image; a program for updating the execution procedure based on the calculated first similarity and second similarity;

Images