Image processing apparatus, image forming apparatus, and image processing method

Abstract

To provide an image processing apparatus, an image forming apparatus, and an image processing method that can effectively prevent information leakage from a document.SOLUTION: An image processing apparatus 300 according to an aspect of the present invention comprises authentication units 313, 314 and an output unit 316. In response to receiving a request for outputting an AR object corresponding to an AR marker shown on a recording medium from an augmented reality device D having an imaging function, the authentication units 313, 314 perform authentication using a picked-up image transmitted from the augmented reality device D. In response to the authentication units 313, 314 performing the authentication, the output unit 316 outputs the AR object related to the output request to the augmented reality device D.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus, an image forming apparatus, and an image processing method. [Background technology]

[0002] In recent years, there has been growing interest in the risk of information leakage from various documents, and new technologies have been developed to prevent information leakage from documents.

[0003] For example, Patent Document 1 discloses an image processing technology that converts confidential information in a document into AR markers, allowing only users who view the document through an augmented reality device (such as AR glasses) that can read the AR markers to view the confidential information in the document. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-004346 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the conventional image processing technology described above, anyone can view confidential information simply by wearing an augmented reality device, leaving room for improvement in document security.

[0006] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a secure image processing apparatus, image forming apparatus, and image processing method that can effectively prevent information leakage from documents. [Means for solving the problem]

[0007] According to one aspect of the present invention, an image processing device includes an authentication unit and an output unit. In response to receiving, from an augmented reality device having an image capture function, a request to output an AR object corresponding to an AR marker displayed on a recording medium, the authentication unit attempts authentication using the captured image transmitted from the augmented reality device. In response to authentication by the authentication unit, the output unit outputs the AR object related to the output request to the augmented reality device.

[0008] According to one aspect of the present invention, an image forming apparatus includes an image forming unit and a communication unit capable of communicating with the image processing device, wherein the image forming unit forms, on a recording medium, an image represented by document image data generated by scanning a document. The communication unit is capable of communicating with the image processing device. The image forming unit forms, on the recording medium, an image represented by replaced image data transmitted from the image processing device and in which the confidential information is replaced with the AR marker.

[0009] According to one aspect of the present invention, an image processing method is executed by a computer having a processor, and the processor executes an authentication step and an output step. In the authentication step, in response to receiving, from an augmented reality device having an image capture function, a request to output an AR object corresponding to an AR marker displayed on a recording medium, authentication is attempted using a captured image transmitted from the augmented reality device. In the output step, in response to authentication by the authentication unit being successful, the AR object related to the output request is output to the augmented reality device. [Effects of the Invention]

[0010] According to the image processing device, image forming device, and image processing method of the present invention, it is possible to effectively prevent information leakage from documents. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic diagram of an image processing system according to an embodiment of the present invention. [Figure 2] 2 is a block diagram showing the hardware configuration of the augmented reality device shown in FIG. 1. FIG. [Figure 3A] FIG. 2 is a block diagram showing a hardware configuration of the image processing device shown in FIG. [Figure 3B] FIG. 2 is a block diagram showing a functional configuration of the image processing device shown in FIG. [Figure 4] FIG. 2 is a schematic perspective view of the image forming apparatus shown in FIG. [Figure 5] FIG. 5 is a block diagram of the image forming apparatus shown in FIG. [Figure 6] FIG. 5 is a schematic perspective view of the image reading device shown in FIG. [Figure 7] 2 is a flowchart of an output process of an AR object using the image processing system shown in FIG. 1. [Figure 8] 1. FIG. 4 is a flowchart of output processing of replaced image data using the image processing system shown in FIG. [Figure 9] 2 is a flow diagram of an output process of updated image data using the image processing system shown in FIG. 1. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0012] An embodiment of an image processing system 1 according to the present invention will now be described with reference to the drawings. In the drawings, the same or corresponding parts will be designated by the same reference characters and description thereof will not be repeated.

[0013] (1) Overall configuration of image processing system 1 First, with reference to Fig. 1, the overall configuration of an image processing system 1 (hereinafter simply referred to as system 1) according to the present invention will be described. Fig. 1 is a schematic diagram of system 1 according to one embodiment of the present invention. As shown in Fig. 1, system 1 includes a user terminal U, an augmented reality device D, an image forming apparatus 200, and a server 300. The user terminal U, the augmented reality device D, the image forming apparatus 200, and the server 300 are communicatively connected to one another via a network NW. The server 300 is an example of an "image processing apparatus" in the present invention.

[0014] System 1 is a system that uses AR (Augmented Reality) technology to convert at least a portion of information on a page (recording medium) into an AR object that can only be viewed through an augmented reality device D. That is, by performing the processing described below, system 1 can replace confidential information contained in existing manuscript image data with AR markers and then form an image on the page (recording medium) showing the AR markers.

[0015] For example, in the example shown in the lower part of Fig. 1, replaced image data is generated based on the document image data of the original document, in which confidential information C is replaced with AR markers M. In the following description, the recording medium on which the replaced image in which confidential information is replaced with AR markers is formed is referred to as the replaced document.

[0016] When viewing the replaced manuscript with the AR marker M displayed, the user wears an augmented reality device D (e.g., AR glasses) and views the replaced image while holding the manuscript. At this time, characters or designs corresponding to confidential information are displayed as AR objects generated by AR technology in the real space (on the manuscript) seen through the augmented reality device. The configuration of such a system 1 will be described in detail below.

[0017] (1-1) User terminal U The user terminal U is an information processing terminal used by a user. The user terminal U is realized by a tablet terminal, a smartphone, a desktop PC (Personal Computer), a laptop PC, or the like that has a processor and is compatible with the system 1. In this embodiment, a tablet terminal will be described as an example of the user terminal U. In the example of FIG. 1, one user terminal U is connected to the network NW, but multiple user terminals U may also be connected to the network NW.

[0018] (1-2) Augmented reality device D Next, the configuration of the augmented reality device D will be described with reference to Fig. 2. The augmented reality device D is a terminal device that implements augmented reality technology. The augmented reality technology is a technology that displays preset virtual information (AR objects) superimposed on real space.

[0019] An example of the augmented reality device D is AR glasses (Augmented Reality Glass). AR glasses are eyeglass-type wearable devices that display text and images that represent AR objects superimposed on the real-world scenery in front of the user.

[0020] Fig. 2 is a block diagram showing the hardware configuration of the augmented reality device D shown in Fig. 1. As shown in Fig. 2, the augmented reality device D mainly includes a display D1, a sensor D2, a processor D3, a battery D4, a communication unit D5, and a user interface (UI) D6.

[0021] The display D1 constitutes a screen for displaying AR objects and other information (e.g., information related to operations) to the user wearing the device. The display D1 is formed integrally with the lenses of the augmented reality device D. The display D1 is transparent, and displays virtual information (AR objects) superimposed on the real world without obstructing the view through the lenses of the user wearing the device.

[0022] The sensor D2 detects the environment around the user wearing the device and the user's movements, and provides that information to the AR system. In this embodiment, the augmented reality device D includes at least a camera capable of capturing images and videos. The augmented reality device D may also include other sensors, such as a gyroscope, a magnetic sensor, a distance sensor, and a position sensor.

[0023] The processor D3 is a computer built into the augmented reality device D and processes sensor data (e.g., images captured by a camera). The processor D3 also generates and displays AR objects output from the server 300. The battery D4 is an internal power source that supplies the power necessary for the operation of the augmented reality device D.

[0024] The communication unit D5 is a communication module that transmits and receives various types of data by wireless communication via the network NW. Various wireless communication technologies such as wireless LAN or short-range wireless communication can be applied to the communication unit D5.

[0025] The user interface (UI) D6 is an input device that accepts operations from the user. The user interface D6 uses functions such as operation buttons, voice recognition, and gesture recognition.

[0026] (1-3) Hardware configuration of server 300 Next, the hardware configuration of the server 300 will be described with reference to FIG. 3. The server 300 is an information processing device that processes images using AR technology. The server 300 includes various computers such as a personal computer and a server computer (for example, a web server, an application server, a database server, or a combination thereof). In this embodiment, the server 300 will be described as an example of a cloud server that makes one or more virtualized servers available via a network.

[0027] The server 300 performs various calculation processes (described later) to convert the information included in the document image that has been designated as confidential by the user into an AR object, and then generates and outputs an identification image (AR marker) associated with the AR object.

[0028] In addition, in response to receiving a request from the augmented reality device D to output an AR object based on an AR marker, the server 300 performs authentication regarding the output of the AR object as described below, and then outputs the AR object related to the output request to the augmented reality device D.

[0029] 3A is a block diagram showing the hardware configuration of the server 300. As shown in FIG. 3A, the server 300 includes a processor 310, a memory 320, a storage 330, a communication IF 340, and an input / output IF 350.

[0030] The processor 310 is hardware for executing a set of instructions written in a program stored in the storage 330. The processor 310 includes a central processing unit (CPU). The processor may also include an application specific integrated circuit (ASIC).

[0031] The memory 320 is for temporarily storing programs and data to be processed by the programs, and is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0032] The storage 330 is a storage device for saving programs and data, such as a flash memory or a hard disk drive (HDD). The programs stored in the storage 330 include, for example, the following programs. OS (Operating System) programs Applications that process information (e.g. web browsers)

[0033] The data stored in storage 330 includes, for example, the following data: Databases referenced in information processing Data obtained by performing information processing (i.e., the results of performing information processing)

[0034] The communication IF 340 is an interface for inputting and outputting signals so that the server 300 can communicate with external devices.

[0035] The input / output IF 350 functions as an interface with an input device for accepting input operations via the user terminal U and an output device for presenting information to the user terminal U.

[0036] (2) Functional configuration of server 300 Next, the functional configuration of server 300 will be described with reference to Fig. 3B. Fig. 3B is a block diagram showing the functional configuration of server 300. As shown in Fig. 3B, server 300 functions as a communication unit 301, a storage unit 302, and a control unit 303.

[0037] (2-1) Communications Department 301 The communication unit 301 performs processing for the server 300 to communicate with external devices.

[0038] (2-2) Storage section 302 The storage unit 302 stores data and programs used by the server 300. The storage unit 302 stores, for example, the following data and programs: ·Object Database (Object DB) 3021 Output Condition Database (Output Condition DB) 3022

[0039] The object DB 3021 is a database in which at least the following information is stored in association with one another: New records are recorded in the object DB 3021 by the generation process of an AR object, which will be described later. - Identification information of the AR marker (or AR object) AR Marker AR Objects That is, the object DB 3021 stores a plurality of combinations of AR objects and AR markers.

[0040] The identification information is an identifier that can identify a combination of an AR marker and an AR object, and is expressed, for example, by a character string that is a combination of letters and numbers.

[0041] The AR marker is an identification image that can identify an AR object and is represented, for example, by a two-dimensional code. Note that the AR marker may be represented in a data format other than a two-dimensional code. The AR marker includes, for example, identification information and information indicating the display position of the AR marker in the replaced manuscript.

[0042] The AR object is generated based on information contained in the document that has been designated as confidential by the user (including letters, numbers, figures, photographs, etc.), and is information equivalent to this content. Based on an output request from the augmented reality device D, the AR object is projected on the display D1 of the augmented reality device D so as to overlap with the real space (the surface of the replaced manuscript) visible through the display D1.

[0043] The output condition DB 3022 is a database in which at least the following information is stored in association with one another: New records are recorded in the output condition DB 3022 by the generation process of an AR object, which will be described later. - Identification information of the AR marker (or AR object) Output conditions corresponding to the identification information of the AR marker (or AR object)

[0044] The output condition is a predetermined condition required when outputting an AR object related to the identification information. The output condition specifies, for example, the type of object that should appear in the captured image captured by the camera of the augmented reality device D. The object that should appear in the captured image includes the hand holding the replacement manuscript viewed by the user wearing the augmented reality device D. In this case, the server 300 can output the AR object corresponding to the AR marker only when the replacement manuscript on which the AR marker is displayed is held by the user wearing the augmented reality device D.

[0045] (2-3) Control unit 303 The control unit 303 functions as various modules as a result of the processor 310 of the server 300 performing processing according to a program. The control unit 303 functions as a transmission / reception control module 311, a reception module 312, a request module 313, a determination module 314, a generation module 315, and an output module 316.

[0046] The transmission / reception control module 2031 controls the process of the server 300 transmitting and receiving signals to and from external devices in accordance with a communication protocol.

[0047] The reception module 312 receives an output request for an AR object transmitted from the augmented reality device D. The augmented reality device D transmits an output request to the server 300 for the AR object corresponding to the AR marker indicated on the replacement manuscript. The output request includes information about the AR marker. Upon receiving the output request, the reception module 312 inputs the information to the request module 313 to start authentication processing.

[0048] In response to the output request input from the reception module 312, the request module 313 requests the augmented reality device D to provide a captured image including the replacement document. The server 300 then authenticates the user's authority to view the confidential information using the captured image provided by the augmented reality device D that issued the output request. In this way, by requesting the captured image using the request module 313, the state in which the augmented reality device D is being used can be used as information for authentication.

[0049] Specifically, in the system 1, the augmented reality device D captures an image of the replaced manuscript on which the AR marker related to the output request is displayed. Then, the server 300 detects an object included in the captured image and determines whether the object set as the output condition is included in the captured image.

[0050] As described above, in this embodiment, the output condition is that the hand of the user holding the replacement document is reflected in the captured image. Therefore, only users who view the replacement document while wearing the augmented reality device D and holding the replacement document can be authenticated as having the authority to view the document.

[0051] In the system 1, the output condition for the AR object can be set arbitrarily. For example, the output condition may be that both hands of the user appear in the captured image. In this case, the output condition may further be that both hands of the user appear in the captured image in positions that sandwich the document horizontally, as if grasping both ends of the document.

[0052] The determination module 314 shown in FIG. 3B determines whether the captured image provided by the augmented reality device D satisfies a predetermined condition. Specifically, the determination module 314 first performs object detection on the captured image using an object detection model. The object detection model proposes candidate areas for the input captured image and extracts features from the candidate areas. The object detection model then classifies the objects in the candidate areas based on the features and outputs the results. The object detection model is stored in, for example, the storage unit 302.

[0053] The determination module 314 then determines whether the object included in the captured image satisfies the output condition. At this time, the determination module 314 refers to the output condition corresponding to the AR marker related to the output request, among the output conditions defined in the output condition DB 3022. For example, if the output condition defines that the hand of the user holding the replacement document is captured in the image, the determination module 314 determines whether the type of object included in the captured image is a hand.

[0054] If the captured image satisfies the predetermined output conditions, i.e., if the determination module 314 determines that an object defined in the output conditions is captured in the captured image, it authorizes the output of the AR object related to the output request. On the other hand, if the captured image does not satisfy the predetermined output conditions, the determination module 314 denies the authentication of the AR object related to the output request. In this case, the server 300 has not authorized the output of the AR object.

[0055] In this way, the processing of the request module 313 and the determination module 314 in the server 300 attempts authentication using a captured image for the output of an AR object related to the output request transmitted from the augmented reality device D. That is, the request module 313 is an example of a "request unit" included in the "authentication unit" of the present invention, and the determination module 314 is an example of a "determination unit" included in the "authentication unit" of the present invention.

[0056] The output determination by the determination module 314 is repeatedly performed at preset time intervals (for example, every few seconds). That is, if the user lets go of the document while viewing the replacement document containing the AR object, the output of the AR object is stopped. As a result, the AR object is no longer displayed on the display D1 of the augmented reality device D. Therefore, in this embodiment, the AR object is output to the augmented reality device D worn by the user only while the user continues to hold the replacement document in their hand. This makes it possible to effectively prevent information leakage from the document.

[0057] 3B, the generation module 315 converts confidential information included in a range specified by the user into an AR object in the original image data. The generation module 315 then generates an AR marker corresponding to the converted AR object. The newly generated AR marker is associated with the AR object and stored as a new record in the object DB 3021. This allows an AR object corresponding to confidential information to be generated in association with the AR marker.

[0058] In generating an AR object, the generation module 315 analyzes confidential information and formats it for display as an AR object using a platform for AR technology development. For example, when converting text indicating confidential information into an AR object, the generation module 315 applies appropriate textures and colors to the text indicating the text and generates an AR object represented as a 3D model placed in a three-dimensional virtual space.

[0059] When confidential information corresponding to an AR object is updated, the generation module 315 updates the AR object and the AR marker in the object DB 3021. This allows the AR object and the AR marker to be appropriately updated when the confidential information is changed due to correction or the like. The generation module 315 is an example of the "generation unit" of the present invention.

[0060] In response to authentication being permitted by the request module 313 and the determination module 314, the output module 316 outputs the AR object related to the output request to the augmented reality device D. The output module 316 is an example of the "output unit" of the present invention.

[0061] (3) Configuration of Image Forming Apparatus 200 (3-1) Overall structure Next, the configuration of the image forming apparatus 200 will be described with reference to Figures 4 to 6. The image forming apparatus 200 is, for example, a multifunction peripheral having a printer function, a scanner function, and a copy function. The image forming apparatus 200 may also have a facsimile function. In this embodiment, the image forming apparatus 200 forms images by an electrophotographic method.

[0062] Fig. 4 is a schematic perspective view of an image forming apparatus 200 according to one embodiment. Fig. 7 is a block diagram of the image forming apparatus 200 shown in Fig. 5. As shown in Figs. 4 and 5, in this embodiment, the image forming apparatus 200 includes an image reading device 100 and an image forming unit 60.

[0063] The image reading device 100 is disposed above the image forming unit 60. The image reading device 100 includes a document table 10, a cover 30, a stock cover 33, and a document transport unit 90. For ease of understanding, the document transport unit 90 is indicated by a two-dot chain line in FIG. 4.

[0064] The document table 10 is disposed above the image forming unit 60. The document table 10 has a substantially rectangular parallelepiped shape. An original is placed on the document table 10.

[0065] The cover 30 has a thin, approximately rectangular parallelepiped shape. The cover 30 is placed above the platen 10. The cover 30 can be opened and closed relative to the platen 10. In this embodiment, the cover 30 is attached to the platen 10 on the rear side thereof, and can rotate freely around a rotation shaft located at the rear end of the platen 10. Alternatively, the cover 30 may be attached to the platen 10 at the left end thereof, and can rotate freely around a rotation shaft located at the left end of the platen 10.

[0066] The document transport unit 90 is disposed on the upper surface 35 of the cover 30. In this embodiment, the document transport unit 90 is configured integrally with the cover 30. The document transport unit 90 is also called an automatic document feeder or ADF (Auto Document Feeder).

[0067] The stock cover 33 is disposed on the upper surface 35 of the cover 30. The stock cover 33 faces a storage area 35a that is a part of the upper surface 35 of the cover 30. A part of the stock cover 33 is located inside the document transport unit 90.

[0068] The stock cover 33 and the storage area 35a constitute a storage tray 50. That is, the image reading device 100 is equipped with the storage tray 50. The storage tray 50 has an opening 50h. The opening 50h is located on the front side of the storage tray 50. A user can put in and take out an object to be stored through the opening 50h. That is, a user can store an object to be stored inside the storage tray 50, for example. The object to be stored is, for example, a document before or after being read, or an object other than a document (for example, the user's personal belongings).

[0069] The rear side of stock cover 33 is closed. Therefore, when cover 30 is opened relative to platen 10, the objects contained in stock cover 33 can be prevented from falling to the rear side of image forming apparatus 200.

[0070] The image reading device 100 reads an original document using, for example, an original document transport unit 90. The original document transport unit 90 transports the read original document to the upper surface 33u of the stock cover 33. In other words, the original document transported by the original document transport unit 90 is placed on the upper surface 33u of the stock cover 33.

[0071] As shown in FIG. 5, the image forming apparatus 200 further includes a control unit 70, a storage unit 80, an input / output unit 40, an open / close sensor 52, a human sensor 54, and a communication unit 55.

[0072] Typically, the control unit 70 is disposed inside the image forming apparatus 200. The control unit 70 includes an arithmetic element. The arithmetic element includes a processor. For example, the processor includes a central processing unit (CPU). The processor may include an application specific integrated circuit (ASIC).

[0073] The storage unit 80 stores various data, control programs, and various application programs. The storage unit 80 also stores the following information: -Original image data Replacement image data Updated image data

[0074] The document image data is image data (scanned data) of a document that is generated by scanning the surface of the document using the image reading device 100.

[0075] The replaced image data is image data in which the server 300 replaces an area of ​​the original image data that has been designated by the user as confidential information with an AR marker.

[0076] The updated image data is image data in which the AR marker portion of the replacement image data has been updated. Specifically, when the confidential information portion of the document is updated, the AR marker of the replacement image data is updated together with the AR object corresponding to the confidential information portion.

[0077] The storage unit 80 is configured by, for example, a read-only memory (ROM), a random access memory (RAM), and / or a solid state drive (SSD). The storage unit 80 may include an external memory. The external memory is removable media. The storage unit 80 may include, for example, a universal serial bus (USB) memory and / or a secure digital (SD) card as the external memory. The storage unit 80 may also include a cache memory provided in the CPU for temporarily storing data.

[0078] The control unit 70 controls the operation of each component of the image forming device 200 by executing a control program stored in the storage unit 80. Specifically, in response to a selection operation received by the touch panel, the control unit 70 executes a process corresponding to the selected command item. In this way, the control unit 70 controls the image reading device 100, the input / output unit 40, the open / close sensor 52, the human presence sensor 54, the communication unit 55, the storage unit 80, the document transport unit 90, and the image forming unit 60.

[0079] The reading unit 20 generates document image data based on the document reading results as a result of the control unit 70 executing a control program. At this time, the control unit 70 performs optical character recognition processing on the document image data.

[0080] (3-2) Image reading device 100 Next, details of the image reading device 100 will be described with reference to Fig. 6. Fig. 6 is a schematic perspective view of the image reading device 100 shown in Fig. 4. Note that in order to avoid complicating the drawing, the document transport unit 90 and the image forming section 60 are omitted from Fig. 6.

[0081] 6, document table 10 includes contact glass 11, light-transmitting portion 12, and reading slit 13. On document table 10, contact glass 11, light-transmitting portion 12, and reading slit 13 are located apart from one another. For example, contact glass 11 is located in front of light-transmitting portion 12. Furthermore, reading slit 13 is located to the left of contact glass 11 and light-transmitting portion 12.

[0082] The contact glass 11 is disposed on the upper surface of the document table 10. The contact glass 11 is translucent. A document to be read is placed on the contact glass 11.

[0083] The light-transmitting portion 12 is disposed on the upper surface of the document table 10. The light-transmitting portion 12 has light-transmitting properties. The light-transmitting portion 12 is preferably transparent. The light-transmitting portion 12 is disposed, for example, behind the contact glass 11.

[0084] The reading slit 13 is disposed on the upper surface of the document table 10. The reading slit 13 is translucent. The reading slit 13 is used to read the document transported by the document transport unit 90. The reading slit 13 has a first scan sensor (not shown) therein that forms part of the reading section 20. The first scan sensor is an image sensor that reads the back side of the document transported by the document transport unit 90.

[0085] The reading unit 20 is disposed inside the document table 10. The reading unit 20 reads an image of a document placed on the contact glass 11 by a user. The reading unit 20 has a second scan sensor (not shown) disposed therein. The second scan sensor is disposed below the contact glass 11. The second scan sensor is an image sensor used to read the document placed on the contact glass 11.

[0086] The image sensor may be, for example, a charge coupled device (CCD) or a contact image sensor.

[0087] The cover 30 can be opened and closed freely relative to the document table 10. That is, the cover 30 opens and closes relative to the contact glass 11 included in the document table 10. For example, the cover 30 may be provided with a support member that allows the cover 30 to be opened and closed. When the cover 30 is in the "open state," the top surface of the document table 10 is exposed. On the other hand, when the cover 30 is in the "closed state," the top surface of the document table 10 is covered by the cover 30.

[0088] The cover 30 has a light-shielding portion 31 and a light-transmitting portion 32. The light-shielding portion 31 blocks light. The light-shielding portion 31 is formed from, for example, a colored resin.

[0089] The light-transmitting portion 32 has light-transmitting properties. The light-transmitting portion 32 is formed of a light-transmitting resin material such as an acrylic resin. Therefore, light that enters from one side of the light-transmitting portion 32 exits from the other side. Note that the light-transmitting portion 32 may be glass or hollow as long as it has light-transmitting properties.

[0090] The input / output unit 40 is disposed on the front side of the side of the document table 10. A user inputs instructions via the input / output unit 40. The input / output unit 40 accepts the user's instructions. In this embodiment, the user's instructions include the type of job and the number of documents indicating the number of documents.

[0091] For example, when a user inputs an instruction to start a scan job via input / output unit 40, reading unit 20 reads the document and captures an image of the document. Furthermore, when the user inputs an instruction to output a replacement document along with scanning the document, processing described below is executed, and image forming device 200 outputs a page on which an image representing the replacement image data is formed.

[0092] The input / output unit 40 also outputs information to the user. For example, the input / output unit 40 notifies the user of necessary information regarding the control of the image reading device 100 and the image forming unit 60.

[0093] The input / output unit 40 includes a display unit 41 and an input unit 42. The display unit 41 displays an operation screen or the results of various processes. The display unit 41 has a display. For example, the display includes a liquid crystal display or an organic EL display.

[0094] The input unit 42 includes, for example, various keys for specifying the type and content of a job. The input unit 42 includes a keyboard and a mouse. Alternatively, the input unit 42 may include a touch sensor. Operation instructions for the image reading device 100 and the image forming unit 60 are input to the input unit 42.

[0095] The open / close sensor 52 is attached to the document table 10. In this embodiment, the open / close sensor 52 is attached to an area of ​​the top surface of the document table 10 that overlaps with the cover 30 when the cover 30 is closed. More specifically, the open / close sensor 52 is disposed on the same surface as the contact glass 11. The open / close sensor 52 detects whether the cover 30 is open or closed.

[0096] The human presence sensor 54 detects whether or not a person is present within the detection range of the human presence sensor 54. The human presence sensor 54 is attached to the platen 10. The human presence sensor 54 transmits a detection signal to the control unit 70 (see FIG. 5). For example, the human presence sensor 54 is a pyroelectric infrared sensor, and detects far infrared rays (radiant heat) emitted from the human body.

[0097] The communication unit 55 performs processing for the image forming apparatus 200 to communicate with an external device (for example, the server 300). The communication unit 55 transmits document image data to the server 300 and receives replaced image data transmitted from the server 300 in which confidential information has been replaced with an AR object.

[0098] (3-3) Image forming unit 60 Next, the configuration of the image forming unit 60 will be described with reference to Fig. 5. The image forming unit 60 performs predetermined editing by forming an image on a recording medium. The recording medium is, for example, paper such as a manuscript. The image forming unit 60 forms images indicated by the manuscript image data, replacement image data, and update image data on the paper surface (recording medium).

[0099] As shown in FIG. 5, the image forming unit 60 includes a transfer unit 61, an image carrier 62, a charging unit 63, an exposure unit 64, a developing unit 65, a fixing unit 66, a discharge unit 67, a storage unit 68, and a conveying unit 69.

[0100] The control unit 70 controls the image forming unit 60 to form a specified toner image on the document. That is, the control unit 70 controls the transport unit 69 so that the transport unit 69 transports the document stored in the storage unit 68. The storage unit 68 stores documents before images are formed on them. The documents are, for example, plain paper, recycled paper, thin paper, thick paper, coated paper, or overhead projector (OHP) sheets.

[0101] In addition, the control unit 70 controls the image carrier 62, the charging unit 63, and the exposure unit 64 so that the charging unit 63 charges the rotating image carrier 62 and the exposure unit 64 exposes the rotating image carrier 62 to light to form an electrostatic latent image based on the image.

[0102] The control unit 70 also controls the developing unit 65 so that the developing unit 65 forms a toner image by developing the electrostatic latent image on the image carrier 62. Furthermore, the control unit 70 controls the conveying unit 69 and the transfer unit 61 so that the toner image formed on the image carrier 62 is transferred to the document via the intermediate transfer body of the transfer unit 61.

[0103] Furthermore, the control unit 70 controls the conveying unit 69 and the fixing unit 66 so as to fix the toner image transferred onto the document onto the document. The control unit 70 controls the heating roller and pressure roller of the fixing unit 66 to rotate, and also controls the heating roller to heat to a predetermined temperature.

[0104] Furthermore, the control unit 70 controls the conveying unit 69 to discharge the document onto which the toner image has been fixed from the discharge unit 67 to the outside of the image forming apparatus 200. In this manner, the image forming unit 60 forms a predetermined image on the paper surface and outputs the paper surface on which the image has been formed.

[0105] (4) System 1 processing Next, the processing of the system 1 will be described with reference to FIGS.

[0106] (4-1) AR object output processing First, the output processing of an AR object using the system 1 will be described with reference to Fig. 7. The output processing of an AR object is a procedure in which the server 300 outputs an AR object corresponding to an AR marker displayed on a replacement manuscript to the augmented reality device D. Fig. 7 is a flow diagram of the output processing of an AR object using the system 1.

[0107] 7, in the AR object output process using the server 300, first, the augmented reality device D recognizes the AR marker (step S101). Specifically, when a user wearing the augmented reality device D visually recognizes the page of the replaced manuscript on which the AR marker is displayed, the augmented reality device D detects the AR marker displayed on the page.

[0108] Next, the augmented reality device D makes an output request for the AR object (step S102). Specifically, the augmented reality device D uses the identification information included in the detected AR marker to send an output request for the AR object corresponding to the AR marker to the server 300. The output request includes the identification information capable of identifying the AR object and the display position in the replaced manuscript.

[0109] Next, the server 300 accepts an output request (step S201). Specifically, the reception module 312 of the server 300 accepts the output request transmitted from the augmented reality device D via the transmission / reception control module 311. The reception module 312 inputs the accepted output request to the request module 313.

[0110] Next, the server 300 requests the captured image (step S202). Specifically, the request module 313 of the server 300 transmits, via the transmission / reception control module 311, a signal requesting the augmented reality device D to transmit the captured image including the document with the AR marker indicated thereon.

[0111] Next, the augmented reality device D acquires a captured image (step S103). Specifically, the augmented reality device D uses a camera to capture an image in front of the user, including the original image, and acquires the captured image. The captured image includes the replaced image being viewed by the user and the state of its surroundings. The captured images may be acquired as a captured video by continuously acquiring them over time.

[0112] When capturing the captured image, the augmented reality device D may display a message on the display D1 informing the user that the captured image will be captured by the camera. This can encourage the user to assume a posture that allows the replaced image to be appropriately included in the camera's angle of view.

[0113] Next, the augmented reality device D transmits the captured image (step S104). Specifically, the augmented reality device D transmits the acquired captured image to the server 300 via the communication unit D5.

[0114] Next, the server 300 performs an output determination (step S203). Specifically, the determination module 314 of the server 300 inputs the captured image to the object detection model, thereby obtaining information on the type of object included in the captured image output from the object detection model. Then, the determination module 314 refers to the output condition DB 3022 and determines whether the type of object included in the captured image corresponds to the type of object specified as the output condition for the AR object related to the output request.

[0115] Here, if the captured image does not satisfy the output conditions, the determination module 314 does not output the AR object by the output module 316. In this case, the server 300 determines that the user who is wearing the augmented reality device D and viewing the replaced manuscript does not have the authority to view the manuscript. Note that if the output of the AR object is not authorized, an error message indicating that the AR object cannot be output may be output to the augmented reality device D.

[0116] On the other hand, in step S203, if the captured image satisfies the output condition, that is, if the user's hand appears in the captured image together with the document, the determination module 314 authorizes the output of the AR object.

[0117] Next, the server 300 outputs the AR object (step S204). Specifically, the output module 316 of the server 300 refers to the object DB 3021 to identify an AR object corresponding to the AR marker related to the output request. The output module 316 transmits the identified AR object to the augmented reality device D via the transmission / reception control module 311.

[0118] Next, the augmented reality device D displays the AR object (step S105). Specifically, the augmented reality device D displays the AR object transmitted from the server 300 on the display D1. At this time, the augmented reality device D displays the AR object on the display D1 so that it overlaps with the display position of the AR marker in the document being viewed by the user, in the user's field of view. This allows the user, while viewing the replacement manuscript, to visually recognize the AR object displayed at a position overlapping the AR marker displayed on the replacement manuscript.

[0119] In this way, in system 1, it is not enough for a user to simply wear augmented reality device D and view the replacement manuscript; only users who hold and view the replacement manuscript can view the confidential information corresponding to the AR object. Therefore, only users who know the specified output conditions and view the replacement manuscript in accordance with those conditions can obtain the confidential information, effectively preventing information leakage from the document. This completes the AR object output process by system 1.

[0120] (4-2) Output process of replaced image data Next, a description will be given of a process for outputting replaced image data by system 1 with reference to Fig. 8. The process for outputting replaced image data is a process in which, when a user scans a document using image forming device 200, replaced image data in which confidential information specified by the user is replaced with an AR object is formed on a paper surface (recording medium) and output. In this process, the conversion of confidential information into an AR object is performed by server 300.

[0121] Fig. 8 is a flow diagram of the replaced image data output process by system 1. As shown in Fig. 8, in the replaced image data output process, image forming device 200 first scans an original (step S111). Specifically, image reading device 100 in image forming device 200 scans the surface of the original. Image reading device 100 generates original image data for the scanned original image.

[0122] Next, the image forming apparatus 200 accepts the designation of the confidentiality range (step S112). Specifically, the image forming apparatus 200 accepts the designation of an area of ​​the document image data to be converted into an AR object as the confidentiality range, by a user's operation on the input unit 42. Note that the designation of the confidentiality range may be accepted by a user's operation on the user terminal U.

[0123] Next, the image forming apparatus 200 transmits a conversion request (step S113). Specifically, the control unit 70 of the image forming apparatus 200 transmits the conversion request to the AR object to the server 300 via the communication unit 55. The conversion request to the AR object includes, for example, the following information: -Original image data - Information indicating the extent of confidential information in the original image data

[0124] Next, the server 300 receives a conversion request (step S211). Specifically, the reception module 312 of the server 300 receives the conversion request to the AR object transmitted from the image forming apparatus 200. The reception module 312 inputs the conversion request to the generation module 315.

[0125] Next, the server 300 performs replacement with an AR object (step S212). Specifically, the generation module 315 of the server 300 generates a corresponding AR object based on the information included in the confidential information range in the manuscript image data. The generation module 315 then generates an AR marker corresponding to the converted AR object and replaces the confidential information range in the manuscript image data with the AR marker. As a result, replacement image data is generated.

[0126] Next, server 300 outputs the replaced image data (step S213). Specifically, output module 316 of server 300 transmits the generated replaced image data to image forming apparatus 200 via transmission / reception control module 311.

[0127] Next, image forming apparatus 200 prints the replaced image data (step S114). Specifically, image forming unit 60 of image forming apparatus 200 forms the replaced image data transmitted from server 300 on a sheet of paper. This makes it possible for image forming apparatus 200 to output a sheet of paper on which an image of the replaced image data, in which confidentiality ranges included in the document have been replaced with AR markers, has been formed, thereby improving user convenience in handling confidential information. This completes the replaced image data output process by system 1.

[0128] (4-3) Output process of updated image data Next, an output process of an updated image by the system 1 will be described with reference to Fig. 9. The output process of updated image data is a process of updating the AR object and the AR marker based on the changes made by the user to the replacement image data generated by the above-described process when the user changes confidential information.

[0129] FIG. 9 is a flow diagram of the process of outputting an updated image by the system 1. As shown in FIG. 9, in the process of outputting an updated image by the system 1, first, the user terminal U accepts a correction instruction (step S121). Specifically, the user operates the user terminal U to make necessary corrections to the confidential information of the replacement image data that has already been obtained. The user terminal U has a function of editing the replacement image data. That is, the user terminal U has a function of switching the display of the AR marker shown in the replacement image data to the display of the corresponding confidential information by switching the replacement image data from a viewing mode to an editing mode, and accepting necessary edits.

[0130] Next, the user terminal U corrects the confidential information (step S122). Specifically, the processor of the user terminal U corrects the confidential information in the original image data in accordance with the editing operation from the user.

[0131] Next, the user terminal U transmits an update request for the AR object (step S123). Specifically, in response to an instruction to end the update being input by the user, the user terminal U transmits an update request to the server 300 to reflect the modification content of the confidential information in the AR object. The update request for the AR object includes at least the following information: Replacement image data Information about the modifications and their location

[0132] Next, the server 300 receives an update request (step S221). Specifically, the reception module 312 of the server 300 receives the update request for the AR object transmitted from the user terminal U. The reception module 312 inputs the update request to the generation module 315.

[0133] Next, the server 300 updates the AR object (step S222). Specifically, the generation module 315 of the server 300 references the corrected information from the confidential information and updates the AR object stored in the object DB 3021 to reflect the corrected content. The generation module 315 then updates the AR marker stored in the object DB 3021 along with updating the AR object. The generation module 315 replaces the AR marker in the replacement image data with the updated AR marker. As a result, updated image data in which the AR marker portion of the replacement image data has been updated is generated.

[0134] Next, the server 300 outputs the updated image data (step S223). Specifically, the output module 316 of the server 300 transmits the updated image data to the user terminal U via the transmission / reception control module 311.

[0135] Next, the user terminal U displays the updated image data (step S124). Specifically, the user terminal U displays the updated image data transmitted from the server 300. This allows the user to confirm the updated information as an AR object. This completes the updated image data output process by the system 1.

[0136] (5) Other Examples Next, a description will be given of other examples of the system 1. In the following description, the same configurations as those in the above-described embodiment will not be described.

[0137] (5-1) Example 1: Example of using document holding as an output condition The output condition for the AR object may be a specific way in which the user holds the document. For example, a special holding style that is not typically used when viewing a document may be set as the output condition, such as holding the top right corner of the replacement document with the left hand or holding the bottom left corner of the replacement document with the right hand. In this case, only those who know such a holding style can be authenticated to output the AR object by the server 300, thereby providing a more secure image processing technology.

[0138] (5-2) Example 2: Granting output authority to the first viewer The system 1 may also restrict the output of AR objects to those who have acquired the replacement manuscript. In this case, the terminal number of the augmented reality device D that first sent an output request for the AR object to the server 300 is newly defined as an output condition. After that, it becomes impossible to authenticate output requests from augmented reality devices D other than the augmented reality device D used by the first viewer.

[0139] As a result, anyone other than the user who first viewed the replacement manuscript will not be able to see the AR object shown in the replacement manuscript, even if they view it while wearing the augmented reality device D. In this case, when the server 300 receives an output request from another augmented reality device D, it may notify the augmented reality device D used by the first viewer of this fact.

[0140] The first viewer who receives the notification can then operate the user interface D6 of the augmented reality device D to specify whether or not to permit an output request from the other augmented reality device D. In this case, if the output is permitted by the first viewer, the AR object is output to the other user's augmented reality device D. This allows the first viewer to grant viewing authority to the extent he or she desires, providing convenience regarding the extent to which confidential information is shared.

[0141] (5-3) Example 3: Example of specifying output stop conditions Furthermore, in the system 1, an output stop condition for the AR object may be set and the AR object output authentication may be maintained continuously until the output stop condition is met, instead of repeatedly determining output at each time interval. In this case, for example, once it is confirmed that the document is being held in the hand, the AR object can be viewed continuously until a specific cancellation event (output stop condition) occurs, not just while the document is being held in the hand.

[0142] In this embodiment, the output stop conditions include, for example, the following. The user removes the augmented reality device D. -Augmented reality device D powered off - Specific operations from the user interface In this way, by allowing continuous viewing until the output stop condition is met, user convenience can be ensured in cases where multiple people wish to view a document while circulating it at the same time.

[0143] (5-4) Example 4: Example of specifying position information in output conditions Furthermore, in the system 1, location information may be specified as an output condition for the AR object. In this case, the output condition specifies an area in which the replaced manuscript can be viewed, such as a specific room in an office. The viewable area can be set arbitrarily by the user or the manager of confidential information.

[0144] When the viewing location is specified in the output conditions as in this embodiment, the augmented reality device D transmits the user's location information, determined by the following method, to the server 300 together with an output request. Sensing results from the position sensor included in sensor D2 - Signal strength between the communication unit D5 and the access point during wireless communication

[0145] The determination module 314 then compares the determined user location information with the viewable area defined as the output condition to attempt authentication regarding the output of the AR object, thereby enabling more secure management of confidential information.

[0146] (5-5) Example 5: Peeping prevention function The system 1 may also have a function to prevent peeping. That is, the server 300 can reject any output request when multiple people are viewing a document through the display D1 of the augmented reality device D. Specifically, the server 300 can set in advance in the output determination by the determination module 314 to not accept multiple output requests at the same time.

[0147] In this case, the AR object is displayed on the augmented reality device D used by the original viewer who is holding the document in his / her hand while viewing it, but the AR object is not displayed on the augmented reality device D used by a person peering at the document from the periphery. The original viewer is notified of whether the AR object can be displayed to other users, and if the original viewer gives permission, the person peering can also view it.

[0148] (5-6) Example 6: Example of reusing the AR marker before update Furthermore, in the system 1, when an AR object is updated (see FIG. 9), a change may be made to link the updated AR object to the AR marker before the update in the object DB 3021. This allows the updated AR object to be viewed from the replacement manuscript in which the AR marker before the update is displayed, improving user convenience.

[0149] In this way, the system 1 allows the AR object to be viewed only when a preset output condition is met, such as when the user is holding a document in their hand. This eliminates the need to grant or manage viewing permissions, as compared to conventional image processing technologies, and effectively prevents information leaks due to peeking.

[0150] (6) Variations Modifications of the present invention will be described below.

[0151] In the above embodiment, AR glasses have been described as an example of the augmented reality device D, but this is not limiting. The augmented reality device D is not limited to AR glasses, and various terminal devices applicable to AR technology (for example, smartphones, tablet terminals, head-mounted displays, AR projectors, smart glasses, etc.) can be used.

[0152] The server 300 may also set other conditions as output conditions. For example, the action of the user using the augmented reality device D may be defined as an output condition. Also, for example, multiple types of output conditions may be set so that the output condition based on the captured image and the output condition based on the user's action are used together.

[0153] The same output condition may be set for all AR objects, or may be set individually for each AR object. In this case, when a conversion request is made by the image forming apparatus 200, the user may set the output condition for the AR object related to the conversion request. When the output condition is set by the user, the user may select an appropriate output condition from a plurality of output condition options prepared in advance, taking into account the confidentiality of the confidential information.

[0154] 8, the image reading device 100 transmits the document image data generated by the scan process to the server 300. However, the present invention is not limited to this. In other words, in the process of converting to an AR object, only the information designated by the user as confidential information to be converted to an AR object from the document image data may be transmitted to the server 300.

[0155] (7) Other variations The image forming apparatus 200 may be a copier, a printer, a facsimile, a scanner, or a multifunction device that combines the functions of these.

[0156] 4, the image reading device 100 includes the document transport unit 90. However, the image reading device 100 does not necessarily have to include the document transport unit 90.

[0157] The image forming apparatus 200 may include an inkjet image forming unit. The inkjet image forming unit divides an image to be printed into pixels, charges ink particles with a voltage proportional to the position information of each pixel, and deflects them in an electrostatic field to make them reach the document, thereby forming an image.

[0158] The embodiments of the present invention have been described above with reference to the drawings (FIGS. 1 to 9). However, the present invention is not limited to the above embodiments and can be embodied in various forms without departing from the spirit and scope of the present invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above embodiments. For example, some components may be omitted from all components shown in the embodiments. The drawings mainly show each component in a schematic manner for ease of understanding, and the thickness, length, number, spacing, etc. of each component shown may differ from the actual components due to the convenience of drawing. Furthermore, the materials, shapes, dimensions, etc. of each component shown in the above embodiments are merely examples and are not particularly limited, and various modifications are possible within a scope that does not substantially deviate from the effects of the present invention. [Industrial Applicability]

[0159] The present invention provides an image processing apparatus, an image forming apparatus, and an image processing method, and has industrial applicability. [Explanation of symbols]

[0160] 1. Image processing system 20 Reading unit 70 Control Unit 80 Storage section 100 Image reader 200 Image forming device 300 Servers (Image Processing Devices) 312 Reception Module 313 Request module (request part, authentication part) 314 Judgment module (judgment unit, authentication unit) 315 Generation Module (Generation Unit) 316 Output module (output section) D Augmented reality devices U User terminal

Claims

1. an authentication unit that attempts authentication using a captured image transmitted from an augmented reality device having an image capturing function in response to receiving an output request for an AR object corresponding to an AR marker indicated on a recording medium from the augmented reality device; an output unit that outputs the AR object related to the output request to the augmented reality device in response to authentication by the authentication unit; An image processing device comprising:

2. a request unit that requests the augmented reality device to provide the captured image including the recording medium in response to receiving the output request; a determination unit that determines whether the captured image provided from the augmented reality device satisfies a predetermined condition that has been defined in advance; Furthermore, The determination unit When the captured image satisfies the predetermined condition, the output unit outputs the AR object related to the output request to the augmented reality device; The image processing device according to claim 1 , wherein, when the captured image does not satisfy the predetermined condition, the output unit does not cause the AR object related to the output request to be output to the augmented reality device.

3. The image processing device according to claim 1 , wherein the predetermined condition is that the captured image includes a hand of the user holding the recording medium.

4. 4. The image processing device according to claim 3, wherein the predetermined conditions further define how the recording medium is held by the hand.

5. 3. The image processing device according to claim 1, further comprising a generation unit that converts confidential information included in a range specified by a user into the AR object for document image data generated by scanning a document, and generates the AR marker corresponding to the AR object.

6. The image processing device according to claim 5 , wherein the generation unit updates the AR object and the corresponding AR marker in response to an update of the confidential information corresponding to the AR object.

7. an image forming unit that forms an image represented by document image data generated by scanning a document on a recording medium; a communication unit capable of communicating with the image processing device according to claim 5; The image forming apparatus according to claim 5 , wherein the image forming unit forms, on the recording medium, an image indicated by replaced image data that is transmitted from the image processing apparatus according to claim 5 and in which the confidential information is replaced with the AR marker.

8. 1. An image processing method implemented by a computer having a processor, comprising: The image processing method includes: an authentication step of attempting authentication using a captured image transmitted from an augmented reality device having an imaging function in response to receiving a request to output an AR object corresponding to an AR marker indicated on a recording medium from the augmented reality device; an output step of outputting the AR object related to the output request to the augmented reality device in response to authentication being performed in the authentication step; An image processing method that performs

Images