Terminal device, image formation system, program, and print control method

Abstract

To provide a terminal device capable of adjusting the print size of image data in an easy-to-understand mode, an image formation system, a program, and a print control method, etc.SOLUTION: A terminal device includes: an image acquisition unit which acquires image data; an imaging unit which acquires a captured image obtained by capturing a region including a place scheduled to be arranged; an image processing unit which generates superimposed image data obtained by rescaling image data on the basis of the distance to the place scheduled to be arranged and generates an AR image where the superimposed image data is displayed, superimposing it on the captured image; an operation unit which accepts a size change operation of the superimposed image data in the AR image; and a communication unit which communicates with an image formation device. The image processing unit determines print size information on the basis of the size change operation. The communication unit transmits the print size information to the image formation device.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a terminal device, an image forming system, a program, a print control method, and the like. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there is known a method for setting the print size of image data in an image forming apparatus. For example, the print settings include an item for setting the print magnification.

[0003] Furthermore, Patent Document 1 discloses a method of displaying the printed state of a stamp or the like on a sheet of paper using AR (Augmented Reality). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2015-102909 A Summary of the Invention [Problem to be solved by the invention]

[0005] The technique in Patent Document 1 adjusts the position when placing a stamp or the like on paper, but does not indicate the state in which the printed result of image data is placed in the intended placement location, nor does it take into account the distance between the user (terminal device) and the intended placement location.

[0006] According to some aspects of the present disclosure, it is possible to provide a terminal device, an image forming system, a program, a print control method, and the like that adjusts the print size of image data in an easy-to-understand manner. [Means for solving the problem]

[0007] One aspect of the present disclosure includes an image acquisition unit that acquires image data to be printed, an imaging unit that acquires a captured image of an area including a planned placement location of a print result of the image data, an image processing unit that generates superimposed image data by scaling the image data at a magnification corresponding to the size when the print result of the standard print size is placed at the planned placement location based on the distance to the planned placement location and a standard print size of the image data, and generates an AR image in which the superimposed image data is superimposed and displayed on the captured image, an operation unit that accepts a resize operation of the superimposed image data in the AR image, and a communication unit that communicates with an image forming device, wherein the image processing unit performs a process of determining print size information representing the print size of the image data based on the resize operation, and the communication unit is related to a terminal device that transmits the print size information to the image forming device.

[0008] Another aspect of the present disclosure relates to an image forming system including an image forming device and a terminal device, wherein the terminal device acquires image data to be printed, acquires a captured image capturing an area including a planned placement location of a print result of the image data, generates superimposed image data by scaling the image data at a magnification corresponding to a size of the print result of the standard print size when placed at the planned placement location based on a distance to the planned placement location and a standard print size of the image data, generates an AR image superimposing and displaying the superimposed image data on the captured image, accepts a resize operation of the superimposed image data in the AR image, determines print size information representing a print size of the image data based on the resize operation, and transmits the print size information to the image forming device, and the image forming device prints the image data based on the print size information.

[0009] Yet another aspect of the present disclosure relates to a program that causes a computer to execute processes of acquiring image data to be printed, acquiring a captured image that captures an area including a planned placement location of a print result of the image data, generating superimposed image data by scaling the image data at a magnification corresponding to the size of the print result of the standard print size when placed at the planned placement location based on the distance to the planned placement location and a standard print size of the image data, generating an AR image that superimposes and displays the superimposed image data on the captured image, accepting a resizing operation of the superimposed image data in the AR image, determining print size information representing the print size of the image data based on the resizing operation, and transmitting the print size information to an image forming device that prints the image data.

[0010] Yet another aspect of the present disclosure relates to a print control method which acquires image data to be printed, acquires a captured image capturing an area including a location where a print result of the image data is planned to be placed, generates superimposed image data by scaling the image data at a magnification corresponding to the size of the print result of the standard print size when placed at the planned location based on the distance to the planned location and a standard print size of the image data, generates an AR image in which the superimposed image data is displayed superimposed on the captured image, accepts a resize operation of the superimposed image data in the AR image, determines print size information representing the print size of the image data based on the resize operation, and transmits the print size information to an image forming device which prints the image data. [Brief description of the drawings]

[0011] [Figure 1] 1 is a configuration example of an image forming system including a terminal device. [Diagram 2] 1 is a configuration example of an image forming apparatus. [Diagram 3] 2 is a configuration example of a terminal device. [Figure 4] FIG. 4 is a sequence diagram illustrating the processing of the present embodiment. [Diagram 5]FIG. 1 is a diagram for explaining the positional relationship between a planned placement location and a terminal device (user). [Figure 6A] 1 is an example of a captured image. [Figure 6B] 1 is an example of a captured image. [Figure 7] 1 is a schematic diagram showing the relationship between a subject size (actual size), a size on an image, and a distance to the subject. [Figure 8A] This is an example of an AR image. [Figure 8B] This is an example of an AR image. [Figure 9] FIG. 13 is a diagram showing an example of a user's resizing operation on an AR image. [Figure 10] FIG. 11 is a sequence diagram illustrating another example of the image acquisition process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the present embodiment will be described with reference to the drawings. In the drawings, the same or equivalent elements are given the same reference numerals, and duplicated explanations are omitted. Note that the present embodiment described below does not unduly limit the contents described in the claims. In addition, all of the configurations described in the present embodiment are not necessarily essential configurations of the present disclosure.

[0013] 1. System configuration example FIG. 1 is a diagram illustrating a configuration of an image forming system 10 including a terminal device 200 according to the present embodiment. As shown in FIG. 1, the image forming system 10 according to the present embodiment includes an image forming device 100 and a terminal device 200. The image forming device 100 is, for example, an MFP (Multifunction Peripheral). However, a device other than an MFP may be used as the image forming device 100. Note that the system configuration is not limited to that shown in FIG. 1, and various modifications such as omitting some components and adding other components are possible. The fact that modifications such as omissions and additions are possible is also true for the drawings described later, such as FIG. 2 and FIG. 3.

[0014] The image forming apparatus 100 is, for example, a printer including a printing function. However, the image forming apparatus 100 may be an apparatus having a scanning function in addition to the printing function. The image forming apparatus 100 may also be a device having other functions such as a facsimile function or a copy function.

[0015] The terminal device 200 is a device used by a user who uses the image forming system 10 according to this embodiment, and is, for example, a mobile terminal device such as a smartphone. However, the terminal device 200 may be a tablet terminal, a wearable device, or a PC (Personal Computer), and various modifications of the specific aspects are possible.

[0016] The image forming apparatus 100 and the terminal apparatus 200 are connected, for example, via a network. The network here is, for example, a public communication network such as the Internet. However, the network may be an intranet or the like, and various specific modifications are possible. For example, the terminal apparatus 200 transmits a print instruction for image data to the image forming apparatus 100, and the image forming apparatus 100 executes a print process for the image data.

[0017] 2 is a diagram showing an example of the configuration of image forming apparatus 100. Image forming apparatus 100 includes a processing unit 110, a storage unit 120, a communication unit 130, a display unit 140, and an image forming unit 150. However, image forming apparatus 100 may have a scanning function or the like, and may include, for example, an image reading unit (not shown).

[0018] The processing unit 110 is configured by the following hardware. The hardware may include at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the hardware may be configured by one or more circuit devices or one or more circuit elements mounted on a circuit board. The one or more circuit devices may be, for example, an integrated circuit (IC) or a field-programmable gate array (FPGA). The one or more circuit elements may be, for example, a resistor, a capacitor, etc.

[0019] The processing unit 110 may be realized by the following processor. The image forming apparatus 100 of the present embodiment includes a memory that stores information and a processor that operates based on the information stored in the memory. The information is, for example, a program and various data. The processor includes hardware. The processor can be various processors such as a central processing unit (CPU), a graphics processing unit (GPU), and a digital signal processor (DSP). The memory may be a semiconductor memory such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), a register, a magnetic storage device such as a hard disk drive (HDD), or an optical storage device such as an optical disk device. For example, the memory stores computer-readable instructions, and the processor executes the instructions to realize the function of the processing unit 110 as processing. The instructions here may be instructions of an instruction set constituting a program, or instructions that instruct the hardware circuit of the processor to operate.

[0020] The processing unit 110 controls each unit of the image forming apparatus 100 shown in Fig. 2. For example, the processing unit 110 performs communication control, data reading and writing control, display control, etc. The processing unit 110 may also perform printing control, scanning control, facsimile transmission and reception control, etc.

[0021] The storage unit 120 is a work area for the processing unit 110, and stores various information. The storage unit 120 can be realized by various types of memory, and the memory may be a semiconductor memory, a register, a magnetic storage device, or an optical storage device.

[0022] The communication unit 130 is an interface for performing communication via a network. The image forming apparatus 100 is connected to a network via the communication unit 130. The communication unit 130 includes, for example, an antenna, an RF (radio frequency) circuit, and a baseband circuit. The communication unit 130 operates according to the control by the processing unit 110. The communication unit 130 may also include a processor for communication control different from the processing unit 110. The communication unit 130 may be an interface for performing communication according to, for example, IEEE802.11, and specifically, performs communication according to WiFi (registered trademark). However, the communication unit 130 may be a communication interface for performing wireless communication according to another method, or may be a communication interface for performing wired connection. The image forming apparatus 100 of this embodiment may also be capable of communication by facsimile. For example, the communication unit 130 may include a telephone line connection unit for performing communication using a public switched telephone network. The communication unit 130 of the present embodiment may include any one of connection units with different communication methods, such as a wireless LAN connection unit, a wired LAN connection unit, and a telephone line connection unit, or may include two or more of these.

[0023] The display unit 140 is a display that displays video information. The display unit 140 can be realized by, for example, a liquid crystal display or an organic EL display. The display unit 140 may be a UI (User Interface) for using functions of the image forming apparatus 100, such as a print function or a scan function.

[0024] The image forming unit 150 is realized by, for example, a print engine. The print engine includes a mechanical configuration for printing on a print medium. For example, the print engine includes a transport roller that transports the print medium along the transport direction and a transport motor that drives the transport roller. If the image forming device 100 is an inkjet printer, the print engine includes a carriage on which a print head is provided and a carriage motor that drives the carriage in a main scanning direction perpendicular to the transport direction. The print engine forms an image on the print medium by ejecting ink from a print head that reciprocates in the main scanning direction while transporting the print medium in the transport direction. However, the image forming device 100 of this embodiment can be applied to various types of image forming devices, and the configuration of the print engine is not limited to this. For example, the image forming device 100 may be an electrophotographic image forming device such as a laser printer.

[0025] 3 is a diagram showing an example of the configuration of the terminal device 200. The terminal device 200 includes a processing unit 210, a storage unit 220, a communication unit 230, a display unit 240, an operation unit 250, and an imaging unit 260.

[0026] The processing unit 210 is configured with hardware including at least one of a circuit for processing digital signals and a circuit for processing analog signals. For example, the hardware can be configured with one or more circuit devices mounted on a circuit board or one or more circuit elements.

[0027] The processing unit 210 may be realized by various processors such as a CPU, a GPU, a DSP, etc. The terminal device 200 of this embodiment includes a memory that stores information and a processor that operates based on the information stored in the memory. For example, the memory stores computer-readable instructions, and the processor executes the instructions to realize the function of the processing unit 210 as processing.

[0028] The processing unit 210 includes an image acquisition unit 211 and an image processing unit 212. The image acquisition unit 211 acquires image data to be printed. For example, the image acquisition unit 211 may acquire the image data by reading the image data from the storage unit 220 of the terminal device 200. The image data stored in the storage unit 220 may be data captured by the imaging unit 260. Alternatively, the image data stored in the storage unit 220 may be data created by application software (word processing software, spreadsheet software, etc.) running on the terminal device 200. However, as will be described later with reference to FIG. 10, the image acquisition unit 211 may acquire the image data from the image forming device 100.

[0029] The image processing unit 212 performs a process of determining the print size information of the image data. The print size information here is information that determines the actual size when the image data is printed on a print medium. For example, the print size information may be information that specifies the length in the vertical direction and the length in the horizontal direction in units such as mm or inches. Alternatively, the print size information may be information that specifies the magnification with respect to the standard print size (such as A4 size). Alternatively, if the resolution on the print medium (such as information determined in units such as dpi) is known, the actual size at the time of printing is specified when the number of pixels in the vertical direction and the number of pixels in the horizontal direction of the image data are determined. Therefore, the print size information may be information that specifies the number of pixels in the vertical direction and the horizontal direction of the image data and the resolution at the time of printing. Alternatively, if the resolution at the time of printing is fixed, the print size information may be information that specifies the number of pixels in the vertical direction and the horizontal direction of the image data. Note that by using the aspect ratio of the image data, one of the information in the vertical direction and the information in the horizontal direction may be omitted. In addition, various modifications are possible for the specific format of the print size information.

[0030] The image processing unit 212 displays an AR image IM3 corresponding to the distance between the terminal device 200 and the planned placement location on the display unit 240, and performs processing to determine print size information based on a size change operation by the user using the AR image IM3. A specific processing flow will be described later with reference to Figs. 4 to 9.

[0031] The storage unit 220 is a work area for the processing unit 210, and stores various information. The storage unit 220 can be realized by various types of memory, and the memory may be a semiconductor memory, a register, a magnetic storage device, or an optical storage device.

[0032] The communication unit 230 is an interface for performing communication via a network. The terminal device 200 is connected to the image forming device 100 via the communication unit 230. For example, the communication unit 230 transmits print size information determined by the image processing unit 212 to the image forming device 100.

[0033] The communication unit 230 includes, for example, an antenna, an RF circuit, and a baseband circuit. The communication unit 230 may be an interface that performs communication according to, for example, IEEE802.11, and more specifically, performs communication according to WiFi. However, the communication unit 230 may be a communication interface that performs wireless communication according to another method, or may be a communication interface that performs a wired connection.

[0034] The display unit 240 is a display that displays video information. The display unit 240 can be realized by, for example, a liquid crystal display, an organic EL display, etc. The display unit 240 displays, for example, a captured image IM1 described later with reference to Figs. 6A and 6B, and an AR image IM3 described later with reference to Figs. 8A and 8B.

[0035] The operation unit 250 is an interface that accepts user operations. The operation unit 250 may be an interface such as a button provided on the terminal device 200, or may be another operation interface. The display unit 240 and the operation unit 250 may be a touch panel that is integrally configured. For example, the operation unit 250 accepts a size change operation of superimposed image data IM2, which will be described later, in the AR image IM3.

[0036] The imaging unit 260 has an image sensor that captures a predetermined imaging range and outputs a captured image IM1. The captured image IM1 here may be a still image or a moving image. The captured image IM1 may be color or monochrome. For example, the imaging unit 260 captures and outputs a captured image IM1 that captures an area including a planned location for arranging the print result of the image data, as described later with reference to FIG. 5.

[0037] According to the method of the present embodiment, it is possible to allow the user to confirm the state in which the print result of the image data is placed at the planned placement location by using the AR image IM3. The print result of the image data is a print medium on which the image data is printed, and may be a poster or other notice. Specifically, as described later with reference to Figs. 5 to 8, the image processing unit 212 adjusts the size of the superimposed image data IM2 according to the distance D from the planned placement location. In this way, it is possible to allow the user to confirm the size of the image data when it is printed with the current size setting in correspondence with the real space, so that the user can easily adjust the size. As a result, it is possible to reduce the operational burden on the user. In addition, since reprinting due to a setting error is suppressed, consumption of ink, toner, print media, etc. can be suppressed.

[0038] 2. Processing flow 4 is a sequence diagram illustrating the process of this embodiment. First, in step S101, a user performs a process of selecting image data to be printed using the operation unit 250 of the terminal device 200. The user here is a user of the terminal device 200 who intends to print image data using the image forming device 100 and arrange the printout result in a predetermined planned arrangement location.

[0039] For example, prior to the processing of step S101, the user may operate the terminal device 200 to launch a print application. The print application is software for executing the processing of the terminal device 200, which will be described below. For example, the user may use the print application to select image data to be printed. Alternatively, the user may use an application other than the print application (e.g., an image viewer) to select image data and perform a print operation, and the print operation may be used as a trigger to launch the print application.

[0040] In step S102, the image acquisition unit 211 of the terminal device 200 acquires image data to be printed. For example, the image acquisition unit 211 performs a process of reading, from the storage unit 220, the image data specified by the user operation in step S101.

[0041] In step S103, the terminal device 200 starts up the imaging unit 260. For example, the processing unit 210 operates according to a print application to start up the imaging unit 260. The print application starts a process of displaying, for example, a captured image IM1 captured by the imaging unit 260 on the display unit 240 of the terminal device 200 in real time.

[0042] In step S104, the user performs a shooting operation using the imaging unit 260. For example, the shooting here is shooting a moving image, and the user performs an operation to adjust the position and orientation of the imaging unit 260 (terminal device 200) so that a desired range falls within the imaging range (angle of view) of the imaging unit 260.

[0043] 5 is a diagram showing a specific example of the photographing operation in step S104, and shows the positional relationship between the intended placement location and the user (terminal device 200). For example, the user of the terminal device 200 captures an image of the intended placement location where a poster or the like that is a printout of image data will be placed. In the example of FIG. 5, the user is considering posting the printout on the door of a room, so the user captures an image of an area including the door. For example, as shown in FIG. 5, the user stands at a position a distance D away from the door, and performs an action of holding the terminal device 200 so that the optical axis of the imaging unit 260 is directed in the direction of the door.

[0044] In step S105, the imaging unit 260 acquires a captured image IM1. FIGS. 6A and 6B are diagrams showing specific examples of the captured image IM1. FIG. 6A is an example of a captured image IM1a when the area including almost the entire door is the imaging range. FIG. 6B is an example of a captured image IM1b when only the center part of the door is the imaging range. For example, the captured image IM1a in FIG. 6A represents a captured image IM1 when the distance D between the user and the door is relatively small, and the captured image IM1b in FIG. 6B represents a captured image IM1 when the distance D between the user and the door is relatively large. In this embodiment, it is sufficient that the area in which the user wants to place the print result is included in the captured image IM1, and the specific imaging range can be modified in various ways.

[0045] Also, in step S105, the display unit 240 displays the captured image IM1 acquired by the imaging unit 260. For example, the imaging unit 260 captures a moving image at a constant frame rate, and the display unit 240 displays the moving image.

[0046] In step S106, the terminal device 200 performs a process of measuring the distance between the terminal device 200 and the subject captured in the captured image IM1. For example, the terminal device 200 may include a distance measurement sensor that measures the distance to the subject. The distance measurement sensor here is, for example, a ToF (Time of Flight) type sensor that measures the distance D based on the time from when a laser light is projected to when the reflected light of the laser light is received by the subject. However, various types of distance measurement sensors are known, and they can be widely applied in this embodiment. For example, various modifications are possible, such as the imaging unit 260 of the terminal device 200 including a stereo camera, and the processing unit 210 performing a stereo matching process to measure the distance.

[0047] In step S107, the image processing unit 212 generates superimposed image data IM2 by scaling the image data to an appropriate size. For example, the image processing unit 212 acquires a reference print size of the image data. The reference print size here refers to a reference actual size when the image data is printed on a print medium. The reference print size may be a fixed size such as A4 (210 mm x 297 mm) or A3 (29.7 cm x 42 cm), or may be a size that varies depending on the number of pixels of the image data.

[0048] The image processing unit 212 generates superimposed image data IM2 by scaling the image data at a magnification corresponding to the size when the print result of the standard print size is placed at the intended placement location, based on the distance D to the intended placement location and the standard print size of the image data.

[0049] For the sake of convenience, the following description will be given assuming that the standard print size is A4 size. In this case, the image processing unit 212 estimates how the image data will appear on the captured image IM1 when the printout is placed at the intended placement location, assuming that the image data is printed in A4 size. The size of A4 size in real space is known to be 210 mm×297 mm, but the size on the captured image IM1 (e.g., the number of pixels in the vertical and horizontal directions) varies depending on how the captured image IM1 is captured.

[0050] FIG. 7 is a schematic diagram showing the relationship between the subject size and the size on the imaging element. The subject size indicates the actual size in real space. The size on the imaging element corresponds to the size on the image. As shown in FIG. 7, the distance between the lens 261 and the imaging element 262 included in the imaging unit 260 is the focal length. The focal length is information known from the configuration of the imaging unit 260 and the zoom magnification during imaging. The distance from the subject to the imaging unit 260 (lens 261 in a narrow sense) corresponds to the distance D in FIG. 5. As is clear from FIG. 7, the actual size, the size on the image, the distance D, and the focal length satisfy the following formula (1). The following formula (1) can be transformed into the following formula (2). Actual size = (distance D × size on image) / (focal length) ... (1) Image size = (focal length x actual size) / (distance D) ... (2)

[0051] In the process of step S107, it is known that the actual size of the printout is A4 size. Also, the distance D has been acquired in step S106. Also, the focal length is known information as described above. Therefore, in step S107, based on the above formula (2), the image processing unit 212 can determine the image size, which indicates how large the A4 size printout will appear on the captured image IM1, assuming that the A4 size printout is placed in the intended placement location.

[0052] As can be seen from the above formula (2), when the focal length and actual size are constant, the on-image size is inversely proportional to the distance D. Therefore, when considering captured image IM1a in Fig. 6A where the distance D is large, the on-image size of the printed result will be small, and when considering captured image IM1b in Fig. 6B where the distance D is small, the on-image size of the printed result will be large. The image processing unit 212 resizes the image data to obtain the calculated on-image size, and sets the image data resulting from the resizing as superimposed image data IM2.

[0053] In step S108, the image acquisition unit 211 performs a process of generating an AR image IM3 by superimposing the superimposed image data IM2 on the captured image IM1. FIGS. 8A and 8B are examples of the AR image IM3. As shown in FIG. 8A, the captured image IM1a is superimposed with relatively small superimposed image data IM2a as the superimposed image data IM2. The AR image IM3 in which the superimposed image data IM2a is superimposed on the captured image IM1a is set as the AR image IM3a. As shown in FIG. 8B, the captured image IM1b is superimposed with relatively large superimposed image data IM2b as the superimposed image data IM2. The AR image IM3 in which the superimposed image data IM2b is superimposed on the captured image IM1b is set as the AR image IM3b. The superimposed position of the superimposed image data IM2 may be the center of the captured image IM1 or may be another position determined in advance. The superimposed position of the superimposed image data IM2 may be changed based on a user operation.

[0054] In step S109, the image processing unit 212 performs processing to display the generated AR image IM3 on the display unit 240. Specifically, the display unit 240 starts displaying the captured image IM1 in step S105, but switches the display target to the AR image IM3 in step S109. As a result, the AR image IM3 shown in FIG. 8A, FIG. 8B, etc. is presented to the user.

[0055] By displaying the AR image IM3, it is possible to clearly show the user what size the printed result of the image data will be in real space before printing. As a result, when the image data is printed and placed in the intended location, the user can judge whether the size of the printed result itself is appropriate, or whether the size of the images and text included in the printed result is appropriate.

[0056] If there is no problem with the size of the superimposed image data IM2 on the AR image IM3, the user is likely to instruct printing as is. On the other hand, if there is a problem with the size of the superimposed image data IM2, the user is likely to want to change the print size (actual size) of the image data from the standard print size (e.g., A4). In this embodiment, the user's size change operation is accepted using the AR image IM3.

[0057] It is assumed that if the size of the superimposed image data IM2 in the AR image IM3 is excessively small or large, the user may find it difficult to perform a resizing operation. Therefore, the user is not prevented from changing the distance between the terminal device 200 and the intended placement location, separate from the resizing operation. In this case, the actual size (print size) remains fixed, and the distance D varies, so the size on the image, that is, the size of the superimposed image data IM2, is changed. For example, for a user who is considering performing a resizing operation to enlarge the size, it is conceivable that the superimposed image data IM2a before the operation is too small in the state of FIG. 8A, and it is difficult to perform a sufficient enlargement operation. In this case, the user performs an operation to shorten the distance D from the door while holding the terminal device 200, separate from the magnification change operation. In this way, for example, the AR image IM3 changes to the image shown in FIG. 8B, making the resizing operation easier. As can be seen from the above explanation, the distance measurement is not limited to being performed only once in step S106, and may be performed continuously. Also, instead of the operation to change the distance, the user may perform an operation to change the zoom magnification of the imaging unit 260. In this case, since the focal length varies while the actual size remains fixed, the size of the superimposed image data IM2 is changed.

[0058] In step S110, the user performs a size change operation on the superimposed image data IM2 on the AR image IM3 using the operation unit 250. Fig. 9 is a diagram showing an example of a size change operation. For example, the operation unit 250 is a touch panel, and the user performs a pinch-in or pinch-out operation as a size change operation. However, the size change operation is not limited to this, and may be another operation using a touch panel, or may be an operation using an operation interface different from a touch panel.

[0059] In the example of Fig. 9, a resize operation is performed to enlarge the superimposed image data IM2b corresponding to the standard print size (A4 size), thereby obtaining superimposed image data IM2c having a larger image size. Note that the resize operation may be an operation to reduce the size of the superimposed image data IM2.

[0060] In step S111, the image processing unit 212 performs a process of determining the actual size (print size) corresponding to the resized superimposed image data IM2. Specifically, the image processing unit 212 obtains the actual size by using the size of the superimposed image data IM2 after the resizing operation as the on-image size in the above formula (1). For example, if the area of ​​the superimposed image data IM2 is doubled by the resizing operation, the actual size is changed from A4 size to A3 size. As described above, in this embodiment, the distance D and the focal length may change during the operation, but by performing a calculation using the above formula (1), it is possible to obtain the actual size while taking these changes into consideration. The resizing operation by the user is not limited to one time, and may be performed repeatedly until the user's desired size is obtained. The image processing unit 212 uses information representing the actual size as print size information.

[0061] In step S112, communication unit 230 transmits the image data and the print size information in association with each other to image forming apparatus 100. Communication unit 130 of image forming apparatus 100 receives the image data and the print size information.

[0062] In step S113, the image forming apparatus 100 determines print medium information including at least one of the size and type of print medium used to print the image data based on the print size information received from the terminal device 200. For example, if the image forming apparatus 100 supports a plurality of print media of different sizes, the processing unit 110 of the image forming apparatus 100 selects the smallest print medium among the plurality of print media that satisfies the condition that the size is larger than the size corresponding to the print size information. In this way, the print result is not divided into a plurality of media, and it is possible to reduce the margin area. The image forming apparatus 100 may also print the image data by combining a plurality of print media that are equal to or smaller than the print size. In this case, the processing unit 210 may select a print medium whose area when a predetermined number of print media are combined is larger than the print size and whose margin area is the smallest.

[0063] Furthermore, the image forming apparatus 100 may change the type of print medium depending on the print size of the image data. For example, the processing unit 110 may select plain paper when the print size is equal to or smaller than a predetermined size, and may use coated paper or high-quality paper when the print size is equal to or larger than a predetermined size. In this way, it becomes possible to automatically select a print medium with excellent visibility when printing a large notice such as a poster.

[0064] 9 illustrates an example in which the image forming device 100 determines the print medium information, but the present invention is not limited to this. For example, the image processing unit 212 of the terminal device 200 may determine print medium information including at least one of the size and type of the print medium used to print the image data based on a size change operation. In this case, the communication unit 230 transmits the print medium information to the image forming device 100 in addition to the print size information. In this way, it becomes possible to execute various print settings on the terminal device 200.

[0065] When terminal device 200 determines print medium information, communication unit 130 may acquire in advance information indicating the type of print medium supported by image forming device 100, which is the communication target. Communication unit 130 may receive information such as the type of print medium that can be printed from image forming device 100. Alternatively, communication unit 130 may receive model number information from image forming device 100, and perform processing to identify information on print media that image forming device 100 can print based on the model number and information such as a catalog.

[0066] In step S113, the image forming unit 150 (print engine) of the image forming apparatus 100 executes printing processing of the image data according to the print size information and the print medium information. Note that setting of the print medium information is not essential and may be omitted. In this case, the image forming apparatus 100 prints the image data using a print medium designated by default or a print medium designated by the user.

[0067] 3. Variations In the above, an example has been described in which data stored in the storage unit 220 of the terminal device 200 is selected as image data to be printed. However, the method of acquiring image data is not limited to this. For example, a user may print image data stored in a storage medium such as a Universal Serial Bus (USB) memory. In this case, the user connects the storage medium to a connection interface of the image forming device 100, and the image forming device 100 acquires the image data to be printed via the connection interface. In this case, too, it is effective to perform the above-mentioned processing related to the terminal device 200 in order to allow the user to perform a size change operation taking into account the size of the printed result.

[0068] It is also conceivable that a user may obtain (e.g., create) data to be printed using a device such as a PC (Personal Computer) and transmit the image data to the image forming device 100 via a network such as a LAN. If a device such as a PC is too large to be carried around or does not have an imaging unit 260, it may be difficult for the device to display the AR image IM3 or accept a resize operation. In this case as well, it is effective to perform the above-described processing related to the terminal device 200 in order to allow the user to perform a resize operation that takes into account the size of the print result.

[0069] However, in the above example using a USB memory, a PC, or the like, image data to be printed is transmitted to the image forming apparatus 100 from a device other than the terminal apparatus 200, and therefore the image data needs to be shared with the terminal apparatus 200.

[0070] FIG. 10 is a sequence diagram for explaining the processing of this modified example, and in particular, a diagram for explaining the processing for acquiring image data by the image acquiring unit 211 of the terminal device 200. In FIG.

[0071] First, in step S201, the user registers (transmits) image data to be printed to the image forming apparatus 100. Step S201 may be executed using, for example, a USB memory as described above, or may be executed using a device other than the terminal apparatus 200, such as a PC.

[0072] In step S202, the processing unit 110 of the image forming apparatus 100 performs processing to display a display object including information for identifying the acquired image data on the display unit 140. The display object here is, for example, a QR code (registered trademark), but other information may be used. The display object may include the image data itself, or may include a network address indicating the location of the image data on the network. For example, when the image forming apparatus 100 and the terminal device 200 are connected to the same network, the image data can be accessed from the terminal device 200 by using the network address of the storage unit 120 of the image forming apparatus 100 (specifically, an address indicating the hierarchical structure up to the image data).

[0073] In step S203, the user launches an application of the terminal device 200 and performs an image capturing operation. The application may be a print application or another camera application. The terminal device 200 launches the application based on the user's operation. This transitions to a state in which the image capturing unit 260 can capture an image.

[0074] In step S204, imaging unit 260 captures an image of a display object displayed on image forming apparatus 100. In step S205, image acquisition unit 211 acquires image data from image forming apparatus 100 based on the analysis processing result of the display object. The analysis processing here refers to reading information included in the display object by processing the display object according to the rules of the object. The analysis processing may be executed by image acquisition unit 211 or another component of processing unit 210.

[0075] In this way, even when image data stored in a device other than the terminal device 200 is to be printed, the image acquisition unit 211 of the terminal device 200 can acquire the image data without significantly increasing the burden on the user. As a result, the image processing unit 212 can execute the creation of superimposed image data IM2 and the creation of an AR image IM3, so that print size information can be appropriately determined regardless of the device that created the image data.

[0076] Specifically, after the process of step S205, the terminal device 200 determines the print size information by performing the processes from step S103 onward in Fig. 4. Furthermore, the image forming device 100 prints the image data using the determined print size information.

[0077] In the embodiment described above using steps S101 and S102 in FIG. 4, image data in the terminal device 200 is the printing target. In this case, the communication unit 230 of the terminal device 200 transmits the image data to the image forming device 100 in association with the print size information (step S112). When printing image data stored in the terminal device 200, it is necessary to transmit the image data from the terminal device 200 to the image forming device 100, and by clearly indicating the correspondence between the image data and the print size information, it becomes possible to print at an appropriate size. In step S112, it is assumed that the image data and the print size information are transmitted simultaneously, but as long as the correspondence is clear, it is not prevented that the transmission timings are separated.

[0078] On the other hand, when the image acquisition unit 211 acquires image data by the process shown in FIG. 10, the image data is already stored in the image forming apparatus 100. Therefore, there is little need to transmit the image data itself in step S112. Therefore, the communication unit 230 of the terminal device 200 may transmit information identifying the image data to the image forming apparatus 100 in association with the print size information. The information identifying the image data here may be identification information uniquely assigned to each of the multiple image data stored in the image forming apparatus 100, a file name, or other information. In this way, it becomes possible for the image forming apparatus 100 to appropriately determine the image data to be printed using the print size information. In addition, since there is no need to transmit the image data itself, it becomes possible to reduce the communication load.

[0079] The method of the present embodiment described above is not limited to being applied to the terminal device 200. For example, the method of the present embodiment can be applied to an image forming system 10 including the image forming device 100 and the terminal device 200. The terminal device 200 of the image forming system 10 acquires image data to be printed, acquires a captured image IM1 capturing an area including a planned arrangement location of the print result of the image data, generates superimposed image data IM2 by scaling the image data at a magnification corresponding to the size when the print result of the standard print size is arranged at the planned arrangement location based on the distance to the planned arrangement location and the standard print size of the image data, generates an AR image IM3 that displays the superimposed image data IM2 superimposed on the captured image IM1, accepts a size change operation of the superimposed image data IM2 in the AR image IM3, determines print size information representing the print size of the image data based on the size change operation, and transmits the print size information to the image forming device 100. The image forming device 100 of the image forming system 10 prints the image data based on the print size.

[0080] In addition, a part or all of the processing performed by the terminal device 200 (or, in a narrow sense, each part of the processing unit 210) of this embodiment may be realized by a program. The program according to this embodiment can be stored in a non-transitory information storage medium (information storage device), which is, for example, a medium readable by a computer. The information storage medium can be realized by, for example, an optical disk, a memory card, a HDD, or a semiconductor memory. The semiconductor memory is, for example, a ROM. The image forming device 100 performs various processing of this embodiment based on the program stored in the information storage medium. That is, the information storage medium stores a program for making a computer function as each part of the terminal device 200. The computer is a device equipped with an input device, a processing unit, a storage unit, and an output unit. Specifically, the program according to this embodiment is a program for making a computer execute each step described above with reference to FIG. 4, FIG. 10, etc.

[0081] For example, the program causes the computer to execute the following processes: acquire image data to be printed, acquire an captured image IM1 capturing an area including the intended placement location of the print result of the image data, generate superimposed image data IM2 by scaling the image data at a magnification corresponding to the size when the print result of the standard print size is placed in the intended placement location based on the distance to the intended placement location and the standard print size of the image data, generate an AR image IM3 superimposing and displaying the superimposed image data IM2 on the captured image IM1, accept a resize operation of the superimposed image data IM2 in the AR image IM3, determine print size information representing the print size of the image data based on the resize operation, and send the print size information to image forming device 100 which prints the image data.

[0082] The technique of this embodiment can also be applied to a print control method including the following steps. The print control method includes the steps of acquiring image data to be printed, acquiring a captured image IM1 capturing an area including a planned location for arranging a print result of the image data, generating superimposed image data IM2 by scaling the image data at a magnification corresponding to the size when a print result of the standard print size is arranged at the planned location based on the distance to the planned location and the standard print size of the image data, generating an AR image IM3 that displays the superimposed image data IM2 superimposed on the captured image IM1, accepting a size change operation of the superimposed image data IM2 in the AR image IM3, determining print size information representing the print size of the image data based on the size change operation, and transmitting the print size information to an image forming apparatus 100 that prints the image data.

[0083] Although the present embodiment has been described in detail as above, it will be easily understood by those skilled in the art that many modifications are possible without substantially departing from the novel matters and effects of the present embodiment. Therefore, all such modifications are intended to be included in the scope of the present disclosure. For example, a term described at least once in the specification or drawings together with a different term having a broader meaning or synonymy can be replaced with that different term anywhere in the specification or drawings. In addition, all combinations of the present embodiment and modifications are also included in the scope of the present disclosure. In addition, the configurations and operations of the terminal device, image forming device, etc. are not limited to those described in the present embodiment, and various modifications are possible. [Explanation of symbols]

[0084] 10...image forming system, 100...image forming device, 110...processing unit, 120...storage unit, 130...communication unit, 140...display unit, 150...image forming unit, 200...terminal device, 210...processing unit, 211...image acquisition unit, 212...image processing unit, 220...storage unit, 230...communication unit, 240...display unit, 250...operation unit, 260...imaging unit, 261...lens, 262...imaging element, D...distance, IM1, IM1a, IM1b...captured image, IM2, IM2a, IM2b, IM2c...superimposed image data, IM3, IM3a, IM3b...AR image

Claims

1. an image acquisition unit that acquires image data to be printed; an imaging unit that captures an image of an area including a location where a print result of the image data is to be arranged; an image processing unit that generates superimposed image data by scaling the image data at a magnification corresponding to a size when the print result of the standard print size is arranged at the intended arrangement location based on a distance to the intended arrangement location and a standard print size of the image data, and generates an AR (Augmented Reality) image in which the superimposed image data is superimposed on the captured image; an operation unit that accepts a size change operation of the superimposed image data in the AR image; A communication unit that communicates with the image forming apparatus; Including, The image processing unit includes: determining print size information representing a print size of the image data based on the size change operation; The communication unit is A terminal device that transmits the print size information to the image forming device.

2. In claim 1, The imaging unit includes: capturing an image of a display object displayed on the image forming device; The image acquisition unit includes: A terminal device that acquires the image data from the image forming apparatus based on a result of the analysis process of the display object.

3. In claim 1 or 2, The image processing unit includes: determining print medium information including at least one of a size and a type of a print medium to be used for printing the image data based on the resizing operation; The communication unit is A terminal device that transmits the print medium information to the image forming device.

4. In claim 1, The communication unit is A terminal device transmits the image data or information for specifying the image data to the image forming device in association with the print size information.

5. An image forming apparatus; A terminal device; Including, The terminal device Obtain image data to be printed; Acquire a captured image of an area including a location where a print result of the image data is to be arranged; generating superimposed image data by scaling the image data at a magnification corresponding to a size when the print result of the standard print size is arranged at the intended arrangement location based on a distance to the intended arrangement location and a standard print size of the image data; and generating an AR (Augmented Reality) image in which the superimposed image data is superimposed on the captured image; receiving a size change operation of the superimposed image data in the AR image; determining print size information representing a print size of the image data based on the resizing operation; Transmitting the print size information to the image forming device; The image forming apparatus includes: Printing the image data based on the print size information. Image forming system.

6. In claim 5, The image forming apparatus includes: An image forming system that determines print medium information including at least one of the size and type of print medium to be used for printing the image data based on the print size information, and prints the image data based on the print size information and the print medium information.

7. Obtain image data to be printed; Acquire a captured image of an area including a location where a print result of the image data is to be arranged; generating superimposed image data by scaling the image data at a magnification corresponding to a size when the print result of the standard print size is arranged at the intended arrangement location based on a distance to the intended arrangement location and a standard print size of the image data; and generating an AR image in which the superimposed image data is displayed superimposed on the captured image; In the AR (Augmented Reality) image, a size change operation of the superimposed image data is received; determining print size information representing a print size of the image data based on the resizing operation; transmitting the print size information to an image forming apparatus that prints the image data; A program that causes a computer to carry out a process.

8. Obtain image data to be printed; Acquire a captured image of an area including a location where a print result of the image data is to be arranged; generating superimposed image data by scaling the image data at a magnification corresponding to a size when the print result of the standard print size is arranged at the intended arrangement location based on a distance to the intended arrangement location and a standard print size of the image data; and generating an AR image in which the superimposed image data is displayed superimposed on the captured image; In the AR (Augmented Reality) image, a size change operation of the superimposed image data is received; determining print size information representing a print size of the image data based on the resizing operation; transmitting the print size information to an image forming apparatus that prints the image data; Printing control method.

Images