Program, information processing device, and control method for the information processing device.

Abstract

This can improve the accuracy of ink level detection. [Solution] The present invention provides a program that enables communication between an information processing device and a printing device having a window provided to allow a user to view the recording material stored in the storage section, to function as an imaging control means for controlling the imaging of the window, a first acquisition means for acquiring edges indicating the outline of the window from edges detected from the image of the window acquired by the imaging control means based on predetermined conditions, a second acquisition means for acquiring a feature quantity indicating the proportion of recording material occupying the storage section corresponding to the window based on the edges indicating the outline of the window acquired by the first acquisition means, and a control means for controlling the setting of the feature quantity on the printing device.

Description

Technical Field

[0001] The present disclosure relates to a program, an information processing apparatus, and a control method for an information processing apparatus.

Background Art

[0002] As one of the ink remaining amount detection methods in an inkjet printer, "dot count remaining detection" is known. This method measures the amount of ejected ink and the amount of ink consumed in the ejection recovery operation by counting the number of ejection times and the number of suction recovery operations, and detects the remaining amount of ink in the ink tank.

[0003] Patent Document 1 discloses a technique for correcting the remaining amount of ink managed by a printing apparatus by resetting the ink consumption amount calculated from the ink consumption history at the time of ink replacement and the ink consumption amount temporarily stored at the time of resetting the remaining ink amount due to ink replacement. Patent Document 2 discloses a technique for setting the remaining amount of ink in a printing apparatus based on a captured image of an ink remaining amount window of an ink tank in order to appropriately manage the remaining amount of ink replenished to the printing apparatus.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the method of estimating the remaining amount from the ink consumption history as in Patent Document 1, there is a risk that the error will become large. Further, in the method of Patent Document 2, it may be difficult to appropriately extract the ink remaining amount window when analyzing the captured image and perform appropriate remaining amount analysis. [Means for solving the problem]

[0006] A program according to one aspect of the present disclosure is characterized in that an information processing device, which is communicatively connected to a printing device having a window provided so that a user can view recording material contained in a storage section, functions as an imaging control means for controlling imaging of the window; a first acquisition means for acquiring edges indicating the outline of the window from edges detected from the image of the window acquired by the imaging control means based on predetermined conditions; a second acquisition means for acquiring a feature quantity indicating the proportion of the recording material occupying the storage section corresponding to the window based on the edges indicating the outline of the window acquired by the first acquisition means; and a control means for controlling setting the feature quantity to the printing device. [Effects of the Invention]

[0007] According to this disclosure, the accuracy of ink level detection can be improved. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example of a hardware configuration. [Figure 2] This is a schematic diagram showing an example of a printing device and an information processing device. [Figure 3] An example of a software configuration. [Figure 4] This is a schematic diagram illustrating an example of refilling ink in a printing machine. [Figure 5] This is a magnified view of the ink tank section. [Figure 6] This flowchart shows an example of a process performed by an information processing device. [Figure 7] This figure shows an example of a display screen shown on the display unit of an information processing device. [Figure 8] This is a diagram used to explain image processing. [Figure 9] This is a diagram used to explain image processing. [Figure 10] This is a diagram used to explain image processing. [Figure 11] It is a diagram for explaining image processing. [Figure 12] It is a diagram for explaining image processing. [Figure 13] It is a diagram for explaining image processing. [Figure 14] It is a diagram for explaining image processing. [Figure 15] It is a diagram showing an example of a display screen displayed on a display unit of an information processing apparatus. [Figure 16] It is a flowchart showing an example of processing performed by an information processing apparatus. [Figure 17] It is a diagram showing an example of a display screen displayed on a display unit of an information processing apparatus. [Figure 18] It is a diagram showing an example of a display screen displayed on a display unit of an information processing apparatus. [Figure 19] It is a diagram showing an example of a display screen displayed on a display unit of an information processing apparatus.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the matters of the present disclosure, and not all combinations of features described in the following embodiments are essential for the solution means of the present disclosure. For the same configuration, the same reference numerals will be used for description.

[0010] In the prior art, when an information processing apparatus performs color analysis to detect the remaining amount of ink in an ink tank from imaging data, preparations such as holding the size of the ink tank and the position of the ink tank with respect to the printing apparatus in the application in advance are required. In addition, in order for the information processing apparatus to detect the remaining amount of ink from the captured image, there is a need for the user to take a picture of the ink remaining amount window in accordance with the guide, or there may be an incorrect detection of the remaining amount of ink when the picture taking fails. Thus, it may be difficult for the information processing apparatus to accurately detect the ink remaining amount window and for the user to easily set the remaining amount of ink.

[0011] Therefore, in the prior art, there is a risk that the accuracy of ink remaining amount detection and the usability in the setting operation of the ink remaining amount will decrease. Hereinafter, embodiments for improving the accuracy of ink remaining amount detection and usability will be described.

[0012] <<First Embodiment>> In this embodiment, a printing system having a printing device (recording device) and an information processing device which is an external device of the printing device will be described as an example. The printing device in this embodiment will be described by taking an inkjet printer as an example. Note that the recording method of the printer may be an electrophotographic method.

[0013] The information processing device in this embodiment will be described by taking a smartphone as an example. Although details will be described later, in this embodiment, an example of managing the remaining amount of ink (recording material, recording agent) of the printing device using an image captured using the imaging function of the information processing device will be described. For this reason, the information processing device will be described by taking a smartphone as an example, but the information processing device may be a notebook PC or a tablet terminal equipped with a camera. Note that the information processing device may acquire an image captured by another camera device which is a device different from the information processing device, and the processing described later may be performed. That is, the information processing device may not have a photographing function.

[0014] FIG. 1 is a block diagram showing an example of the hardware configuration of the printer 100 and the smartphone 150. FIG. 2 is a schematic diagram showing an example of the printer 100 and the smartphone 150. FIG. 3 is a block diagram showing an example of the software configuration of the printer 100 and the smartphone 150. Hereinafter, the configurations of the printer 100 and the smartphone 150 will be described respectively.

[0015] <Configuration of Printer> Figure 1(a) is a block diagram showing an example of the hardware configuration of printer 100. Printer 100 includes a CPU 101, ROM 102, RAM 103, network communication unit 104, network connection unit 105, display unit 106, operation unit 107, scanner unit 108, printer unit 109, and ink tank unit (storage unit) 110.

[0016] The CPU 101 is a central processing unit for controlling the various parts of the printer 100. The ROM 102 stores various program codes. The RAM 103 temporarily stores image data and performs buffering, for example, when executing operations such as printing.

[0017] The network communication unit 104 connects to a router (not shown) via the network connection unit 105. In other words, the network communication unit 104 is configured to connect to a network such as the Internet via the network connection unit 105 and communicate with other devices. The RAM 103 also stores image data and other data received by the network communication unit 104.

[0018] The display unit 106 is composed of, for example, a liquid crystal display and is capable of displaying characters, graphics, and indicators. The display unit 106 is not limited to a liquid crystal display; it can also be composed of LEDs or other display devices. The information displayed by the display unit 106 may include, for example, the settings information of the printer 100 or information about the ink contained in each ink tank. The information about ink may include, for example, information about the remaining amount of ink in the ink tank or information about ink replenishment instructions. The operation of the display unit 106 is controlled by the CPU 101.

[0019] The operation unit 107 includes switches or hard keys for the user to perform various input operations. Alternatively, the display unit 106 may function as a touch panel, receiving input operations from the user.

[0020] In the printer 100, the scanner function is implemented by the scanner unit 108, and the printing function is implemented by the printer unit 109. Ink is stored in the ink tank unit 110.

[0021] As shown in Figure 2, the ink tank section 110 is configured such that at least its front surface is made of a transparent material such as plastic or glass, allowing the user to see the ink inside. A portion of the ink tank cover 401, which will be described later, is provided so as to overlap with the front portion of the ink tank section 110 from which the remaining ink level can be seen, and is also referred to as the ink level window. In this embodiment, the ink level window is made of a transparent material.

[0022] As will be described later, the ink tank section 110 is equipped with an ink inlet 402 (see Figure 4) and is configured so that the user can replenish the ink from an ink bottle 403 (see Figure 4).

[0023] The printer unit 109 has an inkjet head, and ink is supplied to the inkjet head from the ink tank unit 110 via a tube. The printer unit 109 prints an image on a recording medium such as printing paper using an inkjet method, based on image data received from an external source or image data read from the scanner unit 108. Furthermore, the printer unit 109 also manages ink information for the ink tank, including information on the remaining ink level (e.g., ink level value), and paper information, including information on the number of sheets of paper loaded.

[0024] Ink level information is managed to properly notify the user when it is time to replace the ink. For example, the ink level value will show 100% immediately after the ink has been replaced and 0% when the ink needs to be replaced. If the ink level is not set correctly in the printer unit 109, the printed area may appear faded even though the printer recognizes that there is still ink remaining. Conversely, if the ink consumption dot count is not reset to 0 after the ink has been replaced, the printer may issue an ink-out warning or error to the user even though there is still ink available.

[0025] Ink level information may be stored and managed in non-volatile memory (HDD) or the like (not shown). The scanner unit 108 optically reads the document placed on the document glass 201 (Figure 2), converts it into electronic data, and then transmits the image data, which has been converted into a specified file format, to an external device via the network or stores it in a storage area such as an HDD. The copy function is achieved by the scanner unit 108 reading the document placed on the document glass 201 and transferring the generated image data to the printer unit 109. The printer unit 109 then stores the image data on a storage medium based on that image data. Each of the above-mentioned units is interconnected by the bus 111, and can receive and transfer data to and from each other.

[0026] Figure 3(a) shows an example of the software configuration of printer 100. The control program of printer 100 is broadly divided into three blocks. These three blocks are the application layer 310 which manages applications, the middleware layer 320 which controls the device via various interfaces (I / F), and the operating system 330 which manages overall control.

[0027] The roles of each of the three blocks are explained below. The operating system (OS) 330 provides the basic functions for the control unit, such as the CPU 101, to execute control programs. The middleware layer 320 consists of a group of software that controls the interfaces with each physical device in the printer 100. The middleware layer 320 includes a printer control module 321, a scanner control module 322, an operation unit control module 323, and an image processing module 324.

[0028] The printer control module 321 controls the printer interface and manages printing and general operations. The scanner control module 322 controls the scanner interface. The operation unit control module 323 controls the operation unit interface. The image processing module 324 controls the image data. Printing is performed when the image processing module 324 passes the controlled image data to the printer control module 321. When using the copy function, the image processing module 324 reads the scanned image data from a scan image data memory (not shown) where the scanned image is stored, and similarly passes it to the printer control module 321 to realize the copy function.

[0029] The application layer 310 includes a job management application 311 and a sensor control application 312. The application layer 310 realizes the functions that the printer 100 provides to the user, such as printing, by operating each device via the respective middleware. For example, when the operation unit control module 323 detects a print execution instruction from the user via the operation unit 107, the print execution instruction is notified to the application layer 310. In the application layer 310, upon receiving the print execution instruction, the job management application 311 executes the print.

[0030] The job management application 311 uses the printer control module 321 of the middleware layer 320 to execute printing operations using the inkjet head. The job management application 311 controls the content displayed on the operation unit 107 via the operation unit control module 323 as needed, and controls the operation of the printer part via the printer control module 321. The sensor control application 312 controls various sensors.

[0031] <Smartphone Configuration> Figure 1(b) is a block diagram showing an example of the hardware configuration of a smartphone 150. The smartphone 150 includes a CPU 151, ROM 152, RAM 153, network communication unit 154, network connection unit 155, display unit 156, operation unit 157, imaging unit 158, control unit 159, and image processing unit 160.

[0032] The CPU 151 is a central processing unit for controlling the various parts of the smartphone 150. The ROM 152 stores various program codes. The RAM 153 is a temporary storage area used as work memory. Additionally, image data captured by the smartphone's camera and applications downloaded to the smartphone are stored in the RAM 153 or other non-volatile memory (not shown) within the smartphone 150.

[0033] The network communication unit 154 connects to a router (not shown) via the network connection unit 155. In other words, the network communication unit 154 is configured to connect to a network such as the Internet via the network connection unit 155 and communicate with other devices. Furthermore, the smartphone 150 can make calls with other mobile devices via the network communication unit 154.

[0034] The display unit 156 is composed of a liquid crystal display or the like and is capable of displaying characters, graphics, and indicators. The operation unit 157 is composed of a touch panel for the user to perform various input operations and is configured to accept user input operations. The CPU 151 performs display control to display characters, graphics, and indicators on the display unit 156.

[0035] The imaging unit 158 ​​captures an image. The CPU 151 performs imaging control to capture an image of the subject via the imaging unit 158 ​​using a printer control application described later. The display unit 156 can also display a preview image from the imaging unit 158. The control unit 159 exchanges data with the image processing unit 160 and instructs the image processing unit 160 to perform processing. The image processing unit 160 performs various image processing. For example, the image processing unit 160 can perform processing to obtain the ink level value from the image obtained when the imaging unit 158 ​​captures the ink level window of the printer 100. Note that in Figure 1(b), the CPU 151, control unit 159, and image processing unit 160 are shown as separate blocks, but all of these may be implemented by the CPU 151. Each of the above-mentioned units is interconnected by the bus 161, and can receive and pass data to each other.

[0036] In this embodiment, the smartphone 150 performs a process to capture an image of the ink level window of the printer 100 at the user's request. The smartphone 150 then estimates the ink level of the ink tank 110 from the captured image, obtains an ink level value, and sends the converted ink level value to the printer 100. Further details will be described later.

[0037] Figure 3(b) shows an example of the software configuration of smartphone 150. The control program of smartphone 150 is broadly divided into three blocks. These three blocks are the application layer 340, which manages applications; the middleware layer 350, which controls the device via various interfaces; and the operating system 360, which manages overall control. The role of each will be explained below.

[0038] The operating system 360 provides the basic functions for control units such as the CPU 151 to execute the smartphone control program. The middleware layer 350 consists of a set of software that controls the interfaces with each physical device in the smartphone 150. The middleware layer 350 includes a smartphone control module 351, a communication control module 352, an operation unit control module 353, an imaging unit control module 354, and an image processing module 355.

[0039] The smartphone control module 351 controls the smartphone 150 and manages all aspects of its operation. The communication control module 352 controls the communication interface. The operation control module 353 controls the operation interface. The imaging unit control module 354, which controls the imaging unit 158, is used to capture images of the ink level window of the printer 100. The image processing module 355, which controls the image data, is used for processes such as analyzing the captured image data to obtain ink level values.

[0040] The application layer 340 implements the functions of a smartphone by operating each device through its respective middleware. The application layer 340 has a smartphone application 341. The smartphone application 341 of the application layer 340 of the smartphone is launched and operated via the operation unit 157.

[0041] In this embodiment, a printer control application is used as the smartphone application 341. The printer control application has the function of sending a command to the printer 100 to print an image held by the smartphone 150. The printer control application also has the function of displaying the ink level information of the printer 100 on the smartphone application. In this embodiment, the printer control application is configured to activate the imaging unit 158 ​​(camera). For example, based on the user's operation, the printer control application performs a process to capture an image of the ink level window of the printer 100. The printer control application then performs a process to obtain the ink level value from the captured image of the ink level window via the image processing module 355. Details will be described later.

[0042] <How to refill the ink in the printer> Figure 4 is a schematic diagram showing an example of replenishing ink (liquid), a consumable, in a printer 100, which is an example of a printing device. When the ink tank cover 401 of the printer 100 is opened, the user can access the ink tank section 110.

[0043] In the example shown in Figure 4, the four ink tanks 110 are located on the right side of the front of the printer 100, as viewed from the user operating the printer 100. Specifically, the cyan ink tank 110C, magenta ink tank 110M, yellow ink tank 110Y, and black ink tank 110BK are arranged from left to right as viewed from the user.

[0044] The basic configuration of each ink tank section in this embodiment is substantially the same. Furthermore, although the ink tank section 110 and the ink tank cover 401 are located on the front of the printer 100, the installation configuration is not limited to this. For example, the ink tank cover 401 may be located elsewhere on the printer 100, as long as it allows the user to visually check the remaining ink level in the ink tank section 110. In the following description, each ink tank section will be denoted by reference numeral 110 when described comprehensively.

[0045] An ink inlet 402 is provided at the top of each ink tank section 110. The user opens the cap that is blocking the ink inlet 402, inserts the tip of the ink bottle 403 into the ink inlet, and supplies ink from the ink bottle 403 to the ink tank section 110. As mentioned above, the ink level window is part of the ink tank cover 401, but the configuration of the ink level window is not limited to this. For example, the configuration including the front surface of the ink tank section 110 may be considered the ink level window.

[0046] As will be described later, the front surface of the ink tank section 110 in this embodiment has a scale that serves as an indicator of the remaining ink level, but it is not limited to this. For example, the scale that serves as an indicator of the remaining ink level may be provided in the ink level window of the ink tank cover 401. Details of the scale will be described later.

[0047] In this embodiment, the printer 100 is described as a color printer that ejects multiple inks of different colors to record a color image on a recording medium. However, the printer 100 is not limited to this example and may be a monochrome printer that performs printing using black ink, or a printer that can be equipped with five or more inks. Furthermore, the ink tank section 110 is not limited to being concentrated on one side (right side) of the front, but may be separated and arranged on both sides (left and right) of the front.

[0048] Figure 5 is an enlarged view of the ink tank section 110 of the printer 100. The example in Figure 5 is a schematic diagram showing the external appearance of the ink tank section 110, including the ink tank cover 401. In this embodiment, the ink tank cover 401 is provided with an ink level window 501 that overlaps with the transparent front portion of the ink tank section 110. The ink level window 501 is part of the ink tank cover 401 that allows the user to visually check the ink level of each ink tank. The user can visually check the actual amount of ink stored in each ink tank section 110 through the ink level window 501. A scale 503 that shows the ink level in steps is provided so that it can be seen by the user through the ink level window 501. In this embodiment, the scale 503 is described as being provided on the ink tank section 110 side, but it is not limited to this example. The scale 503 may also be provided on the ink tank cover 401 on the printer 100 main body side. The ink level 502 represents the actual amount of ink stored in each ink tank. In other words, the user can visually check the ink level in each ink tank by looking at the ink level in each ink tank and the scale 503.

[0049] <Smartphone Processing> Figure 6 is a flowchart showing an example of processing performed by the smartphone 150 in this embodiment. The processing shown in Figure 6 is achieved when the CPU 151 of the smartphone 150 reads the control program and the printer control application program stored in the ROM 152 into the RAM 153, and the CPU 151 executes them. Note that some or all of the functions of the steps in Figure 6 may be implemented by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process means that it is a step in the flowchart (the same applies to flowcharts in this specification hereafter).

[0050] The process shown in Figure 6 is one of the processes executed when a user launches a printer control application, which is a smartphone application 341. Each process shown in Figure 6 is performed by the launched printer control application. For simplicity of explanation, the printer control application, which is a smartphone application 341, will be simply referred to as application 341 below. The printer control application can be any application that can exchange information with the printer 100, and it can be a dedicated application or not. The process in Figure 6 starts when application 341 is launched by the user.

[0051] In S601, the CPU 151 connects the smartphone 150 and the printer 100 via application 341. For example, the smartphone 150 and the printer 100 are wirelessly connected via a router (not shown).

[0052] In S602, the CPU 151 receives a selection request from the user for the ink level imaging function via application 341. As mentioned above, application 341 has various functions, such as a printing function that sends a print command to the printer 100, but here we will explain it assuming that the user has selected the ink level imaging function.

[0053] In S603, the CPU 151 displays the ink level capture screen on the display unit 156 of the smartphone 150 via application 341. Figure 7 shows an example of the ink level capture screen. This screen is for the user to take a picture of the ink level. Specifically, this screen is for the user to take a picture of the ink level window, which is positioned to overlap with the front of the ink tank.

[0054] The message display area 701 displays messages related to taking pictures of the ink level window. The ink level display section 702 is an area that displays the ink level values ​​acquired by the image processing module 355 in steps S604-S611, which will be described later. As shown in Figure 7, if the image processing module 355 has not acquired the ink level values ​​corresponding to each ink tank, the ink level display section 702 displays an initial value. The ink level display section 702 displays a hyphen as the initial value. The ink level display section 702 displays the remaining amount of each ink tank 110 in percentage.

[0055] The ink levels for the cyan ink tank 110C, the magenta ink tank 110M, and the yellow ink tank 110Y are displayed on 702C, 702M, and 702Y, respectively. The ink level for the black ink tank 110BK is also displayed on 702BK, similar to cyan, magenta, and yellow.

[0056] The preview image display unit 703 displays the image (video) acquired from the lens and image sensor of the imaging unit 158. The guidelines 704 are displayed overlaid on the preview image display unit 703. By referring to the overlaid guidelines 704 (markers), the user can easily determine the position and orientation of the smartphone 150 when photographing the ink level windows of all colors.

[0057] The OK button 705 is a button that accepts an instruction to set the ink level value displayed on the ink level display unit 702 to the printer 100. The Cancel button 706 is a button that accepts an instruction to end the capture of the ink level window. If the Cancel button 706 is pressed by the user, the CPU 151 closes the ink level capture screen shown in Figure 7 by application 341.

[0058] In S604, the CPU 151 acquires an image (video) from the lens and image sensor of the imaging unit 158 ​​via the image processing module 355 using application 341. S604 to S611 are processes performed each time an image (video) is acquired, and are performed at a maximum rate of FLAMES PER SECOND (FPS) per second of the smartphone 150. Figure 8 shows an example of an acquired image of the ink level window.

[0059] In S605, the CPU 151, via application 341, crops the area of ​​the ink level window captured in S604 through the image processing module 355, specifically the area corresponding to guideline 704. The cropped area in S605 is the same region as guideline 704 and is highly likely to contain the ink level window. It is desirable that guideline 704 is prominently displayed in the center of the captured image acquired by the imaging unit 158. For example, guideline 704 is displayed as a square with sides equal to 90% of the length of the shorter side of the rectangular captured image. The image cropped in S605 is treated as the region containing the ink level window in subsequent processing. Figure 9 shows an image containing the region of the ink level window cropped from the captured image in Figure 8.

[0060] In S606, the CPU 151 detects edges via the image processing module 355 using application 341. In this embodiment, an edge refers to a boundary between adjacent colors in an image that has a brightness difference of a predetermined threshold or more. For example, the CANNY method is a known algorithm for detecting edges in an image. In S606, the image processing module 355 can obtain the detected edges as an array of positions for each consecutive edge.

[0061] Figure 10 shows an image obtained by converting the captured image cropped in S605 into a binary image. The white lines in Figure 10 indicate edges detected in S606. As shown in Figure 10, edges different from the edges indicating the outline of the ink level window may be detected. For example, in Figure 10, edge 1001 is a linear noise edge. Edge 1002 is an edge representing a shadow. Edge 1003 is an edge indicating the outline of a button and is an edge that is in contact with the frame that constitutes the outer shape of the cropped captured image. Edge 1004 is a linear edge indicating scale 503. Edge 1005 is an edge indicating the outline of the ink level window. Edge 1006 is an edge representing ink stains. Edge 1007 is an edge representing the ink level 502. Edge 1008 is an edge indicating the outline of the display unit that displays the ink color corresponding to the ink tank.

[0062] In edge detection using the S606, the CPU 151 may perform image pre-processing and post-processing to improve edge detection accuracy. For example, because the detected edge lines may be broken, the CPU 151 may not be able to obtain the outline of the ink level window in the processing described later. Therefore, the CPU 151 can perform morphological transformation (such as dilation or stenosis) as post-processing to connect the broken lines of fine edges. Alternatively, color space modification may be performed as pre-processing.

[0063] In S607, the CPU 151 acquires a predetermined edge via the image processing module 355 using application 341. In this embodiment, a predetermined edge is an edge that satisfies the following conditions. The specific conditions are listed below. The first condition is that the end of the edge is closed. The second condition is that the edge does not touch the frame that constitutes the outline of the captured image. The third condition is that the second edge is one of the edges that satisfies the first condition and is not surrounded by the first edge.

[0064] Therefore, in S606, of the edges detected from the captured image, edges 1001 and 1004 are removed according to the first condition. Then, edge 1003 is removed according to the second condition. Furthermore, edges 1006 and 1007 are removed according to the third condition. As a result, in S607, the CPU 151 acquires edges 1002, 1005, and 1008 as predetermined edges. Figure 11 shows the predetermined edges acquired in S607.

[0065] In S608, the CPU 151, via application 341 and image processing module 355, acquires edges indicating the outline of the ink level window. Specifically, the CPU 151 acquires, from among predetermined edges acquired in S607, edges that have an area greater than or equal to a predetermined threshold relative to the area cut out in S605 as edges indicating the outline of the ink level window. Figure 12 shows the edges indicating the outline of the ink level window acquired in S608. For example, in S608, the CPU 151 removes edges 1002 and 1008, which have an area of ​​less than 10% of the cut-out area, from the image shown in Figure 11. Then, the CPU 151 acquires four edges, including edge 1005, as edges indicating the outline of the ink level window. That is, since the area enclosed by each of the four edges including edge 1005 is 10% or more of the area cut out in S605, the CPU 151 acquires these edges.

[0066] In S609, the CPU 151 determines, via the image processing module 355, whether or not an edge indicating the outline of the ink level window was acquired in S608 by the application 341. If an edge was acquired after processing in S607 and S608, the CPU 151 determines that an edge indicating the outline of the ink level window was acquired (Yes) and proceeds to S610. In this embodiment, the CPU 151 can acquire four edges indicating the outline of the ink level window from the captured image shown in Figure 8 via S607 and S608, and therefore proceeds to S610.

[0067] In S609, if the CPU 151 cannot obtain an edge from the captured image, it determines that it has not obtained an edge indicating the outline of the ink level window (No), returns to S604, and reads the next image (frame).

[0068] Figure 13 shows an example of an image obtained by photographing the ink level window from a different position than the position from which the image shown in Figure 8 was taken. In the image shown in Figure 13, a part of the ink level window extends outside the guideline, so the CPU 151 may not be able to obtain the edge indicating the outline of the ink level window from the image through processing in S605-S608. In such cases, in S609, the CPU 151 returns to S604 and reads the next image.

[0069] In S610, the CPU 151, using application 341, obtains the ink level value for each ink level window corresponding to an edge from the captured image in S608. The ink level value can be determined, for example, in the captured image shown in Figure 9, by deriving the ratio of the area occupied by the area enclosed by the edge indicating the amount of ink stored to the area enclosed by the edge indicating the contour of the ink level window. Alternatively, the ink level value can be determined from the relative position of the edge indicating the liquid level of ink stored in the ink tank relative to the edge indicating scale 503.

[0070] Figure 14 shows an example of an image extracted from the captured image shown in Figure 9, specifically the region enclosed by the outline of the ink level window. As a method for extracting the ink portion within the outline of the ink level window of the extracted image, for example, a method is known in which upper and lower limits are set for the color space parameters, and a group of pixels having a color within this range is extracted. In this embodiment, the ink portion within the outline of the ink level window represents the amount of ink stored and is sometimes referred to as a feature quantity. For example, if the ink color is cyan, the hue is set to a range of 0 to 180. The saturation is set to a range of 155 to 255. Furthermore, the brightness is set to a range of 0 to 255. In other words, a group of pixels having parameters with these ranges set for cyan is extracted. Then, in S610, the CPU 151 can obtain the ink portion 1402 by extracting these pixel groups. For example, if the ink portion 1402 is extracted from the cropped image shown in Figure 14, the CPU 151 can obtain an ink level value where the proportion of the ink portion 1402 in the ink level window image 1401 corresponding to edge 1005 is 80 percent.

[0071] In S611, the CPU 151 displays the ink level value obtained in S610 via application 341. For example, the CPU 151 displays the ink level value obtained in S610 in percentage units on the ink level display unit 702 in Figure 7. If the CPU 151 could not obtain the ink level value, it displays a hyphen on the ink level display unit 702.

[0072] In S612, the CPU 151 determines whether the cancel button 706 was pressed by the application 341. If the CPU 151 determines that the cancel button 706 was pressed (Yes), it proceeds to S616, closes the ink level capture screen shown in Figure 7, and ends this flow. If the CPU 151 determines that the cancel button 706 was not pressed (No), it proceeds to S613.

[0073] In S613, the CPU 151 determines whether the OK button 705 was pressed by the application 341. If the CPU 151 determines that the OK button 705 was pressed (Yes), it proceeds to S614. If the CPU 151 determines that the OK button 705 was not pressed (No), it returns to S604 and loads the next image.

[0074] In S614, the CPU 151 determines, based on application 341, whether the ink level value acquisition result is correct. Figure 15 shows an example of the ink level capture screen after the OK button 705 is pressed in S613. Figure 15 shows an example where the ink level value acquired in S610 is displayed.

[0075] In S614, CPU151 does not read images (frames). In S614, when CPU151 detects that the OK button 705 is pressed in S613, it displays a message to the user to confirm whether the ink level value acquisition result is correct, as shown in Figure 15. Specifically, CPU151 prompts the user to confirm whether the ink level value acquisition result is correct by displaying a confirmation message 1501 in the message display area 701 that reads, "Press OK to reflect this in the printer's ink level."

[0076] In S614, if the CPU 151 detects that the OK button 705 has been pressed, it determines that the ink level value has been obtained correctly (Yes) and proceeds to S615. If the CPU 151 detects that the Cancel button 706 has been pressed, it determines that the ink level value has been obtained incorrectly (No) and returns to S604 to load the next image (frame).

[0077] In S615, the CPU 151 controls the printer 100 to set the ink level via application 341. For example, the CPU 151 sends the result of acquiring the ink level value to the printer 100 via application 341. In addition to the result of acquiring the ink level value, the CPU 151 may also send an instruction to the printer 100 to set the ink level.

[0078] When printer 100 receives the result of acquiring the ink level value, it replaces the ink level value it holds with the received ink level value. Alternatively, printer 100 may correct the ink level value it has stored using the received ink level value. For example, it is unlikely that the amount of ink actually stored in the ink tank will change drastically compared to the ink level value held by printer 100. Therefore, in S615, printer 100 can perform a correction process that prevents replacement if the value before replacement is very large compared to the ink level value set for itself.

[0079] In S616, the CPU 151 terminates the processing of this flowchart after the ink level capture screen is closed by application 341. This concludes the explanation of the processing flow in smartphone 150.

[0080] According to this embodiment, the ink level set in the printing device can be appropriately managed by updating the ink level setting using an image of the ink level window captured by a smartphone, which is an external device of the printing device. Furthermore, since the ink level window can be accurately identified by capturing an image with a smartphone according to the guidelines, the ink level value can be obtained without impairing operability. In other words, according to this embodiment, ink level detection can be improved and the usability of setting the ink level can be improved.

[0081] Furthermore, because it can accurately acquire ink level values, it can improve ordering accuracy in systems that automatically order ink bottles (consumables) based on the remaining ink level in the ink tank. In addition, it becomes possible to accurately notify printer users of ink-out errors.

[0082] <<Second Embodiment>> An embodiment in which a warning message is displayed to the user when an edge indicating the outline of the ink level window cannot be obtained is described.Figure 16 shows a flowchart showing the process executed by the CPU 151 of the smartphone 150 in this embodiment.Figure 16 is a flowchart in which the process of displaying a warning message is added to the flowchart shown in Figure 6.If the CPU 151 cannot obtain an edge from the captured image obtained via the image processing module 355 in S609, it proceeds to S1601.

[0083] In S1601, the CPU 151 displays a warning message to the user so that application 341 can obtain the outline of the ink level window. Specifically, the CPU 151 displays the warning message in the message display area 701, as shown in Figure 17 or Figure 18.

[0084] Figure 17 shows an example where a message prompting the user to adjust the shooting position of the smartphone 150 so that the smartphone 150 can acquire the edge showing the outline of the ink level window is displayed in the message display area 701. For example, in the situation shown in Figure 17, part of the ink level window is outside the guideline, so the edge is not acquired as the outline of the ink level window by S607. Therefore, the user needs to take the picture so that it is inside the guideline.

[0085] Figure 18, similar to Figure 17, shows an example where a message prompting the user to adjust the shooting position on the smartphone 150 is displayed in the message display area 701. In the situation shown in Figure 18, the size of the ink level window is small relative to the area inside the guideline, so the edge of the ink level window may not be acquired as the outline by S608. Therefore, the user needs to take a picture of the ink level window at a larger size.

[0086] In the example screen shown in Figure 17, the S607 does not capture the outline of the ink level window. Therefore, a warning message, "Please photograph the ink level window so that it fits within the guidelines," is displayed in the message display area 701 to prompt the user to adjust the position of the ink level window when taking the photograph.

[0087] In the example screen shown in Figure 18, the S608 does not capture the outline of the ink level window. Therefore, a warning message, "Please take a larger picture of the ink level window so that it fits within the guidelines," is displayed in the message display area 701 to prompt the user to adjust the position of the ink level window when taking the picture.

[0088] According to this embodiment, if a portion of the ink level window is outside the guideline, or if the size of the ink level window is small relative to the area inside the guideline, the user can be prompted to adjust the position of the ink level window during shooting. This allows for efficient acquisition of the ink level value.

[0089] <<Other Embodiments>> This section describes an embodiment in which the CPU 151 displays the outline of the ink level window acquired in S608 to the user via application 341. Figure 19 shows an example in which the acquired ink level window outline is displayed on the ink level capture screen of Figure 7. The ink level window portion 1901 shows an example of the display of the acquired ink level window outline. The CPU 151 overlays the ink level window outline portion acquired in S608 onto the captured image acquired in S604. By displaying the ink level window outline portion, the user can confirm which part of the captured image was acquired as the ink level window.

[0090] In the embodiments described above, the explanation mainly focused on an example where the printing device is an inkjet printer, but the printing device may also be an electrophotographic printer. Therefore, in the embodiments described above, ink was used as an example of the recording material, but toner may also be used as the recording material. That is, in the embodiments described above, the amount of ink was used as an example of the amount of recording material, but the amount of recording material may also be the amount of toner.

[0091] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0092] The disclosure of this embodiment includes configurations represented by the following example programs, example information processing devices, and example control methods for information processing devices.

[0093] <Configuration 1> A printing device having a window that allows the user to view the recording material stored in the storage compartment, and an information processing device that is communicatively connected to the printing device, An imaging control means for controlling the imaging of the aforementioned window, A first acquisition means that acquires an edge indicating the outline of the window from among the edges detected from the captured image of the window acquired by the imaging control means, based on predetermined conditions, A second acquisition means acquires a feature quantity indicating the proportion of the recording material occupying the housing corresponding to the window, based on the edge indicating the outline of the window acquired by the first acquisition means, A program characterized by its ability to function in a certain way.

[0094] <Configuration 2> The program according to configuration 1, characterized in that the predetermined condition is that the end of the edge indicating the outline of the window is in a closed state.

[0095] <Structure 3> The program according to configuration 1 or 2, characterized in that the predetermined condition is that the edge indicating the outline of the window does not touch the frame that constitutes the outer shape of the captured image.

[0096] <Structure 4> The program according to any one of configurations 1 to 3, characterized in that the predetermined condition is that the edge indicating the outline of the window is a first edge with a closed end and is not surrounded by a second edge with a closed end.

[0097] <Composition 5> The program according to any one of configurations 1 to 4, characterized in that the predetermined condition is that the size of the region enclosed by the edge representing the outline of the window is greater than or equal to a predetermined threshold with respect to the size of the captured image.

[0098] <Composition 6> The program according to any one of configurations 1 to 5, characterized in that, when the first acquisition means acquires an edge indicating the outline of the window, the information processing device is made to function as a display control means for controlling the display unit to overlay the outline onto the captured image.

[0099] <Composition 7> The program according to configuration 6, characterized in that, if the first acquisition means does not acquire an edge indicating the outline of the window, the display control means performs control to display a message prompting the user to adjust the shooting position of the window.

[0100] <Structure 8> The program according to any one of configurations 1 to 7, characterized in that the information processing device functions as a control means for controlling the setting of the feature quantities acquired by the second acquisition means to the printing device.

[0101] <Composition 9> The program according to configuration 8, characterized in that the control means performs control to transmit the feature quantities acquired by the second acquisition means to the printing device.

[0102] <Composition 10> An information processing device that is communicatively connected to a printing device having a window provided so that the user can view the recording material stored in the storage section, An imaging control means for controlling the imaging of the aforementioned window, A first acquisition means that acquires an edge indicating the outline of the window from among the edges detected from the captured image of the window acquired by the imaging control means, based on predetermined conditions, A second acquisition means acquires a feature quantity indicating the proportion of the recording material occupying the housing corresponding to the window, based on the edge indicating the outline of the window acquired by the first acquisition means, Control means for controlling the setting of the aforementioned feature quantities in the printing device. An information processing device characterized by comprising:

[0103] <Composition 11> A control method for an information processing device that is communicatively connected to a printing device having a window provided so that the user can view the recording material stored in the storage compartment, An imaging control step that controls the imaging of the aforementioned window, A first acquisition step involves acquiring an edge that represents the outline of the window from among the edges detected from the captured image of the window acquired in the aforementioned imaging control step, based on predetermined conditions. A second acquisition step involves acquiring a feature quantity that indicates the proportion of the recording material occupying the housing corresponding to the window, based on the edge indicating the outline of the window acquired in the first acquisition step, A control step that performs control to set the aforementioned feature quantities in the printing device. A control method for an information processing device, characterized by having the following features. [Explanation of Symbols]

[0104] 100 Printers 150 smartphones

Claims

1. A printing device having a window that allows the user to view the recording material stored in the storage compartment, and an information processing device that is communicatively connected to the printing device, An imaging control means for controlling the imaging of the aforementioned window, A first acquisition means that acquires an edge indicating the outline of the window from among the edges detected from the captured image of the window acquired by the imaging control means, based on predetermined conditions, A second acquisition means acquires a feature quantity indicating the proportion of the recording material occupying the housing corresponding to the window, based on the edge indicating the outline of the window acquired by the first acquisition means. A program characterized by its ability to function in a certain way.

2. The program according to claim 1, characterized in that the predetermined condition is that the end of the edge indicating the outline of the window is in a closed state.

3. The program according to claim 1, characterized in that the predetermined condition is that the edge indicating the outline of the window does not touch the frame that constitutes the outer shape of the captured image.

4. The program according to claim 1, characterized in that the predetermined condition is that the edge indicating the outline of the window is a first edge with a closed end and is not surrounded by a second edge with a closed end.

5. The program according to any one of claims 2 to 4, characterized in that the predetermined condition is that the size of the region enclosed by the edge representing the outline of the window is greater than or equal to a predetermined threshold with respect to the size of the captured image.

6. The program according to claim 1, characterized in that, when the first acquisition means acquires an edge indicating the outline of the window, the information processing device is made to function as a display control means that performs control to display the outline superimposed on the captured image on the display unit.

7. The program according to claim 6, characterized in that, if the first acquisition means does not acquire an edge indicating the outline of the window, the display control means performs control to display a message prompting the user to adjust the shooting position of the window.

8. The program according to claim 1, characterized in that the information processing device functions as a control means for controlling the setting of the feature quantities acquired by the second acquisition means to the printing device.

9. The program according to claim 8, characterized in that the control means performs control to transmit the feature quantities acquired by the second acquisition means to the printing device.

10. An information processing device that is communicatively connected to a printing device having a window provided so that the user can view the recording material stored in the storage section, An imaging control means for controlling the imaging of the aforementioned window, A first acquisition means that acquires an edge indicating the outline of the window from among the edges detected from the captured image of the window acquired by the imaging control means, based on predetermined conditions, A second acquisition means acquires a feature quantity indicating the proportion of the recording material occupying the housing corresponding to the window, based on the edge indicating the outline of the window acquired by the first acquisition means, Control means for controlling the setting of the aforementioned feature quantities in the printing device. An information processing device characterized by comprising:

11. A control method for an information processing device that is communicatively connected to a printing device having a window provided so that the user can view the recording material stored in the storage compartment, An imaging control step that controls the imaging of the aforementioned window, A first acquisition step involves acquiring an edge that represents the outline of the window from among the edges detected from the captured image of the window acquired in the aforementioned imaging control step, based on predetermined conditions. A second acquisition step involves acquiring a feature quantity that indicates the proportion of the recording material occupying the housing corresponding to the window, based on the edge indicating the outline of the window acquired in the first acquisition step, A control step that performs control to set the aforementioned feature quantities in the printing device. A control method for an information processing device, characterized by having the following features.

Images