Information processing device, control method for information processing device, program, and information processing system

Abstract

Improves the accuracy of identifying the correct ink color for each ink window. [Solution] The information processing device is characterized by comprising: an acquisition means for acquiring ink tank information relating to a plurality of ink tanks loaded into a recording device; an imaging means for photographing a plurality of ink windows corresponding to the ink tanks loaded into the recording device; a determination means for determining a number indicating the order of the plurality of ink windows photographed by the imaging means based on the image taken by the imaging means, which includes the ink windows and a specific part of the recording device; an identification means for identifying the type of ink tank corresponding to the ink window included in the image based on the determined number and the ink tank information; an extraction means for analyzing the image taken by the imaging means of the plurality of ink windows to extract the remaining amount of ink contained in the ink tank corresponding to each ink window; and a display control means for displaying the extracted remaining amount of ink for each ink tank on a display unit.

Description

Technical Field

[0006] , , , ,

[0001] The present disclosure relates to an information processing apparatus, a control method for the information processing apparatus, a program, and an information processing system.

Background Art

[0002] In a recording apparatus capable of replenishing ink in an ink tank, there is a recording apparatus provided with an ink window through which the remaining amount of ink in the ink tank can be visually confirmed. In Patent Document 1, a technique has been proposed in which a recording apparatus is photographed with a camera provided in an information processing apparatus which is an external device through an ink window, and the remaining amount of ink is calculated by analyzing the photographed data to update the ink remaining amount setting of the recording apparatus.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventionally, depending on the method of photographing the ink window, it may not be possible to determine which ink window among a plurality of ink windows the photographed ink window is located, and the ink color of the ink window may not be specified.

[0005] Therefore, an object of the present disclosure is to improve the accuracy of specifying the ink color corresponding to the ink window.

Means for Solving the Problems

[0006] An information processing device according to one aspect of the present disclosure is characterized by comprising: acquisition means for acquiring ink tank information relating to a plurality of ink tanks loaded into a recording device; shooting means for photographing a plurality of ink windows corresponding to the ink tanks loaded into the recording device; determination means for determining a number indicating the order of the plurality of ink windows photographed by the shooting means based on a photographed image including the ink windows and a specific location in the recording device; identification means for identifying the type of ink tank corresponding to the ink window included in the photographed image based on the determined number and the ink tank information; derivation means for analyzing the photographed image of the plurality of ink windows photographed by the shooting means to derive the remaining amount of ink contained in the ink tank corresponding to each ink window; and display control means for displaying the derived remaining amount of ink on a display unit for each ink tank. [Effects of the Invention]

[0007] According to this disclosure, the accuracy of identifying the ink color corresponding to the ink window can be improved. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram showing the external appearance of an image forming apparatus and a mobile terminal. [Figure 2] This block shows the hardware configuration of an image forming apparatus and a mobile terminal. [Figure 3] This is a block diagram showing the software configuration of an image forming apparatus and a mobile terminal. [Figure 4] This is a schematic diagram showing the method of injecting ink into the ink tank section of an image forming apparatus. [Figure 5] This is a diagram illustrating an example of how it appears on a mobile device. [Figure 6] This is an example of detecting the edge of the screen where the aircraft is located. [Figure 7] This is a diagram showing ink tank information. [Figure 8] This is a diagram showing the display screen of a mobile device. [Figure 9] This is a diagram showing the display screen of a mobile device. [Figure 10] It is a flowchart showing the execution of image analysis of the remaining ink amount in the ink tank. [Figure 11] It is a flowchart for determining the ink color of the ink window. [Figure 12] It is a schematic diagram showing the appearances of the image forming apparatus and the mobile terminal. [Figure 13] It is an example of detection of the screen edge of the machine body. [Figure 14] It is a schematic diagram showing the appearances of the image forming apparatus and the mobile terminal. [Figure 15] It is a diagram showing ink tank information. [Figure 16] It is an example of detection of the screen edge of the machine body.

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present disclosure will be described in detail. Note that the components described in this embodiment show forms as examples of the present disclosure, and the scope of this disclosure is not limited only to them.

[0010] <Embodiment 1> (Configuration of Printer) Hereinafter, the configuration of an inkjet printer 100 (hereinafter referred to as the printer 100), which is an example of a recording apparatus, will be described with reference to FIGS. 1 to 3.

[0011] FIG. 1 is a schematic diagram showing an example of the printer 100 and the smartphone 150, which is an information processing apparatus, in the information processing system of this embodiment. The printer 100 described in this embodiment includes an ink tank unit 110 for storing ink on the front surface. A plurality of types of ink tanks are loaded in the ink tank unit 110. The ink tank unit 110 is configured such that at least the front surface is made of a transmissive member such as plastic or glass, and the ink inside is configured to be visible to the user. The front surface portion of the ink tank unit 110 where the remaining ink amount is visible is also referred to as an ink window.

[0012] FIG. 2 is a block diagram showing an example of the hardware configuration of the printer 100 and the smartphone 150. First, the hardware configuration of the printer 100 will be described.

[0013] FIG. 2(a) is a diagram showing an example of the hardware configuration of the printer 100. The printer 100 includes a CPU 101, a ROM 102, a RAM 103, a network communication unit 104, a network connection unit 105, a display unit 106, an operation unit 107, a scanner unit 108, a printer unit 109, and an ink tank unit 110. The CPU 101 is a central processing unit for controlling each unit within the printer 100. The ROM 102 stores various program codes. Image data and the like are temporarily stored or buffered in the RAM 103 during the execution of each service. The ROM 102 and the RAM 103 are also referred to as a storage unit.

[0014] The network communication unit 104 is connected to a router via the network connection unit 105. That is, the network communication unit 104 connects to a network such as the Internet via the network connection unit 105 to perform communication. Image data and the like received by the network communication unit 104 are also stored in the RAM 103.

[0015] The display unit 106 is constituted by, for example, a liquid crystal display and is capable of displaying characters, graphics, indicators, and the like. The display unit 106 is not limited to a liquid crystal display and can also be constituted using an LED or other displays. Information displayed by the display unit 106 includes, for example, setting information for the printer 100 and information regarding the ink contained in each ink tank. Further, information regarding the ink includes, for example, remaining ink amount information and ink replenishment instruction information contained in the ink tank. Note that the driving of the display unit 106 is display-controlled by the CPU 101. The operation unit 107 includes switches, hard keys, and the like for the user to perform various input operations. Also, it is possible to adopt a form in which the display unit 106 functions as an operation unit for performing input operations and the like, such as a touch panel.

[0016] In the printer 100, the scanner function is implemented in the scanner unit 108, and the printing function is implemented in the printer unit 109. Ink is stored in the ink tank unit 110. The ink tank unit 110 is equipped with an ink inlet 402, and ink can be refilled from an ink bottle 403. The printer unit 109 has an inkjet head, and ink is supplied to the inkjet head from the ink tank unit 110 via a tube. The printer unit 109 also prints images on a recording medium such as printing paper using an inkjet method, based on image data received from an external source, image data read from the scanner unit 108, etc. Furthermore, the printer unit 109 also manages ink information for the ink tank, including ink level information, and paper information, including the number of sheets of paper loaded.

[0017] The scanner unit 108 optically reads the document placed on the document glass 201 and converts it into electronic data. It then converts the image data into a specified file format and transmits it to an external device via the network or stores it in a storage area such as an HDD. The copying function is achieved by scanning the document placed on the document glass 201 with the scanner unit 108, transferring the generated image data to the printer unit 109, and the printer unit 109 storing the image data on a storage medium. Each of the above units is interconnected by the bus 111, enabling them to send and receive data from each other.

[0018] Figure 3 shows the software configurations of printer 100 and smartphone 150. First, the software configuration of printer 100 will be described.

[0019] Figure 3(a) shows the software configuration of printer 100. Printer 100 has an application layer 320 for managing applications, a middleware layer 330 for controlling the device via various I / Fs, and an operation system 340 for managing overall control.

[0020] The operation system 340 provides the basic functions for executing control programs in the control unit. The middleware layer 330 consists of software that controls the interface with the printer or each physical device. The printer control module 331, which controls the printer interface, is responsible for printing operations and overall functionality. The middleware layer 330 also includes a scanner control module 332 that controls the scanner interface, an operation unit control module 333 that controls the operation unit interface, and an image processing module 334 that controls image data. The printer 100 passes the controlled image data to the printer control module 331 and performs printing. When using functions such as copying, the image data is read from the scanned image data memory where the scanned image is stored and similarly passed to the printing unit.

[0021] The application layer 320 implements functions such as printing provided by this printing device to the user by operating each device via middleware. For example, when the operation unit control module 333 detects that the user has performed a print operation via the operation unit, the application layer 320 is notified of this. The application layer 320 then executes the print job using the job management application 321.

[0022] The job management application 321 uses the printer control module 331 of the middleware layer 330 to execute printing operations using the ink head. If necessary, it controls the content displayed on the control panel via the control panel control module 333 and the operation of the printer via the printer control module 331. The main unit information management module 335 manages information such as model-specific information, ink levels, and the total number of ink tanks. It also manages information about the relationship between the position and color of the ink tanks. When the ink color determination module 336 receives position information obtained from the analysis results of the ink window 202 image data sent from the smartphone 150, it queries the main unit information management module 335 for information about the relationship between the position and color of the ink tanks. This allows the color of the captured ink tank to be determined.

[0023] (Smartphone configuration) Figure 2(b) is a block diagram showing an example of the hardware configuration of a smartphone 150. The smartphone 150 includes a CPU 151, ROM 152, RAM 153, network communication unit 154, network connection unit 155, display unit 156, operation unit 157, imaging unit 158, and image processing unit 160. The CPU 151 is a central processing unit for controlling each part of the smartphone 150. The ROM 102 stores various program codes. The RAM 153 stores image data captured by the smartphone's camera and applications downloaded to the smartphone.

[0024] The network communication unit 154 connects to the router via the network connection unit 155. That is, the network communication unit 154 connects to a network such as the Internet via the network connection unit 155 and performs communication. Furthermore, it is possible to make calls with other mobile terminals via the network communication unit 154.

[0025] The display unit 156 is composed of a liquid crystal display or the like, and is capable of displaying characters, graphics, and indicators. The operation unit 157 is a touch panel that allows the user to perform various input operations, and the display unit 156 functions as the operation unit for input operations. The imaging unit 158 ​​is capable of capturing images, and the captured images can be processed by the image processing unit 160. The imaging unit 158 ​​also captures images of the ink window, converts the captured ink level image to an ink level value, and transmits the converted ink level value to the printer body.

[0026] Figure 3(b) shows the software configuration of smartphone 150.

[0027] The smartphone 150 has an application layer 350 for managing applications, a middleware layer 330 for controlling the device via various interfaces, and an operation system 370 for managing overall control.

[0028] The operation system 370 provides basic functions for executing the smartphone control program in the control unit. The middleware layer 330 consists of a group of software that controls the interface with the smartphone and various physical devices. The smartphone control module 361 controls the overall operation of the smartphone. It also includes an operation unit control module 363 that controls the operation unit interface and an imaging unit control module 364 that controls the camera. The imaging unit control module 364 is used to capture the ink level window of the printer body, which is a feature of the present invention. Furthermore, it also includes an image processing module 365 that controls the image data. This is used when analyzing the captured ink level image data and converting it into an ink level value, which is a feature of the present invention.

[0029] The application layer 360 implements the functions of the smartphone provided to the user by operating each device via various middleware. The smartphone application 351 of the smartphone's application layer 320 operates via the control unit. In this embodiment, the user uses a printer control application as the smartphone application 351. The functions of the printer control application include issuing print commands for images on the smartphone. It also displays the printer's ink level information on the smartphone application. In this embodiment, the printer control application activates the camera to take a picture of the ink window on the printer body. The captured image of the ink level in the ink window is converted into a numerical value indicating the percentage of remaining ink.

[0030] (How to refill the ink in the printer) Figure 4 is a schematic diagram showing a method for replenishing liquid in a printer 100, which is an example of an image forming apparatus. Opening the ink tank cover 301 of the printer 100 provides access to the ink tank section 110. Ink is supplied to the ink tank section 110 from the ink bottle 303 by opening the cap on the ink inlet 302 after opening the ink tank cover 301. The ink tank section 110 is also called the loading section.

[0031] In this embodiment, the printer 100 is configured as a color printer that ejects multiple types of ink of different colors to record a color image on a recording medium. For this reason, four types of ink tanks are provided to store cyan, magenta, yellow, and black inks. Figure 4 shows the ink tank 110C which contains cyan liquid, the ink tank 110M which contains magenta liquid, the ink tank 110Y which contains yellow liquid, and the ink tank 110Bk which contains black liquid. The basic configuration of each ink tank in this embodiment is substantially the same.

[0032] (Mobile device app) Figure 5 shows the home screen of the application used to check the machine status. This application has three functions: ink tank photography, ink level, and connection settings. The title section 501 is where the screen title is written, and in this screen, the application name is listed as "Remote Connection". The ink tank photography function button 502 is a button that starts the function to analyze the ink level from the captured data. When this button is pressed, photography is performed through the imaging unit 158. The ink level display function button 503 is a button that starts the function to display the ink level of the printer 100. When this button is pressed, the system queries the main unit information management module 335 and displays the ink level on the display unit 156. The connection settings function button 504 is a button that starts the function to change the settings of the printer 100's communication unit. When this button is pressed, the communication settings in the main unit information management module 335 can be changed via communication.

[0033] (Example of detecting the aircraft's edge using image processing) Figure 6 shows an example of detecting the edge of the aircraft from a captured image. If the captured data is as shown in Figure 6(a), first, areas with large color differences in surrounding pixels are identified within the captured data. For the identified areas with large color differences, lines are drawn to indicate whether the area is enclosed by lines of a predetermined size. If the enclosed area is of a predetermined size, that area is determined to be an ink window. This determination allows the areas of ink tanks 110M, 110Y, and 110Bk and line 601 to be detected. Next, line 601 is analyzed to determine if it is a line that extends outside the upper part of the captured data. In the case of this captured data, line 601 is a line that extends outside the data, so it is determined to be the edge of the aircraft (the edge of the recording device). After that, image processing is performed on ink tanks 110M, 110Y, and 110Bk as ink tanks to analyze the remaining ink levels.

[0034] If the captured data is like that shown in Figure 6(b), then areas with large color differences within the image are identified, similar to Figure 6(a). Upon identification, the regions of ink tanks 110C, 110M, and 110Y can be detected, but no lines indicating the edges are detected. Since there are no lines indicating the edges in the captured data, it is determined that there are no edges on the aircraft. Subsequently, image processing is performed on ink tanks 110C, 110M, and 110Y as if they were ink tanks, and the remaining ink levels are analyzed.

[0035] If the captured data is as shown in Figure 6(c), then, as in Figure 6(a), areas with large color differences within the screen are identified as lines. As a result, all of these lines extend outside the captured data and are treated as the aircraft's edge. In this embodiment, if a line forms a predetermined size, it is determined to be a region; therefore, in this example, it is determined that there is no ink tank, and consequently, the capture is deemed to have failed.

[0036] (Relationship between the order of ink tanks and ink colors) Figure 7 shows information 700, which indicates the correspondence between the order of ink tanks and ink colors stored in the main unit information management module 335. Information 700 is also called ink tank information. The relationship between the order of ink tanks and ink colors is different for each model. In this embodiment, the ink tanks are numbered from right to left from the end of the machine (machine end), and these numbers are treated as ink tank positions 701. By matching these ink tank positions 701 with the corresponding ink color information 702, the order of the ink tanks and the ink colors are associated. It should be noted that the number of ink tanks in order shown in information 700 can also be considered as the total number of ink tanks loaded in the printer 100. Therefore, it can be considered that information 700 also includes information indicating the total number of ink tanks.

[0037] In this embodiment, the ink tanks are arranged from right to left as viewed from the printer in the order of Bk, Y, and MC. Therefore, when the position of the ink tank is determined to be 1, it indicates that it is located closest to the right as viewed from the printer 100. Conversely, when the position of the ink tank is determined to be 4, it means that it is located furthest to the left (farthest from the right edge) as viewed from the printer 100.

[0038] (Example of display on a mobile device) Figure 8 shows an example of a mobile device display. Each screen in Figure 8 is displayed on the display unit 156. Figure 8(a) is a screen instructing the user to take a picture of the ink window 202. The title field 801 shows the screen title and indicates to the user what the current screen is. In this screen, it shows the screen name "Taking a picture of the ink window". The message area 802 is an area for displaying messages to the user. In this screen, a message is displayed instructing the user to take a picture of the ink tank area, specifically the ink window 202. The shooting preview area 803 is an area that displays the image being captured and shows the user what kind of image will be taken. The image captured by the imaging unit 158 ​​is displayed in the area enclosed by the dotted line. The OK button 804 is a button that starts shooting when pressed by the user. The captured data is confirmed on the screen currently displayed in the shooting preview area 803. The CANCEL button 805 is a button that the user presses to interrupt the processing on this screen. When this button is pressed, the user returns to the screen that was displayed before this screen.

[0039] Figure 8(b) shows the screen displaying the shooting results. The title field 811 indicates the screen title, showing the user what the current screen is. In this screen, it displays the screen name "Shooting Results". The message area 812 is an area for displaying messages to the user. In this screen, it displays a message asking the user if they want to start the analysis on the screen displayed in the shooting results display area 813. The shooting results display area 813 is the area in Figure 8(a) that displays the image captured by the imaging unit 158 ​​when the OK button 804 is pressed. The OK button 814 is a button that starts the analysis of the shooting data when pressed by the user. The shooting data to be analyzed is the same data as the screen displayed in Figure 8(a). The CANCEL button 815 is a button that the user presses to interrupt the processing on this screen. When this button is pressed, the screen returns to the screen in Figure 8(a).

[0040] Figure 8(c) shows the screen displaying the analysis results. Specifically, it is the screen that is accessed when the OK button 814 is pressed in the screen shown in Figure 8(b). The title field 811 shows the screen title, indicating to the user what the current screen is. In this screen, it shows the screen name as "Analysis Results". The message area 812 is the area for displaying messages to the user. In this screen, it displays a message indicating that the ink level has been detected and asking the user whether to reflect the ink level in the main unit. The shooting result display area 823 is the area in Figure 8(a) that displays the image captured by the imaging unit 158 ​​when the OK button 804 is pressed. The OK button 824 is a button that, when pressed, reflects the analysis results. When the OK button 824 is pressed, the smartphone application 351 sends the ink level analysis results to the main unit information management module 335 for reflection. The CANCEL button 815 is a button that the user presses if there is a problem with the analysis results on this screen. If the CANCEL button 815 is pressed, it is necessary to return to the screen shown in Figure 8(a) and retake the image. The color label 826 is an area that displays the color information transmitted from the ink color determination module 336 to the smartphone application 351. By displaying it on top of the data of the identified ink tank, it indicates what color the target ink tank is. The remaining amount display unit 827 is an area that displays the remaining amount of each ink tank obtained from the analysis results. The remaining amount of the ink tanks is displayed as an analysis result obtained by the image processing module 365.

[0041] Figure 8(d) is a display screen showing the shooting results, similar to Figure 8(b). The shooting data shown in Figure 8(d) is data obtained when the user takes a picture without including the edges of the aircraft.

[0042] Figure 8(e) is a display screen showing the shooting results, similar to Figure 8(c). Unlike Figure 8(c), Figure 8(e) fails to analyze because the aircraft's end is not present in the captured data. In this case, only the message area 812 changes, displaying a message requesting that the camera be pulled back and the shooting be repeated.

[0043] Figure 9 shows an example of a mobile device display. Figure 9(a) is the same as Figure 8(e), showing the display screen of the shooting result. Since the aircraft edge is not present in the shooting data shown in Figure 9(a), the analysis has failed. Also, unlike Figure 8(e), the area of ​​the ink window in the shooting data is small (below a predetermined threshold), so it is determined that it is difficult to zoom out the camera and take another picture of the edge. This is because if the field of view is zoomed out any further, it will become impossible to detect the ink window in the shooting data. Therefore, the message area 812 displays a message requesting that the right side of the shooting data be photographed with the camera. In this embodiment, the message area shows a message to photograph the right side, but in reality, it is not limited to this. For example, if the information processing device recognizes, based on some information, that the aircraft edge is on the left side of the currently displayed ink window, the message area may display an instruction to photograph the left side.

[0044] Figure 9(b), like Figure 8(a), shows a screen instructing the user to photograph the ink window 202. Figure 9(b) shows that after the display of Figure 9(a), the user has photographed a new ink window and the end of the machine. In this way, by moving the camera in accordance with the message to photograph the right (or left) part of the photographed data, the currently displayed (identified) ink window goes out of the field of view. That is, by moving the field of view as in Figure 9(b), the ink tank shown in Figure 9(a) goes out of the field of view. Therefore, when displaying a message like Figure 9(a), the data is temporarily saved when photographing the currently displayed ink tank. The remaining ink level is displayed based on the number assigned to the target image, which is based on the temporarily saved photographed data (target image) and the photographed data taken again according to the message (second photographed image), as well as the information 700 shown in Figure 7. Details will be described later in the flowchart of Figure 10.

[0045] Figure 9(c) is a display screen showing the shooting results, similar to Figure 8(b). The data displayed in the shooting result display area 813 at this time is the shooting data captured by the imaging unit 158 ​​when the OK button 804 was pressed, as in Figure 9(b).

[0046] Figure 9(d) is a display screen showing the analysis results, similar to Figure 8(c). The shooting data displayed in the shooting result display area 823 at this time is the same shooting data that was displayed in Figure 9(a). In this embodiment, the positional relationship of the ink tanks can be determined based on the temporarily saved shooting data of the ink tanks (the shooting data in Figure 9(a)) and the shooting data including the edges (the data in Figure 8(c)). Therefore, it is possible to display the remaining amount of ink in the ink tanks corresponding to the ink windows in the target image.

[0047] Figure 9(e) shows an example of a screen that displays the ink level on the display unit 156. This screen is for the user to check the ink level. The title unit 831 is where the title of the screen is written, and in this screen, it says "Ink Level Display" to show the ink level to the user. The color specification unit 832 contains the color information of the ink tank unit 110 and indicates which color corresponds to the ink level display of the ink tank unit described later. The ink level unit 833 displays the remaining amount of each ink tank unit 110 in percentage. This ink level is obtained by querying the ink level information recorded in the main unit information management module 335. The ink level illustration unit 834 displays the ink level obtained by the smartphone application 351 querying the main unit information management module 335 in a memory divided into 10 sections.

[0048] Figure 9(f) shows an example of a screen displaying the ink level on the display unit 156. This screen is used when the user checks the connection settings of the printer 100. The title section 841 is where the screen title is written, and in this case, "Connection Settings" is written to indicate to the user that it is a connection settings function. The Wi-Fi function button 842 is a button for changing the Wi-Fi connection settings. Pressing this button changes the Wi-Fi connection settings via communication in the main unit information management module 335. The Wireless Direct function button 843 is a button for changing the Wireless Direct connection settings. Pressing this button changes the Wireless Direct connection settings via communication in the main unit information management module 335. The Setting Information Confirmation function button 844 is a button for confirming the connection settings. Pressing this button changes the Wireless Direct connection settings via communication in the main unit information management module 335.

[0049] (Flowchart for determining ink tank color) Figure 10 is a flowchart showing the process from having the user actually photograph the ink window 202 to displaying the correct ink level to the user. Each step in the flowchart in Figure 10 is performed by the CPU 151 of the smartphone 150 executing a control program that it has read from the ROM 152 and loaded into the RAM 153. This process also starts when the CPU 151 detects that the ink tank photograph function button 502 has been pressed by the user in the smartphone application 351. In the following flowchart, each step is executed by the various modules of the smartphone 150 functioning under the instructions of the CPU 151, but the CPU 151 is considered the main processor. In the following explanation, the symbol "S" represents a step.

[0050] In S1001, the CPU 151 displays an instruction to photograph the ink window on the display unit 156. This means that a screen like the one shown in Figure 8(a) is displayed. In S1002, the CPU 151 uses the imaging unit 158 ​​to photograph the ink window in response to the pressing of the OK button 804 shown in Figure 8(a).

[0051] In S1003, the CPU 151 displays the captured image taken in S1002 on the display unit 156. That is, it displays the screen shown in Figure 8(b). The message area 812 in Figure 8(b) displays a message indicating that the captured image will be analyzed.

[0052] In S1004, the CPU 151 determines whether the user has pressed the OK button 814 on the screen shown in Figure 8(b). That is, it determines whether it has received an instruction to analyze the captured image. If it determines that it has received an instruction to analyze the image, the CPU 151 proceeds to S1005; otherwise, it returns to S1003.

[0053] In S1005, the CPU 151 analyzes the captured image. In S1006, the CPU 151 determines whether an ink window was detected in the captured image. The method for detecting the ink window is as described above in Figure 6. If the CPU 151 determines that an ink window was detected, it proceeds to S1008; if it determines that it was not detected, it proceeds to S1007.

[0054] In S1007, the CPU 151 displays a message in message area 812 indicating that it could not detect the ink window and will take another picture of it. After that, the CPU 151 returns to S1002. In S1008, the CPU 151 determines whether the edge of the printer 100 was detected in the captured image. The method for detecting the edge of the printer 100 is as described above in Figure 6. If the CPU 151 determines that the edge of the printer 100 was detected, it proceeds to S1013; if it determines that it was not detected, it proceeds to S1009.

[0055] In S1009, CPU151 determines whether the ink window in the captured image is within the detection limit. The detection limit refers to the threshold area of ​​the ink window contained in the captured image. If the area of ​​the ink window is greater than the threshold, the ink window can be detected even if the image is taken slightly further away from the current field of view. On the other hand, if the area of ​​the ink window is less than or equal to the threshold, further reducing the field of view will prevent the ink window from being detected. In situations where the aircraft edge has not been detected, it is necessary to determine whether the captured image is within the detection limit in order to detect the aircraft edge, and to perform processing according to the determination result. In this embodiment, as an example, the area indicating the detection limit is assumed to be such that the captured image contains at least three ink windows. That is, for example, a captured image containing one or two ink windows will be determined not to be within the detection limit. In this case, an instruction to reduce the field of view and re-shoot will be given in the processing described later. If the CPU 151 determines that the ink windows in the captured data are within the detection limit, it proceeds to S1010; if it determines that they are not within the detection limit, it proceeds to S1012.

[0056] In S1012, the CPU 151 displays a message to the display unit 156 instructing it to take a picture of the edge of the aircraft and prompting the user to take another picture. That is, it displays the screen shown in Figure 8(e). Since the area of ​​the ink window is not at the detection limit (there is sufficient area to detect the ink window), the message area 812 in Figure 8(e) displays an instruction to zoom out and take another picture. After that, the CPU 151 returns to S1002.

[0057] In S1010, the CPU 151 temporarily stores information indicating that the detection limit has been reached, along with the captured data, in the image processing unit 160. It also notifies the user that there is no detection limit. In S1011, the CPU 151 prompts the user to retake the image by displaying a message in the message area 812 on the display unit 156 of the smartphone application 351, for example, to take a picture of the right side of the captured image. That is, it displays the screen shown in Figure 9(a). When the user presses the OK button 814 on the screen shown in Figure 9(a), the screen transitions to the screen shown in Figure 9(b). If another part is captured due to the message displayed in S1011, the image of the other part is called the second captured image. The reason for taking the second captured image is to detect the edge of the printer 100. Also, as explained above, the reason for temporarily storing the captured data in S1010 is that the ink window in the first captured image (target image) is outside the field of view due to the message in S1011. Since the information the user wants to know is solely the ink level in the ink window shown in the target image, the target image is temporarily saved to prevent it from being discarded due to reshooting. In the process described later in this flowchart, the process for identifying the ink color of the ink window shown in the target image is explained based on the target image and a second image showing the edge of the aircraft. After that, CPU 151 returns to S1002.

[0058] Based on the message displayed in S1012 and S1011, the user takes a second image in S1002. This process of taking the second image is repeated until the ink window is detected in S1006 and the end of the aircraft is detected in S1008.

[0059] In S1013, the CPU 151 determines whether the detected aircraft end is the left or right end. The method for determining which end it is is as described above in Figure 6. The determined end information is stored in the image processing unit 160.

[0060] In S1014, the CPU 151 determines whether the target image was within the detection limit. This can be determined by checking whether information indicating that the image was within the detection limit was stored in S1010. If the CPU 151 determines that the target image is within the detection limit, it proceeds to S1015; otherwise, it proceeds to S1016.

[0061] In S1015, the CPU 151 obtains the number of ink windows in the temporarily saved target image. In S1016, the CPU 151 sends to the printer 100 information indicating that the target image was within the detection limit, information on the left and right ends of the machine, and the total number of ink windows in the target image and the number of ink windows newly detected in the second image.

[0062] In S1017, the CPU 151 requests the printer 100 to perform ink color determination processing. The color determination module of the printer 100, upon receiving the request, executes the ink color determination processing. Details will be described later using Figure 11.

[0063] After the ink color is determined, in S1018 the CPU 151 receives the determined ink color information from the printer 100 and analyzes (derives) the remaining ink amount. In S1019 the CPU 151 displays the analyzed ink amount on the display unit 156. That is, the CPU 151 displays a screen on the display unit 156 showing the analysis results as shown in Figure 9(d). After that, the CPU 151 terminates the processing of this flowchart.

[0064] Figure 11 shows the details of the ink color determination process in S1017 of Figure 10. This process begins when the CPU 101 of the printer 100 receives a request for ink color determination from the smartphone 150. In the following flowchart, each process is executed by the functioning of each module of the printer at the instruction of the CPU 101, but the CPU 101 is considered the main entity performing the process. In the following explanation, the symbol "S" represents a step.

[0065] In S1101, the CPU 101 of the printer 100 acquires ink tank information (information 700). In S1102, the CPU 101 determines whether the information received from the smartphone contains information indicating that the image was within the detection limit. That is, it determines whether the target image was within the detection limit. If it was within the detection limit, the CPU 101 proceeds to S1106; otherwise, it proceeds to S1103.

[0066] In S1103, the CPU 101 determines whether the aircraft edge visible in the target image is to the right, based on the aircraft edge information received from the smartphone 150. If it determines that it is to the right, the CPU 101 proceeds to S1104; if it determines that it is not to the right (i.e., to the left), it proceeds to S1105.

[0067] In S1104, CPU101 assigns numbers to the ink windows in the target image in order from the edge, corresponding to the number of ink windows. That is, it assigns numbers sequentially from right to left. In S1105, CPU101 assigns numbers to the ink windows in the target image in reverse order from the total number of ink tanks, starting from the edge.

[0068] In S1106, the CPU 101 determines whether the aircraft end shown in the second image is to the right based on the aircraft end information received from the smartphone 150. If it determines that it is to the right, the CPU 101 proceeds to S1107; if it determines that it is not to the right (i.e., to the left), it proceeds to S1108.

[0069] In S1107, CPU101 assigns numbers to the ink windows of the target image in order of the number of ink tanks in the target image. Then, it adds the number of ink windows included in the second captured image to the assigned numbers.

[0070] In S1108, CPU101 assigns numbers to the ink windows of the target image in reverse order, calculated by subtracting the number of ink tanks shown in the second image from the total number of ink tanks.

[0071] In S1109, the CPU 101 determines the ink color corresponding to the information 700 and the number assigned to the ink window in the target image. Specifically, it refers to the information 700 and identifies the type of ink window so that the number indicating the location of the ink tank matches the number assigned to the ink window in the above process. After that, it notifies the smartphone 150 of the determined ink color, and the CPU 101 terminates the processing of this flowchart.

[0072] Here, for example, if the target image has three ink windows and includes the end of the machine, the total number of ink tanks is four, so the three ink windows are numbered 4, 3, and 2 in reverse order from left to right. Based on the assigned numbers and information 700, the three ink windows are identified as cyan, magenta, and yellow from left to right. The above corresponds to processing S1105 and S1109. Furthermore, consider a case where the target image contains three ink windows but does not include the aircraft end, and the user takes a second image that includes a new ink window and the aircraft end detected to the right of that ink window. This corresponds to S1107. Therefore, after numbering the ink windows of the target image from right to left, the number 1, which is the number of the newly detected ink window, is added. As a result, the ink windows of the target image are numbered 2, 3, 4 from right to left. Based on these numbers and information 700, the ink windows of the target image are identified as yellow, magenta, and cyan from right to left. The above is a flowchart for determining the ink color of the ink window in this embodiment. In this embodiment, the case described was one in which the edge of the aircraft was captured in either the first or second shot. However, the user will not necessarily capture the edge of the aircraft in response to messages to capture other parts or messages describing the edge. It is also possible that the user may retake the image multiple times if the edge of the aircraft cannot be confirmed in the captured image. In such cases, the numbering method described in Figure 11 may not necessarily be correct, and this embodiment only describes a method for determining the ink color of the ink window included in the target image with one or two shots.

[0073] As described above, this embodiment improves the accuracy of identifying the ink color corresponding to the ink window. Specifically, the captured ink window can be identified based on the position of the printer 100's body edge. Furthermore, the method of identifying the type of ink window is changed depending on whether the printer 100's body edge is included in the captured image of the ink window, and whether the printer 100's body edge is to the right or left of the ink window. This improves the accuracy of identifying the captured ink window.

[0074] <Embodiment 2> In this embodiment, we consider a configuration where there are two ink tanks on the left and two on the right.

[0075] Figure 12 shows the external appearance of the main body of this embodiment. In this embodiment, there are ink windows 202 on both sides (referred to as 202R or 202L, respectively). The other configurations are the same as in Embodiment 1. The order of the ink colors in this case is Bk, Y, M, C from right to left, which is the same as in Embodiment 1. Therefore, the information 700 relating the color information of the ink tanks and the position of the ink tanks shown in Figure 7 is the same as that shown in Embodiment 1.

[0076] (Example of aircraft end detection and color determination procedure in Embodiment 2) Figure 13 shows an example of detecting the edge of the aircraft from an image of the printer 100 shown in Figure 12.

[0077] If the captured data is as shown in Figure 13(a), first, areas with large color differences in surrounding pixels are identified within the captured data. For the identified areas with large color differences, lines are drawn to indicate the areas, and it is determined whether the lines enclose an area of ​​a predetermined size. If the enclosed area is of a predetermined size, it is determined that the area is an ink window. Analysis based on this determination allows the areas of ink tanks 110Y and 110Bk and the line indicated by line 1301 to be detected. Next, it is analyzed whether these lines extend outside the upper part of the captured data. In the case of this captured data, 1301 is a line that extends outside the data, so it is recognized as the edge of the aircraft. Subsequently, image processing is performed on ink tanks 110Y and 110Bk as ink tanks to analyze the remaining ink levels. In this case of captured data, line 1301 is determined to be the right edge. In this case, the right edge and two ink tanks are detected, so it is determined that ink tank positions 1 and 2 have been detected from ink tank position 701 in Figure 7. Therefore, based on the information 700 regarding the relationship between the order of the ink tanks and the ink colors, the ink colors are determined to be Bk and Y.

[0078] If the captured data is as shown in Figure 13(b), then areas with large color differences within the screen are identified, similar to Figure 13(a). For the identified areas, the areas with large color differences are indicated with lines, and it is determined whether the lines enclose an area of ​​a predetermined size. If the enclosed area is of a predetermined size, it is determined that the area is an ink window. Analysis based on this determination allows the areas of ink tanks 110C and 110M and line 1302 to be detected. In this case, since line 1302 is a line that extends outside the data, it is recognized as the end of the aircraft. Subsequently, image processing is performed on ink tanks 110C and 110M as ink tanks, and the remaining ink level is analyzed. In this case, the leftmost area and two ink tanks are detected, so it is determined that positions 3 and 4 are detected from the ink tank position 701 in Figure 7. Therefore, from the information 700 relating the order of the ink tanks to the ink color, the ink colors are determined to be M and C.

[0079] <Embodiment 3> In this embodiment, we consider a configuration where there is one ink tank on the left and three on the right.

[0080] Figure 14 shows the external appearance of the main body of this embodiment. In this embodiment, there are ink windows 202 on both sides (referred to as 202R or 202L, respectively). The other configurations are the same as in Embodiment 1. Note that the order of the ink colors differs from Embodiments 1 and 2, being Y, M, C, and Bk from right to left.

[0081] (Relationship between the order of ink tanks and ink colors) Figure 15 shows information 1500 regarding the relationship between the order of the ink tanks and the ink color, which is stored in the main unit information management module 335. The relationship between the order of the ink tanks and the ink color is different for each model. In this embodiment, the ink tanks are numbered sequentially from right to left from the end of the machine, and this order is treated as the ink tank position 1501. By combining this ink tank position 1501 with the corresponding ink color information 1502, the information 1500 regarding the relationship between the order of the ink tanks and the ink color is obtained.

[0082] (Example of aircraft end detection and color determination procedure in Embodiment 3) Figure 16 shows an example of detecting the edge of the aircraft from an image taken of the printer 100 shown in Figure 14.

[0083] If the captured data is as shown in Figure 16(a), first, areas with large color differences in surrounding pixels are identified within the captured data. For the identified areas with large color differences, lines are drawn to indicate the areas, and it is determined whether the lines enclose an area of ​​a predetermined size. In this case, if the enclosed area is of a predetermined size, it is determined that the area is an ink window. By analyzing based on this determination, the areas of ink tanks 110C and 110M can be recognized. On the other hand, in the case of this captured data, analysis to see if there are any lines extending outside the top of the captured data revealed that no lines were detected, and the end of the aircraft was not recognized. Therefore, a reshoot is requested. This illustrates the case where NO is determined in S1006 in Figure 10.

[0084] If the captured data is as shown in Figure 16(b), then, as in Figure 16(a), the first step is to identify areas within the captured data where the color difference of surrounding pixels is large. For the identified areas, the areas with large color differences are indicated by lines, and it is determined whether the lines enclose an area of ​​a predetermined size. If the enclosed area is of a predetermined size, it is determined that the area is an ink window. Based on this determination, the areas of ink tanks 110C, 110M, and 110Y and line 1601 can be detected. In this case, since line 1601 is a line that extends outside the data, it is recognized as the edge of the machine. Subsequently, image processing is performed on ink tanks 110C, 110M, and 110Y as ink tanks, and the remaining ink level is analyzed. In this case, line 1601 is determined to be the right edge. In this case, the right edge and three ink tanks are detected, so it is determined that ink tank positions 1, 2, and 3 have been detected from the ink tank position 1501 included in the information 1500 in Figure 15. Therefore, based on the information 1500 regarding the relationship between the order of the ink tanks and the ink colors, the ink colors are determined to be Y, M, and C.

[0085] If the captured data is as shown in Figure 16(c), then, as in Figure 16(a), the first step is to identify areas within the captured data where the color difference of surrounding pixels is large. For the identified areas, the areas with large color differences are indicated by lines, and it is determined whether the lines enclose an area of ​​a predetermined size. If the enclosed area is of a predetermined size, it is determined that the area is an ink window. By analyzing the data based on this determination, the area of ​​ink tank 110Bk and the line indicated by line 1602 can be detected. In this case, since line 1602 is a line that extends outside the data, it is recognized as the end of the machine. Subsequently, image processing is performed on ink tanks 110C, 110M, and 110Y as ink tanks, and the remaining ink level is analyzed. In this case, line 1602 is determined to be the left end. In this case, the left end and one ink tank are detected, so it is determined that ink tank position 4 has been detected, based on ink tank position 1501 in Figure 15. Therefore, based on the information 1500 regarding the relationship between the order of the ink tanks and the ink color, the ink color is determined to be Bk.

[0086] <Other Embodiments> The above-described embodiment explains a configuration in which a smartphone's camera photographs the ink window, sends the image to the printer, the printer identifies the color of the ink window, and returns that information to the smartphone. However, the embodiment is not limited to this configuration. For example, the smartphone may perform the entire process. In such a case, the smartphone can acquire the ink tank information, as explained in Figure 7, in advance. Alternatively, after the smartphone has performed the series of processes, the printer may update the ink level by sending the resulting ink level information to the printer.

[0087] Furthermore, although the above embodiment was described assuming that there are four types of ink tanks to be loaded, it is not limited to this. In the case of a printer that can load five or more types of ink tanks, when identifying an ink window, if the edge of the printer body is visible in the image of the ink window, the ink window can be identified by assigning a number to the ink window based on the edge, as in the above embodiment. If the "edge of the printer body" is not visible in the image of the ink window (target image), a re-shooting instruction will be displayed. Even if the edge can be detected from other images afterward, the relationship between the target image and the edge of the printer body may not be clear. In such cases, an instruction to photograph another location (another ink window) should be issued until the relationship between the target image and the edge of the printer body becomes clear.

[0088] In the above-described embodiment, the machine edge is detected to identify the ink window, and a number is assigned based on the machine edge. However, the invention is not limited to this. For example, a number may be assigned to the ink window based on a mark placed near the ink window, and the ink color may be identified using the ink tank information and the number. Note that the mark is just one example; other special codes or marks may also be used.

[0089] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0090] Furthermore, this disclosure includes the following components. (Composition 1) An acquisition means for acquiring ink tank information relating to multiple ink tanks loaded into a recording device, A photographing means for photographing a plurality of ink windows corresponding to the ink tanks loaded in the recording device, A determination means that determines a number indicating the order of the multiple ink windows photographed by the photographing means, based on the photographed image including the ink windows and a specific location in the recording device, A means for identifying the type of ink tank corresponding to the ink window included in the captured image, based on the determined number and the ink tank information, A means for analyzing the images of the plurality of ink windows captured by the aforementioned photographic means and deriving the remaining amount of ink contained in the ink tank corresponding to each ink window, A display control means that displays the remaining amount of the extracted ink for each ink tank on a display unit, An information processing device characterized by comprising: (Configuration 2) The information processing device according to configuration 1, characterized in that the ink tank information includes the type of ink tank loaded into the recording device, a number indicating the position of each ink tank, and the total number of ink tanks. (Composition 3) The information processing apparatus according to configuration 1, characterized in that the aforementioned specific location is the end of the recording device. (Composition 4) The determination means identifies areas in the captured image where the color difference from surrounding pixels is large as lines. The information processing apparatus according to configuration 1, characterized in that it detects an area of ​​a predetermined size enclosed by the aforementioned lines as an ink window. (Composition 5) The information processing apparatus according to configuration 4, characterized in that, when the determination means detects the region as an ink window in the captured image, and further detects a line extending outside the captured image, the line extending outside is identified as the end of the recording device. (Composition 6) If the ink tank loading section is located on the right side of the recording device, the determination means identifies the outgoing line as the end of the recording device when the outgoing line is detected on the right side of the area. The information processing apparatus according to configuration 5, characterized in that, when the ink tank loading section is located on the left side of the recording device, the determination means identifies the line exiting to the outside as the end of the recording device when the line exiting to the outside is detected on the left side of the area. (Composition 7) If the end of the recording device is detected to the right of the ink window, the determination means assigns numbers to the ink windows in order from the rightmost one. The information processing apparatus according to configuration 6, characterized in that, when the end of the recording device is detected to the left of the ink window, the determination means assigns numbers to the total number of ink tanks in reverse order from the left ink window. (Composition 8) The information processing apparatus according to configuration 7, characterized in that the identifying means refers to the ink tank information and identifies the type of ink tank such that the number indicating the location of the ink tank matches the number of the ink window assigned by the determination means. (Composition 9) The information processing apparatus according to configuration 1, wherein the determination means determines a number indicating the order of arrangement based on a plurality of captured images, at least one of which is a captured image containing the specific area captured by the shooting means. (Composition 10) If the image captured by the aforementioned shooting means does not include the edge, The information processing apparatus according to configuration 3, characterized in that the display control means displays a message on the display unit indicating that the field of view should be reduced and the ink window re-photographed if the area of ​​the ink window in the captured image is greater than a predetermined threshold, and displays a message indicating that the captured image should be saved as a target image and another part should be re-photographed if the area of ​​the ink window in the captured image is less than or equal to a predetermined threshold. (Composition 11) If the second image re-captured by the aforementioned imaging means includes the edge of the recording device, The information processing apparatus according to configuration 10, characterized in that the determination means determines the number of the ink window included in the target image based on the target image, the second captured image, and the ink tank information. (Composition 12) If the end of the recording device is detected to the right of the ink window, The information processing apparatus according to configuration 11, characterized in that the determination means assigns numbers to the ink windows included in the target image from the right, and adds the number of ink windows newly detected in the second captured image to the assigned numbers. (Composition 13) If the end of the recording device is detected to the left of the ink window, The information processing apparatus according to configuration 11, wherein the determination means assigns numbers to the ink windows included in the target image in reverse order from left to right, based on the number obtained by subtracting the number of ink windows newly detected in the second captured image from the number indicating the total number of ink tanks. (Composition 14) The information processing device according to configuration 1, characterized in that the information processing device is a portable terminal. (Composition 15) An acquisition step to acquire ink tank information regarding multiple ink tanks loaded into a recording device, A shooting step of photographing a plurality of ink windows corresponding to the ink tanks loaded in the recording device, A determination step in which, based on the captured image including the ink windows and a specific location in the recording device captured in the aforementioned shooting step, a number indicating the order in which the plurality of ink windows captured in the aforementioned shooting step are determined; Based on the determined number and the ink tank information, the identification step involves identifying the type of ink tank corresponding to the ink window included in the captured image. A derivation step involves analyzing the images of the multiple ink windows captured in the aforementioned shooting step to derive the remaining amount of ink contained in the ink tank corresponding to each ink window, A display control step that displays the remaining amount of the extracted ink for each ink tank on the display unit, A control method for an information processing device, characterized by comprising the following: (Composition 16) A program that operates in an information processing device, wherein the information processing device An acquisition means for acquiring ink tank information relating to multiple ink tanks loaded into a recording device, A photographing means for photographing a plurality of ink windows corresponding to the ink tanks loaded in the recording device, A determination means that determines a number indicating the order of the multiple ink windows photographed by the photographing means, based on the photographed image including the ink windows and a specific location in the recording device, A means for identifying the type of ink tank corresponding to the ink window included in the captured image, based on the determined number and the ink tank information, A means for analyzing the images of the plurality of ink windows captured by the aforementioned photographic means and deriving the remaining amount of ink contained in the ink tank corresponding to each ink window, A display control means that displays the remaining amount of the extracted ink for each ink tank on a display unit, A program characterized by being designed to function as such. (Composition 17) An information processing system having a recording device and a portable terminal, The recording device is An acquisition means for acquiring ink tank information relating to the ink tank loaded in the recording device, A receiving means that receives from the mobile terminal information indicating the edge of the recording device and information indicating the number of ink windows in the target image, A determination means for determining the number of the ink window included in the target image based on the information indicating the end and the ink tank information, A means for identifying the type of ink tank corresponding to the ink window included in the target image, based on the number determined above and the ink tank information, Equipped with, The aforementioned mobile terminal is A means for photographing at least one ink window, If the captured target image including the ink window includes the edge of the recording device, a first determination means determines whether the edge is the left or right edge as information indicating the edge of the recording device, A transmission means for transmitting information indicating the aforementioned end and information indicating the number of ink windows in the target image to the recording device, A receiving means for receiving from the recording device the type of ink tank corresponding to the ink window included in the target image, which has been identified by the recording device, A display control means that displays the type of ink tank and the remaining amount of ink corresponding to the ink window included in the target image on the display unit, An information processing system characterized by comprising the following features.

Claims

1. An acquisition means for acquiring ink tank information relating to multiple ink tanks loaded into a recording device, A photographing means for photographing a plurality of ink windows corresponding to the ink tanks loaded in the recording device, A determination means that determines a number indicating the order of the multiple ink windows photographed by the photographing means, based on the photographed image including the ink windows and a specific location in the recording device, A means for identifying the type of ink tank corresponding to the ink window included in the captured image, based on the determined number and the ink tank information, A means for analyzing the images of the plurality of ink windows captured by the aforementioned photographic means and deriving the remaining amount of ink contained in the ink tank corresponding to each ink window, A display control means that displays the remaining amount of the extracted ink for each ink tank on a display unit, An information processing device characterized by comprising:

2. The information processing apparatus according to claim 1, characterized in that the ink tank information includes the type of ink tank loaded into the recording device, a number indicating the position of each ink tank, and the total number of ink tanks.

3. The information processing apparatus according to claim 1, characterized in that the aforementioned specific location is the end of the recording device.

4. The determination means identifies areas in the captured image where the color difference from surrounding pixels is large as lines. The information processing apparatus according to claim 1, characterized in that it detects an area of ​​a predetermined size enclosed by the aforementioned lines as an ink window.

5. The information processing apparatus according to claim 4, characterized in that, when the determination means detects the region as an ink window in the captured image and further detects a line extending outside the captured image, the line extending outside is identified as the end of the recording device.

6. If the ink tank loading section is located on the right side of the recording device, the determination means identifies the outgoing line as the end of the recording device when the outgoing line is detected on the right side of the area. The information processing apparatus according to claim 5, wherein, if the ink tank loading section is located on the left side of the recording device, the determination means identifies the outgoing line as the end of the recording device when the outgoing line is detected on the left side of the area.

7. If the end of the recording device is detected to the right of the ink window, the determination means assigns numbers to the ink windows in order from the rightmost one. The information processing apparatus according to claim 6, characterized in that, when the end of the recording device is detected to the left of the ink window, the determination means assigns numbers in reverse order from the left ink window to indicate the total number of ink tanks.

8. The information processing apparatus according to claim 7, characterized in that the identifying means refers to the ink tank information and identifies the type of ink tank such that the number indicating the location of the ink tank matches the number of the ink window assigned by the determination means.

9. The information processing apparatus according to claim 1, wherein the determination means determines a number indicating the order of arrangement based on a plurality of captured images, at least one of which is a captured image containing the specific area captured by the shooting means.

10. If the image captured by the aforementioned shooting means does not include the edge, The information processing apparatus according to claim 3, characterized in that the display control means displays a message on the display unit indicating that the field of view should be reduced and the ink window re-photographed if the area of ​​the ink window in the captured image is greater than a predetermined threshold, and displays a message indicating that the captured image should be saved as a target image and another part should be re-photographed if the area of ​​the ink window in the captured image is less than or equal to a predetermined threshold.

11. If the second image re-captured by the aforementioned imaging means includes the edge of the recording device, The information processing apparatus according to claim 10, characterized in that the determination means determines the number of the ink window included in the target image based on the target image, the second captured image, and the ink tank information.

12. If the end of the recording device is detected to the right of the ink window, The information processing apparatus according to claim 11, characterized in that the determination means assigns numbers to the ink windows included in the target image from the right, and adds the number of ink windows newly detected in the second captured image to the assigned numbers.

13. If the end of the recording device is detected to the left of the ink window, The information processing apparatus according to claim 11, wherein the determination means assigns numbers to the ink windows included in the target image in reverse order from left to right, based on the number obtained by subtracting the number of ink windows newly detected in the second captured image from the total number of ink tanks.

14. The information processing device according to claim 1, characterized in that the information processing device is a portable terminal.

15. An acquisition step to acquire ink tank information regarding multiple ink tanks loaded into a recording device, A shooting step of photographing a plurality of ink windows corresponding to the ink tanks loaded in the recording device, A determination step in which, based on the captured image including the ink windows and a specific location in the recording device captured in the aforementioned shooting step, a number indicating the order of the plurality of ink windows captured in the aforementioned shooting step is determined; Based on the determined number and the ink tank information, the identification step involves identifying the type of ink tank corresponding to the ink window included in the captured image. A derivation step involves analyzing the images of the multiple ink windows captured in the aforementioned shooting step to derive the remaining amount of ink contained in the ink tank corresponding to each ink window, A display control step that displays the remaining amount of the extracted ink for each ink tank on the display unit, A control method for an information processing device, characterized by comprising the following:

16. A program that operates in an information processing device, wherein the information processing device An acquisition means for acquiring ink tank information relating to multiple ink tanks loaded into a recording device, A photographing means for photographing a plurality of ink windows corresponding to the ink tanks loaded in the recording device, A determination means that determines a number indicating the order of the multiple ink windows photographed by the photographing means, based on the photographed image including the ink windows and a specific location in the recording device, A means for identifying the type of ink tank corresponding to the ink window included in the captured image, based on the determined number and the ink tank information, A means for analyzing the images of the plurality of ink windows captured by the aforementioned photographic means and deriving the remaining amount of ink contained in the ink tank corresponding to each ink window, A display control means that displays the remaining amount of the extracted ink for each ink tank on a display unit, A program characterized by being designed to function as such.

17. An information processing system having a recording device and a portable terminal, The recording device is An acquisition means for acquiring ink tank information relating to the ink tank loaded in the recording device, A receiving means that receives from the mobile terminal information indicating the edge of the recording device and information indicating the number of ink windows in the target image, A determination means for determining the number of the ink window included in the target image based on the information indicating the end and the ink tank information, A means for identifying the type of ink tank corresponding to the ink window included in the target image, based on the determined number and the ink tank information, Equipped with, The aforementioned mobile terminal is A means for photographing at least one ink window, If the captured target image including the ink window includes the edge of the recording device, a first determination means determines whether the edge is the left or right edge as information indicating the edge of the recording device, A transmission means for transmitting information indicating the aforementioned end and information indicating the number of ink windows in the target image to the recording device, A receiving means for receiving from the recording device the type of ink tank corresponding to the ink window included in the target image, which has been identified by the recording device, A display control means that displays the type of ink tank and the remaining amount of ink corresponding to the ink window included in the target image on the display unit, An information processing system characterized by comprising the following features.

Images