Terminal device, information processing method, program, and information processing system

Abstract

To provide a terminal device, an information processing method, a program, and an information processing system that execute information processing using a library suited to a browser.SOLUTION: In an information processing system, a terminal device for performing predetermined image processing via a browser includes a verification processing unit that uses verification information to verify whether a browser can perform predetermined image processing on an imaging subject, a library acquisition control unit that acquires a first image processing library described in a first programming language that executes the predetermined image processing when it is determined as a result of the verification that the processing is executable, and acquires a second image processing library described in a second programming language that executes the predetermined image processing when it is determined that the processing is not executable, and an image processing execution unit for executing the predetermined image processing using the first image processing library or the second image processing library, and a display control unit for causing a result of the predetermined image processing to be displayed on a screen of the terminal device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a terminal device, an information processing method, a program, and an information processing system. [Background technology]

[0002] Technologies have been developed that work with cloud services from external vendors such as Google (registered trademark) and Office365 (registered trademark) to upload documents scanned by image forming devices and print documents stored in cloud storage. In addition, a platform has been developed that can create highly flexible and functional applications by combining these technologies with functions such as OCR (Optical Character Recognition / Reader), image cropping, and barcode authentication.

[0003] This application is called a workflow application, and can be executed not only on the operation terminal of the image forming apparatus, but also from a web browser of a mobile terminal (an example of a terminal device). Non-Patent Document 1 discloses an application for smartphones that can correct images taken with the smartphone. Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when a workflow application is executed from a mobile terminal, there is a demand for image processing such as keystone correction of images captured by the mobile terminal. In addition, when considering using a mobile terminal as a substitute for an image forming device, it is an essential function to extract a document from an image captured by the mobile terminal and perform keystone correction.

[0005] For example, the Microsoft (registered trademark) OneDrive (registered trademark) application can take an image on a mobile terminal, perform keystone correction and processing on the taken image, and save the taken image as a single PDF file on OneDrive. There are already many applications with such information processing functions as keystone correction. For example, a browser-based application, which is an application that can be executed by a browser, can realize a desired information processing function by loading a library for information processing.

[0006] However, some image processing libraries cannot be used depending on the browser of the mobile device, and there is a problem that the information processing speed is slowed down when using an image processing library that can be used on all browsers.

[0007] The present invention has been made in consideration of the above, and aims to provide a terminal device, an information processing method, a program, and an information processing system that can execute information processing using an image processing library suitable for a browser. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems and achieve the object, the present invention is a terminal device that performs a predetermined image processing via a browser, and includes a verification processing unit that performs a verification process to determine whether or not the browser is able to perform the predetermined image processing on a subject based on a shooting operation on the subject, using verification information for verifying whether or not the browser is able to perform the predetermined image processing on the subject; an acquisition control unit that acquires a first image processing library written in a first programming language that executes the predetermined image processing if it is determined as a result of the verification process that the browser is able to execute the predetermined image processing, and acquires a second image processing library written in a second programming language that executes the predetermined image processing if it is determined that the browser is not able to execute the predetermined image processing; an image processing unit that executes the predetermined image processing using the first image processing library or the second image processing library; and a display control unit that displays the result of the predetermined image processing executed based on the first image processing library or the second image processing library on a screen of the terminal device. Effect of the Invention

[0009] According to the present invention, it is possible to perform information processing using an image processing library suited to a browser. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of a configuration of an information processing system according to the present embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a terminal device of the information processing system according to the present embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of a hardware configuration of the server device 2 in the information processing system according to the present embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a display sequence of a UI screen in the terminal device of the information processing system according to the present embodiment. [Diagram 5]FIG. 5 is a sequence diagram showing an example of the flow of predetermined image processing in the information processing system according to the present embodiment. [Figure 6] FIG. 6 is a sequence diagram showing another example of the flow of predetermined image processing in the information processing system according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of a terminal device, an information processing method, a program, and an information processing system will be described in detail with reference to the accompanying drawings.

[0012] Fig. 1 is a diagram showing an example of a configuration of an information processing system according to the present embodiment. As shown in Fig. 1, the information processing system according to the present embodiment includes a terminal device 1 and a server device 2. The terminal device 1 and the server device 2 are connected via a network NW such as the Internet.

[0013] 1, the server device 2 has a receiving unit 201, a transmitting unit 202, and a library storage unit 203. The library storage unit 203 stores an image processing library that is a library including predetermined image processing programs. Specifically, the library storage unit 203 stores a first image processing library and a second image processing library.

[0014] Here, the predetermined image processing (one example of the predetermined processing) is a processing performed via the terminal device 1 and set in advance, and includes, for example, keystone correction. Keystone correction is an image processing in which an object photographed by the camera of the terminal device 1 is accommodated within a rectangle displayed on the screen of the terminal device 1 and represented by a predetermined frame line. In addition, the image processing will be described below as an example of the predetermined processing, but the content of the processing is not limited thereto. For example, the processing may be a processing in which any of conversion, deformation, analysis, and information extraction is performed on any of data such as image data, video data, text data, and music data.

[0015] Here, the first image processing library is an image processing library written in a first program language that executes a predetermined image processing. The first image processing library is a library written in the same program language (an example of a first program language) as verification information for verifying whether the terminal device 1, more specifically, the web browser of the terminal device 1, can execute a predetermined image processing on a shooting target. The first image processing library may be a library executed by a web browser (an example of a browser). In this embodiment, the program language for writing the first image processing library may be WebAssembly (hereinafter, referred to as WASM). In the following, verifying whether software including a web browser owned by the terminal device 1 can execute a predetermined process may be expressed as verifying whether the terminal device 1 can execute a predetermined process.

[0016] Here, the second image processing library is an image processing library written in a second program language that executes a predetermined image processing. The second image processing library is a library written in a program language (an example of a second program language) different from the verification information. The second image processing library may be a library executed by a Web browser (an example of a browser). In this embodiment, the program language that writes the second image processing library may be Java (registered trademark) Script (registered trademark, hereinafter referred to as JS). Note that image processing using the first image processing library may be able to shorten the processing time more than image processing using the second image processing library. For this reason, when a Web browser is capable of executing image processing using the first image processing library, it may be preferable to use the first image processing library rather than the second image processing library.

[0017] The receiving unit 201 receives the result of the predetermined image processing on the subject from the terminal device 1 (i.e., an image on which the predetermined image processing has been performed in the terminal device 1). The transmitting unit 202 transmits the first image processing library or the second image processing library to the terminal device 1. The server device 2 may have a functional unit that executes a workflow including a desired process on the result of the predetermined image processing on the subject received by the receiving unit 201. As an example, the workflow may include a process of storing the result of the predetermined image processing on the subject in a storage area in the server device 2, and a process of transmitting the result of the predetermined image processing on the subject to a desired destination such as an external server, an email address, or a fax number. At this time, the server device 2 may determine the destination by analyzing the result of the predetermined image processing on the subject. However, the server device 2 may realize a part or all of the workflow process by cooperating with an external server.

[0018] As shown in Fig. 1, the terminal device 1 has a display control unit 101, a verification processing unit 102, a library acquisition control unit 103, an image processing execution unit 104, a verification program storage unit 105, and an image processing library storage unit 106. One or more of the functional units of the display control unit 101, the verification processing unit 102, the library acquisition control unit 103, the image processing execution unit 104, the verification program storage unit 105, and the image processing library storage unit 106 can be realized by a Web browser. For example, a part or all of the functions realized by one or more functional units may be realized by a program running on the Web browser. Here, when the terminal device 1 has a plurality of Web browsers, one or more functional units can be realized for each Web browser.

[0019] The verification program storage unit 105 stores verification information. Here, the verification information is information for verifying whether or not the terminal device 1 can execute a predetermined image processing on an object to be photographed (for example, an image photographed by the terminal device 1). In this embodiment, the verification information may be a simple program written in WASM, which is an example of a programming language for writing a first image processing library. The image processing library storage unit 106 stores the first image processing library and the second image processing library acquired from the server device 2.

[0020] The verification processing unit 102 is an example of a verification processing unit that uses verification information to verify whether the terminal device 1 (Web browser) can execute a predetermined image processing based on the shooting operation of the object to be shot. In this embodiment, the verification processing unit 102 determines that the terminal device 1 (Web browser) can execute the predetermined image processing if it can execute a program written in WASM, which is an example of the verification information.

[0021] On the other hand, when a program written in WASM, which is an example of the verification information, cannot be executed, the verification processing unit 102 determines that a predetermined image processing cannot be executed in the terminal device 1 (Web browser). Furthermore, the verification processing unit 102 may execute the verification processing when the use of a camera that captures an image of a subject or a CMOS sensor 405 (see FIG. 2), which is an example of a capture unit, is permitted during a photographing operation.

[0022] The library acquisition control unit 103 is an example of an acquisition control unit that acquires a first image processing library for executing the predetermined image processing when it is determined that the terminal device 1 (Web browser) is capable of executing the predetermined image processing as a result of the verification process by the verification processing unit 102. On the other hand, the library acquisition control unit 103 acquires a second image processing library for executing the predetermined image processing when it is determined that the terminal device 1 (Web browser) is not capable of executing the predetermined image processing. In this embodiment, the library acquisition control unit 103 acquires the first image processing library and the second image processing library from the server device 2. Then, the library acquisition control unit 103 stores the acquired first image processing library and the second image processing library in the image processing library storage unit 106. The library acquisition control unit 103 also functions as an example of a transmission unit that transmits the result of the predetermined image processing performed by the image processing execution unit 104, which will be described later, on the subject to be photographed to the server device 2.

[0023] The image processing execution unit 104 is an example of an image processing unit that executes a predetermined image processing using a first image processing library or a second image processing library. That is, when the library acquisition control unit 103 acquires the first image processing library, the predetermined image processing is executed using the first image processing library, and when the library acquisition control unit 103 acquires the second image processing library, the predetermined image processing is executed using the second image processing library. This allows the predetermined image processing to be executed on the image of the subject to be photographed using an image processing library executable on a Web browser, thereby realizing a data processing function in a browser-based application.

[0024] Here, the predetermined image processing includes keystone correction, binarization processing for converting an image into two colors, conversion processing to grayscale, etc. Keystone correction is a correction for fitting an object photographed by the CMOS sensor 405 (see FIG. 2) of the terminal device 1 into a rectangle represented by a predetermined frame line displayed on the screen of the terminal device 1. In other words, keystone correction is a process for correcting the distortion of the object to a square.

[0025] The display control unit 101 is an example of a display control unit that displays the result of the predetermined image processing executed based on the first image processing library or the second image processing library on a display 418 (see FIG. 2), which is an example of a display unit (or screen) of the terminal device 1. For example, the display control unit 101 may display on the display 418 of the terminal device 1 a subject that has been subjected to predetermined image processing based on the first image processing library or the second image processing library.

[0026] Furthermore, the display control unit 101 further displays a screen including an operation item for accepting an editing operation (for example, correction of a boundary of a subject included in an image) for the result of the predetermined image processing from the operator of the terminal device 1 on the display 418 of the terminal device 1. When an operation item is operated, the display control unit 101 displays the screen for the editing operation on the same screen as the screen displaying the result of the predetermined image processing or on a different screen.

[0027] Fig. 2 is a diagram showing an example of a hardware configuration of a terminal device of the information processing system according to the present embodiment. As shown in Fig. 2, the terminal device 1 such as a smartphone includes a CPU 401, a ROM 402, a RAM 403, an EEPROM 404, a CMOS sensor 405, an image sensor I / F 406, an acceleration / direction sensor 407, a media I / F 409, and a GPS receiver 411.

[0028] Among these, the CPU 401 controls the operation of the entire terminal device 1. The ROM 402 stores the CPU 401 and programs such as an IPL used to drive the CPU 401. The RAM 403 is used as a work area for the CPU 401. The EEPROM 404 reads or writes various data such as smartphone programs under the control of the CPU 401. The CMOS (Complementary Metal Oxide Semiconductor) sensor 405 is a type of built-in imaging means that captures an image of a subject (mainly a self-portrait) under the control of the CPU 401 to obtain image data. Note that the imaging means may be a CCD (Charge Coupled Device) sensor or the like instead of a CMOS sensor. The imaging element I / F 406 is a circuit that controls the driving of the CMOS sensor 405. The acceleration / direction sensor 407 is various sensors such as an electronic magnetic compass and a gyrocompass that detects geomagnetism, and an acceleration sensor. The media I / F 409 controls the reading or writing (storing) of data from or to a recording medium 408 such as a flash memory. The GPS receiver 411 receives GPS signals from GPS satellites.

[0029] The terminal device 1 also has a long-distance communication circuit 412, a CMOS sensor 413, an image sensor I / F 414, a microphone 415, a speaker 416, an audio input / output I / F 417, a display 418, an external device connection I / F (Interface) 419, a short-distance communication circuit 420, an antenna 420a of the short-distance communication circuit 420, and a touch panel 421.

[0030] Among these, the long-distance communication circuit 412 is a circuit that communicates with other devices via a communication network. The CMOS sensor 413 is a type of built-in imaging means that captures an image of a subject and obtains image data under the control of the CPU 401. The imaging element I / F 414 is a circuit that controls the driving of the CMOS sensor 413. The microphone 415 is a built-in circuit that converts sound into an electric signal. The speaker 416 is a built-in circuit that converts an electric signal into physical vibration to generate sound such as music and voice. The sound input / output I / F 417 is a circuit that processes input and output of sound signals between the microphone 415 and the speaker 416 under the control of the CPU 401. The display 418 is a type of display means such as a liquid crystal or organic EL (Electro Luminescence) that displays an image of a subject, various icons, etc. The external device connection I / F 419 is an interface for connecting various external devices. The short-distance communication circuit 420 is a communication circuit such as NFC (Near Field Communication) and Bluetooth (registered trademark). The touch panel 421 is a type of input means that allows a user to operate the terminal device 1 by pressing the display 418.

[0031] The terminal device 1 also includes a bus line 410. The bus line 410 is an address bus, a data bus, etc. for electrically connecting the components such as the CPU 401 shown in FIG.

[0032] Fig. 3 is a diagram showing an example of a hardware configuration of the server device 2 of the information processing system according to the present embodiment. As shown in Fig. 3, the server device 2 is constructed by a computer, and includes a CPU 501, a ROM 502, a RAM 503, an HD 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516.

[0033] Among these, the CPU 501 controls the operation of the entire server device 2. The ROM 502 stores programs such as IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls reading or writing of various data from the HD 504 according to the control of the CPU 501. The display 506 displays various information such as a cursor, a menu, a window, a character, or an image. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using a communication network. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG. 3.

[0034] Moreover, the keyboard 511 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513 as an example of a removable recording medium. Note that this is not limited to DVD-RW, and may be DVD-R, etc. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515 such as a flash memory.

[0035] 4 is a diagram showing an example of a display sequence of a UI screen in a terminal device of the information processing system according to the present embodiment. First, when a user issues an instruction to start an application running in a web browser of the terminal device 1, the display control unit 101 displays an application screen A for accepting various operations on a display unit such as the display 418.

[0036] Then, when a user performs a photographing operation on the application screen A, the display control unit 101 displays a document photographing screen B on the display 418, which allows the user to instruct photographing of the subject. The user can also instruct photographing multiple images of the subject continuously on the document photographing screen B. Then, when the subject is photographed by the CMOS sensor 405 (an example of a photographing unit or camera), the display control unit 101 displays a page confirmation screen C on the display 418, which includes the results of predetermined image processing on the subject.

[0037] Thereafter, when the completion of shooting is instructed on the page confirmation screen C, or when the completion of shooting is instructed on the boundary manual correction screen D that allows manual correction of the trapezoid boundary included in the shooting result of the shooting object, the display control unit 101 displays an application screen E on the display 418 that allows an instruction to send the shooting result of the shooting object to the server device 2. On the application screen E, an email address to which image data that is the shooting result of the shooting object is to be sent, and a storage destination such as the server device 2 or a folder registered on the server device 2 can be set. Information on these sending destinations and storage destinations are examples of setting information. However, the setting information included in the application screen E is not limited to this. The image processing execution unit 104 sends the shooting result of the shooting object to the server device 2 according to the instruction on the application screen E. Here, the image processing execution unit 104 can send the image (shooting result of the shooting object) on which a predetermined image processing has been performed to the server device 2.

[0038] The display control unit 101 may display the application screen A or the page confirmation screen C including an operation item for accepting an editing operation (for example, the above-mentioned procedural correction) for the result of the predetermined image processing from the user (operator) of the terminal device 1. Then, when the operation item is operated, the display control unit 101 may display a boundary manual correction screen D, which is an example of a screen for the editing operation, on the same screen as the page confirmation screen C, which is an example of a screen displaying the result of the predetermined image processing, or on a different screen.

[0039] However, if the CMOS sensor 405 fails to start despite being started in accordance with an instruction to photograph the subject on the document photographing screen B, the display control unit 101 displays a camera startup error dialog F on the display 418, which instructs the user to upgrade the web browser or obtain the photographing results of the subject by other means. Here, if an instruction to upgrade the web browser is given, the display control unit 101 displays the document photographing screen B on the display 418 again after the web browser is upgraded.

[0040] On the other hand, when an instruction is given to acquire the photographing result of the subject by another means, the display control unit 101 causes the display 418 to display an OS (Operating System) dependent screen G capable of instructing to acquire the photographing result of the subject. In the OS dependent screen G, a desired image can be selected from image data stored in the terminal device 1 or an external device capable of communicating with the terminal device 1. In addition, in the OS dependent screen G, the photographing result of the subject may be acquired by starting a camera application or the like and taking a picture with a camera. After that, the display control unit 101 causes the display 418 to display a page confirmation screen C including the acquired photographing result of the subject.

[0041] 5 is a sequence diagram showing an example of a flow of a predetermined image processing in the information processing system according to the present embodiment. If the terminal device 1 has multiple Web browsers, the following processing can be executed for each Web browser. The following processing can be executed by a Web browser that is instructed to be started by a user among the multiple Web browsers. When an instruction is given to start an application that runs in the Web browser of the terminal device 1 (step S521), the display control unit 101 causes the display 418 to display an application screen A that accepts various operations (step S522).

[0042] Then, when an instruction to move to document capture screen B, which can instruct CMOS sensor 405 to capture an image, is given on application screen A (step S523), verification processing unit 102 uses the verification information to execute a verification process to determine whether terminal device 1 can execute a predetermined image processing (step S524). In this embodiment, verification processing unit 102 verifies whether WASM is available. For example, verification processing unit 102 may verify whether WASM is available based on whether a simple WASM program, which is an example of verification information, can be executed.

[0043] If WASM is available, the library acquisition control unit 103 acquires and reads a first image processing library (for example, a keystone correction library written in WASM) from the server device 2 (step S525). On the other hand, if WASM is unavailable, the library acquisition control unit 103 acquires and reads a second image processing library (for example, a keystone correction library written in JS) from the server device 2 (step S526). The verification processing unit 102 also determines whether or not use of the CMOS sensor 405 is permitted (step S527). In this embodiment, the verification processing unit 102 may determine whether or not use of the CMOS sensor 405 is permitted based on whether or not a video (image) captured by the CMOS sensor 405 can be displayed in a Web browser.

[0044] When the use of the CMOS sensor 405 is permitted, the display control unit 101 displays the document photographing screen B on the display 418 (step S528). The image processing execution unit 104 stores the photographed image in a memory such as the RAM 43 each time the CMOS sensor 405 photographs an image (step S529), and executes a predetermined image processing (for example, a process of recognizing a trapezoid included in the image and storing its vertex coordinates) on the image using a first image processing library (for example, a keystone correction library) (step S530). At this time, the image processing execution unit 104 may generate an image that has been subjected to keystone correction on the photographed image, and store the generated image in a memory such as the RAM 43. However, the image processing execution unit 104 may store setting information required for the keystone correction, and generate an image that has been subjected to keystone correction at the timing when an instruction to transition to the page confirmation screen C is received and at the timing when the start button is pressed on the application screen E. Next, the display control unit 101 causes the display 418 to display the document photographing screen B again (step S531).

[0045] Next, when the user instructs to transition to a page confirmation screen C including an image captured by the CMOS sensor 405 (step S532), the display control unit 101 causes the display 418 to display the page confirmation screen C including an image that has been subjected to predetermined image processing (step S533). Next, when the user instructs to transition to a boundary manual correction screen D that allows manual correction of a trapezoid boundary included in the image captured by the CMOS sensor 405 (step S534), the display control unit 101 causes the display 418 to display the boundary manual correction screen D (step S535).

[0046] When the manual correction of the trapezoid boundary is performed on the boundary manual correction screen D (step S536), the image processing execution unit 104 updates the coordinates of the vertices of the trapezoid included in the image (step S537). Next, the display control unit 101 causes the display 418 to display the boundary manual correction screen D including the image of the trapezoid with the updated coordinates of the vertices (step S538). After that, when the completion button is pressed on the boundary manual correction screen D (step S539), the image processing execution unit 104 ends the predetermined image processing for the image (step S540).

[0047] Next, when the start button is pressed on the application screen E (step S541), the library acquisition control unit 103 transmits the image on which the predetermined image processing has been performed to the server device 2 (step S542). At this time, the terminal device 1 transmits the setting information input on the application screen E to the server device 2. This allows the server device 2 to execute a workflow based on the image on which the predetermined image processing has been performed and the setting information. Since the server device 2 can execute a plurality of workflows, the terminal device 1 may transmit information specifying the workflow to the server device 2. Here, which workflow is to be executed can be set by the application screen A. For example, the application screen A can select the workflow to be linked. Also, by inputting a URL into a web browser, it may be possible to link with a workflow corresponding to the URL.

[0048] Fig. 6 is a sequence diagram showing another example of the flow of predetermined image processing in the information processing system according to the present embodiment, in which the same processing as in Fig. 5 will not be described.

[0049] If it is determined in step S527 that use of the CMOS sensor 405 is not permitted, the display control unit 101 causes the display 418 to display the document shooting screen B again (step S528). Furthermore, the display control unit 101 causes the display 418 to display a camera startup error dialog F notifying that an error has occurred in starting up the CMOS sensor 405 (step S601).

[0050] Next, when the camera startup error dialogue F instructs to acquire an image by another means (step S602), the display control unit 101 causes the display 418 to display an OS-dependent screen G including a menu for selecting an image acquisition means (step S603). Next, when an image file is selected on the OS-dependent screen G (step S604), the image processing execution unit 104 executes a predetermined image processing on the selected image using the loaded image processing library of the first image processing library and the second image processing library (step S605). Thereafter, the display control unit 101 causes the display 418 to display a manual boundary correction screen D including the image on which the predetermined image processing has been executed (step S606).

[0051] Then, when the complete button is pressed on the boundary manual correction screen D (step S539), the image processing execution unit 104 ends the predetermined image processing on the image (step S540).

[0052] In this way, according to the information processing system of this embodiment, a predetermined image processing can be performed on an image of a subject to be photographed using an image processing library that can be executed on a Web browser, thereby realizing a data processing function in a browser-based application.

[0053] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module designed to execute each function described above.

[0054] The devices described in the examples are merely illustrative of one of a number of computing environments for implementing the embodiments disclosed herein. In one embodiment, the server device 2 includes a number of computing devices, such as a server cluster. The computing devices are configured to communicate with each other via any type of communication link, including a network, shared memory, etc., to perform the processes disclosed herein. Similarly, the terminal device 1 may include a number of computing devices configured to communicate with each other.

[0055] Furthermore, the terminal device 1 and the server device 2 can be configured to share the disclosed processing steps, e.g., FIG. 1, in various combinations. For example, the processes performed by the terminal device 1 can be performed by the server device 2. Similarly, the functions of the server device 2 can be performed by the terminal device 1. Also, the elements of the terminal device 1 and the server device 2 can be integrated into one server device or separated into multiple devices.

[0056] In addition, the terminal device 1 or the server device 2 is not limited to a smartphone or a server as long as it is a device equipped with a communication function. The terminal device 1 or the server device 2 may be, for example, a PJ (Projector), an IWB (Interactive White Board: a white board having an electronic blackboard function capable of mutual communication), an output device such as a digital signage, a HUD (Head Up Display) device, an industrial machine, an imaging device, a sound collecting device, a medical device, a network home appliance, an automobile (Connected Car), a notebook PC (Personal Computer), a mobile phone, a smartphone, a tablet terminal, a game machine, a PDA (Personal Digital Assistant), a digital camera, a wearable PC, a desktop PC, or the like.

[0057] The programs executed by the terminal device 1 and the server device 2 of this embodiment are provided by being pre-installed in the ROM 402, 502, etc. The programs executed by the terminal device 1 and the server device 2 of this embodiment may be provided by being recorded in a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD) in the form of an installable or executable file.

[0058] Furthermore, the programs executed by the terminal device 1 and the server device 2 of the present embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the programs executed by the terminal device 1 and the server device 2 of the present embodiment may be provided or distributed via a network such as the Internet.

[0059] The program executed by the terminal device 1 of this embodiment has a modular configuration including the above-mentioned units (display control unit 101, verification processing unit 102, library acquisition control unit 103, and image processing execution unit 104), and in terms of actual hardware, a processor such as CPU 401 reads out the program from the above-mentioned ROM 402 and executes it, thereby loading the above-mentioned units onto a main memory device, and the display control unit 101, verification processing unit 102, library acquisition control unit 103, and image processing execution unit 104 are generated on the main memory device.

[0060] The program executed by the server device 2 in this embodiment has a modular structure including the above-mentioned units (receiving unit 201, transmitting unit 202), and in terms of actual hardware, a processor such as CPU 501 reads out the program from the above-mentioned ROM 502 and executes it, thereby loading the above-mentioned units onto a main memory device, and the receiving unit 201 and transmitting unit 202 are generated on the main memory device.

[0061] For example, aspects of the present invention are as follows.

[0062] <1> a verification processing unit that performs a verification process to determine whether or not the browser can execute the predetermined image processing on a subject based on a photographing operation performed on the subject, using verification information for verifying whether or not the browser can execute the predetermined image processing on the subject; and an acquisition control unit that acquires a first image processing library written in a first program language for executing the predetermined image processing when it is determined that the browser is capable of executing the predetermined image processing as a result of the verification process, and acquires a second image processing library written in a second program language for executing the predetermined image processing when it is determined that the browser is not capable of executing the predetermined image processing; an image processing unit that executes the predetermined image processing using the first image processing library or the second image processing library; a display control unit that displays a result of the specified image processing executed based on the first image processing library or the second image processing library on a screen of the terminal device.

[0063] <2> The first program language is the same program language as the verification information, and the second program language is a program language different from the verification information. <1> A terminal device as described in.

[0064] <3> the acquisition control unit acquires the first image processing library and the second image processing library from a server device storing the first image processing library and the second image processing library; <1> or <2> A terminal device as described in.

[0065] <4> The first image processing library or the second image processing library includes a library of the image processing unit that accommodates the subject to be photographed in a rectangle represented by a predetermined frame line displayed on a screen of the terminal device. <1> from <3> 2. A terminal device according to claim 1 .

[0066] <5> The programming language for writing the first image processing library is WebAssembly, and the programming language for writing the second image processing library is Java Script. <2> A terminal device as described in.

[0067] <6> the display control unit further displays, on a screen including an operation item for accepting an editing operation on the result of the predetermined image processing from an operator of the terminal device, and when the operation item is operated, displays a screen for the editing operation on the same screen as the screen displaying the result of the predetermined image processing or on a different screen. <1> from <5> 2. A terminal device according to claim 1 .

[0068] <7> the verification process is executed when permission to use a camera for performing the photographing operation is granted at the time of the photographing operation; <1> from <6> 2. A terminal device according to claim 1 .

[0069] <8> The image processing device further includes a transmission unit that transmits a result of the predetermined image processing to a server device. <1> from <7> 2. A terminal device according to claim 1 .

[0070] <9> An information processing method executed in a terminal device that performs a predetermined image processing via a browser, comprising: a verification processing unit, based on a photographing operation on a photographing target, using verification information for verifying whether the predetermined image processing can be performed on the photographing target via the browser, to execute a verification process for verifying whether the terminal device can execute the predetermined image processing; an acquisition control unit, when it is determined as a result of the verification process that the browser is capable of executing the predetermined image processing, acquiring a first image processing library written in a first program language for executing the predetermined image processing, and when it is determined that the browser is not capable of executing the predetermined image processing, acquiring a second image processing library written in a second program language for executing the predetermined image processing; an image processing unit executing the predetermined image processing using the first image processing library or the second image processing library; a step of displaying, by a display control unit, a result of the predetermined image processing executed based on the first image processing library or the second image processing library on a screen of the terminal device; An information processing method comprising:

[0071] <10> Computer, a verification processing unit that performs a verification process to verify whether or not the browser can execute a predetermined image processing on a subject to be photographed, using verification information for verifying whether or not the terminal device can execute the predetermined image processing on the subject to be photographed via the browser, based on a photographing operation on the subject to be photographed; an acquisition control unit that acquires a first image processing library written in a first program language for executing the predetermined image processing when it is determined that the browser is capable of executing the predetermined image processing as a result of the verification process, and acquires a second image processing library written in a second program language for executing the predetermined image processing when it is determined that the browser is not capable of executing the predetermined image processing; an image processing unit that executes the predetermined image processing using the first image processing library or the second image processing library; a display control unit that displays a result of the predetermined image processing executed based on the first image processing library or the second image processing library on a screen of the terminal device; A program to make it function as such.

[0072] <11> An information processing system for performing a predetermined image processing via a browser, a verification processing unit that performs a verification process to verify whether or not the browser can execute a predetermined image processing on the subject to be photographed, using verification information for verifying whether or not the terminal device can execute the predetermined image processing on the subject to be photographed via the browser, based on a photographing operation on the subject to be photographed; an acquisition control unit that acquires a first image processing library written in a first program language for executing the predetermined image processing when it is determined that the browser is capable of executing the predetermined image processing as a result of the verification process, and acquires a second image processing library written in a second program language for executing the predetermined image processing when it is determined that the browser is not capable of executing the predetermined image processing; an image processing unit that executes the predetermined image processing using the first image processing library or the second image processing library; a display control unit that displays a result of the predetermined image processing executed based on the first image processing library or the second image processing library on a screen of the terminal device; An information processing system comprising: [Explanation of symbols]

[0073] 1 Terminal equipment 2. Server equipment 101 Display control unit 102 Verification processing unit 103 Library Acquisition Control Unit 104 Image processing execution unit 105 Verification program storage unit 106 Image processing library storage unit 201 Receiving unit 202 Transmitter 203 Library Storage 401,501 CPU 402,502 ROM 403,503 RAM 405 CMOS sensor 418 Display [Prior art documents] [Non-patent literature]

[0074] [Non-Patent Document 1] GiXo, “I tried Microsoft Office Lens | A smartphone photo app that automatically corrects the reflection of tilted whiteboards”, [online], April 13, 2016, GiXo, [Retrieved October 26, 2022], Internet<URL:https: / / www.gixo.jp / blog / 8325 / >

Claims

1. A terminal device that performs a predetermined image processing via a browser, a verification processing unit that performs a verification process to verify whether or not the browser can execute the predetermined image processing on the subject to be photographed, using verification information for verifying whether or not the predetermined image processing on the subject to be photographed can be executed via the browser, based on a photographing operation on the subject to be photographed; an acquisition control unit that acquires a first image processing library written in a first program language for executing the predetermined image processing when it is determined that the browser is capable of executing the predetermined image processing as a result of the verification process, and acquires a second image processing library written in a second program language for executing the predetermined image processing when it is determined that the browser is not capable of executing the predetermined image processing; an image processing unit that executes the predetermined image processing using the first image processing library or the second image processing library; a display control unit that displays a result of the predetermined image processing executed based on the first image processing library or the second image processing library on a screen of the terminal device; A terminal device comprising:

2. the first programming language is the same programming language as the verification information, The terminal device according to claim 1 , wherein the second programming language is a programming language different from the programming language of the verification information.

3. The terminal device according to claim 1 , wherein the acquisition control unit acquires the first image processing library and the second image processing library from a server device that stores the first image processing library and the second image processing library.

4. The terminal device according to claim 1 , wherein the first image processing library or the second image processing library includes a library of an image processing unit that contains the subject to be photographed in a rectangle represented by a predetermined frame line displayed on the screen of the terminal device.

5. The programming language for writing the first image processing library is WebAssembly; The terminal device according to claim 2 , wherein the second image processing library is Java Script.

6. The terminal device of claim 1, wherein the display control unit further displays on a screen including an operation item that accepts editing operations on the result of the specified image processing from an operator of the terminal device, and when the operation item is operated, displays the screen for the editing operation on the same screen as the screen displaying the result of the specified image processing or on a separate screen.

7. The terminal device according to claim 1 , wherein the verification process is executed when permission to use a camera for performing the photographing operation is granted at the time of the photographing operation.

8. The terminal device according to claim 1 , further comprising a transmission unit that transmits a result of the predetermined image processing to a server device.

9. An information processing method executed in a terminal device that performs a predetermined image processing via a browser, comprising: a verification processing unit, based on a photographing operation on a photographing target, using verification information for verifying whether the predetermined image processing can be performed on the photographing target via the browser, to perform a verification process to determine whether the browser can perform the predetermined image processing; an acquisition control unit, when it is determined as a result of the verification process that the browser is capable of executing the predetermined image processing, acquiring a first image processing library written in a first program language for executing the predetermined image processing, and when it is determined that the browser is not capable of executing the predetermined image processing, acquiring a second image processing library written in a second program language for executing the predetermined image processing; an image processing unit executing the predetermined image processing using the first image processing library or the second image processing library; a step of displaying, by a display control unit, a result of the predetermined image processing executed based on the first image processing library or the second image processing library on a screen of the terminal device; An information processing method comprising:

10. Computer, a verification processing unit that performs a verification process to verify whether or not the browser can execute a predetermined image processing on a subject to be photographed, using verification information for verifying whether or not the terminal device can execute the predetermined image processing on the subject to be photographed via the browser, based on a photographing operation on the subject to be photographed; an acquisition control unit that acquires a first image processing library written in a first program language for executing the predetermined image processing when it is determined that the browser is capable of executing the predetermined image processing as a result of the verification process, and acquires a second image processing library written in a second program language for executing the predetermined image processing when it is determined that the browser is not capable of executing the predetermined image processing; an image processing unit that executes the predetermined image processing using the first image processing library or the second image processing library; a display control unit that displays a result of the predetermined image processing executed based on the first image processing library or the second image processing library on a screen of the terminal device; A program to make it function as such.

11. An information processing system for performing a predetermined image processing via a browser, a verification processing unit that performs a verification process to verify whether or not the browser can execute a predetermined image processing on the subject to be photographed, using verification information for verifying whether or not the terminal device can execute the predetermined image processing on the subject to be photographed via the browser, based on a photographing operation on the subject to be photographed; an acquisition control unit that acquires a first image processing library written in a first program language for executing the predetermined image processing when it is determined that the terminal device is capable of executing the predetermined image processing as a result of the verification process, and acquires a second image processing library written in a second program language for executing the predetermined image processing when it is determined that the browser is not capable of executing the predetermined image processing; an image processing unit that executes the predetermined image processing using the first image processing library or the second image processing library; a display control unit that displays a result of the predetermined image processing executed based on the first image processing library or the second image processing library on a screen of the terminal device; An information processing system comprising:

Images