Terminal device, information processing method, program, and information processing system
The system optimizes image processing on mobile browsers by dynamically selecting between WebAssembly and JavaScript libraries based on browser capability, ensuring efficient trapezoidal correction and other image processing tasks.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- RICOH CO LTD
- Filing Date
- 2025-01-23
- Publication Date
- 2026-07-07
AI Technical Summary
Existing image processing libraries for mobile browsers may not be available or result in slower processing speeds, limiting the ability to perform functions like trapezoidal correction on captured images.
A terminal device and system that determines the capability of the browser to perform image processing using a verification process, and selectively acquires and utilizes either a first image processing library written in WebAssembly or a second library in JavaScript to execute image processing, ensuring efficient performance.
Enables efficient image processing, such as trapezoidal correction, on mobile browsers by utilizing libraries suitable for the browser's capabilities, enhancing processing speed and functionality.
Smart Images
Figure 0007885889000001 
Figure 0007885889000002 
Figure 0007885889000003
Abstract
Description
Technical Field
[0001] The present invention relates to a terminal device, an information processing method, a program, and an information processing system.
Background Art
[0002] Technologies have been developed to cooperate with external vendor cloud services such as Google (registered trademark) and Office365 (registered trademark), upload documents scanned by an image forming apparatus, and print documents stored in cloud storage. Further, platforms have been developed that can create highly flexible and functional applications by combining these technologies with functions such as OCR (Optical Character Recognition / Reader), image cropping, and barcode authentication.
[0003] This application is called a workflow application and can be executed not only on the operation terminal of an image forming apparatus but also from a web browser of a mobile terminal (an example of a terminal device). Non-Patent Document 1 discloses an application for a smartphone that can correct an image captured by the smartphone.
Summary of the Invention
Problems to be Solved by the Invention
[0004] By the way, when executing a workflow application from a mobile terminal, there is a desire to perform image processing such as trapezoidal correction on an image captured by the mobile terminal. Considering using a mobile terminal as an alternative to an image forming apparatus, it is an essential function to extract a document from an image captured by the mobile terminal and perform trapezoidal correction.
[0005] For example, Microsoft's OneDrive application allows users to take pictures on a mobile device, perform trapezoidal correction and other processing on the captured images, and save the captured images as a single PDF file to OneDrive. Many applications with such information processing functions, such as trapezoidal correction, already exist. For example, browser-based applications, which can be executed in a browser, can achieve the desired information processing functions by loading information processing libraries.
[0006] However, some image processing libraries may not be available depending on the browser on a mobile device. On the other hand, using an image processing library that is available on all browsers can result in slower information processing speeds.
[0007] The present invention has been made in view of the above, and aims to provide a terminal device, an information processing method, a program, and an information processing system that can perform information processing using an image processing library suitable for a browser. [Means for solving the problem]
[0008] To solve the above-mentioned problems and achieve the objective, the present invention provides a terminal device that performs predetermined image processing via a browser, comprising: a verification processing unit that determines whether the browser is capable of performing the predetermined image processing; an image processing unit that, if the verification processing unit determines that the browser is capable of performing the predetermined image processing, performs the predetermined image processing using a first image processing library that performs the predetermined image processing in the browser; and a display control unit that displays the result of the predetermined image processing on the screen of the terminal device. The verification processing unit determines whether the browser is capable of performing the predetermined image processing by executing a program written in the same programming language as the first image processing library. ru. [Effects of the Invention]
[0009] The present invention has the effect of enabling information processing using an image processing library suitable for browsers. [Brief explanation of the drawing]
[0010] [Figure 1] Figure 1 shows an example of the configuration of the information processing system according to this embodiment. [Figure 2] Figure 2 shows an example of the hardware configuration of a terminal device for the information processing system according to this embodiment. [Figure 3] Figure 3 shows an example of the hardware configuration of the server device 2 of the information processing system according to this embodiment. [Figure 4] Figure 4 shows an example of the display sequence of the UI screen in the terminal device of the information processing system according to this embodiment. [Figure 5] Figure 5 is a sequence diagram showing an example of a predetermined image processing flow in the information processing system according to this embodiment. [Figure 6] Figure 6 is a sequence diagram showing another example of a predetermined image processing flow in the information processing system according to this embodiment. [Modes for carrying out the invention]
[0011] The following describes in detail embodiments of the terminal device, information processing method, program, and information processing system with reference to the attached drawings.
[0012] Figure 1 shows an example of the configuration of an information processing system according to this embodiment. As shown in Figure 1, the information processing system according to this embodiment includes a terminal device 1 and a server device 2. The terminal device 1 and the server device 2 are connected via a network NW such as the Internet.
[0013] As shown in Figure 1, the server device 2 has a receiving unit 201, a transmitting unit 202, and a library storage unit 203. The library storage unit 203 stores an image processing library, which is a library containing a predetermined image processing program. Specifically, the library storage unit 203 stores a first image processing library and a second image processing library.
[0014] Here, the predetermined image processing (an example of predetermined processing) is performed via the terminal device 1 and is a pre-configured process, including, for example, trapezoidal correction. Trapezoidal correction is an image processing that places the object captured by the camera of the terminal device 1 within a rectangle displayed on the screen of the terminal device 1 and represented by a predetermined frame. Furthermore, although the image processing described below is an example of predetermined processing, the content of the processing is not limited to this. For example, it may be a process that performs any of the following on data such as image data, video data, text data, or music data: transformation, modification, analysis, or information extraction.
[0015] Furthermore, the first image processing library here is an image processing library written in a first programming language that executes predetermined image processing. The first image processing library is also a library written in the same programming language (an example of the first programming language) as the verification information used by terminal device 1, or more specifically, the Web browser of terminal device 1, to verify whether predetermined image processing can be executed on the target object. The first image processing library may also be a library executed by a Web browser (an example of a browser). In this embodiment, the programming language used to describe the first image processing library may be WebAssembly (hereinafter referred to as WASM). In the following, the verification of whether the software including the Web browser of terminal device 1 can execute predetermined processing may be expressed as "terminal device 1 verifies whether predetermined processing can be executed."
[0016] Furthermore, the second image processing library here is an image processing library written in a second programming language that performs predetermined image processing. The second image processing library is also a library written in a programming language different from the verification information (an example of a second programming language). The second image processing library may also be a library executed by a web browser (an example of a browser). In this embodiment, the programming language used to describe the second image processing library may be Java(registered trademark)Script (registered trademark; hereinafter referred to as JS). Note that image processing using the first image processing library may reduce processing time compared to image processing using the second image processing library. Therefore, if a web browser is capable of executing image processing using the first image processing library, it may be preferable to use the first image processing library rather than the second.
[0017] The receiving unit 201 receives the result of a predetermined image processing on the target object from the terminal device 1 (i.e., the image on which the predetermined image processing has been performed in the terminal device 1). The transmitting unit 202 transmits either the first image processing library or the second image processing library to the terminal device 1. The server device 2 may also have a function unit that executes a workflow including desired processing on the result of the predetermined image processing on the target object received by the receiving unit 201. The workflow may include, for example, the process of storing the result of the predetermined image processing on the target object in a memory area within the server device 2, or the process of sending the result of the predetermined image processing on the target object to a desired destination such as an external server, email address, or fax number. In this case, the server device 2 may determine the destination by analyzing the result of the predetermined image processing on the target object. However, the server device 2 may implement part or all of the workflow processing by cooperating with an external server.
[0018] As shown in FIG. 1, the terminal device 1 includes a display control unit 101, a verification processing unit 102, a library acquisition control unit 103, an image processing execution unit 104, a verification program storage unit 105, and an image processing library storage unit 106. Note that one or more of the display control unit 101, the verification processing unit 102, the library acquisition control unit 103, the image processing execution unit 104, the verification program storage unit 105, and the image processing library storage unit 106 can be realized by a web browser. For example, a program operating on the web browser may realize some or all of the functions realized by one or more functional units. Here, when the terminal device 1 has a plurality of web browsers, one or more functional units can be realized for each web browser.
[0019] The verification program storage unit 105 stores verification information. Here, the verification information is information for verifying whether the terminal device 1 can execute predetermined image processing on a shooting target (for example, an image captured by the terminal device 1). In the present embodiment, the verification information may be a simple program described in WASM, which is an example of a programming language for describing a first image processing library. The image processing library storage unit 106 stores the first image processing library and the second image processing library acquired from the server device 2.
[0020] The verification processing unit 102 is an example of a verification processing unit that executes a verification process of whether the terminal device 1 (web browser) can execute predetermined image processing using the verification information based on a shooting operation of a shooting target. In the present embodiment, when the verification processing unit 102 can execute a program described in WASM, which is an example of the verification information, it determines that the terminal device 1 (web browser) can execute predetermined image processing.
[0021] On the other hand, when the verification processing unit 102 determines that a program described in WASM, which is an example of verification information, cannot be executed, it determines that the terminal device 1 (Web browser) cannot execute predetermined image processing. Further, when the use of the CMOS sensor 405 (see FIG. 2), which is an example of a camera or imaging unit that executes imaging of the imaging target during the imaging operation, is permitted, the verification processing unit 102 may execute verification processing.
[0022] The library acquisition control unit 103 is an example of an acquisition control unit that acquires a first image processing library for executing predetermined image processing when the verification processing unit 102 determines that the terminal device 1 (Web browser) can execute the predetermined image processing as a result of the verification processing. On the other hand, when the verification processing unit 102 determines that the terminal device 1 (Web browser) cannot execute the predetermined image processing, the library acquisition control unit 103 acquires a second image processing library for executing the predetermined image processing. In the present embodiment, the library acquisition control unit 103 acquires the first image processing library and the second image processing library from the server device 2. Then, the library acquisition control unit 103 stores the acquired first image processing library and second image processing library in the image processing library storage unit 106. Further, the library acquisition control unit 103 also functions as an example of a transmission unit that transmits the result of the predetermined image processing on the imaging target by the image processing execution unit 104 described later to the server device 2.
[0023] The image processing execution unit 104 is an example of an image processing unit that executes predetermined image processing using the first image processing library or the second image processing library. That is, when the library acquisition control unit 103 acquires the first image processing library, it executes the predetermined image processing using the first image processing library, and when the library acquisition control unit 103 acquires the second image processing library, it executes the predetermined image processing using the second image processing library. As a result, since the predetermined image processing can be executed on the image of the imaging target by the image processing library executable on the Web browser, the data processing function in the browser-based application can be realized.
[0024] Here, the predetermined image processing includes keystone correction, binarization to convert the image to two colors, and conversion to grayscale. Keystone correction is a correction that fits the object captured by the CMOS sensor 405 (see Figure 2) of the terminal device 1 into a rectangle represented by a predetermined frame displayed on the screen of the terminal device 1. In other words, keystone correction is a process that corrects the distortion of the object to a rectangle.
[0025] The display control unit 101 is an example of a display control unit that displays the results of predetermined image processing performed based on the first or second image processing library on a display 418 (see Figure 2), which is an example of a display unit (or screen) of the terminal device 1. For example, the display control unit 101 may display on the display 418 of the terminal device 1 an object that has undergone predetermined image processing based on the first or second image processing library.
[0026] Furthermore, the display control unit 101 displays on the display 418 of the terminal device 1 a screen that includes an operation item for accepting an editing operation on the result of a predetermined image processing (for example, correction of the boundary of a subject to be photographed included in the image) from the operator of the terminal device 1. When an operation item is operated, the display control unit 101 displays the editing operation screen on the same screen as the screen displaying the result of the predetermined image processing, or on a separate screen.
[0027] Figure 2 shows an example of the hardware configuration of a terminal device of the information processing system according to this embodiment. As shown in Figure 2, the terminal device 1, such as a smartphone, is equipped with a CPU 401, ROM 402, RAM 403, EEPROM 404, CMOS sensor 405, image sensor I / F 406, acceleration / direction sensor 407, media I / F 409, and GPS receiver 411.
[0028] Of these, the CPU 401 controls the operation of the entire terminal device 1. The ROM 402 stores the CPU 401 and programs used to drive the CPU 401, such as the IPL. The RAM 403 is used as the work area for the CPU 401. The EEPROM 404 reads or writes various data, such as smartphone programs, according to the control of the CPU 401. The CMOS (Complementary Metal Oxide Semiconductor) sensor 405 is a type of built-in imaging means that captures an image of a subject (mainly a self-portrait) according to the control of the CPU 401 to obtain image data. Note that it may be an imaging means such as a CCD (Charge Coupled Device) sensor instead of a CMOS sensor. The image sensor I / F 406 is a circuit that controls the driving of the CMOS sensor 405. The acceleration / direction sensor 407 is a type of sensor such as an electronic magnetic compass and gyrocompass that detect the Earth's magnetic field, and an acceleration sensor. The media I / F 409 controls the reading or writing (storage) of data to or from the recording medium 408, such as flash memory. The GPS receiver 411 receives GPS signals from GPS satellites.
[0029] Furthermore, the terminal device 1 includes a long-range communication circuit 412, a CMOS sensor 413, an image sensor interface 414, a microphone 415, a speaker 416, an audio input / output interface 417, a display 418, an external device connection interface 419, a short-range communication circuit 420, an antenna 420a for the short-range communication circuit 420, and a touch panel 421.
[0030] Of these, the long-range communication circuit 412 is a circuit that communicates with other devices via a communication network. The CMOS sensor 413 is a type of built-in imaging means that captures an image of a subject and obtains image data according to the control of the CPU 401. The image sensor interface 414 is a circuit that controls the driving of the CMOS sensor 413. The microphone 415 is a built-in circuit that converts sound into electrical signals. The speaker 416 is a built-in circuit that converts electrical signals into physical vibrations to produce sounds such as music and speech. The sound input / output interface 417 is a circuit that processes the input and output of sound signals between the microphone 415 and the speaker 416 according to the control of the CPU 401. The display 418 is a type of display means such as liquid crystal or organic EL (electroluminescence) that displays images of subjects and various icons. The external device connection interface 419 is an interface for connecting various external devices. The short-range communication circuit 420 is a communication circuit such as NFC (Near Field Communication) and Bluetooth (registered trademark). The touch panel 421 is a type of input means that allows the user to operate the terminal device 1 by pressing the display 418.
[0031] Furthermore, terminal device 1 is equipped with a bus line 410. The bus line 410 is an address bus and a data bus, etc., for electrically connecting each component, such as the CPU 401 shown in Figure 2.
[0032] Figure 3 shows an example of the hardware configuration of the server device 2 of the information processing system according to this embodiment. As shown in Figure 3, the server device 2 is built by a computer and includes a CPU 501, ROM 502, RAM 503, HD 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, bus line 510, keyboard 511, pointing device 512, DVD-RW (Digital Versatile Disk Rewritable) drive 514, and media I / F 516.
[0033] Of these, the CPU 501 controls the operation of the entire server device 2. The ROM 502 stores programs used to drive the CPU 501, such as the IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 509 is an interface for data communication using a communication network. The bus line 510 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 3.
[0034] The keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, and various instructions. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting processing targets, moving the cursor, etc. The DVD-RW drive 514 controls the reading or writing of various data to the DVD-RW 513, which is an example of a removable recording medium. Note that it is not limited to DVD-RW, but may also be DVD-R, etc. The media I / F 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.
[0035] Figure 4 shows an example of the display sequence of the UI screen in the terminal device of the information processing system according to this embodiment. First, when a user instructs the launch of an application that runs in the web browser of the terminal device 1, the display control unit 101 displays the application screen A, which accepts various operations, on a display unit such as the display 418.
[0036] Then, when the user performs a shooting operation on the application screen A, the display control unit 101 displays a document shooting screen B on the display 418, which allows the user to instruct the system to shoot the target object. The user can also instruct the system to shoot multiple targets consecutively on the document shooting screen B. Then, when the CMOS sensor 405 (an example of a shooting unit or camera) has captured the target object, the display control unit 101 displays a page confirmation screen C on the display 418, which includes the results of predetermined image processing on the target object.
[0037] Subsequently, if the camera is instructed to complete shooting on the page confirmation screen C, or if the camera is instructed to complete shooting on the boundary manual correction screen D, which allows manual correction of the trapezoidal boundary included in the shooting result of the subject, the display control unit 101 displays an application screen E on the display 418 that allows the user to instruct the transmission of the shooting result of the subject to the server device 2. On the application screen E, the user can set the email address to which the image data, which is the shooting result of the subject, will be sent, and the storage location, such as the server device 2 or a folder registered on the server device 2. This information regarding the destination and storage location is just an example of the setting information. However, the setting information included on the application screen E is not limited to this. The image processing execution unit 104 transmits the shooting result of the subject to the server device 2 according to the instructions on the application screen E. Here, the image processing execution unit 104 can transmit an image (the shooting result of the subject) on which predetermined image processing has been performed to the server device 2.
[0038] The display control unit 101 may also display an application screen A or a page confirmation screen C that includes an operation item for accepting editing operations (for example, the procedural correction described above) on the results of a predetermined image processing from the user (operator) of the terminal device 1. When such an operation item is performed, the display control unit 101 may display a boundary manual correction screen D, which is an example of an editing operation screen, on the same screen as or a different screen from the page confirmation screen C, which is an example of a screen displaying the results of a predetermined image processing.
[0039] However, if the CMOS sensor 405 fails to start despite being activated according to the instructions to photograph the subject on document shooting screen B, the display control unit 101 displays a camera startup error dialog F on the display 418 instructing the user to update the web browser or obtain the shooting result of the subject by other means. If the user is instructed to update the web browser, the display control unit 101 displays document shooting screen B on the display 418 again after the web browser has been updated.
[0040] On the other hand, if the user is instructed to obtain the image of the target object by other means, the display control unit 101 displays an OS (Operating System)-dependent screen G on the display 418 that allows the user to instruct the acquisition of the image of the target object. On the OS-dependent screen G, the user can select a desired image from image data stored in the terminal device 1 or an external device that can communicate with the terminal device 1. Alternatively, on the OS-dependent screen G, the user may also acquire the image of the target object by launching a camera application or the like and taking a picture with the camera. After that, the display control unit 101 displays a page confirmation screen C on the display 418 that includes the acquired image of the target object.
[0041] Figure 5 is a sequence diagram showing an example of a predetermined image processing flow in the information processing system according to this embodiment. If the terminal device 1 has multiple web browsers, the following processing can be performed for each web browser. The following processing can be performed by the web browser that the user instructs to start from among the multiple web browsers. When the terminal device 1 instructs to start an application running in its web browser (step S521), the display control unit 101 displays the application screen A, which accepts various operations, on the display 418 (step S522).
[0042] Then, when the application screen A is instructed to move to the document shooting screen B, which allows the CMOS sensor 405 to be instructed to take a picture (step S523), the verification processing unit 102 uses the verification information to perform a verification process to determine whether the terminal device 1 is capable of performing a predetermined image processing (step S524). In this embodiment, the verification processing unit 102 verifies whether WASM is available. For example, the verification processing unit 102 may verify whether WASM is available by determining whether it can execute a simple WASM program, which is an example of verification information.
[0043] If WASM is available, the library acquisition control unit 103 acquires and reads a first image processing library (for example, a trapezoidal correction library written in WASM) from the server device 2 (step S525). On the other hand, if WASM is not available, the library acquisition control unit 103 acquires and reads a second image processing library (for example, a trapezoidal correction library written in JS) from the server device 2 (step S526). The verification processing unit 102 also determines whether or not the use of the CMOS sensor 405 is permitted (step S527). In this embodiment, the verification processing unit 102 may also determine whether or not the use of the CMOS sensor 405 is permitted by checking whether or not the video (image) captured by the CMOS sensor 405 can be displayed in a web browser.
[0044] If the use of the CMOS sensor 405 is permitted, the display control unit 101 displays the document capture screen B on the display 418 (step S528). The image processing execution unit 104 stores the captured image in memory such as RAM 43 each time a capture is performed by the CMOS sensor 405 (step S529), and performs predetermined image processing on the image using a first image processing library (for example, a trapezoidal correction library) (for example, a process that recognizes a trapezoid contained in the image and stores its vertex coordinates) (step S530). At this time, the image processing execution unit 104 may generate an image with trapezoidal correction applied to the captured image and store the generated image in memory such as RAM 43. However, the image processing execution unit 104 may also store the setting information necessary for trapezoidal correction and generate the image with trapezoidal correction applied when it receives an instruction to transition to the page confirmation screen C, and when the start button is pressed on the application screen E. Next, the display control unit 101 displays the document capture screen B on the display 418 again (step S531).
[0045] Next, when the user instructs to transition to page confirmation screen C, which includes an image captured by the CMOS sensor 405 (step S532), the display control unit 101 displays page confirmation screen C, which includes an image after predetermined image processing has been performed, on the display 418 (step S533). Next, when the user instructs to transition to boundary manual correction screen D, which allows manual correction of the trapezoidal boundary included in the image captured by the CMOS sensor 405 (step S534), the display control unit 101 displays boundary manual correction screen D on the display 418 (step S535).
[0046] When manual correction of the trapezoidal boundary is performed on the manual boundary correction screen D (step S536), the image processing execution unit 104 updates the coordinates of the trapezoidal vertices included in the image (step S537). Next, the display control unit 101 displays the manual boundary correction screen D, which includes the updated trapezoidal image with the updated vertex coordinates, on the display 418 (step S538). After that, when the complete button is pressed on the manual boundary correction screen D (step S539), the image processing execution unit 104 terminates the predetermined image processing on the image (step S540).
[0047] Next, when the start button is pressed on the application screen E (step S541), the library acquisition control unit 103 sends the image after performing predetermined image processing to the server device 2 (step S542). At this time, the terminal device 1 sends the setting information entered on the application screen E to the server device 2. This allows the server device 2 to execute a workflow based on the image after performing predetermined image processing and the setting information. Since the server device 2 can execute multiple workflows, the terminal device 1 may also send information to the server device 2 that identifies the workflow. The workflow to be executed can be set on the application screen A. For example, the workflow to be linked can be selected on the application screen A. Alternatively, the user can link to a workflow corresponding to a URL by entering a URL in a web browser.
[0048] Figure 6 is a sequence diagram showing another example of a predetermined image processing flow in the information processing system according to this embodiment. In Figure 6, the same processing as in Figure 5 is omitted from the explanation.
[0049] If, in step S527, it is determined that the use of the CMOS sensor 405 is not permitted, the display control unit 101 displays the document capture screen B on the display 418 again (step S528). Furthermore, the display control unit 101 displays a camera startup error dialog F on the display 418, notifying that an error occurred during the startup of the CMOS sensor 405 (step S601).
[0050] Next, if the camera startup error dialog F instructs the user to acquire an image by another means (step S602), the display control unit 101 displays an OS-dependent screen G on the display 418 that includes a menu for selecting the image acquisition means (step S603). Then, if an image file is selected on the OS-dependent screen G (step S604), the image processing execution unit 104 performs predetermined image processing on the selected image using the loaded image processing library from the first and second image processing libraries (step S605). After that, the display control unit 101 displays a boundary manual correction screen D on the display 418 that includes the image on which the predetermined image processing has been performed (step S606).
[0051] Then, when the "Complete" button is pressed on the manual boundary correction screen D (step S539), the image processing execution unit 104 terminates the predetermined image processing on the image (step S540).
[0052] Thus, according to the information processing system of this embodiment, predetermined image processing can be performed on the image of the target object using an image processing library that can be executed on a web browser, thereby realizing data processing functions in a browser-based application.
[0053] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), FPGAs (Field Programmable Gate Arrays), and conventional circuit modules designed to execute each of the functions described above.
[0054] The apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, server apparatus 2 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including a network or shared memory, and perform the processing disclosed herein. Similarly, terminal apparatus 1 may include multiple computing devices configured to communicate with each other.
[0055] Furthermore, terminal device 1 and server device 2 can be configured to share the disclosed processing steps, for example, Figure 1, in various combinations. For example, a process performed by terminal device 1 may be performed by server device 2. Similarly, the functions of server device 2 may be performed by terminal device 1. Also, each element of terminal device 1 and server device 2 may be combined into a single server device or divided among multiple devices.
[0056] Note that terminal device 1 or server device 2 is not limited to smartphones or servers, as long as it is a device equipped with communication capabilities. Terminal device 1 or server device 2 may be, for example, output devices such as PJ (Projector), IWB (Interactive White Board), digital signage, HUD (Head Up Display) devices, industrial machinery, imaging devices, sound collection devices, medical equipment, networked home appliances, automobiles (Connected Car), notebook PCs (Personal Computers), mobile phones, smartphones, tablet devices, game consoles, PDAs (Personal Digital Assistants), digital cameras, wearable PCs, or desktop PCs.
[0057] The programs executed by terminal device 1 and server device 2 in this embodiment are provided pre-installed in ROMs 402, 502, etc. The programs executed by terminal device 1 and server device 2 in this embodiment may also be provided as installable or executable files recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R, or DVD (Digital Versatile Disk).
[0058] Furthermore, the programs executed by the terminal device 1 and server device 2 of this embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading them via the network. Alternatively, the programs executed by the terminal device 1 and server device 2 of this embodiment may be provided or distributed via a network such as the Internet.
[0059] The program executed by the terminal device 1 of this embodiment has a modular configuration that includes the above-mentioned parts (display control unit 101, verification processing unit 102, library acquisition control unit 103, and image processing execution unit 104). In actual hardware, a processor such as the CPU 401 reads the program from the ROM 402 and executes it, loading the above-mentioned parts onto the main memory, and generating the display control unit 101, verification processing unit 102, library acquisition control unit 103, and image processing execution unit 104 on the main memory.
[0060] The program executed by the server device 2 of this embodiment has a modular configuration that includes the above-described parts (receiving unit 201, transmitting unit 202). In actual hardware, a processor such as the CPU 501 reads the program from the ROM 502 and executes it, loading the above-described parts into the main memory, and creating the receiving unit 201 and transmitting unit 202 in the main memory.
[0061] Examples of the present invention are as follows:
[0062] <1> A terminal device that performs predetermined image processing via a browser, comprising: a verification processing unit that performs a verification process to determine whether the browser can perform the predetermined image processing on the target object, based on a shooting operation on the target object, using verification information for verifying whether the browser can perform the predetermined image processing on the target object; If the verification process determines that the browser is capable of performing the predetermined image processing, the acquisition control unit acquires a first image processing library written in a first programming language that performs the predetermined image processing; if the browser determines that it is not capable of performing the predetermined image processing, the acquisition control unit acquires a second image processing library written in a second programming language that performs the predetermined image processing. An image processing unit that performs the predetermined image processing using the first image processing library or the second image processing library, A terminal device comprising: a display control unit that displays the result of the predetermined image processing performed based on the first image processing library or the second image processing library on the screen of the terminal device.
[0063] <2> The first programming language is the same programming language as the verification information, and the second programming language is a different programming language from the verification information. <1> The terminal device described above.
[0064] <3> The acquisition control unit acquires the first image processing library and the second image processing library from the server device storing the first image processing library and the second image processing library. <1> or <2> The terminal device described above.
[0065] <4> The first image processing library or the second image processing library includes a library for the image processing unit that stores the target to be photographed within a rectangle represented by a predetermined frame displayed on the screen of the terminal device. <1> from <3> A terminal device as described in any one of the following.
[0066] <5> The programming language used to describe the first image processing library is WebAssembly, and the second image processing library is JavaScript. <2> The terminal device described above.
[0067] <6> The display control unit further displays an operation item on a screen that accepts an editing operation on the result of the predetermined image processing from the operator of the terminal device, and when the operation item is operated, it displays the editing operation screen on the same screen as the screen displaying the result of the predetermined image processing or on a separate screen. <1> from <5> A terminal device as described in any one of the following.
[0068] <7> The verification process is performed when permission is granted to use the camera used to perform the shooting operation. <1> from <6> A terminal device as described in any one of the following.
[0069] <8> The system further comprises a transmission unit that transmits the results of the predetermined image processing to a server device. <1> from <7> A terminal device as described in any one of the following.
[0070] <9> An information processing method performed on a terminal device that performs predetermined image processing via a browser, The verification processing unit performs a verification process to determine whether the terminal device can perform the predetermined image processing, using verification information to verify whether the predetermined image processing can be performed on the target via the browser, based on the shooting operation on the target. The acquisition control unit, if it determines as a result of the verification process that the browser is capable of performing the predetermined image processing, acquires a first image processing library written in a first programming language that performs the predetermined image processing; and if it determines that the browser is not capable of performing the predetermined image processing, acquires a second image processing library written in a second programming language that performs the predetermined image processing. The image processing unit performs the predetermined image processing using the first image processing library or the second image processing library, The display control unit performs the step of displaying the result of the predetermined image processing performed based on the first image processing library or the second image processing library on the screen of the terminal device, Information processing methods including
[0071] <10> Computers, A verification processing unit that, based on a shooting operation on the target to be photographed, uses verification information to verify whether the terminal device can perform a predetermined image processing on the target via a browser, and performs a verification process to determine whether the browser can perform the predetermined image processing. If the verification process determines that the browser is capable of performing the predetermined image processing, the acquisition control unit acquires a first image processing library written in a first programming language that performs the predetermined image processing; if the browser determines that it is not capable of performing the predetermined image processing, the acquisition control unit acquires a second image processing library written in a second programming language that performs the predetermined image processing. An image processing unit that performs the predetermined image processing using the first image processing library or the second image processing library, A display control unit that displays the result of the predetermined image processing performed based on the first image processing library or the second image processing library on the screen of the terminal device, A program to make it work.
[0072] <11> An information processing system that performs predetermined image processing via a browser, A verification processing unit that, based on a shooting operation on the target to be photographed, uses verification information to verify whether the terminal device can perform predetermined image processing on the target via the browser, and performs verification processing to determine whether the browser can perform predetermined image processing. If the verification process determines that the browser is capable of performing the predetermined image processing, the acquisition control unit acquires a first image processing library written in a first programming language that performs the predetermined image processing; if the browser determines that it is not capable of performing the predetermined image processing, the acquisition control unit acquires a second image processing library written in a second programming language that performs the predetermined image processing. An image processing unit that performs the predetermined image processing using the first image processing library or the second image processing library, A display control unit that displays the result of the predetermined image processing performed based on the first image processing library or the second image processing library on the screen of the terminal device, An information processing system equipped with the following features. [Explanation of Symbols]
[0073] 1 Terminal device 2 Server devices 101 Display Control Unit 102 Verification Processing Unit 103 Library Acquisition Control Unit 104 Image Processing Execution Unit 105 Verification Program Storage Unit 106 Image Processing Library Storage Unit 201 Receiver 202 Transmitter 203 Library Storage Unit 401,501 CPU 402,502 ROM 403,503 RAM 405 CMOS sensor 418 displays [Prior art documents] [Non-patent literature]
[0074] [Non-Patent Document 1] GiXo, “I tried using Microsoft Office Lens | A smartphone photo app that automatically corrects tilt and reflection on whiteboards,” [online], April 13, 2016, GiXo, [Retrieved October 26, 2022], Internet<URL:https: / / www.gixo.jp / blog / 8325 / >
Claims
1. A terminal device that performs predetermined image processing via a browser, A verification processing unit that determines whether the browser is capable of performing the predetermined image processing, If the verification processing unit determines that the browser is capable of performing the predetermined image processing, the browser will execute the predetermined image processing using a first image processing library, A display control unit that displays the result of the predetermined image processing on the screen of the terminal device, Equipped with, The verification processing unit is a terminal device that determines whether the browser is capable of performing the predetermined image processing by executing a program written in the same programming language as the first image processing library.
2. The terminal device according to claim 1, wherein if the image processing unit determines that the browser cannot perform the predetermined image processing, it performs the predetermined image processing using a second image processing library that performs the predetermined image processing in the browser.
3. The aforementioned first image processing library is written in the first programming language, The terminal device according to claim 2, wherein the second image processing library is written in a second programming language different from the first programming language.
4. The first programming language is the same programming language as the program, The terminal device according to claim 3, wherein the second programming language is a programming language different from the program.
5. The terminal device according to any one of claims 2 to 4, further comprising an acquisition control unit that acquires the first image processing library and the second image processing library from a server device storing the first image processing library and the second image processing library.
6. The terminal device according to any one of claims 2 to 4, wherein the first image processing library or the second image processing library includes a library for the image processing unit that stores the object to be photographed in a rectangle represented by a predetermined frame line displayed on the screen of the terminal device.
7. The programming language used to describe the aforementioned first image processing library is WebAssembly. The terminal device according to any one of claims 2 to 4, wherein the second image processing library is JavaScript.
8. The terminal device according to any one of claims 1 to 4, wherein the display control unit further displays an operation item on a screen that accepts an editing operation on the result of the predetermined image processing from the operator of the terminal device, and when the operation item is operated, the editing operation screen is displayed on the same screen as the screen displaying the result of the predetermined image processing or on a separate screen.
9. The terminal device according to claim 4, wherein the processing based on the program is executed when permission is granted to use the camera that performs the shooting operation on the object to be photographed.
10. The terminal device according to any one of claims 1 to 4, further comprising a transmission unit for transmitting the results of the predetermined image processing to a server device.
11. An information processing method performed on a terminal device that performs predetermined image processing via a browser, The verification processing unit includes the step of determining whether the browser is capable of performing the predetermined image processing, If the image processing unit determines that the browser is capable of performing the predetermined image processing, it performs the predetermined image processing using a first image processing library that performs the predetermined image processing in the browser, The display control unit performs the step of displaying the result of the predetermined image processing on the screen of the terminal device, Includes, The verification processing unit determines whether the browser is capable of performing the predetermined image processing by executing a program written in the same programming language as the first image processing library, which is an information processing method.
12. Computers, A verification processing unit that determines whether the browser is capable of performing a predetermined image processing, If the verification processing unit determines that the browser is capable of performing the predetermined image processing, the image processing unit executes the predetermined image processing using a first image processing library that performs the predetermined image processing in the browser. A display control unit that displays the result of the predetermined image processing on the screen of a terminal device, and make it work The verification processing unit is a program that determines whether the browser is capable of performing the predetermined image processing by executing a program written in the same programming language as the first image processing library.
13. An information processing system that performs predetermined image processing via a browser, A verification processing unit that determines whether the browser is capable of performing the predetermined image processing, If the verification processing unit determines that the browser is capable of performing the predetermined image processing, the browser will execute the predetermined image processing using a first image processing library, A display control unit that displays the result of the predetermined image processing on the screen of a terminal device, Equipped with, The verification processing unit is an information processing system that determines whether the browser is capable of performing the predetermined image processing by executing a program written in the same programming language as the first image processing library.