Image processing device, image processing system, image processing method, program, and recording medium

Abstract

Provided is a technology which makes it possible to ensure uniqueness in a virtual space, with respect to a real printed object, and to appropriately manage the handling of the same. An image processing device 10 according to one embodiment of the present invention comprises a processor 104. The processor 104: acquires information 111 related to an association, in which a first image 6A and meta information pertaining to the result of reading a real printed object 5 are associated; and draws a second image 521 in a virtual space on the basis of the information 111 related to the association.

Description

Image processing device, image processing system, image processing method, program, and recording medium

[0001] One embodiment of the present invention relates to an image processing device, an image processing system, an image processing method, a program, and a recording medium that ensure uniqueness in a virtual space with respect to a real print and enable appropriate management of its handling.

[0002] While high-performance digital cameras have become widespread, instant cameras with printing capabilities, which allow users to easily enjoy the unique quality of photographs printed on photographic paper, are once again garnering attention. Unlike digital images, which can be easily copied and shared, photographs taken and printed with such instant cameras (hereinafter referred to as prints) are unique, capturing the moment of the photo and offering the potential to become treasured keepsakes for the photographer and the subject. However, because such prints are single images, they are difficult to easily share with related parties. For example, when a photographer posts a print of a photo from an event or family gathering on a social networking service (SNS) as a fond memory, the photographer photographs the print with a smartphone, converts it into a digital image, and posts it. As described above, technologies for appropriately digitizing physical objects and handling them in virtual spaces on a network are already known. Patent Document 1 (JP-A-2005-102666) is one example of such a technology. The technology described in Patent Literature 1 relates to a method for transporting a physical object into a virtual world, the method comprising detecting property and structure data of the physical object and transmitting the property and structure data to a computing device, where the property and structure data is automatically converted into code that generates the virtual world object when instantiated in the virtual world. This makes it possible to provide a system and method for transporting real physical objects into a virtual world.Furthermore, the technology described in Patent Document 2 relates to an information processing device that generates virtual space data usable in a virtual space from scan information of the appearance of an object acquired by a three-dimensional scanner device configured to acquire three-dimensional shape and color information of the object, the information processing device including: a determination means for determining the type of the object; an acquisition means for acquiring frame data corresponding to the determined type of object, the frame data being configured to map texture information obtained from the scan information; a first screen for adjusting a coordinate system set for first scan information generated by scanning the object with the three-dimensional scanner device to a coordinate system set for the frame data, the first screen including a display of the first scan information, a receiving means for receiving an instruction, including a first operation instruction of at least one of movement and rotation, for the first scan information displayed on the first screen; and a first generation means for generating the virtual space data, including texture information to be mapped onto the frame data, from second scan information obtained after displaying the first screen is terminated in response to the instruction, and the frame data. This enables information about the appearance of an object to be acquired with high accuracy and used in a virtual space. Furthermore, the technology described in Patent Document 3 is a technology in a virtual space communication system that realizes photographing by operating an avatar in a virtual space with a sense of realism similar to photographing in the real world, and relates to a virtual space communication system that includes a server device that provides a virtual space in which a character that acts as the user's avatar is placed, and a user terminal that is connected to the server device via a network and that the user uses to operate the character, the virtual space has a virtual camera that the character uses to take photographs within the virtual space, and the user terminal is connected to a printing device that prints images, and the virtual space communication system is characterized in that, based on the user's operation of the character, the images taken within the virtual space by the virtual camera are obtained from the server device and printed on the printing device.

[0003] JP 2010-131764 A JP 2013-128649 A JP 2009-176025 A

[0004] By adopting the technologies of the above-mentioned patent documents, it becomes possible to digitize real prints so that they can be handled in virtual space, etc. In addition, in the virtual space, it is also possible to enjoy taking pictures in the space and printing the taken images on real photo paper using a printer in the real world.

[0005] However, as virtual space has become more advanced, there is a demand for a seamless experience between the virtual and real worlds. For example, even if you own a physical photo print in the real world, it is not easy to obtain an exact copy of that same photo. This is because the process of copying results in degradation of image quality and changes in color. This is why a photo print that is "your own precious photo" has meaning. However, once this photo is digitized and placed in a virtual space, the nature of digital data makes it easy to obtain an exact copy. In this case, the meaning of the photo's existence as "your own precious photo" is diminished, resulting in a sense of separation between the virtual and real worlds.

[0006] On the other hand, if the digitization, copying, printing, etc. of printed materials is allowed too easily and this is not managed properly, there may be concerns that the value and rights (e.g., copyright and portrait rights) of the printed materials and their subjects may be damaged.

[0007] One embodiment of the present invention has been made in consideration of the above circumstances, and aims to provide an apparatus, system, method, program, and recording medium that ensures uniqueness in virtual space for real printed materials and enables appropriate management of their handling.

[0008] The above object is achieved by an image processing device according to any one of [1] to [7] below. [1] An image processing device having a processor, wherein the processor acquires information relating to the association that associates a first image with meta information relating to the results of scanning a real printed matter, and renders a second image in a virtual space based on the information relating to the association, wherein the first image is an image included in the scanned result. [2] The image processing device according to [1], wherein the processor controls the rendering function of the second image in the virtual space to a certain level or less. [3] The image processing device according to [1], wherein the processor stores information relating to the association between the first image and the meta information in a blockchain on a network. [4] The image processing device according to [3], wherein the processor adds information restricting the printing operation of the second image on a real printer to the information relating to the association stored in the blockchain. [5] The image processing device according to [3], wherein the processor adds information indicating that ownership of the second image has been transferred from the user's source to the user's destination, to the information relating to the association in the blockchain, upon the transfer of the second image between users in the virtual space. [6] The image processing device according to [1], wherein the processor, based on information relating to the association of one image from a group of moving or still images, executes, in a virtual space, playback of a moving image corresponding to the association or continuous drawing of a group of images as a second image. [7] The image processing device according to [6], wherein the processor, when a holder of a single image plays back a moving image or continuously draws a group of images in the virtual space, controls the target of playback or drawing to be all or part of the moving image or group of images.

[0009] The above object can also be achieved by an image processing system described in [8] below: [8] An image processing system including an image processing device having a processor, wherein the processor of the image processing device acquires information relating to the association that associates a first image with meta information relating to the results of reading a real printed matter, and renders a second image in a virtual space based on the information relating to the association, and the first image is an image included in the results of the reading.

[0010] The above object can also be achieved by an image processing method described in [9] below: [9] An image processing method comprising: a step of acquiring, by a processor, information relating to the association that associates a first image with meta information relating to the results of reading a real printed matter; and a step of drawing, by the processor, a second image in a virtual space based on the information relating to the association, wherein the first image is an image included in the results of the reading.

[0011] The above object can also be achieved by the program described in

[10] below:

[10] A program for causing a computer to execute each step included in the image processing method described in [9] above.

[0012] The above object can also be achieved by the recording medium described in

[11] below:

[11] A computer-readable recording medium having recorded thereon a program for causing a computer to execute each step included in the image processing method described in [9] above.

[0013] According to one embodiment of the present invention, there are provided an apparatus, a method, a program, and a recording medium that ensure uniqueness in a virtual space with respect to a real printed matter and enable appropriate management of its handling.

[0014] FIG. 1 is a diagram showing an example of a network configuration including an image processing apparatus according to an embodiment of the present invention. FIG. 2 is a diagram showing a conceptual example of a digitization operation of a printed material according to an embodiment of the present invention. FIG. 3 is a diagram showing an example of the hardware configuration of an information processing apparatus according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of the hardware configuration of an image processing apparatus according to an embodiment of the present invention. FIG. 5 is a diagram showing an example of the configuration of an image information DB according to an embodiment of the present invention. FIG. 6 is a diagram showing an example of the configuration of a management DB according to an embodiment of the present invention. FIG. 7 is a diagram showing an example of the hardware configuration of an imaging device according to an embodiment of the present invention. FIG. 8 is a diagram showing an example of the configuration of a distributed ledger system according to an embodiment of the present invention. FIG. 9 is a diagram showing an example of the hardware configuration of a server according to an embodiment of the present invention. FIG. 10 is a diagram showing an example of the configuration of an image DB according to an embodiment of the present invention. FIG. 11 is a flow diagram of an image processing method (part 1) according to an embodiment of the present invention. FIG. 12 is a diagram showing an example of an output (part 1) according to an embodiment of the present invention. FIG. 13 is a diagram showing an example of an output (part 2) according to an embodiment of the present invention. FIG. 14 is a diagram showing an example of an output (part 3) according to an embodiment of the present invention. FIG. 15 is a diagram showing an example of an object table according to an embodiment of the present invention. FIG. 16 is a diagram showing example of management information according to an embodiment of the present invention.

[0015] Specific embodiments of the present invention will be described below. For ease of explanation, the following description may be given in terms of a GUI (Graphic User Interface). Furthermore, since the basic data processing technologies (communication / transmission technologies, data acquisition technologies, data recording technologies, data processing / analysis technologies, machine learning technologies, image processing technologies, visualization technologies, etc.) required to realize the present invention are well-known technologies, a description thereof will be omitted.

[0016] In addition, in this specification, the concept of "device" includes not only a single device that performs a specific function, but also a combination of multiple devices that exist independently and in a distributed manner but cooperate (link) to perform a specific function.

[0017] Furthermore, in this specification, a "user" is a user who performs various processes such as outputting the printout (including photographing the subject) to be processed by the image processing device of the present invention, and scanning the printout, and who can view the processing results obtained by the functions of the image processing device of the present invention or other devices (for example, VR images in which the printout is digitized and reproduced in a virtual space, and related images, etc.) and print them out using a real printer, etc.

[0018] Furthermore, in this specification, the term "person" refers to an entity that performs a specific action, and includes individuals, groups, corporations such as companies, and organizations, as well as computers and devices that constitute artificial intelligence (AI). Artificial intelligence (AI) realizes intelligent functions such as inference, prediction, and judgment using hardware and software resources. The algorithm of the artificial intelligence is arbitrary, and examples include expert systems, case-based reasoning (CBR), Bayesian networks, and subsumption architectures.

[0019] <<One Embodiment of the Present Invention>> In one embodiment of the present invention (hereinafter, this embodiment), an image processing system 3 is assumed that is configured with an image processing device 10, a photographing device 20, a user terminal 30, a distributed ledger system 40, and a server 50, all connected to a network 1 shown in Fig. 1. Of these, the photographing device 20 is an instant camera with a printing function, and as shown in Fig. 2, is a camera that is capable of printing a photograph 6 on dedicated film upon photographing a subject, and generating and outputting a print 5.

[0020] A user starts a predetermined application on an image processing device 10 such as a smartphone and performs a scan operation on one or more of the prints 5. This scan operation acquires an image 5A of the prints 5, including an image 6A of a photograph 6, and the image 5A is displayed on a UI 122 of the application. The user performs various editing operations on the UI 122, mainly on the image quality, size, etc. of the image 6A of the photograph 6.

[0021] The image processing device 10 stores and manages, as image information 111, information relating to the identification information of one or more images 5A obtained by the scanning operation (e.g., an ID printed on the printout 5 or a hash value of the image 5A) and meta information such as the date and time of the scanning operation, its own identification information, the user's identification information, and the identification information of the photographing device 20. The image 5A data is uploaded from the image processing device 10 to a server 50 on the network 1. The server 50 stores the image 5A data obtained from the image processing device 10 in an image DB 511 and prepares for image output in a virtual space in response to access from the image processing device 10 (or a user terminal 30). Note that when a person other than the user who operated the image processing device 10 views an image output in a virtual space by the server 50, the person may access the server 50 from a user terminal 30 other than the image processing device 10 and view the image after a predetermined authentication process.

[0022] It is also possible to imagine a case in which the photographing device 20 has a function for automatically generating an image 5A of the printout 5 and a communication function over the network 1. In that case, if the photographing device 20 and the image processing device 10 are connected via a wired or wireless network 1, the photographing device 20 can output the printout 5, automatically generate the image 5A of the printout 5, and upload it to the image processing device 10 (without a scanning operation by the user).

[0023] As a form of connection between the photographing device 20 and the image processing device 10 via the network 1, a form in which the internal bus wiring of the image processing device 10 and the interface of the photographing device 20 are directly connected can also be envisaged.

[0024] Furthermore, various situations can be envisioned, such as a case where the photographer who performs various operations related to photographing with the photographing device 20 and outputting the printout 5, the registrant who performs the scanning operation on the printout 5 with the image processing device 10, and the viewer who views the output image in the virtual space generated by the server 50 are all different people or all the same person. For example, such a situation applies to a situation where, at an event venue where a talent is appearing, a photographer from the talent's agency photographs the talent using the photographing device 20, another photographer scans the printout 5 output from the photographing device 20, and spectators at the event access the server 50 with their user terminals 30 to view VR images, etc., based on the image 5A of the printout 5. The image processing system 3 of this embodiment can handle either situation.

[0025] Meanwhile, the user who will be the viewer operates the image processing device 10 (or the user terminal 30) and notifies the server 50 of an output request for the corresponding image 5A, using meta-information such as the user's own identification information or information on the registration date and time of the printout 5 as a key. The server 50 identifies the image 5A corresponding to the meta-information indicated by this output request and generates, for example, a VR image based on this. The server 50 responds via the network 1 to output this VR image in a virtual space for the user. Note that, as will be described in detail later, in this virtual space, it is also possible to control the transfer of the VR image corresponding to the image 5A between users and its printing in the real world. Information for such control is managed as management information (described later) by the image processing device 10 or the distributed ledger system 40.

[0026] Note that information relating to the association between the identification information and meta information of the printed matter 5 can be considered a type of sensitive information due to the nature of the printed matter 5. Therefore, it is more appropriate for the image processing device 10 to store this information in the distributed ledger of the distributed ledger system 40 to prevent unauthorized manipulation such as tampering. If the information is tampered with, it could lead to a situation where an unauthorized person could obtain the image 5A of the printed matter 5 and freely perform various actions such as copying, printing, or selling it to a third party. On the other hand, if the information is managed in a distributed ledger, tampering is difficult, and the occurrence of the above-mentioned situation can be appropriately avoided.

[0027] In order to perform the above-described functions, the image processing device 10 in this embodiment has functional units such as a storage unit 11, a control unit 12, an input unit 13, an output unit 14, a communication unit 15, and an image capturing unit 16 (see FIG. 4). The specific functions of these functional units will be described in order below.

[0028] [Configuration Example of Image Processing Device] Next, a configuration example of an image processing device 10 according to this embodiment will be described with reference to FIG. 3 . FIG. 3 is a diagram illustrating an example of the hardware configuration of an information processing device according to one embodiment of the present invention. The image processing device 10 is a terminal that scans a printout 5 output from a photographing device 20 and acquires an image 5A. Specifically, it is an information processing device operated by the owner of the printout 5. Such an image processing device 10 can be a mobile device such as a smartphone or tablet device carried by a user, or a fixed terminal for acquiring images located in various facilities such as train stations, commercial buildings, and event venues. Note that while the image processing device 10 and the server 50 are shown as separate devices in the example of FIG. 1 , a configuration in which the image processing device 10 and the server 50 are integrated is also possible. Therefore, the image processing device 10 may also include the image DB 511 and the image generation engine 521 that the server 50 includes.

[0029] As shown in FIG. 3, the information processing device 100 constituting the image processing device 10 includes an auxiliary storage device 101, a main storage device 103, a processor 104, an input device 105, an output device 106, a communication device 107, and an image capturing device 108.

[0030] Of these, the auxiliary storage device 101 is composed of, for example, a flash memory, an HDD (Hard Disc Drive), an SSD (Solid State Drive), an FD (Flexible Disc), an MO disk (Magneto-Optical disc), a CD (Compact Disc), a DVD (Digital Versatile Disc), an SD card (Secure Digital card), or a USB memory (Universal Serial Bus memory).

[0031] The auxiliary storage device 101 may be built into the computer main body constituting the image processing device 10, or may be attached to the computer main body in an external format. Alternatively, the auxiliary storage device 101 may be configured as a NAS (Network Attached Storage) or the like.

[0032] The auxiliary storage device 101 may also be an external device, such as a server 50, capable of communicating with one of the computers constituting the image processing device 10 via the network 1. The server 50 in this embodiment is a storage destination for the image 5A obtained from the printout 5 by the image processing device 10, and is a device that can function as a VR distribution server that receives access from the image processing device 10 and the user terminal 30 and responds with the target VR image in the virtual space. The main storage device 103 is composed of semiconductor memory, such as a ROM (Read Only Memory) and a RAM (Random Access Memory).

[0033] The processor 104 is composed of, for example, a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), an MCU (Micro Controller Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a TPU (Tensor Processing Unit), or an ASIC (Application Specific Integrated Circuit).

[0034] The input device 105 is a device that accepts input operations by a user of the image processing device 10, and is configured, for example, by a touch panel, a keyboard, a microphone, etc. The output device 106 is configured, for example, by a display, a speaker, etc. This output device 106 may function as a device that outputs a VR image for the user.

[0035] The communication device 107 may be configured, for example, by a network interface card or a communication interface board. The computer configuring the image processing device 10 can communicate with other devices connected to a network 1 such as the Internet or a mobile communication line via the communication device 107. The photographing device 108 is a device used to scan the printout 5, and is specifically assumed to be a digital camera.

[0036] Furthermore, the information processing device 100 constituting the image processing device 10 has installed therein, as software, a program 102 such as a program for an operating system (OS) and an application that implements the image processing method of the present invention. The processor 104 reads and executes the program 102, and the program 102 cooperates appropriately with other devices constituting the image processing system 3, thereby implementing the functions of the memory unit 11, control unit 12, input unit 13, output unit 14, communication unit 15, and image capture unit 16 ( FIG. 4 ). Some of the functions may be realized using artificial intelligence (AI). Details of each of the above-mentioned functional units are described below.

[0037] (Storage Unit) The storage unit 11 stores information used in or processed by each process in the image processing device 10 in the auxiliary storage device 101. The information stored here includes at least a device ID 110, image information 111, and management information 112. Of these, the device ID 110 is identification information for the image processing device 10, such as a serial number or MAC address.

[0038] 5, the image information 111 is a collection of records in which identification information and meta information of the prints 5 are associated with each other using a management ID as a key. The records included in this image information 111 include meta information for one or more prints 5 for one management ID. In other words, when a set of multiple prints 5 output by the photographing device 20 is scanned by the photographing unit 16 of the image processing device 10, the identification information of the multiple prints 5 is linked to one management ID.

[0039] The meta-information includes information such as the device, scan user, device owner, date and time, and hash value. Among these, "device" is the serial number printed on the photographing device 20, and is an example of a value acquired by scanning the photographing device 20 with the photographing unit 16. Furthermore, "scan user" is an identification number printed on an authentication medium (e.g., a specified membership card, employee ID card, driver's license, etc.) of the person who performed the scan operation on the printed material 5, and is an example of a value acquired by scanning the photographing device 20 with the photographing unit 16.

[0040] The "Device Owner" is an identification number or the like printed on an authentication medium (e.g., a specified membership card, employee ID card, driver's license, etc.) of the owner of the photographing device 20, and is, for example, a value acquired by scanning the photographing device 20 with the photographing unit 16. When the photographing device 20 is transferred and the device owner changes, the value in the "Device Owner" field is updated with a value acquired again by scanning the authentication medium presented by the new owner.

[0041] Furthermore, the "date and time" is the date and time when the scan operation was performed on the target printed matter 5, and is, for example, a date and time value obtained from a clock function or the like provided in the processor 104 of the image processing device 10. Furthermore, the "hash value" is a hash value obtained by applying a hash function to the image 5A obtained by scanning the printed matter 5. Therefore, the processor 104 of the image processing device 10 can call and use a hash function included in the program 102 or a hash function included in another device on the network 1.

[0042] The management information 112 is information for managing each print 5 that holds meta-information and the like in the image information 111. The management information 112 is a collection of records that associate information such as a management ID, print, device, scanner user, device owner, function control, and control status. The items from management ID to device owner are the same as the items in the record of the image information 111. Meanwhile, the "function control" field specifies the maximum number of prints and the drawing resolution limit in the virtual space that are controlled for the image 5A of the print 5. The "control status" field specifies the current status of the items specified in the "function control" field. This information, for example, includes the details of processing (e.g., the number of prints) performed by the device owner on the image 5A of the print 5 in the virtual space generated and managed by the server 50, and is updated by the image processing device 10 each time it is acquired from the server 50.

[0043] The image information 111 and management information 112 may be transmitted by the control unit 12 to the distributed ledger system 40 via the network 1 and stored in the block chain 421 (see FIG. 8 ) in the distributed ledger 42. By using the distributed ledger system 40 as a storage destination for various information, it is possible to effectively prevent the information from being tampered with by a malicious third party or the like.

[0044] (Control Unit) The control unit 12 is a functional unit implemented by the processor 104 executing the program 102, and mainly executes the image processing method of this embodiment. The control unit 12 starts up the application 121 in response to a user instruction, and, under the control of the application 121, executes the photographing of the print 5 by the photographing unit 16 and the uploading of the photographic image 6A of the photograph 6 included in the image 5A obtained by the photographing.

[0045] This upload targets the photographic image 6A and meta information, and is performed by the communication unit 15 to the server 50 via the network 1. Meanwhile, the server 50 associates the photographic image 6A and meta information uploaded from the image processing device 10 with the identification information of the printout 5 and stores them in the image DB 511. It also generates a VR image by providing this image 6A to a predetermined image generation engine 521 (see FIG. 10 ), and stores this in the image DB 511.

[0046] The application 121 photographs the print 5 using the photographing unit 16, extracts an image area from the image 6A that corresponds to the photograph 6, and performs appropriate processing such as rectangularization and glare removal on the image area to obtain the photographic image 6A (first image).

[0047] Furthermore, the control unit 12 receives a user instruction from the input unit 13 or a user instruction from the user terminal 30 via the communication unit 15, and instructs the server 50 to distribute a VR image (second image) in the virtual space. This user instruction includes meta information (stored in the image information 111) managed regarding the printout 5 corresponding to the above-mentioned photographic image 6A. The server 50 identifies the VR image in the image DB 511 using the meta information as a key, and distributes display data of the virtual space including this to the image processing device 10 or the user terminal 30. For example, the control unit 12 of the image processing device 10 displays the VR image in the virtual space distributed from the server 50 on the output unit 14 or in response to the user terminal 30.

[0048] The VR image is a three-dimensional image generated by the image generation engine 521, for example, by pasting a photographic image 6A onto one side of a three-dimensional object (a thin print object) of the image 5A of the print 5.

[0049] [Configuration Example of Photographing Device] As already mentioned, the photographing device 20 is an instant camera with a printing function. As shown in Fig. 7, the photographing device 20 includes a storage unit 21, a control unit 22, an input / output unit 23, a photographing unit 24, a communication unit 25, and a printing unit 26. The storage unit 21 may be implemented as a main storage device such as a memory, and may hold a device ID 211.

[0050] This device ID is notified to the image processing device 10 via the communication unit 25 and is set as a value in the "Device" column in the image information 111 or the management information 112 as identification information for the photographing device 20. Alternatively, the device ID is printed on the surface of the photographing device 20, scanned by the photographing unit 16 of the image processing device 10, and extracted from the scan data. Note that, in the example shown in Fig. 7, the photographing device 20 includes the communication unit 25 and the input / output unit 23, but these are not essential elements.

[0051] The control unit 22 also controls the photographing device 20, and executes the photographing unit 24 to take a photograph of a subject and the printing unit 26 to print the photograph in response to a user operation on the input / output unit 23.

[0052] [Configuration example of a distributed ledger system] An example configuration of a distributed ledger system 40 is shown in Figure 8. The distributed ledger system 40 in this embodiment is a system used to securely manage various sensitive information such as image information 111 indicating the association between a printed matter 5 and its meta information, and management information 112 for use in controlling usage of an image 5A of the printed matter 5, etc. This distributed ledger system 40 is a system made up of multiple nodes 41, one of which is the image processing device 10, the user terminal 30, or the server 50. Any of the image processing device 10, the user terminal 30, and the server 50 may constitute the distributed ledger system 40.

[0053] Below is an overview of the distributed ledger system 40. In the distributed ledger system 40, the validity of data to be processed (hereinafter referred to as a transaction) is verified by specific nodes called miners, and then a confirmation process is carried out by calculating a specific hash value called proof of work. The confirmed transactions are then compiled into a single block and recorded in a distributed ledger 42 called a blockchain 421.

[0054] The transactions handled by the distributed ledger system 40 of this embodiment are assumed to be, for example, information indicating the association between an image 6A of a printed matter 5 handled by the image processing device 10 or the user terminal 30 and its meta information, i.e., image information 111, and management information 112 for controlling the use of the image 6A, etc. Therefore, each time the image processing device 10 or the user terminal 30, which is a node, generates the information, a transaction is issued, and after appropriate consensus building between the nodes, the information is stored in the blockchain 421 of the distributed ledger 42.

[0055] The distributed ledger system 40 employed in this embodiment is assumed to be a so-called permission-based system (also called a consortium-based system). This is because participants in the distributed ledger system 40 in this embodiment are pre-registered users (individuals or businesses) who have completed the required user registration for an image processing service corresponding to the image processing method of the present invention (an image processing service that focuses on generating and registering an image 5A of a printout 5 printed by an image capture device 20, and controlling its use in virtual spaces, etc.). Of course, this assumption is merely an example, and other distributed ledger system configurations may be adopted as appropriate.

[0056] Furthermore, the ledger data held in the distributed ledger 42 may use various data structures, such as a relational database (RDB) or a key-value store (KVS), as appropriate. These data structures can be defined by users of the distributed ledger system 40.

[0057] In distributed ledger management using blockchain, it is usually necessary to trace the blockchain to obtain the (latest) state information (latest image information 111 and management information 112). However, this is inefficient, so there is a method of caching the latest state information separately from the blockchain (for example, "Hyperledger Fabric," [online], [searched March 31, 2017], Internet <URL: http: / / hyperledger-fabric.readthedocs.io / en / latest / >).

[0058] Therefore, in this embodiment, it is assumed that the latest state information 422 is managed on the distributed ledger 42. As a result, the image processing device 10 and the user terminal 30, which are nodes of the distributed ledger system 40, can obtain identification information, etc. of the relevant printed matter 5 using, for example, meta information, etc. related to the printed matter 5 as a key. By performing a search in the image DB 511 of the server 50 using the identification information, etc. obtained here as a key, it is possible to identify a VR image generated based on the image 5A of the printed matter 5, and to play it in a virtual space on the image processing device 10 or the user terminal 30.

[0059] The image processing device 10 and user terminal 30, which are nodes, update this state information each time a transaction is issued or stored in accordance with the image processing method of this embodiment. Note that the above-mentioned distributed ledger technology will not be repeated in the following explanation. Therefore, unless particularly necessary, specific explanations and illustrations of distributed ledger technology will be omitted as appropriate.

[0060] [Server Configuration Example] Next, the server 50 will be described with reference to Fig. 9. The server 50 is a so-called VR distribution server, and includes at least a storage unit 51, a control unit 52, and a communication unit 53. Note that an operation form may be adopted in which the server 50 also functions as the image processing device 10 of this embodiment, or the image processing device 10 also functions as the server 50.

[0061] Of the above, the storage unit 51 stores an image DB 511. As illustrated in Fig. 10 , the image DB 511 is a collection of records including scanned images (data of images 5A (including images 6A) related to printed matter 5) uploaded from the image processing device 10, their meta information, and VR images in a virtual space generated based on the images 5A.

[0062] The control unit 52 also includes an OS (Operating System) that controls various processes in the server 50, as well as an image generation engine 521. The image generation engine 521 is an engine that combines an image 5A or the like uploaded from the image processing device 10 with a predetermined template object to generate, for example, a three-dimensional VR image. An example of the template object is a rectangular plate-shaped object with a certain thickness. In this case, the image generation engine 521 attaches and arranges the image 5A or the like on one side of the plate-shaped object to generate a three-dimensional VR image as a whole.

[0063] Furthermore, the communication unit 53 accesses the network 1 and performs data communication with the image processing device 10 and the user terminal 30. The communication unit 53 receives, for example, an image 5A uploaded by the image processing device 10 via the network 1 and passes it to the control unit 52. The control unit 52 generates the above-mentioned VR image from the image 5A using the image generation engine 521 and stores it in the image DB 511.

[0064] [Example of Image Processing Method According to the Present Embodiment] Next, as an example of the operation of the image processing device 10 according to the present embodiment, an image processing flow using the same device will be described. The image processing flow described below uses the image processing method of the present invention. In other words, each step in the image processing flow described below corresponds to a component of the image processing method of the present invention. Note that the flow below is merely an example, and some steps in the flow may be deleted, new steps may be added, or the execution order of two steps in the flow may be reversed, as long as it does not deviate from the spirit of the present embodiment.

[0065] The steps in the image processing flow according to this embodiment are performed by the processor 104 included in the image processing device 10 or other devices in the order shown in Fig. 11. That is, in each process in the image processing flow, the processor 104 executes the processing corresponding to each step in Fig. 11 among the data processing defined in the image processing program.

[0066] (Acquiring an image of a printed matter and generating a VR image) To be more specific, in the image processing flow according to this embodiment, first, the control unit 12 of the image processing device 10 executes a reading operation of the printed matter 5 using the photographing unit 16 (S1). Note that the printed matter 5 is output by the photographing device 20 and is presented to the photographing unit 16 of the image processing device 10 by, for example, a user of the photographing device 20.

[0067] 2, and displays this image 5A on the UI 122 of the application 121 (S2). At this time, the user can edit the image quality, size, etc. on the UI 122 as needed, and finally click the OK button.

[0068] In response to the click, the control unit 12 of the image processing device 10 uses the management ID assigned to the image 5A as a key and stores information associating the identification information of the printout 5 with meta information in the image information 111 of the storage unit 11 (S3). Note that, among the meta information, the device ID is identification information such as the serial number of the photographing device 20, the scan user ID is identification information of the user who operated the photographing device 20, and the device owner ID is identification information of the owner of the photographing device 20. Furthermore, the date and time information is information on the date and time when the scan operation was performed, and the hash value is a hash value obtained by applying a hash function to the image 5A.

[0069] The control unit 12 of the image processing device 10 then uploads the data of the image 5A to the server 50 via the communication unit 15 (S4). During this upload, at least meta-information about the printout 5 is sent to the server 50 along with the data of the image 5A (scanned image). The server 50 then receives the data of the image 5A from the image processing device 10 and applies the data of the image 5A to the image generation engine 521 to generate a VR image 7A (S5). The server 50 then stores the VR image 7A and the meta-information as a set in the image DB 511 (S6).

[0070] The VR image 7A may be either a still image or a video, and may be generated in various forms, such as a single still image, multiple still images, or continuous playback data of multiple still images. For example, if the meta information, etc., of multiple still images 5A of multiple prints 5 is registered in the image information 111 as a single project, the image generation engine 521 generates the continuous playback data by editing the multiple still images 5A so that they are displayed sequentially at regular intervals. Alternatively, the image generation engine 521 may generate a single video by providing a video generation instruction including one or multiple still images 5A to a generation AI provided by an external device on the network 1, and acquire the video.

[0071] Suppose that the user then issues an instruction to the image processing device 10 or the user terminal 30 to view the VR image 7A. This instruction can take the form of, for example, receiving from the user meta-information (items other than the hash value) related to the desired image, as shown in screen 1000 of FIG. 12 . Alternatively, as shown in screen 1010 of FIG. 13 , when the user scans a printout 5 corresponding to the desired image, the image processing device 10 or user terminal 30 that performed the scan can automatically apply an image 5A of the printout 5 to a hash function to obtain a hash value, which serves as the instruction. Alternatively, when the image processing device 10 or user terminal 30 performs a predetermined operation (e.g., launching an application, launching the device itself, accepting the tap of a specific button, etc.) or accesses the server 50, it can read out identification information of the device or user (scan user or device owner) from its own storage unit, etc., and use the information as the instruction, even without an explicit instruction from the user. When the image processing device 10 or the user terminal 30 automatically acquires various meta-information as described above, the meta-information may be set in each field of the screen 1000 in Fig. 12 and presented to the user in advance. In this case, the instruction may be executed after accepting editing operations such as additions, corrections, and deletions from the user.

[0072] In response to the instruction, the image processing device 10 or the user terminal 30 notifies the server 50 of a request to output the VR image 7A in the virtual space (S7). This output request includes each value of the meta information and a hash value. Meanwhile, the server 50 identifies the VR image in the image DB 511 using each value of the meta information and the hash value included in the output request as a key, and responds with display data for the VR image 7A (S8).

[0073] The image processing device 10 or the user terminal 30 acquires the display data of the VR image 7A included in the response from the server 50 and, for example, renders the VR image 7A on the UI 122 of the application 121 (S9). An example of the rendering form of the VR image 7A is shown in screen 1020 of Fig. 14. In the example shown here, the VR image 7A is obtained by pasting an image 6A of the original photograph 6 onto one surface 7A2 of a print object 7A1 having a predetermined thickness.

[0074] (Detailed Control of Drawing, etc.) The control unit 12 of the image processing device 10 may also control the drawing function of the VR image 7A in the virtual space to a certain level or less. For example, if one user wants to view each of the images 6A related to multiple printouts 5 as a VR image 7A on a single occasion, the image processing device 10 or the user terminal 30 may preferably control the VR images 7A after a certain number (e.g., the second image and thereafter) to be displayed in the virtual space at a resolution lower than a specified value.

[0075] Such control is executed according to the attributes of the subject in image 6A of photograph 6 included in print 5, or the attributes of the user who acquired print 5. In this case, image processing device 10, for example, performs image recognition of the subject in image 6A of photograph 6 and identifies the person and the control content by comparing the feature amount of the subject with subject table T1 ( FIG. 15 ). Subject table T1 is a table that is held by image processing device 10 or held by any device on network 1 and available to image processing device 10.

[0076] The subject table T1 is a table that specifies the image features and control details for each subject person. The intention of specifying the control details for each subject person can be, for example, to prevent the avatar of a person who is famous in the virtual space (automatically determined based on the number of followers managed by a predetermined system) or the avatar of an individually specified person from being able to view the VR image 7A, which is valuable content, without limit.

[0077] As another example of the drawing control, when the owner of the print 5 plays a video or draws a series of images in a virtual space, it is possible to control the target of the playback or drawing to be all or part of the video or image group. In this case, the image processing device 10 also controls the drawing in accordance with the content shown in the "Control Content" column in the subject table T1, such as "Only the first half of the video, etc. can be drawn."

[0078] The holder of the print 5 may be a person of various positions. For example, at an event featuring an idol, the subject of the print 5 photographed and printed during the event may be the idol, but the holder of the print 5 may be the talent agency to which the idol belongs. In the case of an underground idol or freelance talent not managed by a talent agency, both the subject and the holder of the print 5 are the idol or talent. Furthermore, the owners of the photographing equipment 20, image processing device 10, and user terminal 30 may be persons of various positions, such as those who only have the right to use them, or those who have both the right to use them and ownership. Therefore, the scope of application of the image processing method of this embodiment includes the above-mentioned various types of persons and business types (individuals, corporations). (Control using blockchain)

[0079] Although the control unit 12 of the image processing device 10 is assumed to store the image information 111 and management information 112 in the blockchain 421 of the distributed ledger system 40 on the network 1, the image 5A of the printed matter 5 and the image 6A of the photograph 6 may also be stored in the blockchain 421.

[0080] Furthermore, the control unit 12 of the image processing device 10 may add information restricting the printing operation of the VR image 7A on a real printer to the management information 112 stored in the blockchain 421. In this case, the image processing device 10 stores the values ​​of each item of "function control" and "control status" indicated in the management information 112 (see FIG. 6) in the blockchain 421, linking them to the identification information or meta information of the corresponding print object 5. Securely managing information related to the control of such print operations accurately prevents a situation in which an image 6A of a valuable subject is printed without limit, thereby preventing damage to the rights and value of the subject, the print object 5, and the photograph 6.

[0081] Furthermore, the control unit 12 of the image processing device 10 may add information to the image information 111 or management information 112 in the blockchain 421 indicating that ownership of the VR image 7A has been transferred from the user who transferred it to the user who received it, in association with the transfer of the VR image 7A between users, i.e., between avatars, in a virtual space. Addition of such information may be performed, for example, by providing a "Device Owner" field in the management information 112 with a field for inputting the user ID of the transfer source and the user ID of the transfer destination (see the management information 112A in FIG. 16 ), and, when the owner is changed, setting the ID of the previous owner in the "Original" field and the ID of the new owner in the "Current" field. Alternatively, addition of such information may be performed by connecting the user ID of the transfer source and the user ID of the transfer destination with an arrow as the value of the "Device Owner" field in the management information 112.

[0082] In addition, the human user in the above description acts as an avatar in the virtual space. Therefore, it is the avatar that transfers the VR image 7A in the virtual space. Of course, the person who owns the transferred VR image 7A is the human user, and the various processes associated with the transfer are performed by the human user. Therefore, the concept of user in this invention can include both real people and avatars. Furthermore, ownership in this invention is defined as an exclusive right in the virtual space, and possession right is defined as an exclusive right in the real world.

[0083] <<Other Embodiments>> Specific embodiments of the present invention have been described above, but the above-described embodiments are merely examples given to facilitate understanding of the present invention and are not intended to limit the present invention. That is, the present invention may be modified or improved from the embodiments described below without departing from the spirit of the present invention. Furthermore, the present invention includes equivalents thereof. Furthermore, embodiments of the present invention may include a combination of the above-described embodiments with one or more of the following modifications.

[0084] (Regarding the Computer Constituting the Image Processing Device) In the above embodiment, the image processing device of the present invention is configured by a computer directly used by a user, such as a user-owned terminal (client terminal). However, this is not limited to this, and the image processing device of the present invention may also be configured by a computer indirectly available to a user, such as a server computer. Here, the server computer may be, for example, a server computer for a cloud service, specifically, a server computer for an ASP (Application Service Provider), SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service). In this case, when necessary information is input into the client terminal, the server computer performs various processes (calculations) based on the input information, and the calculation results are output on the client terminal. In other words, the functions of the server computer constituting the image processing device of the present invention can be used on the client terminal.

[0085] (Regarding Processor Configuration) The processor included in the image processing device 10 of the present invention includes various types of processors. The various types of processors include, for example, a CPU, which is a general-purpose processor that executes software (programs) and functions as various processing units. The various types of processors also include a PLD (Programmable Logic Device), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array). Furthermore, the various types of processors also include dedicated electrical circuits, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for performing specific processing.

[0086] Furthermore, one functional unit of the image processing device of the present invention may be configured by one of the various processors described above. Alternatively, one functional unit of the image processing device of the present invention may be configured by a combination of two or more processors of the same or different types, such as a combination of multiple FPGAs, or a combination of an FPGA and a CPU. Furthermore, multiple functional units of the image processing device of the present invention may be configured by one of the various processors, or two or more of the multiple functional units may be combined into a single processor. Furthermore, as in the above-described embodiment, one processor may be configured by a combination of one or more CPUs and software, and this processor may function as multiple functional units.

[0087] Furthermore, a processor may be used that realizes the functions of the entire system including the multiple functional units in the image processing device of the present invention on a single IC (Integrated Circuit) chip, as typified by, for example, an SoC (System on Chip), etc. Furthermore, the hardware configuration of the various processors described above may be an electric circuit (Circuitry) that combines circuit elements such as semiconductor elements.

[0088] 1 Network 3 Image processing system 5 Printed material 5A Printed material image 6 Photo (first image) 6A Photo image 7A VR image (second image) 10 Image processing device 11 Memory unit 110 Device ID 111 Image information 112 Management information 12 Control unit 121 Application 122 Application UI 13 Input unit 14 Output unit 15 Communication unit 16 Photography unit 20 Photography device 21 Memory unit 211 Device ID 22 Control unit 23 Input / output unit 24 Photography unit 25 Communication unit 26 Print unit 30 User terminal 40 Distributed ledger system 41 Node 42 Distributed ledger 421 Blockchain 422 State information 50 Server 51 Memory unit 511 Image DB 52 Control unit 521 Image generation engine 53 Communication unit 100 Information processing device 101 Auxiliary storage device 102 Program 103 Main storage device 104 Processor 105 Input device 106 Output device 107 Communication device 108 Imaging device

Claims

1. An image processing device having a processor, wherein the processor: acquires information relating to association that associates a first image with meta information relating to the results of reading a real printed matter; draws a second image in a virtual space based on the information relating to the association; and the first image is an image included in the results of the reading.

2. The image processing device according to claim 1, wherein the processor controls the rendering function of the second image in the virtual space to a certain level or less.

3. The image processing device according to claim 1, wherein the processor stores information relating to the association between the first image and the meta information in a blockchain on a network.

4. The image processing device according to claim 3, wherein the processor adds information restricting the printing operation of the second image in a real printer to the information about the association stored in the blockchain.

5. The image processing device described in claim 3, wherein the processor, in conjunction with the transfer of the second image between users in the virtual space, adds information to the information regarding the association in the blockchain indicating that ownership of the second image has been transferred from the source of transfer to the destination of transfer between the users.

6. The image processing device according to claim 1, wherein the processor, based on information regarding the association for one image from a group of moving or still images, plays the moving image corresponding to the association or continuously draws the group of images in a virtual space as the second image.

7. An image processing device as described in claim 6, wherein the processor controls the target of said playback or said drawing to be all or part of said video or said group of images when said video is played or said group of images is continuously drawn by the owner of said one image in said virtual space.

8. An image processing system including an image processing device, wherein a processor of the image processing device acquires information relating to the association that associates a first image with meta-information relating to the results of reading a real printed matter, and draws a second image in a virtual space based on the information relating to the association, and the first image is an image included in the results of the reading.

9. An image processing method comprising: a step of acquiring, by a processor, information relating to the association that associates a first image with meta information relating to the results of reading a real printed matter; and a step of drawing, by a processor, a second image in a virtual space based on the information relating to the association, wherein the first image is an image included in the results of the reading.

10. A program for causing a computer to execute each step included in the image processing method according to claim 9.

11. A computer-readable recording medium having recorded thereon a program for causing a computer to execute each step included in the image processing method according to claim 9.

Images