Program, information processing device, and information processing method

Abstract

To provide a program, an information processing device and an information processing method capable of eliminating unnecessary components from image data in view of the utilization scheme of such image data.SOLUTION: A program causes a computer to implement: an image obtaining function of obtaining image data obtained by imaging one or more objects in a real space; a utilization obtaining function of obtaining utilization information relating to a utilization scheme of the image data; an object identifying function of identifying at least either each of the one or more objects represented in the image data or each of one or more components constituting the object; an elimination target identifying function of identifying an elimination target to be eliminated from the image data among the one or more objects and the one or more components based on the utilization information; and a first processing function of performing, on the image data, a first process to eliminate the elimination target so as to generate first processed data.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a program, an information processing device, and an information processing method. [Background technology]

[0002] Conventionally, when image data of an object in real space is used to construct a virtual space, a technique is known in which unnecessary objects or components constituting each of the objects shown in the image data are removed from the image data (for example, Patent Document 1 below). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7076861 Summary of the Invention [Problem to be solved by the invention]

[0004] Even when multiple pieces of image data are taken of the same object, the unnecessary content may differ depending on the scene or situation in which they are used. However, with the above-mentioned conventional technology, it is difficult to remove the unnecessary content based on the usage of the image data.

[0005] Therefore, in order to solve the above problem, the present invention aims to provide a program, an information processing device, and an information processing method that are capable of removing unnecessary information from image data, taking into account how the image data will be used. [Means for solving the problem]

[0006] A program according to one embodiment of the present invention causes a computer to realize an image acquisition function that acquires image data of one or more objects in real space, a usage acquisition function that acquires usage information regarding the usage manner of the image data, an object identification function that identifies each of the one or more objects shown in the image data and at least one of each of one or more components that make up the objects, a removal target identification function that identifies removal targets to be removed from the image data among the one or more objects and the one or more components based on the usage information, and a first processing function that applies first processing to the image data to remove the removal targets and generate first processed data.

[0007] An information processing device according to one embodiment of the present invention includes an image acquisition unit that acquires image data of one or more objects in real space, a usage acquisition unit that acquires usage information regarding the usage manner of the image data, an object identification unit that identifies each of the one or more objects shown in the image data and at least one of each of one or more components that make up the objects, a removal target identification unit that identifies removal targets to be removed from the image data among the one or more objects and one or more components based on the usage information, and a first processing unit that applies first processing to the image data to remove the removal targets and generate first processed data.

[0008] An information processing method according to one embodiment of the present invention involves a computer acquiring image data of one or more objects in real space, acquiring usage information relating to the manner in which the image data is used, identifying each of the one or more objects shown in the image data and at least one of each of one or more components that make up the objects, identifying removal targets from the one or more objects and one or more components that are to be removed from the image data based on the usage information, and performing a first processing operation on the image data to remove the removal targets, thereby generating first processed data. [Effects of the Invention]

[0009] According to the present invention, unnecessary data can be removed from image data in consideration of how the image data will be used. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of a system configuration of a data management system according to an embodiment of the present invention; [Figure 2] 1 is a diagram for explaining an overview of a data management system according to an embodiment of the present invention; [Figure 3] 1 is a diagram for explaining an overview of a data management system according to an embodiment of the present invention; [Figure 4] 1 is a diagram for explaining an overview of a data management system according to an embodiment of the present invention; [Figure 5] 1 is a diagram for explaining an overview of a data management system according to an embodiment of the present invention; [Figure 6] 1 is a diagram for explaining an overview of a data management system according to an embodiment of the present invention; [Figure 7] 1 is a diagram for explaining an overview of a data management system according to an embodiment of the present invention; [Figure 8] FIG. 2 is a diagram illustrating an example of a functional configuration of a server device according to the present embodiment. [Figure 9] 10A and 10B are diagrams illustrating an example of removal condition information and permission information according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of the operation of the server device according to the embodiment. [Figure 11] FIG. 2 is a diagram illustrating an example of a hardware configuration of a server device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] A preferred embodiment of the present invention (hereinafter referred to as "the present embodiment") will be described with reference to the accompanying drawings. In the drawings, components with the same reference numerals have the same or similar configurations.

[0012] In this invention, the terms "unit," "means," "device," or "system" do not simply mean physical means, but also include cases where the functions of the "unit," "means," "device," or "system" are realized by software. Furthermore, the functions of one "unit," "means," "device," or "system" may be realized by a combination of two or more physical means, devices, or software modules, and the functions of two or more "units," "means," "device," or "system" may be realized by a single physical means, device, or software module.

[0013] <1. System configuration> An example of the system configuration of a data management system 1 according to this embodiment will be described with reference to FIG. 1. The data management system 1 is a system for managing data (hereinafter also referred to as "image data") including images of one or more objects (hereinafter also simply referred to as "objects") in real space, and providing the data to users. Users use the image data provided by the data management system 1, for example, to build a virtual space, or to generate map information for car navigation, weather forecasting, or electrical equipment inspection. The images included in the image data may be, for example, moving images or still images.

[0014] In the data management system 1, depending on the usage mode of the image data, the object and one or more components that make up the object (hereinafter simply referred to as "components") that are unnecessary for the user are removed from the image data. Here, the object may be any real object that exists in real space, and may be, for example, a building such as a store in real space, a person, an animal, a plant, a moving object (for example, a train, a car, and / or a drone), etc. Furthermore, the object may be, for example, an object that occupies a certain area in real space (for example, the sky and / or a specific area in a city, etc.).

[0015] The units of the objects and components may be set, for example, according to a user's request, or may be set commonly for multiple users. For example, the unit of the object may be set to "road" with each of "roadway" and "sidewalk" as components, or each of "roadway" and "sidewalk" may be set as the object at will.

[0016] The image data may include two-dimensional or three-dimensional images. The image data may also include information accompanying these images (hereinafter also referred to as "accompanying information"). The accompanying information may include, for example, shooting date and time information indicating the date and time when the image was taken (hereinafter also referred to as "shooting date and time") and / or shooting location information indicating the location where the image was taken (hereinafter also referred to as "shooting location"). In the case of image data, the accompanying information may also include information related to image quality. The information related to image quality may be, for example, the color or brightness of the image. The accompanying information may also include comment information indicating comments (including annotations) added to the image data. If the building is a store, the comment may be information for identifying the store.

[0017] As shown in FIG. 1, the data management system 1 includes a server device 100 that manages and provides image data, a user device 200 used by a user who uses the processed image data (hereinafter also referred to as "processed data") provided by the server device 100, and a licensee device 300 used by a licensee of a usage right (hereinafter also simply referred to as "usage right") indicating the right to use an object depicted in the image data. The server device 100, the user device 200, and the licensee device 300 are connected to each other via a network N. The user who uses the processed data and the licensee of the usage right (hereinafter also simply referred to as "licensor") are users of the data management system 1. Note that the user device 200 and the licensee device 300 are collectively referred to as "user device" unless there is a need to distinguish between them.

[0018] The usage right may be, for example, a right defined in intellectual property law (e.g., copyright law, trademark law, or design law) and / or a law such as a personal information protection law regarding the use of the object. The usage right may also be, for example, a privacy right.

[0019] The licensee may be, for example, a right holder of the usage rights, a person to whom authority has been delegated or transferred from the right holder, or a business operator of a right holder organization.

[0020] The network N is configured by a wireless network or a wired network. Examples of the network N include a mobile phone network, a PHS (Personal Handy-phone System) network, a wireless LAN (a local area network, including communication conforming to IEEE802.11 (so-called WI / Fi (registered trademark))), 3G (3rd Generation), LTE (Long Term Evolution), 4G (4th Generation), 5G (5th Generation), WiMax (registered trademark), infrared communication, visible light communication, Bluetooth (registered trademark), a wired LAN, a telephone line, a power line communication network, a network conforming to IEEE1394, etc.

[0021] [Server device] The server device 100 is an information processing device capable of communicating with user devices and an external system 500. The server device 100 is a device used, for example, by a person who operates the data management system 1 (hereinafter also simply referred to as "operator"). The server device 100 executes a predetermined program (hereinafter also referred to as "server program") to process image data to generate processed data, provide the processed data to users, and otherwise manage image data and processed data.

[0022] [User device] The user device is, for example, an information processing device such as a smartphone, laptop, or head-mounted display, and is capable of outputting processed data to the user, accepting various requests from the user regarding the processing of image data, and communicating with the server device 100.

[0023] For example, an application program (hereinafter referred to as a "data management app") dedicated to the data management system 1 may be installed in the user device as a predetermined program (hereinafter also referred to as a "user program"). The data management app may be, for example, a native app that provides each user with a user interface for using the data management system 1. The user device uses the data management app by installing it from an application store, the server device 100, or the like.

[0024] The user program may be, for example, a data management app, a web browser, or a third-party app implemented using a white label provided for the data management system 1. Also, for example, the data management system 1 may be configured to use both a VR app and a web browser as a user interface.

[0025] [External System] The external system 500 is a so-called third-party system. The external system 500 may be, for example, an online storage service that manages user image data, an SNS service in which user image data is registered, or a system that provides a homepage / blog creation service in which user image data is registered.

[0026] <2. Overview> An overview of a data management system 1 according to this embodiment will be described with reference to FIGS.

[0027] <2-1. Processing of image data> First, an example of processing that the data management system 1 performs on image data will be described with reference to Figures 2 and 3. In this example, the image data is a photographic image D of the sky, buildings such as stores and houses, motorcycles, and moving objects such as cars and buses. In this example, it is also assumed that a user uses the photographic image D to create a virtual space (hereinafter also referred to as "VR").

[0028] (1) As shown in FIG. 2, the server device 100 acquires a photographic image D as image data from a user device, an external system 500, or the storage unit 130.

[0029] (2) When the usage of the photographic image D is to be used as a material for constructing a VR image, the server device 100 specifies the removal condition corresponding to this usage (indicated as "for VR" in FIG. 2) for specifying the removal target. The removal target is the object to be removed from the image data.

[0030] (3) The server device 100 identifies at least one of the objects and components shown in the photographic image D. In this example, the objects are, for example, the sky, buildings such as stores and houses, motorcycles, and moving objects such as cars and buses. In addition, in this example, if the object is a store, the components are, for example, the store's sign, door, or exterior wall, which make up the store. Based on the information indicating the removal conditions identified in (2) above (hereinafter also referred to as "removal condition information"), the server device 100 identifies the objects to be removed from among the one or more objects and components identified. In this example, the removal conditions are set to conditions for removing a moving car, a moving motorcycle, and a blue sky, and these are identified as objects to be removed.

[0031] (4) The server device 100 performs processing (hereinafter also referred to as "first processing") to remove the removal target identified in (3) above from the photographic image D. Specifically, the server device 100 performs image processing on the photographic image D to trim it so as to remove the removal target. The photographic image D subjected to the first processing is generated as first processed data D1.

[0032] (5) As shown in Fig. 3, among the one or more objects shown in the first processed data D1 generated in (4) of Fig. 2, identify the objects to be protected with respect to the usage rights of each of the one or more objects (hereinafter also referred to as "protected objects"). In this example, the server device 100 identifies the nameplate of a house (written as "Japan" in Fig. 3), the signboard of a store (written as "ABC" in Fig. 3), and Tokyo Tower as the protected objects.

[0033] (6) The server device 100 determines whether the protected object identified in (5) above is licensed based on information indicating whether the right to use the real object in real space is licensed (hereinafter also referred to as "license information").

[0034] (7) The server device 100 processes the first processed data D1 (hereinafter also referred to as "second processing") so as not to use the protected object identified in (5) above and determined to be unauthorized in (6) above. Specifically, the server device 100 performs image processing on the first processed data D1 to mask the unauthorized protected object so that it becomes invisible. The server device 100 generates second processed data D2 by performing the second processing on the first processed data D1.

[0035] (8) The server device 100 provides the second processed data D2 generated in (7) above to the user device 200 (in other words, transmits it to the user device 200) in order to construct the VR.

[0036] With the above configuration, the data management system 1 can identify removal conditions corresponding to the manner in which the photographic image D is used and remove unnecessary data based on the identified removal conditions. Therefore, unnecessary data can be removed from the image data in consideration of the manner in which the image data is used.

[0037] With the above configuration, the data management system 1 can use the first processed data from which unnecessary data has been removed to determine whether the protected usage rights are permitted, perform second processing, and execute subsequent processing, thereby improving the efficiency of subsequent processing.

[0038] <2-2. Overall Processing Flow Executed by the Data Management System> Next, referring to FIGS. 4 to 7, an example of the processing flow executed by the data management system 1 will be described.

[0039] As shown in FIGS. 4 to 7, it is assumed that the data management system 1 is composed of three subsystems, for example, A) a data removal system, B) a permission determination system, and C) a content management system. Also, the functions realized by each of the three subsystems are classified as follows. In subsystem 1, cooperation (including information cooperation and / or function cooperation. The same applies hereinafter.) is performed between the classified functions. For example, the functions of the A) data removal system are described by assigning serial numbers such as A1), A2), etc. A) Data Removal System: A1) Image Decomposition Function, A2) Image Processing Function, A3) Removal Target Management Function, A4) User Interface (UI) Function B) Permission Determination System: B1) Permission Determination Function, B2) Masking Setting Function C) Content Management System: C1) License Management DB, C2) Content Management DB, C3) Content Registration Control Function

[0040] <2-2-1. When the Protected Object Without Usage Permission Includes Image Data> Referring to FIG. 4, in the data management system 1, an example of the processing from the acquisition of image data to the output of processed data obtained by processing this image data when the protected object without usage permission includes image data will be described.

[0041] <A4) User Interface (UI) Function> (1) As shown in FIG. 4, the user device 200 requests the server device 100 to register removal conditions specified by the user. The reception unit 116 of the server device 100 receives this registration request.

[0042] <A3) Removal Target Management Function> (2) The reception unit 116 of the server device 100 registers the removal conditions indicated in the registration request received in (1) above in the storage unit 130 as removal condition information.

[0043] <A1) Image decomposition function> (3) The image acquisition unit 111a of the server device 100 acquires image data from an external system 500 or a user device or the like. (4) The object identification unit 112a of the server device 100 identifies at least any one of each of the one or more objects and each of the one or more components constituting the object indicated in the acquired image data.

[0044] <A2) Image processing function> (5) The removal target identification unit 112b of the server device 100 identifies, based on the usage information, a removal target that is a target to be removed from the image data among the one or more objects and the one or more components identified in (4) above.

[0045] The usage information is information regarding the usage mode of the image data. The usage information may include, for example, at least any one of the usage purpose, usage period, and usage method regarding the usage of each of the one or more objects. Also, the usage information may include, for example, removal condition information indicating removal conditions for each usage purpose. The usage purpose may indicate, for example, whether it is for personal use, in other words, whether it is for commercial use. Also, when the usage purpose is commercial use, for example, as a more detailed purpose, it may include construction of a virtual space, or generation of map information for car navigation, weather prediction, or electrical equipment inspection.

[0046] (6) The first processing unit 113a of the server device 100 performs first processing on the image data to remove the removal target identified in (5) above, and generates first processed data. (7) As the first processing, the first processing unit 113a may execute a process of correcting the image data by synthesizing (so-called mask synthesis) an image of another image data or coloring the removed portion in addition to the process of removing the removal target from the image data.

[0047] <B1) Permission determination function> (8) The protection target specifying unit 112c of the server device 100 specifies one or more protection targets that are targets to be protected regarding the right of use among one or more objects shown in the first processed data.

[0048] (9) The permission acquisition unit 111c of the server device 100 acquires permission information from the storage unit 130. The determination unit 114 of the server device 100 determines whether there is a permission for the right of use for each of the one or more protection targets specified in (8) above based on the acquired permission information. In this example, it is assumed that the protection targets determined to have no permission are included in the first processed data.

[0049] <B2) Masking setting function> (10) The second processing unit 113b of the server device 100 performs second processing on the first processed data so as not to use each of the one or more protection targets determined to have no permission in (8) above, and generates second processed data. Specifically, the second processing unit 113b masks each of the one or more protection targets determined to have no permission.

[0050] <B3) Content management DB function> (11) The second processing unit 113b of the server device 100 registers the second processed data generated in (10) above in the storage unit 130 (in other words, stores it in the storage unit 130). (12) The output unit 117 of the server device 100 causes the processed image indicated by the second processed data to be output to the screen of the data management application or the web browser provided in the user device 200 in response to a request from the user.

[0051] <2-2-2. When the image data does not include protection targets without permission for the right of use> Referring to FIG. 5, in the data management system 1, an example of the process from the acquisition of image data to the output of the processed data obtained by processing this image data when the image data does not include protection targets without permission for the right of use will be described. Note that (1) to (8) in this example are the same as (1) to (8) in the example of FIG. 4 above, so the description will be omitted.

[0052] <B1) Permission determination function> (9) As shown in FIG. 5, the permission acquisition unit 111c of the server device 100 acquires permission information from the storage unit 130. The determination unit 114 of the server device 100 determines whether there is a permission for the use right for each of the one or more protected objects specified in (8) above based on the acquired permission information. In this example, it is assumed that the protected objects for which no permission is determined are not included in the first processed data.

[0053] <B3) Content management DB function><00> (10) When one or more protected objects for which no permission is determined in (8) above are not included in the first processed data, the permission acquisition unit 111c of the server device 100 registers the first processed data in the storage unit 130 as it is without processing. (11) The output unit 117 of the server device 100 causes the processed image indicated by the first processed data to be output to the screen of the data management application or the web browser provided in the user device 200 in response to a request from the user.

[0054] <2-2-3. Case of obtaining permission from the licensor for protected objects without permission for the right of use> Referring to FIG. 6, an example of the process in the data management system 1 when obtaining permission from the licensor for protected objects without permission for the right of use will be described.

[0055] <C3) Content registration control function> (1) As shown in FIG. 6, the request unitl 115 of the server device 100 makes a permission request to the licensor to obtain permission for the protected object for which no permission is determined. Specifically, the request unit 115 transmits this permission request to the licensor device 300 of the licensor. (2) The licensor device 300 responds to the server device 100 with a response indicating whether to grant permission for this permission request received from the licensor. The reception unit 116 of the server devicel 100 receives this response.

[0056] <C4) License management DB function> (3) Based on the response received in (2) above, the update unit 116a of the server device 100 updates the license information so that the protected object targeted by the license request changes from having no license to having a license.

[0057] <B1) License determination function> (4) Based on the updated license information, the determination unit 114 of the server device 100 re-determines whether there is a license for the protected object targeted by the license request. In this example, the result of this re-determination indicates that there is a license for this protected object.

[0058] <B2) Masking setting function> (5) When the above response indicates that a license is granted, that is, when it is determined in the re-determination in (4) above that there is a license for the protected object, the second processing unit 113b of the server device 100 processes the second processing data registered in the storage unit 130 so as to return the protected object targeted by the license request to the state before the second processing. Specifically, the second processing unit 113b may remove the masking applied to this protected object and return the second processing data to the first processing data.

[0059] <C2) Content management DB function> (6) The second processing unit 113b of the server device 100 updates and registers in the storage unit 130 so as to update the second processing data registered in the storage unit 130 for the protected object targeted by the license request with the data (for example, the first processing data) that has been returned to the state before the second processing. (7) The output unit 117 of the server device device 100 outputs the processed image indicated by the updated and registered processed data to the screen of the data management application or the web browser provided in the user device 200 in response to a request from the user.

[0060] <2-2-4. When the license for a protected object with a license for the right of use is revoked> Referring to FIG. 7, an example of the processing in the data management system 1 when the license for a protected object with a license for the right of use is revoked by the licensor will be described.

[0061] <C3) Content Registration Control Function> (1) As shown in FIG. 7, the licensor makes a cancellation request to the server device 100 from the licensor device 300 to cancel the license for the protected object for which the license has been granted. The reception unit 116 of the server device 100 receives this cancellation request.

[0062] <C4) License Management DB Function> (2) The update unit 116a of the server device 100 updates the license information so that the protected object targeted by the cancellation request changes from having a license to not having a license based on the cancellation request received in (2) above.

[0063] <B1) License Judgment Function> (3) The determination unit 114 of the server device 100 re - determines whether or not to grant the usage right for the protected object targeted by the license request based on the updated license information. In this example, the result of this re - determination indicates that there is no license for this protected object.

[0064] <B2) Masking Setting Function> (4) When the second processing unit 113b of the server device 100 receives a request to cancel the license, that is, when it is determined in the re - determination in (3) above that there is no license for the protected object, the second processing unit 113b performs second processing on the first processed data registered in the storage unit 130 so as not to use the protected object targeted by the cancellation request. Specifically, the second processing unit 113b masks this protected object.

[0065] [[ID=2३]] <C2) Content Management DB Function> (5) The second processing unit 113b of the server device 100 updates and registers in the storage unit 130 the first processed data that has been second - processed for the protected object targeted by the cancellation request as the second processed data, and updates the registered first processed data with this second processed data. (6) The output unit 117 of the server device 100 outputs the processed image indicated by the updated and registered processed data to the screen of the data management application or the web browser provided in the user device 200 in response to a request from the user.

[0066] <3. Functional configuration> Next, the functional configuration of the server device 100 according to this embodiment will be described with reference to Fig. 8. As shown in Fig. 8, the server device 100 includes a control unit 110, a communication unit 120, and a storage unit 130.

[0067] The control unit 110 includes an acquisition unit 111, an identification unit 112, and a processing unit 113. The control unit 110 may also include, for example, a determination unit 114, a request unit 115, a reception unit 116, and / or an output unit 117.

[0068] [Acquisition Department] The acquisition unit 111 acquires various requests and / or various information from the user device or the external system 500 .

[0069] The acquisition unit 111 includes an image acquisition unit 111a. The image acquisition unit 111a acquires image data of one or more objects in real space from the storage unit 130, the user device, or the external system 500.

[0070] The acquisition unit 111 includes a usage acquisition unit 111b. The usage acquisition unit 111b acquires usage information relating to the usage mode of the image data from the storage unit 130, the user device, or the external system 500.

[0071] The acquisition unit 111 includes a license acquisition unit 111c. The license acquisition unit 111c acquires, from the storage unit 130, the user device, or the external system 500, license information indicating a license from a licensee for a usage right indicating a right related to the use of an actual object in real space.

[0072] [Specific part] The identification unit 112 identifies objects, components, objects to be removed, and / or objects to be protected (hereinafter collectively referred to as "identified objects") from among the objects and / or components shown in the image data or processed data.

[0073] The identification unit 112 may identify the specified object using, for example, image recognition technology. The identification unit 112 may extract features from image data using, for example, pattern recognition technology, and compare the extracted features with a class dictionary of classes (e.g., corresponding to each object or each component) pre-registered in the storage unit 130 to identify the class to which the specified object belongs. The features may be, for example, shape features and / or texture features included in the image. Specifically, the identification unit 112 may calculate the distance between the extracted features and each class and classify the specified object into the class for which the calculated distance is the smallest. Alternatively, the identification unit 112 may extract features from a group of sample images as training data and train a discrimination model that inputs the extracted features and outputs class probabilities using machine learning technology. The identification unit 112 may then extract features from the image data, calculate the class probability of each class using the trained discrimination model, and classify the specified object into the class with the largest class probability. The machine learning technique may be, for example, a support vector machine or a random forest as supervised learning, or a neural network as deep learning. A sample image data group indicating a sample image group to be used as training data may be registered in the storage unit 130. The training data may include, for example, tag information indicating an object or component, or a category thereof.

[0074] For example, when identifying at least one of one or more objects and one or more components constituting the objects, the identification unit 112 may perform region segmentation processing to divide regions constituting the objects or components shown in the image. The identification unit 112 can extract the objects and components from the image data image by the region segmentation processing.

[0075] The identification unit 112 includes an object identification unit 112a. The object identification unit 112a identifies at least one of each of one or more objects shown in the image data and one or more components that make up the object.

[0076] The identification unit 112 includes a removal target identification unit 112b. The removal target identification unit 112b identifies removal targets to be removed from the image data from among one or more objects and one or more components based on the usage information.

[0077] The removal target specifying unit 112b may specify the removal target based on, for example, the removal condition indicated by the removal condition information.

[0078] Here, an example of the data configuration of the removal condition information will be described with reference to FIG. 9(a). As shown in FIG. 9(a), the removal condition information may indicate whether or not each object is to be removed for each purpose of use. When the purpose of use is "car navigation," the removal condition information may indicate that objects such as "sky" and "power lines" are to be removed, while "roads" and "electric poles" are not to be removed. Furthermore, when the purpose of use is "weather forecasting," the removal condition information may indicate that objects such as "sky" are not to be removed, while "roads," "electric poles," and "power lines" are to be removed.

[0079] According to the above configuration, it is possible to remove from the image data the object to be removed that is specified based on the removal conditions set for each purpose of use.

[0080] Returning to FIG. 8, the explanation continues. The identification unit 112 includes a protection target identification unit 112c. The protection target identification unit 112c identifies one or more protection targets that should be protected with respect to the usage right, among one or more objects indicated in the first processed data. The protection target identification unit 112c may identify one or more protection targets based on, for example, license information. The license information may include, for example, at least one of a permitted purpose of use, a permitted usage period, and a prohibited usage mode regarding the use of each of the one or more objects in the usage right.

[0081] Here, an example of the data configuration of the license information will be described with reference to FIG. 9(b). As shown in FIG. 9(b), the license information may include, for each protected object, the category of the object (hereinafter simply referred to as "category"), whether or not permission is granted for each purpose of use (in this example, use in the metaverse and commercial use), the scope of permitted use, the permitted period of use (referred to as "license period" in FIG. 9(b)), prohibited usage patterns (referred to as "prohibited matters" in FIG. 9(b)), and / or remarks (e.g., exceptions, etc.). Furthermore, the unit of the protected object may be an individual object (e.g., Tokyo Tower, which belongs to the category of architectural works) or all objects in the same category (e.g., all objects in the category of signs). With this configuration, categories such as signs that are likely to contain elements protected under the Trademark Act or the Unfair Competition Prevention Act, such as trademarks or logos, can be uniformly identified as protected objects.

[0082] According to the above configuration, it is possible to determine whether or not permission has been granted based on the permitted purpose of use, the permitted period of use, and the prohibited mode of use, thereby achieving more accurate and precise determination.

[0083] [Processing department] Returning to FIG. 8, the explanation will continue. The processing unit 113 processes the image data or the processed data. The processing unit 113 may store part or all of the data before processing separately in the storage unit 130 as a backup, or may overwrite and update it with the processed data, or may delete it after a certain period of time has passed. If the processing unit 113 does not store part or all of the data before processing, it may delete the image itself before processing (for example, an image showing Tokyo Tower) and leave only the accompanying information (for example, information that can identify Tokyo Tower (such as a place name or location coordinates)).

[0084] The processing unit 113 includes a first processing unit 113a. The first processing unit 113a performs first processing on the image data to remove the removal target, and generates first processed data. For example, if the removal target is a region such as the background (e.g., the sky), the first processing unit 113a may perform chromakey processing to separate this region.

[0085] According to the above configuration, the first processing unit 113a can remove unnecessary data from the image data, which is specified by the removal condition corresponding to the usage of the image data. Therefore, it is possible to remove unnecessary data from the image data, taking into account the usage of the image data.

[0086] The processing unit 113 includes a second processing unit 113b. The second processing unit 113b performs second processing on the first processed data so as not to use each of the one or more protected objects determined to be unauthorized, thereby generating second processed data. Specifically, the second processing unit 113b masks each of the one or more protected objects determined to be unauthorized. The masking process performed by the second processing unit 113b may be, for example, a mosaic process, a fill process, a trimming process, a process of assimilation with surrounding objects (e.g., assimilation of Tokyo Tower with surrounding trees), or a process of replacing another image by mask synthesis (e.g., replacement of Tokyo Tower with a steel tower shown in another image).

[0087] According to the above configuration, the second processing unit 113b can process data so as not to use unauthorized protected objects. This eliminates the need for the user to obtain permission for use from the licensee, and makes it possible to provide processed data that is easy for the user to use.

[0088] For example, if the response from the license holder indicates that permission will be granted, the second processing unit 113b may process the second processing data to return the protected object that is the subject of the permission request to the state before the second processing.

[0089] [Judgment section] The determination unit 114 determines whether or not a usage right has been granted for each of one or more protected objects based on the license information. The determination unit 114 may also re-determine whether or not a usage right has been granted for each of one or more protected objects based on the license information updated by the update unit 116a. With this configuration, it is possible to re-determine whether or not a usage right has been granted based on license information updated without a license in response to a license cancellation request from the license holder. This makes it possible to update the license status at the license holder's discretion. This makes it possible to realize a data management system that is easy for the license holder to use.

[0090] [Request part] For a protected object that has been determined by the determining unit 114 to be unlicensed, the requesting unit 115 makes a license request to the licensee to obtain a license for the right to use this protected object.

[0091] [Reception] The receiving unit 116 receives various requests from the user or responses to various requests. For example, the receiving unit 116 receives a response from a license holder indicating whether or not to grant permission in response to a permission request from the requesting unit 115.

[0092] The receiving unit 116 may receive, for example, a cancellation request to cancel the license for a protected subject for which a licensee has granted a usage right.

[0093] The reception unit 116 may include, for example, an update unit 116a. The update unit 116a updates the permission information registered in the storage unit 130 based on the cancellation request received by the reception unit 116, for example.

[0094] [Output section] The output unit 117 causes the user device to output various information. For example, the output unit 117 may generate output information for outputting a processed image included in the processing data based on the processing data, and transmit the output information to the user device.

[0095] [Communications Department] The communication unit 120 transmits and receives various types of information to and from the user device and / or the external system 500 via the network N.

[0096] [Storage] The storage unit 130 stores various types of information, such as image data, processed data, usage information, and / or permission information, which are processed by the data management system 1. For example, the storage unit 130 may store these pieces of information in association with one another. The storage unit 130 may store various types of information using a database management system (DBMS), or may store various types of information using a file system. When a DBMS is used, a table may be provided for each piece of information, and the various pieces of information may be managed by associating these tables with one another.

[0097] <4. Example of operation> Next, an example of the operation of the server device 100 will be described with reference to Fig. 10. Note that the order of the processes shown below is an example and may be changed as appropriate.

[0098] 10, the image acquisition unit 111a of the server device 100 acquires image data (S10). The usage acquisition unit 111b acquires usage information of the acquired image data (S11).

[0099] The object identification unit 112a identifies at least one of each of one or more objects shown in the image data and one or more components constituting the objects (S12). The removal target identification unit 112b identifies removal targets to be removed from the image data from among the one or more objects and one or more components based on the usage information (S13).

[0100] The first processing unit 113a performs a first processing on the image data to remove the specified removal target, and generates first processed data (S14).

[0101] The license acquisition unit 111c acquires license information (S15). The protection target identification unit 112c identifies one or more protection targets that should be protected with respect to usage rights, from among one or more objects indicated in the first processed data (S16). The determination unit 114 determines whether or not each of the one or more protection targets has permission, based on the license information (S17).

[0102] If there is one or more protected objects that are determined to be unlicensed (Yes in S18), the second processing unit 113b performs second processing on the first processed data so as not to use each of these one or more protected objects, and generates second processed data (S19).

[0103] The processing unit 113 registers the processed data in the storage unit 130 (S20).

[0104] <5. Hardware Configuration> Finally, an example of a hardware configuration in which the above-described server device 100 is realized by a computer 800 will be described with reference to Fig. 11. Note that the functions of each device can also be realized by dividing them into multiple devices.

[0105] As shown in FIG. 11, the computer 800 includes a processor 801, a memory 803, a storage device 805, an input I / F unit 807, a data I / F unit 809, a communication I / F unit 811, and a display device 813.

[0106] The processor 801 controls various processes in the computer 800 by executing programs stored in the memory 803. For example, each functional unit included in the control unit 110 of the server device 100 can be realized by the processor 801 executing a program temporarily stored in the memory 803.

[0107] The memory 803 is a storage medium such as a RAM (Random Access Memory), etc. The memory 803 temporarily stores the program code of the program executed by the processor 801 and data required when the program is executed.

[0108] The storage device 805 is a non-volatile storage medium such as a hard disk drive (HDD) or flash memory. The storage device 805 stores an operating system and various programs for implementing the above-mentioned components. In addition, the storage device 805 can also store tables for registering various types of information such as image data, processed data, usage information, license information, or output information, and a DB for managing these tables. Such programs and data are loaded into the memory 803 as needed and can be referenced by the processor 801.

[0109] The input I / F unit 807 is a device for receiving input from a user. Specific examples of the input I / F unit 807 include a keyboard, a mouse, a touch panel, various sensors, and a wearable device. The input I / F unit 807 may be connected to the computer 800 via an interface such as a USB (Universal Serial Bus).

[0110] The data I / F unit 809 is a device for inputting data from outside the computer 800. A specific example of the data I / F unit 809 is a drive device for reading data stored in various storage media. The data I / F unit 809 may be provided outside the computer 800. In this case, the data I / F unit 809 is connected to the computer 800 via an interface such as a USB.

[0111] The communication I / F unit 811 is a device for performing data communication via a network N, either wired or wirelessly, with an external device of the computer 800. The communication I / F unit 811 may be provided outside the computer 800. In this case, the communication I / F unit 811 is connected to the computer 800 via an interface such as a USB.

[0112] The display device 813 is a device for displaying various types of information. Specific examples of the display device 813 include a liquid crystal display, an organic EL (Electro-Luminescence) display, and a display of a wearable device. The display device 813 may be provided outside the computer 800. In this case, the display device 813 is connected to the computer 800 via, for example, a display cable. Furthermore, when a touch panel is adopted as the input I / F unit 807, the display device 813 can be configured as an integral part of the input I / F unit 807.

[0113] The above-described embodiment is merely an example for explaining the present invention, and is not intended to limit the present invention to the embodiment. Furthermore, the present invention can be modified in various ways without departing from the spirit of the present invention. Furthermore, a person skilled in the art can adopt an embodiment in which each element described above is replaced with an equivalent, and such an embodiment is also within the scope of the present invention.

[0114] The components of the server device described in the above embodiment are assumed to realize predetermined processing in cooperation with other hardware by the processor 801 executing a program stored in the storage device 805. In other words, these components are assumed to be software or firmware, as well as corresponding hardware, and in both concepts, they are also referred to as "functions," "means," "parts," "processing circuits," "units," or "modules," and can be interpreted as such.

[0115] [Variations] Although the present invention has been described based on the above embodiment, the following cases are also included in the present invention.

[0116] [Variation 1] In the above embodiment, a client-server system has been described as an example, but the data management system according to the present invention is not limited to this. The data management system according to the present invention may also adopt, for example, a P2P system (star or full-mesh system) implemented only by edge-side user devices. In this case, all or part of the components of the server device 100 may be provided in the user device. The user device may, for example, implement the functions of the functional units provided in the control unit 110 of the server device 100 by executing an installed data management app.

[0117] [Variation 2] In the above embodiment, the storage unit for storing the image data, processed data, usage information, and / or permission information is the storage unit of the server device 100, but the present invention is not limited to this. For example, at least a part of these storage units may be provided in a device of the external system 500. [Explanation of symbols]

[0118] 1...data management system, 100...server device, 110...control unit, 111...acquisition unit, 112...identification unit, 113...processing unit, 114...determination unit, 115...request unit, 116...reception unit, 117...output unit, 120...communication unit, 130...storage unit, 200...user device, 300...licensor device, 500...external system, 800...computer, 801...processor, 803...memory, 805...storage device, 807...input I / F unit, 809...data I / F unit, 811...communication I / F unit, 813...display device.

Claims

1. On the computer, an image acquisition function for acquiring image data of one or more objects including a road consisting of a roadway and a sidewalk in real space; a usage acquisition function for acquiring usage information regarding the usage manner of the image data; an object identification function that identifies at least one of each of one or more objects shown in the image data and one or more components that make up the object, the object identification function identifying each of the roadways and sidewalks that make up one or more roads shown in the image data; a removal target specifying function that specifies a removal target that is a target to be removed from the image data, among the one or more objects and the one or more components, based on removal conditions that are set for each of the usage information; a first processing function of performing a first processing on the image data to remove the removal target and generating first processed data; program.

2. On the computer, an image acquisition function for acquiring image data of one or more objects in real space; a usage acquisition function for acquiring usage information regarding the usage manner of the image data; an object identification function that identifies at least one of each of one or more objects shown in the image data and each of one or more components that make up the object; a removal target specifying function that specifies a removal target to be removed from the image data among the one or more objects and the one or more components based on a removal condition corresponding to the usage information; a first processing function that performs a first processing on the image data to remove the removal target and generates first processed data; a license acquisition function for acquiring license information indicating a license from a licensee for a usage right indicating a right to use an actual object in real space; a protection object specifying function for specifying one or more protection objects that should be protected with respect to the usage right, among the one or more objects indicated in the first processed data; a determination function for determining whether or not the permission exists for each of the one or more protection subjects based on the permission information; a second processing function that performs second processing on the first processed data so as not to use each of the one or more protected subjects for which it is determined that there is no permission, thereby generating second processed data; a receiving function for receiving a cancellation request for canceling the license for the protected subject for which the licensee has granted the right of use; an update function for updating the permission information based on the cancellation request; the determination function determines whether or not the protection target has the permission based on the updated permission information. program.

3. The computer, a license acquisition function for acquiring license information indicating a license from a licensee for a usage right indicating a right to use an actual object in real space; a protection object specifying function for specifying one or more protection objects that should be protected with respect to the usage right, among the one or more objects indicated in the first processed data; a determination function for determining whether or not the permission exists for each of the one or more protection subjects based on the permission information; a second processing function of performing second processing on the first processed data so as not to use each of the one or more protected subjects for which it is determined that there is no permission, thereby generating second processed data; The program according to claim 1.

4. the license information includes at least one of a permitted purpose of use, a permitted period of use, and a prohibited mode of use regarding the use of each of the one or more objects under the usage right; The program according to claim 3.

5. a request function for making a permission request to the licensor to obtain permission for the right of use for the protected subject determined to be unlicensed; a receiving function for receiving a response from the licensee indicating whether or not to grant the license to the license request, When the response indicates that the permission is granted, the second processing function processes the second processed data so as to restore the protected object that is the subject of the permission request to a state before the second processing. The program according to claim 3.

6. an image acquisition unit that acquires image data of one or more objects in real space, including a road consisting of a roadway and a sidewalk; a usage acquisition unit that acquires usage information regarding a usage mode of the image data; an object specifying unit that specifies at least one of each of one or more objects shown in the image data and one or more components that make up the object, the object specifying unit specifying each of the roadways and sidewalks that make up one or more roads shown in the image data; a removal target specifying unit that specifies a removal target to be removed from the image data, among the one or more objects and the one or more components, based on a removal condition that is set for each of the usage information; a first processing unit that performs first processing on the image data to remove the removal target and generates first processed data; Information processing device.

7. The computer Acquire image data of one or more objects in real space, including a road consisting of a roadway and a sidewalk; Acquire usage information regarding the manner in which the image data is used; Identifying each of one or more objects shown in the image data, and at least one of one or more components constituting the objects, and each of the roadways and sidewalks constituting one or more roads shown in the image data; Identifying a removal target based on a removal condition for identifying a removal target that is a target to be removed from the image data, from among the one or more objects and the one or more components set for each of the usage information; performing a first processing on the image data to remove the object to be removed, thereby generating first processed data; Information processing methods.

8. an image acquisition unit that acquires image data of one or more objects in real space; a usage acquisition unit that acquires usage information regarding a usage mode of the image data; an object identification unit that identifies at least one of each of one or more objects shown in the image data and each of one or more components that make up the object; a removal target specifying unit that specifies a removal target to be removed from the image data from among the one or more objects and the one or more components based on a removal condition corresponding to the usage information; a first processing unit that performs first processing on the image data to remove the removal target and generates first processed data; a license acquisition unit that acquires license information indicating a license from a licensee for a usage right indicating a right related to the usage of an actual object in real space; a protection target specifying unit that specifies one or more protection targets that should be protected with respect to the usage right, among the one or more targets indicated in the first processed data; a determination unit that determines whether or not the permission exists for each of the one or more protection targets based on the permission information; a second processing unit that performs second processing on the first processed data so as not to use each of the one or more protected objects for which it is determined that there is no permission, thereby generating second processed data; a receiving unit that receives a cancellation request to cancel the license for the protected subject for which the licensee has granted the right of use; an update unit that updates the permission information based on the cancellation request, the determination unit determines whether or not the protection target has the permission based on the updated permission information. Information processing device.

9. The computer Acquire image data of one or more objects in real space; Acquire usage information regarding the manner in which the image data is used; Identifying at least one of each of one or more objects shown in the image data and each of one or more components constituting the object; Identifying a removal target to be removed from the image data from among the one or more objects and the one or more components based on a removal condition corresponding to the usage information; performing a first process on the image data to remove the object to be removed, thereby generating first processed data; Acquire license information indicating a license from a licensee for a usage right indicating a right to use an actual object in real space; Identifying one or more protection objects that should be protected with respect to the usage right from among the one or more objects indicated in the first processed data; determining whether or not the license is present for each of the one or more protected objects based on the license information; generating second processed data by performing second processing on the first processed data so as not to use each of the one or more protected subjects for which it is determined that there is no permission; receiving a cancellation request to cancel the license for the protected subject matter for which the licensee has granted the right of use; updating the permission information based on the cancellation request, and determining whether or not the permission is granted for the protection object based on the updated permission information; Information processing methods.

Images