Mobile terminal, program, method, and system

Abstract

To improve handling of an image captured by a portable terminal.SOLUTION: A portable terminal, which has imaging functions, includes a control unit that generates an image subjected to mask processing on an area in which at least a part of a photographer's body is captured in a captured image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a mobile terminal, a program, a method, and a system. [Background technology]

[0002] It is conventionally known to generate a 3D model representing a space photographed by a mobile terminal using an image captured by the mobile terminal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-36812 A Summary of the Invention [Problem to be solved by the invention]

[0004] When a photographer takes an image with a mobile device, the photographer may end up appearing in the image captured by the mobile device. In such a case, when generating a 3D model that represents the space captured by the mobile device using the image captured by the mobile device, modeling processing must be performed even in unnecessary areas (i.e., the area where the photographer is captured).

[0005] The present disclosure aims to improve the handling of images captured by mobile terminals. [Means for solving the problem]

[0006] A mobile terminal according to a first aspect of the present disclosure includes: A mobile terminal equipped with a photographing function, The control unit of the mobile terminal An image is generated by performing a mask process on an area in the captured image in which at least a part of the photographer's body is captured.

[0007] A second aspect of the present disclosure is a mobile terminal according to the first aspect, The control unit is Based on the orientation of the mobile device, a region within the image where masking is to be performed is identified.

[0008] A third aspect of the present disclosure is the mobile terminal according to the second aspect, The orientation of the mobile terminal is The position is identified based on information from an acceleration sensor of the mobile terminal.

[0009] A fourth aspect of the present disclosure is a mobile terminal according to any one of the first to third aspects, The control unit is An area in the image to be masked is identified based on the body parts of the photographer that appear in the image and the area in the image in which the body parts appear.

[0010] A fifth aspect of the present disclosure is a mobile terminal according to any one of the first to fourth aspects, The mask process is a process of filling the area with a single color.

[0011] A sixth aspect of the present disclosure is a mobile terminal according to any one of the first to fifth aspects, The mobile terminal transmits the image on which the masking process has been performed to a server.

[0012] A seventh aspect of the present disclosure is a mobile terminal according to any one of the first to sixth aspects, The mobile terminal is a neck-worn wearable terminal.

[0013] A program according to an eighth aspect of the present disclosure, A mobile device equipped with a photography function A step of generating an image in which a mask process is performed on an area in a captured image in which at least a part of the photographer's body is captured Execute the command.

[0014] A method according to a ninth aspect of the present disclosure, comprising: A method executed by a mobile terminal equipped with a photographing function, comprising: A step of generating an image in which a mask process is performed on an area in the captured image in which at least a part of the photographer's body is captured. Includes.

[0015] A system according to a tenth aspect of the present disclosure includes: A system including a mobile terminal and a server, An image is generated by performing a mask process on an area in the captured image in which at least a part of the photographer's body is captured. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram showing an overall configuration according to an embodiment of the present disclosure. [Diagram 2] 1 is an example of a wearable terminal according to an embodiment of the present disclosure. [Diagram 3] FIG. 2 is a diagram illustrating a hardware configuration of a wearable terminal according to an embodiment of the present disclosure. [Figure 4] FIG. 2 is a diagram illustrating a hardware configuration of a server according to an embodiment of the present disclosure. [Diagram 5] FIG. 2 is a functional block diagram of a control unit of a wearable terminal according to an embodiment of the present disclosure. [Figure 6] 11 is a diagram for explaining a method for identifying an area in which a wearer is photographed when a wearable terminal according to an embodiment of the present disclosure is in a specified orientation. FIG. [Figure 7] 13 is a flowchart of a mask process according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0018] <Overall composition> 1 is a diagram showing an overall configuration according to an embodiment of the present disclosure. The image management system 1 includes a mobile terminal carried by a photographer (for example, a wearable terminal 10 worn by a wearer 11) and a server 20. Each of them will be described below.

[0019] <<Wearable devices>> The wearable terminal 10 is an example of a mobile terminal. In this specification, a case where the mobile terminal is a wearable terminal (for example, a neck-worn wearable terminal) will be described, but the present invention can be applied to any mobile terminal equipped with a photographing function, such as a smartphone.

[0020] The wearable terminal 10 is a terminal equipped with a photographing function and used by a wearer 11 (one example of a photographer). For example, the wearable terminal 10 is a neck-hanging type terminal. The wearable terminal 10 can transmit and receive data to and from the server 20 via any network.

[0021] Specifically, the mobile terminal (for example, the wearable terminal 10) generates an image in which a mask process is performed on an area in a captured image in which at least a part of the body of the photographer (for example, the wearer 11) is captured.

[0022] For example, the wearable terminal 10 identifies an area in an image captured by the wearable terminal 10 in which the wearer 11 of the wearable terminal 10 is photographed, masks the area (e.g., fills in the area), and transmits the image captured by the wearable terminal 10 including the masked area to the server 20.

[0023] For example, the area in which the wearer 11 of the wearable terminal 10 is photographed is identified from the orientation of the wearable terminal. That is, the area in the image where mask processing is performed can be identified based on the orientation of the wearable terminal 10. Note that the orientation of the wearable terminal 10 may be identified based on information from an acceleration sensor of the wearable terminal 10.

[0024] <<Server>> The server 20 receives images from the wearable terminal 10 and uses the images. For example, the server 20 uses the images received from the wearable terminal 10 to generate a 3D model representing the space photographed by the wearable terminal 10. The server 20 is composed of one or more computers. The server 20 can transmit and receive data to and from the wearable terminal 10 via any network.

[0025] In addition, at least a part of the processing of the wearable terminal 10 described in this specification may be performed by the server 20, or at least a part of the processing of the server 20 described in this specification may be performed by the wearable terminal 10.

[0026] <Wearable device configuration> Fig. 2 is an example of a wearable terminal 10 according to an embodiment of the present disclosure. For example, the wearable terminal 10 is a terminal having a neck-hanging shape as shown in Fig. 2. For example, the wearable terminal 10 includes an imaging unit (camera) 107 as shown in Fig. 2. Furthermore, the wearable terminal 10 may include a sound input unit (microphone) 105, a sound output unit (speaker) 106, and an operation unit 104 as shown in Fig. 2. Each unit will be described in detail with reference to Fig. 3.

[0027] <Hardware configuration> Hereinafter, an example of the hardware configuration of the wearable terminal 10 will be described with reference to FIG. 3, and an example of the hardware configuration of the server 20 will be described with reference to FIG.

[0028] 3 is a diagram showing a hardware configuration of a wearable terminal 10 according to an embodiment of the present disclosure. The wearable terminal 10 can include a control unit (processor) 101, a storage unit (memory) 102, a communication unit 103, an operation unit 104, a sound input unit (microphone) 105, a sound output unit (speaker) 106, an imaging unit (camera) 107, and various sensors 108. Each of these will be described below.

[0029] The control unit (processor) 101 is a processor that controls the wearable terminal 10. For example, the control unit (processor) 101 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0030] The storage unit (memory) 102 is a memory that stores any data.

[0031] The communication unit 103 connects to an arbitrary network and communicates with other computers (such as the server 20).

[0032] The operation unit 104 is a button or the like that allows the wearer 11 to input instructions to the wearable terminal 10.

[0033] The sound input unit (microphone) 105 collects sound.

[0034] The sound output unit (speaker) 106 outputs sound.

[0035] An imaging unit (camera) 107 captures still images and moving images.

[0036] The various sensors 108 are one or more arbitrary sensors such as an acceleration sensor, a GPS (Global Positioning System), and the like.

[0037] 4 is a diagram showing a hardware configuration of the server 20 according to an embodiment of the present disclosure. The server 20 can include a control unit (processor) 201, a storage unit (memory) 202, and a communication unit 203. Each of these will be described below.

[0038] The control unit (processor) 201 is a processor that controls the server 20. For example, the control unit (processor) 201 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0039] The storage unit (memory) 202 is a memory that stores any data.

[0040] The communication unit 203 connects to an arbitrary network and communicates with other computers (such as the wearable terminal 10).

[0041] 5 is a functional block diagram of the control unit 101 of the wearable terminal 10 according to an embodiment of the present disclosure. The control unit 101 of the wearable terminal 10 includes an area identification unit 111, a masking unit 112, and an image transmission unit 113. The control unit 101 of the wearable terminal 10 functions as the area identification unit 111, the masking unit 112, and the image transmission unit 113 by executing a program.

[0042] The area specifying unit 111 specifies an area in which the wearer 11 of the wearable terminal 10 is captured within an image captured by the wearable terminal 10 (imaging unit 107).

[0043] For example, the region identifying unit 111 identifies a region in the image where mask processing is to be performed based on a body part (e.g., a leg, etc.) of the photographer that appears in the image and the region in the image that the body part appears in. The region identifying unit 111 can identify a body part (e.g., a leg, etc.) of the photographer that appears in the image using an existing technology for detecting a body part in an image.

[0044] Masking unit 112 performs mask processing on the area specified by area specifying unit 111 (for example, by filling the area with a single color).

[0045] [Mask processing] Here, the masking process will be described. Masking unit 112 can fill in the area identified by area identification unit 111. Specifically, masking unit 112 sets all pixels in the area identified by area identification unit 111 to the same color (e.g., black) (e.g., sets the RGB values ​​of all pixels to 0:0:0).

[0046] The image transmission unit 113 transmits the image captured by the wearable terminal 10 (the imaging unit 107), including the area masked by the masking unit 112, to the communication unit 203.

[0047] <Identifying the area in which the wearer is being photographed> Hereinafter, the determination of the area in which the wearer 11 is photographed will be described with reference to FIG.

[0048] Specifically, a method for identifying the area in which the wearer 11 is photographed when the wearable terminal 10 is in a predetermined orientation will be described with reference to FIG.

[0049] [Wearable device orientation] FIG. 6 is a diagram for explaining a method for identifying an area in which the wearer 11 is photographed when the wearable terminal 10 according to an embodiment of the present disclosure is in a predetermined orientation.

[0050] Specifically, when the wearable terminal 10 is in a predetermined orientation (for example, in FIG. 6, when the direction in which the imaging unit (camera) 107 captures an image is perpendicular to the ground and faces the ground), the wearable terminal 10 identifies the area of ​​the wearer 11 in the image and performs mask processing on the area. This will be described in detail below.

[0051] First, a description will be given of how to specify the current orientation of the wearable terminal 10. The area specifying unit 111 specifies the current orientation of the wearable terminal 10. For example, the area specifying unit 111 can specify the current orientation of the wearable terminal 10 by using the function of an acceleration sensor of the wearable terminal 10. Note that the area specifying unit 111 may specify the current orientation of the wearable terminal 10 by using a method other than the function of the acceleration sensor of the wearable terminal 10.

[0052] Next, detection of the wearer's area will be described. When the current orientation of the wearable terminal 10 is a predetermined orientation (for example, in FIG. 6, when the direction in which the imaging unit (camera) 107 captures an image is perpendicular to the ground and faces the ground), the area identifying unit 111 detects the body of the wearer 11 (for example, the hands of the wearer 11, the feet of the wearer 11, etc. (in the example of FIG. 6, the feet)) in the image captured by the wearable terminal 10 (imaging unit 107). The area identifying unit 111 can detect the body of the wearer 11 in the image by using any object detection method. The predetermined orientation may be specified by the wearer 11 and stored in any memory such as the storage unit 102 of the wearable terminal 10 (the wearable terminal 10 refers to the predetermined orientation stored in the memory).

[0053] The current orientation of the wearable terminal 10 may be identified using a Visual SLAM (Simultaneous Localization and Mapping) technique.

[0054] <Processing method> FIG. 7 is a flowchart of a mask process according to an embodiment of the present disclosure.

[0055] In step 11 (S11), the wearable terminal 10 (imaging unit 107) captures an image.

[0056] In step 12 (S12), the wearable terminal 10 (control unit 101) identifies an area in which the wearer 11 of the wearable terminal 10 is photographed within the image photographed in S11.

[0057] In step 13 (S13), the wearable terminal 10 (control unit 101) performs mask processing on the area identified in S12.

[0058] In step 14 (S14), the wearable terminal 10 (control unit 101) transmits to the server 20 the image captured by the wearable terminal 10 (imaging unit 107) including the area masked in S13.

[0059] In this way, even if the wearer 11 of the wearable terminal 10 is captured in an image taken by the wearable terminal 10, the area of ​​the wearer 11 in the image can be masked before being transmitted to the server 20.

[0060] Therefore, for example, when the server 20 generates a 3D model representing the space photographed by the wearable terminal 10 using an image received from the wearable terminal 10, the modeling process of the masked area is not required. Also, when the wearable terminal 10 transmits an image to the server 20 in a remote location, the amount of image data is reduced, so that the amount of data communication between the wearable terminal 10 and the server 20 can be reduced.

[0061] Although the embodiments have been described above, it will be understood that various changes in form and details are possible without departing from the spirit and scope of the claims. [Explanation of symbols]

[0062] 1 Image Management System 10. Wearable Devices 11 Wearer 20 Servers 101 Control unit (processor) 102 Memory unit (memory) 103 Communications Department 104 Operation section 105 Sound input section (microphone) 106 Sound output unit (speaker) 107 Imaging unit (camera) 108 Various sensors 111 Area identification part 112 Masking Section 113 Image transmission unit 201 Control unit (processor) 202 Memory unit (memory) 203 Communications Department

Claims

1. A mobile terminal equipped with a photographing function, The control unit of the mobile terminal When the current orientation of the mobile terminal is a predetermined orientation, an image is generated in which a mask process is performed on an area in the captured image in which at least a part of the photographer's body is captured. Mobile device.

2. The control unit is The mobile terminal according to claim 1 , further comprising: a step of identifying an area in an image where mask processing is to be performed based on an orientation of the mobile terminal.

3. The orientation of the mobile terminal is The mobile terminal according to claim 2 , wherein the mobile terminal is specified based on information from an acceleration sensor of the mobile terminal.

4. The control unit is The mobile terminal according to claim 1 , wherein an area in an image to be masked is identified based on a body part of a photographer that appears in the image and an area in the image in which the body part appears.

5. The mobile terminal according to claim 1 , wherein the masking process is a process of filling the area with a single color.

6. The mobile terminal according to claim 1 , wherein the mobile terminal transmits the image on which the masking process has been performed to a server.

7. The mobile terminal according to claim 1 , wherein the mobile terminal is a neck-worn wearable terminal.

8. A mobile device equipped with a photography function A step of generating an image in which a mask process is performed on an area in an image captured when the current orientation of the mobile terminal is a predetermined orientation, in which at least a part of the photographer's body is captured. A program for executing the above.

9. A method executed by a mobile terminal equipped with a photographing function, comprising: generating an image in which a mask process is performed on an area in the captured image in which at least a part of the photographer's body is captured, when the current orientation of the mobile terminal is a predetermined orientation; The method includes:

10. A system including a mobile terminal and a server, When the current orientation of the mobile terminal is a predetermined orientation, an image is generated in which a mask process is performed on an area in the captured image in which at least a part of the photographer's body is captured. system.

Images