Portable terminal, program, method, and system

The mobile terminal's mask processing of the photographer's area in images taken, based on its orientation and using acceleration sensor data, addresses the inefficiency of including the photographer in 3D model generation, enhancing processing and data transmission efficiency by eliminating unnecessary data.

WO2024247850A1PCT designated stage expired Publication Date: 2024-12-05FAIRY DEVICES INC +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/018884
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-26
Filing Date
2024-05-22
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Conventional methods for generating 3D models from images taken by mobile terminals often include unnecessary areas, such as the photographer's body, which requires processing in these regions, leading to inefficiencies and increased data transmission.

Method used

A mobile terminal equipped with a control unit that performs mask processing on the photographer's body area in the image, specifying the area based on the terminal's orientation, using information from an acceleration sensor, and filling it with a single color, then transmits the masked image to a server for 3D model generation.

Benefits of technology

This approach reduces unnecessary processing and data transmission by masking the photographer's area, allowing for more efficient generation of 3D models and minimizing data communication between the terminal and server.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024018884_05122024_PF_FP_ABST
    Figure JP2024018884_05122024_PF_FP_ABST
Patent Text Reader

Abstract

The present invention improves the handling of an image captured by a portable terminal. A portable terminal according to an embodiment of the present disclosure which is equipped with an imaging function, wherein a control unit of the portable terminal generates an image obtained by subjecting a region in which at least part of the body of the photographer appears in the captured image to mask processing.
Need to check novelty before this filing date? Find Prior Art

Description

Mobile terminal, program, method, and system

[0001] The present disclosure relates to a mobile terminal, a program, a method, and a system.

[0002] 2. Description of the Related Art Conventionally, it is known to generate a 3D model representing a space photographed by a mobile terminal using an image photographed by the mobile terminal.

[0003] Japanese Patent Application Laid-Open No. 2018-36812

[0004] When a photographer takes an image with a mobile device, the photographer may appear in the image captured by the mobile device. In such cases, when generating a 3D model representing the space captured by the mobile device using the image captured by the mobile device, modeling processing must be performed even in unnecessary areas (i.e., the area where the photographer appears).

[0005] The present disclosure aims to improve the handling of images captured by mobile terminals.

[0006] A mobile terminal according to a first aspect of the present disclosure is a mobile terminal equipped with a photographing function, and a control unit of the mobile terminal generates an image in which a mask process has been performed on an area in a photographed image in which at least a part of the photographer's body is captured.

[0007] A second aspect of the present disclosure is the mobile terminal according to the first aspect, wherein the control unit identifies an area in the image where mask processing is to be performed based on an orientation of the mobile terminal.

[0008] A third aspect of the present disclosure is the mobile terminal according to the second aspect, wherein the orientation of the mobile terminal is identified based on information from an acceleration sensor of the mobile terminal.

[0009] A fourth aspect of the present disclosure is a mobile terminal according to any one of the first to third aspects, wherein the control unit identifies an area in the image where masking is to be performed based on a body part of the photographer that appears in the image and an area in the image in which the body part appears.

[0010] A fifth aspect of the present disclosure is the mobile terminal according to any one of the first to fourth aspects, wherein the masking process is a process of filling the area with a single color.

[0011] A sixth aspect of the present disclosure is the mobile terminal according to any one of the first to fifth aspects, wherein the mobile terminal transmits the image on which the masking process has been performed to a server.

[0012] A seventh aspect of the present disclosure is the mobile terminal according to any one of the first to sixth aspects, wherein the mobile terminal is a neck-worn wearable terminal.

[0013] A program according to an eighth aspect of the present disclosure causes a mobile terminal equipped with a photographing function to execute a procedure for generating an image in which a mask process has been applied to an area in a photographed image in which at least a part of the photographer's body is captured.

[0014] A method according to a ninth aspect of the present disclosure is a method executed by a mobile terminal equipped with a photographing function, and includes a step of generating an image in which a mask process has been performed on an area in the photographed image in which at least a part of the photographer's body is captured.

[0015] A system according to a tenth aspect of the present disclosure is a system including a mobile terminal and a server, and generates an image in which a mask process is performed on an area in a captured image in which at least a part of the photographer's body is captured.

[0016] FIG. 1 is a diagram illustrating an overall configuration according to an embodiment of the present disclosure; FIG. 2 is an example of a wearable terminal according to an embodiment of the present disclosure; FIG. 3 is a diagram illustrating a hardware configuration of a wearable terminal according to an embodiment of the present disclosure; FIG. 4 is a diagram illustrating a hardware configuration of a server according to an embodiment of the present disclosure; FIG. 5 is a functional block diagram of a control unit of a wearable terminal according to an embodiment of the present disclosure; FIG. 6 is a diagram for explaining a method of identifying an area in which a wearer is photographed when a wearable terminal according to an embodiment of the present disclosure is in a predetermined orientation; and FIG. 7 is a flowchart of mask processing according to an embodiment of the present disclosure.

[0017] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0018] 1 is a diagram showing the overall configuration according to an embodiment of the present disclosure. The image management system 1 includes a mobile terminal carried by a photographer (for example, a wearable terminal 10 worn by a wearer 11) and a server 20. Each of these components will be described below.

[0019] <<Wearable Terminal>> The wearable terminal 10 is an example of a mobile terminal. In this specification, a case where the mobile terminal is a wearable terminal (for example, a neck-worn wearable terminal) will be described, but the present invention can be applied to any mobile terminal equipped with a photographing function, such as a smartphone.

[0020] The wearable device 10 is a device equipped with a photographing function and used by a wearer 11 (an example of a photographer). For example, the wearable device 10 is a neck-worn device. The wearable device 10 can transmit and receive data to and from a server 20 via any network.

[0021] Specifically, the mobile terminal (e.g., the wearable terminal 10) generates an image in which a mask process is performed on an area in the captured image in which at least a part of the body of the photographer (e.g., the wearer 11) is captured.

[0022] For example, the wearable terminal 10 identifies an area in an image captured by the wearable terminal 10 in which the wearer 11 of the wearable terminal 10 is photographed, masks the area (e.g., fills in the area), and transmits the image captured by the wearable terminal 10 including the masked area to the server 20.

[0023] For example, the area in which the wearer 11 of the wearable device 10 is photographed is identified from the orientation of the wearable device 10. That is, the area in the image where masking is performed can be identified based on the orientation of the wearable device 10. Note that the orientation of the wearable device 10 may be identified based on information from an acceleration sensor of the wearable device 10.

[0024] <<Server>> The server 20 receives images from the wearable terminal 10 and uses the images. For example, the server 20 uses the images received from the wearable terminal 10 to generate a 3D model representing the space photographed by the wearable terminal 10. The server 20 is configured from one or more computers. The server 20 can send and receive data to and from the wearable terminal 10 via any network.

[0025] In addition, at least a part of the processing of the wearable terminal 10 described in this specification may be performed by the server 20, or at least a part of the processing of the server 20 described in this specification may be performed by the wearable terminal 10.

[0026] <Configuration of Wearable Terminal> Fig. 2 shows an example of a wearable terminal 10 according to an embodiment of the present disclosure. For example, the wearable terminal 10 is a terminal having a neck strap shape as shown in Fig. 2. For example, the wearable terminal 10 includes an imaging unit (camera) 107 as shown in Fig. 2. Furthermore, the wearable terminal 10 may include a sound input unit (microphone) 105, a sound output unit (speaker) 106, and an operation unit 104 as shown in Fig. 2. Each unit will be described in detail with reference to Fig. 3.

[0027] <Hardware Configuration> Hereinafter, an example of the hardware configuration of the wearable terminal 10 will be described with reference to FIG. 3, and an example of the hardware configuration of the server 20 will be described with reference to FIG.

[0028] 3 is a diagram illustrating the hardware configuration of a wearable terminal 10 according to an embodiment of the present disclosure. The wearable terminal 10 can include a control unit (processor) 101, a storage unit (memory) 102, a communication unit 103, an operation unit 104, a sound input unit (microphone) 105, a sound output unit (speaker) 106, an imaging unit (camera) 107, and various sensors 108. Each of these will be described below.

[0029] The control unit (processor) 101 is a processor that controls the wearable terminal 10. For example, the control unit (processor) 101 is a central processing unit (CPU), a graphics processing unit (GPU), or the like.

[0030] The storage unit (memory) 102 is a memory that stores any data.

[0031] The communication unit 103 connects to any network and communicates with other computers (such as the server 20).

[0032] The operation unit 104 is a button or the like that allows the wearer 11 to input instructions to the wearable terminal 10 .

[0033] The sound input unit (microphone) 105 collects sound.

[0034] The sound output unit (speaker) 106 outputs sound.

[0035] The imaging unit (camera) 107 captures still images and moving images.

[0036] The various sensors 108 are one or more arbitrary sensors such as an acceleration sensor, a GPS (Global Positioning System), and the like.

[0037] 4 is a diagram illustrating a hardware configuration of the server 20 according to an embodiment of the present disclosure. The server 20 may include a control unit (processor) 201, a storage unit (memory) 202, and a communication unit 203. Each of these will be described below.

[0038] The control unit (processor) 201 is a processor that controls the server 20. For example, the control unit (processor) 201 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0039] The storage unit (memory) 202 is a memory that stores any data.

[0040] The communication unit 203 connects to an arbitrary network and communicates with other computers (such as the wearable terminal 10).

[0041] 5 is a functional block diagram of the control unit 101 of the wearable terminal 10 according to an embodiment of the present disclosure. The control unit 101 of the wearable terminal 10 includes an area identification unit 111, a masking unit 112, and an image transmission unit 113. The control unit 101 of the wearable terminal 10 executes a program to function as the area identification unit 111, the masking unit 112, and the image transmission unit 113.

[0042] The area specifying unit 111 specifies an area in which the wearer 11 of the wearable terminal 10 is captured within an image captured by the wearable terminal 10 (the image capturing unit 107).

[0043] For example, the region identifying unit 111 identifies a region in the image to be masked based on a body part (e.g., a leg) of the photographer that appears in the image and the region in the image that the body part appears in. Note that the region identifying unit 111 can identify a body part (e.g., a leg) of the photographer that appears in the image using an existing technology for detecting a body part in an image.

[0044] The masking unit 112 performs mask processing on the area identified by the area identifying unit 111 (for example, by painting the area with a single color).

[0045] [Masking Process] Here, the masking process will be described. The masking unit 112 can fill in the area identified by the area identification unit 111. Specifically, the masking unit 112 sets all pixels in the area identified by the area identification unit 111 to the same color (e.g., black) (e.g., sets the RGB values ​​of all pixels to 0:0:0).

[0046] The image transmission unit 113 transmits the image captured by the wearable terminal 10 (the imaging unit 107 ) that includes the area masked by the masking unit 112 to the communication unit 203 .

[0047] <Identifying the Area in Which the Wearer is Photographed> Hereinafter, with reference to FIG. 6, how to identify the area in which the wearer 11 is photographed will be described.

[0048] Specifically, a method for identifying the area in which the wearer 11 is photographed when the wearable terminal 10 is in a predetermined orientation will be described with reference to FIG. 6 .

[0049] [Orientation of Wearable Terminal] FIG. 6 is a diagram for explaining a method for identifying an area in which the wearer 11 is photographed when the wearable terminal 10 according to an embodiment of the present disclosure is in a predetermined orientation.

[0050] Specifically, when the wearable device 10 is in a predetermined orientation (for example, in FIG. 6 , when the direction in which the imaging unit (camera) 107 captures an image is perpendicular to the ground and points toward the ground), the wearable device 10 identifies the area of ​​the wearer 11 in the image and performs mask processing on that area. This will be described in detail below.

[0051] First, we will explain how to determine the current orientation of the wearable terminal 10. The area determination unit 111 determines the current orientation of the wearable terminal 10. For example, the area determination unit 111 can determine the current orientation of the wearable terminal 10 using the function of an acceleration sensor of the wearable terminal 10. Note that the area determination unit 111 may determine the current orientation of the wearable terminal 10 using a method other than the function of the acceleration sensor of the wearable terminal 10.

[0052] Next, detection of the wearer's region will be described. When the wearable device 10 is currently facing a predetermined direction (for example, in FIG. 6 , when the direction in which the image capturing unit (camera) 107 captures images is perpendicular to the ground and points toward the ground), the region identifying unit 111 detects the body of the wearer 11 (for example, the hands of the wearer 11, the feet of the wearer 11, etc. (in the example of FIG. 6 , the feet)) in an image captured by the wearable device 10 (image capturing unit 107). The region identifying unit 111 can detect the body of the wearer 11 in the image using any object detection method. The predetermined direction may be specified by the wearer 11 and stored in any memory, such as the storage unit 102 of the wearable device 10 (the wearable device 10 refers to the predetermined direction stored in the memory).

[0053] The current orientation of the wearable terminal 10 may be identified using a Visual SLAM (Simultaneous Localization and Mapping) technique.

[0054] <Processing Method> FIG. 7 is a flowchart of a mask process according to an embodiment of the present disclosure.

[0055] In step 11 (S11), the wearable terminal 10 (imaging unit 107) captures an image.

[0056] In step 12 (S12), the wearable terminal 10 (control unit 101) identifies an area in the image captured in S11 in which the wearer 11 of the wearable terminal 10 is captured.

[0057] In step 13 (S13), the wearable terminal 10 (control unit 101) performs mask processing on the area identified in S12.

[0058] In step S14, the wearable terminal 10 (control unit 101) transmits to the server 20 the image captured by the wearable terminal 10 (imaging unit 107) that includes the area masked in S13.

[0059] In this way, even if the wearer 11 of the wearable terminal 10 is captured in an image taken by the wearable terminal 10, the area of ​​the wearer 11 in the image can be masked before being sent to the server 20.

[0060] Therefore, for example, when the server 20 generates a 3D model representing the space photographed by the wearable device 10 using an image received from the wearable device 10, the server 20 does not need to perform modeling of the masked area. Furthermore, when the wearable device 10 transmits an image to the server 20 located in a remote location, the amount of image data is reduced, and therefore the amount of data communication between the wearable device 10 and the server 20 can be reduced.

[0061] Although the embodiments have been described above, it will be understood that various changes in form and details can be made without departing from the spirit and scope of the claims.

[0062] This international application claims priority based on Japanese Patent Application No. 2023-086964, filed on May 26, 2023, the entire contents of which are incorporated herein by reference.

[0063] REFERENCE SIGNS LIST 1 Image management system 10 Wearable terminal 11 Wearer 20 Server 101 Control unit (processor) 102 Storage unit (memory) 103 Communication unit 104 Operation unit 105 Sound input unit (microphone) 106 Sound output unit (speaker) 107 Imaging unit (camera) 108 Various sensors 111 Area identification unit 112 Masking unit 113 Image transmission unit 201 Control unit (processor) 202 Storage unit (memory) 203 Communication unit

Claims

1. A mobile terminal equipped with a photographing function, wherein a control unit of the mobile terminal generates an image in which a mask process is performed on an area in a photographed image in which at least a part of the photographer's body is captured.

2. The mobile terminal according to claim 1, wherein the control unit identifies an area in the image where masking is to be performed based on the orientation of the mobile terminal.

3. The mobile terminal according to claim 2, wherein the orientation of the mobile terminal is determined based on information from an acceleration sensor of the mobile terminal.

4. A mobile terminal as described in any one of claims 1 to 3, wherein the control unit identifies an area in the image to which masking processing is to be performed based on the body parts of the photographer that are captured in the image and the area in the image in which the body parts are captured.

5. A mobile terminal according to any one of claims 1 to 4, wherein the masking process is a process of filling the area with a single color.

6. The mobile terminal according to any one of claims 1 to 5, wherein the mobile terminal transmits the image on which the masking process has been performed to a server.

7. The mobile terminal according to any one of claims 1 to 6, wherein the mobile terminal is a neck-worn wearable terminal.

8. A program for causing a mobile terminal equipped with a photographing function to execute a procedure for generating an image in which a mask process has been applied to an area in a photographed image in which at least a part of the photographer's body is captured.

9. A method executed by a mobile terminal equipped with a photographing function, comprising the step of generating an image in which a mask process is performed on an area in a photographed image in which at least a part of the photographer's body is captured.

10. A system including a mobile terminal and a server, which generates an image in which a mask process has been applied to areas within a captured image in which at least part of the photographer's body is captured.

Citation Information

Patent Citations

  • Image processing apparatus, image processing method, and imaging apparatus

    JP2013198062A

  • Image capture device

    JP2020016767A

  • Neck hanging type device and remote operation support system

    JP2022119684A

  • Image capture system

    WO2018025825A1