Image processing device, image processing system, image processing method and program

The image processing device automatically identifies and processes specific areas in captured images using pattern detection, addressing the inefficiencies of manual post-processing in conventional systems.

JP7593098B2Active Publication Date: 2024-12-03RICOH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2020212786
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-02-17
Filing Date
2020-12-22
Publication Date
2024-12-03
Estimated Expiration
2040-12-22

AI Technical Summary

Technical Problem

Conventional image processing devices that capture all-direction images require manual post-processing to mask unwanted areas, which is time-consuming and prone to errors.

Method used

An image processing device that detects a predetermined pattern in the captured image to automatically identify and apply image processing to specific areas before capturing, using LED devices or QR codes to define the mask processing region.

Benefits of technology

Enables efficient and accurate pre-designation of processing targets in captured images, reducing the time and effort required for masking unwanted areas.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007593098000001
    Figure 0007593098000001
  • Figure 0007593098000002
    Figure 0007593098000002
  • Figure 0007593098000003
    Figure 0007593098000003
Patent Text Reader

Abstract

To provide an image processing apparatus.SOLUTION: An image processing apparatus 10 includes image input means 15 that receives input of a picked-up image. The image processing apparatus 10 includes detection means 16 that detects the positions of predetermined patterns indicated in the picked-up image input to the image input means 15. The image processing apparatus 10 includes setting means 22 that sets an area to be processed based on the positions of the plurality of predetermined patterns detected by the detection means 16. The image processing apparatus 10 includes image processing means 23 that performs image processing on the area set by the setting means 22.SELECTED DRAWING: Figure 6
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing system, an image processing method, and a program. [Background technology]

[0002] Conventionally, a photographing device that photographs in all directions using a plurality of wide-angle lenses or fisheye lenses is known. When photographing using such a photographing device, there are cases where a part that is not to be shown is photographed, and a masking process is applied to the photographed part by blacking out or blurring it after the fact in image processing. However, the application of the masking process is applied after the fact after the photographing by the photographing device that photographs in all directions is finished. After the photographing by the photographing device that photographs in all directions, it is necessary to search for the relevant part in the image and apply the masking process, which is time-consuming and there is a high possibility that the masking process may be omitted.

[0003] For example, Japanese Patent Application Laid-Open No. 2016-133908 (Patent Document 1) discloses an image generating device that generates a planar image for a cut-out range cut out from a panoramic image for the purpose of reducing the amount of data. The image generating device includes an acquisition means for acquiring a panoramic image, a detection means for detecting an unnecessary region of the panoramic image, a designation means for designating a cut-out range by excluding the unnecessary region detected by the detection means, a cut-out means for performing cut-out based on the region designation by the designation means, a conversion means for converting the cut-out image cut out by the cut-out means into a planar image, and a display means for displaying the planar image converted by the conversion means on a UI. However, Patent Document 1 does not disclose anything from the viewpoint of pre-designating a region to be processed in image processing such as mask processing. Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure has been made in consideration of the above points, and aims to provide an image processing device that enables a user to recognize a pre-specified area of ​​a captured image to be processed and apply image processing to the captured image. [Means for solving the problem]

[0005] In view of the above problems, the present disclosure provides an image processing device having the following features: The image processing device includes an image input means for receiving input of a captured image, a detection means for detecting the position of a predetermined pattern shown in the captured image input to the image input means, a setting means for setting an area to be processed based on the position of the predetermined pattern detected by the detection means, and an image processing means for performing image processing on the area set by the setting means. Effect of the Invention

[0006] With the above configuration, the user can recognize a pre-specified area of ​​a captured image to be processed and apply image processing to the image. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of an image processing system according to an embodiment. [Diagram 2] 10A and 10B are diagrams for explaining an example of a configuration for setting a mask processing region according to the embodiment; [Diagram 3] 13 is a diagram for explaining another example of the configuration for setting a mask processing region according to the embodiment. FIG. [Figure 4] FIG. 1 is a diagram illustrating an example of a hardware configuration of an imaging apparatus according to an embodiment. [Diagram 5] FIG. 1A is a conceptual diagram showing an example of a data structure of each image and a data flow of image processing in generating a spherical image; FIG. 1B is a conceptual diagram showing an example of the data structure of the spherical image in a plan view; and FIG. 1C is a conceptual diagram showing an example of the data structure of the spherical image in a spherical view. [Figure 6] FIG. 2 is a diagram illustrating an example of a functional configuration of an imaging apparatus according to an embodiment. [Figure 7] FIG. 4 is a conceptual diagram illustrating an example of a position information management table according to the embodiment. [Figure 8] 1 is a diagram illustrating functional blocks related to an automatic masking process implemented in the omnidirectional imaging device according to the embodiment, and a flow of the automatic masking process. FIG. [Figure 9] 5 is a flowchart showing an automatic mask process executed by the imaging apparatus according to the embodiment. [Figure 10] 1A to 1C are diagrams illustrating an example of a method for specifying a mask processing area by a photographer or an administrator and an example of an image to which mask processing has been applied according to an embodiment. [Figure 11] 1A and 1B are schematic diagrams illustrating a celestial sphere image before and after application of a mask, the celestial sphere image being captured when two mask processing regions are specified by a photographer or an administrator according to an embodiment; [Figure 12] 11A to 11C are diagrams illustrating variations of automatic mask processing according to an embodiment. [Figure 13] 11A to 11C are diagrams illustrating variations of automatic mask processing according to an embodiment. [Figure 14] 5 is a schematic diagram showing an example of a detection position of a predetermined pattern according to the embodiment; FIG. [Figure 15] 11 is a diagram showing another example of the data structure of the position information management table stored in the storage unit according to the embodiment. FIG. [Figure 16] FIG. 13 is a diagram for explaining an example of an outline of an image processing system according to a modified example of an embodiment. [Figure 17] FIG. 13 is a diagram illustrating an example of a hardware configuration of a PC according to a modified example of the embodiment. [Figure 18] FIG. 13 is a diagram illustrating an example of a functional configuration of an image processing system according to a modified example of an embodiment. [Figure 19] FIG. 11 is a sequence diagram showing an example of a mask process for an imaging device in an image processing system according to a modified example of an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described, but the embodiment of the present invention is not limited to the embodiment described below. Note that in the following embodiment, as an example of an image processing device and an image processing system, an image capturing device is described that includes an image capturing means including two fisheye lenses in its optical system, and has an image processing function that performs distortion correction and projective transformation on two partial images captured by the two fisheye lenses, and performs image stitching to generate a celestial sphere image.

[0009] ●Embodiment● System Configuration Fig. 1 is a diagram showing an example of an image processing system according to an embodiment of the present invention, which performs image processing on a captured image of a subject in a shooting space captured by an imaging device 10.

[0010] The imaging device 10 captures an image of a subject in a shooting space such as a factory or a server room. In this shooting space, there are a subject to be shot and an object to be concealed. The imaging device 10 is an image processing device that performs a mask process on the captured image captured by the subject in the shooting space to conceal the object to be concealed.

[0011] The imaging device 10 is a digital camera capable of capturing an image by capturing an object in a capturing space. The imaging device 10 is, for example, a special digital camera for capturing a 360° panoramic image. The imaging device 10 may be a general digital camera (such as a single-lens reflex camera or a compact digital camera). In this embodiment, the imaging device 10 is described as a digital camera (special imaging device) for capturing a 360° panoramic image. The captured image may be a moving image or a still image, or may be both a moving image and a still image. The captured image may include sound together with the image. The imaging device 10 may be fixed, or may be held by a photographer or a mobile robot and capture images while moving.

[0012] Overview Here, image processing in the image processing system according to the embodiment will be explained in outline with reference to Fig. 2 and Fig. 3. Fig. 2 is a diagram for explaining an example of a configuration for setting a mask processing region according to the embodiment. Note that Fig. 2 is a simplified explanation of the setting of a mask processing region in the image processing system according to the embodiment, and details of functions and the like realized by the image processing system will be explained with reference to drawings and the like described later.

[0013] FIG. 2 shows a configuration for concealing an object J using LED devices 30A to 30E. As shown in FIG. 1, there is a subject (object J) in the shooting space that the photographer does not want to appear in the image. For example, a use is conceivable in which the captured image captured by the imaging device 10 is sent to a remote location to remotely check the on-site video of a factory or a server room. In particular, a method is conceivable in which the photographer walks around the site while shooting a video. In this case, a facility at a specific location (for example, object J) may be set as a concealment target because of its high confidentiality. In this case, as shown in FIG. 2, a plurality of LED devices 30A to 30H (hereinafter, when there is no need to distinguish between them, they will be referred to as LED devices 30) are three-dimensionally arranged around the object J to be protected. In the example shown in FIG. 2, eight LED devices 30A to 30H are arranged.

[0014] The LED devices 30A-30H generate bright spots 35A-35H of a color as a predetermined pattern, respectively. These bright spots 35A-35H become a pattern for specifying a mask processing area. The LED device 30 is an example of a light emitting device that generates bright spots of a predetermined color, and is a device that is placed by a photographer who intends to use the automatic mask processing function to specify a mask processing area in advance. The predetermined pattern is preferably a pattern that can be distinguished from a single-color lamp such as red or green that is used in various electronic devices, and may be configured as a light intensity (blinking) pattern, a color of a bright spot, an arrangement of a plurality of bright spots, or a combination of these.

[0015] The LED device 30 is an example of a pattern generating means that is arranged within the field of view of the imaging device 10 and generates a predetermined pattern. The pattern generating means may be a light emitting device that emits one or more bright spots. The light emitting device may be, in addition to the LED device 30, a wire light or a mesh light in which a light emitting element such as an LED is attached to a wire or mesh. The pattern generating means may be a stand type or a ceiling type as shown in FIG. 2, or a patch type.

[0016] The imaging device 10 calculates the coordinate values ​​of the central positions of the bright spots 35A-35H corresponding to the light emitting sites of the multiple LED devices 30A-30H in the captured image. When the captured image is a moving image, comparing pixel values ​​at the same coordinates between successive images, there is a difference in pixel value at the coordinates corresponding to the blinking LED device 30 depending on whether it is on or off. The imaging device 10 can detect the position in the image corresponding to the blinking LED device 30 by checking the change in pixel value, such as the light intensity (blinking) pattern of the bright spots 35.

[0017] Furthermore, the imaging device 10 can identify the blinking pattern of the LED from the appearance pattern of an image showing a lit LED device 30 and an image showing an unlit LED device 30. By recording a specific blinking pattern in the imaging device 10, the imaging device 10 can detect that the specific blinking pattern is a predetermined pattern generated by a pattern generating means.

[0018] FIG. 3 is a diagram for explaining another example of a configuration for setting a mask processing region according to the embodiment. The example shown in FIG. 3 is a configuration in which a medium on which an image such as QR codes (registered trademark) 40A to 40H (hereinafter, when there is no need to distinguish between them, they will be referred to as QR codes (registered trademark) 40) is printed or displayed is used instead of the LED device 30 as a pattern generating means for generating a predetermined pattern for identifying the region. The imaging device 10 can detect the predetermined pattern by registering specific character information that serves as an identifier for each of the multiple QR codes (registered trademark) 40A to 40H and reading the QR codes (registered trademark) 40A to 40H. The QR code (registered trademark) 40 is installed as a printed or displayed medium. It is assumed that the imaging device 10 moves or changes its orientation during shooting, so it is desirable to have multiple surfaces on which the QR code (registered trademark) 40 can be read so that it can be recognized from multiple viewpoints.

[0019] The image as the predetermined pattern shown in FIG. 3 is not limited to a QR code (registered trademark), and may be a one-dimensional code, a two-dimensional code, or a specific image. The one-dimensional code is a so-called barcode. The two-dimensional code includes, for example, a matrix type and a stack type, such as a QR code (registered trademark), DataMatrix (DataCode), MaxiCode, or PDF417. The specific image is an image including numbers, codes, symbols, illustrations, or the like that can identify the predetermined pattern. When an image is used as the predetermined pattern, the image captured by the imaging device 10 does not need to be a moving image, and may be a still image.

[0020] In this way, in the image processing system, the photographer or the manager of the subject to be photographed (the person in charge of the resident of the room or the base such as a factory that accepts the photographing by the photographer) physically and three-dimensionally arranges a pattern generating means that generates a predetermined pattern around the subject that the photographer wants to hide so that the above-mentioned predetermined pattern is reflected in the captured image. Then, image processing, more specifically mask processing, is performed on the image area of ​​the object surrounded by such a pattern generating means in the captured image. This allows the photographer or manager to specify in advance the area that he or she wants to exclude from the subject to be photographed before shooting. In the following description, an example will be described in which the predetermined pattern is the bright spot 35 generated by the LED device 30, but the predetermined pattern may also be an image as shown in FIG. 3.

[0021] ●Hardware configuration Next, the hardware configuration of the imaging device 10 will be described with reference to FIG. 4. FIG. 4 is a diagram showing an example of the hardware configuration of the imaging device according to the embodiment. Note that components may be added or deleted from the hardware configuration shown in FIG. 4 as necessary. In the following, the imaging device 10 is assumed to be an omnidirectional (all-sphere) imaging device using two imaging elements, but the number of imaging elements may be two or more. In addition, the imaging device does not necessarily have to be a device dedicated to omnidirectional imaging, and an omnidirectional imaging unit may be attached to a normal digital camera, smartphone, or the like to have substantially the same functions as the imaging device 10.

[0022] As shown in FIG. 4, the imaging device 10 is composed of an imaging unit 101, an image processing unit 104, an imaging control unit 105, a microphone 108, a sound processing unit 109, a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112, an SRAM (Static Random Access Memory) 113, a DRAM (Dynamic Random Access Memory) 114, an operation unit 115, an input / output I / F (Interface) 116, a short-range communication circuit 117, an antenna 117a of the short-range communication circuit 117, an electronic compass 118, a gyro sensor 119, an acceleration sensor 120 and a network I / F 121.

[0023] Of these, the imaging unit 101 includes wide-angle lenses (so-called fisheye lenses) 102a and 102b (hereinafter, when there is no need to distinguish between them, they will be referred to as lenses 102) each having an angle of view of 180° or more for forming a hemispherical image, and two imaging elements 103a and 103b provided corresponding to the lenses. The imaging elements 103a and 103b include an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor that converts an optical image by the lenses 102a and 102b into image data of an electrical signal and outputs it, a timing generation circuit that generates horizontal or vertical synchronization signals and pixel clocks for the image sensors, and a group of registers in which various commands and parameters required for the operation of the imaging elements are set.

[0024] The imaging elements 103a and 103b of the imaging unit 101 are each connected to the image processing unit 104 via a parallel I / F bus. Meanwhile, the imaging elements 103a and 103b of the imaging unit 101 are each connected to the imaging control unit 105 via a serial I / F bus (such as an I2C bus). The image processing unit 104, the imaging control unit 105, and the sound processing unit 109 are connected to the CPU 111 via a bus 110. In addition, the ROM 112, the SRAM 113, the DRAM 114, the operation unit 115, the input / output I / F 116, the short-range communication circuit 117, the electronic compass 118, the gyro sensor 119, the acceleration sensor 120, and the network I / F 121 are also connected to the bus 110.

[0025] The image processing unit 104 takes in the image data output from the image sensors 103a and 103b via a parallel I / F bus, performs predetermined processing on each piece of image data, and then synthesizes the image data to create equirectangular projection image data.

[0026] The imaging control unit 105 generally sets commands and the like in the register groups of the imaging elements 103a and 103b using an I2C bus, with the imaging control unit 105 acting as a master device and the imaging elements 103a and 103b acting as slave devices. Necessary commands and the like are received from the CPU 111. The imaging control unit 105 also uses the I2C bus to retrieve status data and the like from the register groups of the imaging elements 103a and 103b and send it to the CPU 111.

[0027] Furthermore, the imaging control unit 105 instructs the imaging elements 103a and 103b to output image data when the shutter button of the operation unit 115 is pressed. Some imaging devices 10 have a preview display function or a function corresponding to video display on a display. In this case, the image data is output from the imaging elements 103a and 103b continuously at a predetermined frame rate (frames / minute).

[0028] As described later, the imaging control unit 105 also functions as a synchronization control means for synchronizing the output timing of image data from the imaging elements 103a and 103b in cooperation with the CPU 111. In this embodiment, the imaging device 10 is not provided with a display unit, but a display unit may be provided. The microphone 108 converts sound into sound (signal) data. The sound processing unit 109 takes in the sound data output from the microphone 108 via an I / F bus and performs a predetermined process on the sound data.

[0029] The CPU 111 controls the overall operation of the imaging device 10 and executes necessary processing. The ROM 112 stores various programs for the CPU 111. The SRAM 113 and the DRAM 114 are work memories, and store programs executed by the CPU 111, data in the middle of processing, and the like. In particular, the DRAM 114 stores image data in the middle of processing by the image processing unit 104 and data of a processed equirectangular projection image.

[0030] The operation unit 115 is a collective term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, etc. A user operates the operation unit 115 to input various shooting modes, shooting conditions, etc.

[0031] The input / output I / F 116 is a general term for an interface circuit (such as a Universal Serial Bus (USB) I / F) with an external medium such as an SD card or a personal computer. The input / output I / F 116 may be wireless or wired. The data of the equirectangular projection image stored in the DRAM 114 is recorded on an external medium via the input / output I / F 116, or transmitted to an external terminal (device) via the input / output I / F 116 as necessary.

[0032] The short-range communication circuit 117 communicates with an external terminal (device) by a short-range wireless communication technology such as NFC (Near Field Communication), Bluetooth (registered trademark), or Wi-Fi (registered trademark) via an antenna 117a provided in the imaging device 10. The short-range communication circuit 117 can transmit data of the equirectangular projection image to the external terminal (device).

[0033] The electronic compass 118 calculates the direction of the imaging device 10 from the earth's magnetism and outputs direction information. This direction information is an example of related information (metadata) according to Exif, and is used for image processing such as image correction of the captured image. The related information also includes data on the shooting date and time of the image and the data capacity of the image data. The gyro sensor 119 is a sensor that detects angle changes (roll angle, pitch angle, yaw angle) accompanying the movement of the imaging device 10. The angle changes are an example of related information (metadata) according to Exif, and are used for image processing such as image correction of the captured image. The acceleration sensor 120 is a sensor that detects acceleration in three axial directions. The imaging device 10 calculates the attitude (angle with respect to the direction of gravity) of its own device (imaging device 10) based on the acceleration detected by the acceleration sensor 120. The imaging device 10 improves the accuracy of image correction by providing the acceleration sensor 120. The network I / F 121 is an interface for performing data communication using a communication network such as the Internet via a router or the like.

[0034] ○Spherical image Here, the generation of a celestial sphere image and the generated celestial sphere image will be described with reference to FIG. 5. FIG. 5(A) is a conceptual diagram showing an example of the data structure of each image and the data flow of image processing in the generation of a celestial sphere image. First, the images directly captured by each of the image capture elements 103a and 103b are images with a field of view covering approximately a hemisphere of the celestial sphere. Light incident on the lens 102 is imaged on the light receiving area of ​​the image capture element 103 according to a predetermined projection method. The image captured here is captured by a two-dimensional image capture element whose light receiving area forms a planar area, and becomes image data expressed in a planar coordinate system. Typically, the obtained image is configured as a fisheye lens image including the entire image circle onto which each shooting range is projected, as shown by "partial image A" and "partial image B" in FIG. 5(A).

[0035] A plurality of partial images captured by the plurality of imaging elements 103a and 103b are corrected for distortion and synthesized to form one omnidirectional image. In the correction and synthesis process, first, each image including each complementary hemispherical portion is generated from each partial image configured as a planar image. Then, the two images including each hemispherical portion are aligned (stitched) based on matching of overlapping regions, and the images are synthesized to generate a omnidirectional image including the entire omnidirectional sphere. Although the images of each hemispherical portion include overlapping regions with other images, in the image synthesis, blending is performed on the overlapping regions as appropriate to form natural seams.

[0036] Fig. 5(B) is a conceptual diagram showing an example of the data structure of a celestial sphere image in a plane, and Fig. 5(C) is a conceptual diagram showing an example of the data structure of a celestial sphere image in a spherical surface. As shown in Fig. 5(B), image data of the celestial sphere image is expressed as an array of pixel values ​​whose coordinates are a vertical angle φ made with respect to a predetermined axis and a horizontal angle θ corresponding to a rotation angle around the predetermined axis. The vertical angle φ is in the range of 0 degrees to 180 degrees (or -90 degrees to +90 degrees), and the horizontal angle θ is in the range of 0 degrees to 360 degrees (or -180 degrees to +180 degrees).

[0037] As shown in Fig. 5(C), each coordinate value (θ, φ) of the omnidirectional format is associated with each point on a sphere representing all directions centered on the shooting point, and the all directions are associated with the omnidirectional image. The planar coordinates of the partial image captured by the fisheye lens and the coordinates on the sphere of the omnidirectional image are associated with each other in a predetermined conversion table. The conversion table is data that is created in advance by a manufacturer or the like according to a predetermined projection model based on the design data of each lens optical system, and is data that converts the partial image into a omnidirectional image.

[0038] Furthermore, the omnidirectional image may be subjected to either or both of zenith correction and rotation correction as appropriate, using information from electronic compass 118, gyro sensor 119, and acceleration sensor 120. Here, zenith correction refers to a process of correcting a omnidirectional image actually taken with its central axis tilted relative to the direction of gravity to a omnidirectional image as if it had been taken with its central axis aligned with the direction of gravity (correction in the roll and pitch directions). Rotation correction refers to a process of further rotating the omnidirectional image corrected by zenith correction so that its central axis is aligned with the direction of gravity, from a reference around the direction of gravity (correction in the yaw direction).

[0039] In the following description, a configuration in which the imaging device 10 acquires a celestial sphere image will be described, but the target of the mask processing function may be an image captured by a normal camera. In addition, the above-mentioned zenith correction and rotation correction will be described as being performed before the captured image is input to the image input unit 15, but they may be performed after the mask processing is performed. Furthermore, the above-mentioned imaging device 10 is an example of an imaging device capable of acquiring a wide-angle image, and the celestial sphere image is an example of a wide-angle image. Here, a wide-angle image is generally an image captured using a wide-angle lens, and is captured with a lens that can capture a wider range than the human eye can sense. In addition, it generally means an image captured with a lens with a focal length of 35 mm or less in 35 mm film equivalent.

[0040] Moreover, each of the above programs may be recorded in a computer-readable recording medium as an installable or executable file and distributed. Examples of the recording medium include a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), a Blu-ray (registered trademark) Disc, an SD card, and a USB memory. The recording medium may be provided domestically or internationally as a program product. For example, the imaging device 10 realizes the image processing method according to the present invention by executing the program according to the present invention.

[0041] ●Function configuration Next, the functional configuration of the imaging device 10 according to the embodiment will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a diagram showing an example of the functional configuration of the imaging device according to the embodiment. Note that Fig. 6 shows the components related to the processing or operation described below.

[0042] The imaging device 10 has a communication unit 11, a reception unit 12, a display control unit 13, an imaging processing unit 14, an image input unit 15, a detection unit 17, a calculation unit 18, an estimation unit 21, a region setting unit 22, an image processing unit 23, an image output unit 24, and a storage / readout unit 29. Each of these units is a function or means realized by operating any of the components shown in Fig. 4 in response to an instruction from the CPU 111 in accordance with a program for the imaging device loaded from the SRAM 113 onto the DRAM 114. The imaging device 10 also has a storage unit 1000 constructed by the ROM 112, the SRAM 113, and the DRAM 114 shown in Fig. 4.

[0043] The communication unit 11 is mainly realized by the processing of the CPU 111, and communicates various data or information with other devices or terminals. The communication unit 11 performs data communication with other devices or terminals via a communication network, for example, using the network I / F 121. The communication unit 11 also performs data communication with other devices or terminals via various cables, for example, using the input / output I / F 116. Furthermore, the communication unit 11 performs data communication with other devices or terminals by short-range wireless communication technology, for example, using the short-range communication circuit 117.

[0044] The reception unit 12 is mainly realized by the processing of the CPU 111 on the operation unit 115, and receives various selections or inputs from the user. The display control unit 13 is mainly realized by the processing of the CPU 111, and displays various images. The display control unit 13 performs image display in, for example, live view (a function for shooting while making it possible to check the shooting state in real time). During live view, the display control unit 13 controls display on an external display connected by a wired or wireless interface according to a standard such as HDMI (registered trademark) or Miracast (registered trademark) or on its own display means. The display control unit 13 can display an automatically detected mask processing area superimposed on the display of an input image on the live view screen. Note that the method of displaying the omnidirectional image is not particularly limited, and the omnidirectional image may be displayed as it is as a two-dimensional flat image, or the omnidirectional image may be pasted on a spherical object and displayed as a projected image when the spherical object is observed from a virtual camera. The photographer can confirm on the live view image that the subject to be protected has been masked.

[0045] The imaging processing unit 14 is mainly realized by the processing of the CPU 111 on the imaging unit 101, the image processing unit 104, the imaging control unit 105, the microphone 108, and the sound processing unit 109, and captures an object such as a landscape and acquires captured image data. The imaging processing unit 14 executes imaging processing, generates data of a celestial sphere image (for example, an equirectangular projection image), and inputs the celestial sphere image data to the image input unit 15. The celestial sphere image captured by the imaging processing unit 14 may be a still image of a single frame, or may be a video consisting of a continuous plurality of frames. The imaging processing unit 14 can output an actual captured image after a shooting instruction is given by pressing an operation unit 115 such as a shutter button, and can also output a real-time image for video confirmation before a shooting instruction is given.

[0046] The image input unit 15 is mainly realized by the processing of the CPU 111, and accepts input of a captured image to be processed. The captured image to be subjected to the mask processing is a spherical image. In the case of a still image, one frame of an image captured at a predetermined time point is input to the image input unit 15. In the case of a moving image, each image of a plurality of frames is input sequentially to the image input unit 15. In the case of still image shooting, each image of a plurality of frames may be input sequentially to the image input unit 15 for image confirmation before shooting.

[0047] The detection unit 17 is realized by the processing of the CPU 111, and detects a predetermined pattern generated by a pattern generating means such as the LED device 30 or the QR code (registered trademark) 40, which is shown in the captured image input by the image input unit 15. For example, when the captured image is a still image, the predetermined pattern to be detected may be a one-dimensional code, a two-dimensional code, or a specific image or a pattern of bright spots in the image. For example, the color of the bright spots and the arrangement of multiple bright spots may be used alone or in combination as the pattern of the bright spots. Note that the pattern of the bright spots includes a single bright spot having a predetermined color. Furthermore, when the captured image is a moving image, the predetermined pattern to be detected includes a one-dimensional code, a two-dimensional code, or a specific image or pattern of bright spots in the image of each frame, as well as a temporal pattern of bright spots in an image sequence spanning multiple frames. Here, the temporal pattern of the bright spots is composed of a light intensity (blinking) pattern of a predetermined frequency of the bright spots across multiple frames, the color of the bright spots, the arrangement of multiple bright spots, or a combination of these. The arrangement of the bright spots may be graphically represented in a predetermined shape such as "○", "×", "□", or "▽".

[0048] Any known technique can be used to recognize one-dimensional codes, two-dimensional codes, and specific images, as well as bright spots and their temporal patterns in an image or image sequence. In addition, in order to recognize images in a celestial sphere image, the imaging device 10 may perform a process of correcting distortions of areas in the image corresponding to one-dimensional codes and two-dimensional codes as preprocessing, if necessary.

[0049] The calculation unit 18 is realized by processing of the CPU 111, and calculates the position of the predetermined pattern detected by the detection unit 17. For example, the calculation unit 18 calculates the position of the predetermined pattern in the image as a coordinate value on the spherical image format. The position of the predetermined pattern may be tracked across multiple frames to calculate the correspondence.

[0050] The estimation unit 21 is realized by the processing of the CPU 111, and estimates vertices that complement a specific shape formed based on the positions of the multiple predetermined patterns detected by the detection unit 17. The estimation unit 21 estimates vertices that are missing in order to form a specific shape determined using the multiple detected predetermined patterns. For example, a case is assumed in which one of the LED devices 30 arranged as shown in FIG. 2 is hidden by an obstacle. If one of the LED devices 30 is hidden, the bright spot 35 generated by the hidden LED device 30 cannot be detected, and there is a risk that the intended area cannot be hidden. Therefore, in this embodiment, vertices that are missing in order to form the area to be processed are estimated from the multiple detected bright spots 35. Then, a mask processing area is set by the multiple detected bright spots 35 and the estimated vertices.

[0051] The region setting unit 22 is realized by the processing of the CPU 111, and sets a mask processing region (region to be processed) for the target image based on the position calculated by the calculation unit 18 of the predetermined pattern detected by the detection unit 17 and the vertices estimated by the estimation unit 21 as necessary. Here, the mask processing region may include a region including a polygon defined by connecting the coordinate values ​​of at least the positions of a plurality of predetermined patterns (including the vertices estimated as necessary), or a region including a convex hull (the smallest convex polygon including all given points) of at least the coordinate values ​​of the positions of a plurality of predetermined patterns (including the vertices estimated as necessary) as points. Note that the mask processing region may be set as a polygon or convex hull itself connecting the positions of a plurality of predetermined patterns and the coordinates of the vertices estimated as necessary, or may be any figure (including figures consisting of any closed curve such as a circle or an ellipse in addition to a polygon) with a predetermined margin provided for the polygonal region. The mask processing region is obtained, for example, as an array of coordinate values ​​(parameters may be included in the case of a curve or the like) on a celestial sphere image format.

[0052] Furthermore, the region setting unit 22 automatically recognizes the position of the predetermined pattern and, if necessary, the estimated vertices as a "surface" defined by the pattern, and sets a mask processing region for the surface. Note that the number of predetermined patterns detected by the detection unit 17 is preferably at least three or more, which is sufficient to define a surface, but is not necessarily limited to three or more, as long as it is possible to draw any two-dimensional figure with a predetermined margin.

[0053] The omnidirectional image includes a range of 360 degrees in the horizontal direction, and both ends of the omnidirectional image are connected to each other. Therefore, the mask processing area may straddle the end, and the mask processing area may be set separately at both ends of the omnidirectional image. In this case, the area setting unit 22 determines the intersection point between the mask processing area and the end boundary, and sets the mask processing area including the determined intersection point on each side.

[0054] Here, the detection unit 17 can determine the group to which the detected predetermined pattern belongs in addition to the coordinate value of the predetermined pattern. In this case, the region setting unit 22 can set different mask processing regions based on the identified group configuration. For example, the above-mentioned predetermined pattern can encode information using a one-dimensional code, a two-dimensional code, a bright spot pattern, or the like, so that multiple groups can be expressed by encoded information (e.g., an identifier). Then, the region setting unit 22 can set one mask processing region for each of the one or more identified groups.

[0055] The image processing unit 23 is realized by the processing of the CPU 111, and performs image processing on a region corresponding to the mask processing region in the captured image input by the image input unit 15. The image processing is a mask processing that irretrievably deletes or deteriorates image information in the mask processing region in the image. The mask processing can include at least one image effect from among blurring, filling with a predetermined color or image pattern (checker pattern, stripe, noise pattern (sandstorm), stamp, etc.), mosaic processing, and overwriting with a two-dimensional or three-dimensional graphic object. It is a mask processing. The mask processing is applied to an image region of the omnidirectional image corresponding to the mask processing region on the omnidirectional image format, for example.

[0056] The image output unit 24 is realized mainly by the processing of the CPU 111, and outputs the processed image that has been processed by the image processing unit 23.

[0057] The storage / readout unit 29 is mainly realized by the processing of the CPU 111, and stores various data (or information) in the storage unit 1000 and reads out various data (or information) from the storage unit 1000. The storage unit 1000 also stores captured image data acquired by the imaging processing unit 14 or processed image data processed by the image processing unit 23. The image data stored in the storage unit 1000 may be configured to be deleted when a predetermined time has passed since it was stored, or data transmitted to another device or terminal may be deleted.

[0058] Location information management table FIG. 7 is a conceptual diagram showing an example of a position information management table according to an embodiment. The position information management table is a table for managing position information indicating the position of a detected predetermined pattern. A position information management DB 1001 configured by a position information management table as shown in FIG. 7 is constructed in the storage unit 1000. This position information management table stores the positions of a plurality of predetermined patterns detected in an image (frame) previously input to the image input unit 15. In the position information management table, coordinates indicating the positions of a plurality of predetermined patterns are managed for each area recognized as the same by tracking.

[0059] Fig. 8 illustrates functional blocks related to the automatic mask processing realized on the imaging device 10 according to the embodiment and a flow of the automatic mask processing. Fig. 8 illustrates, as functional blocks of the imaging device 10, an imaging processing unit 14, an image input unit 15, a region designation position detection unit 16, a region setting unit 22, an image processing unit 23, and an image output unit 24.

[0060] The imaging processing unit 14 executes imaging processing, generates data of a celestial sphere image, and inputs the celestial sphere image data to the image input unit 15. The celestial sphere image generated by the imaging processing unit 14 is input to the image input unit 15. In the embodiment to be described, the imaging device 10 including the imaging processing unit 14 is described as having a function of applying mask processing to the captured celestial sphere image.

[0061] However, the present invention is not limited to this embodiment, and as described in detail below, the automatic mask processing function may be implemented on an image processing device that includes a communication means for communicating with the imaging device 10 as an external imaging device and a reading means for reading data of a removable medium in which the imaging device 10 stores images. In this case, instead of the imaging processing unit 14, a communication means or a reading means is provided, and the spherical image is input to the image input unit 15 from these means.

[0062] In the embodiment to be described, the imaging processing unit 14 performs imaging processing of a celestial sphere image, and the celestial sphere image is input to the image input unit 15. However, the target of the automatic mask processing function is not particularly limited, and an image captured by a normal camera may be the target. In the embodiment to be described, the above-mentioned zenith correction and rotation correction are performed before input to the image input unit 15, but this does not prevent them from being performed after the automatic mask processing.

[0063] The image input unit 15 receives an input of an image to be subjected to mask processing. In the embodiment to be described, the image to be subjected to mask processing is a spherical image. The image input to the image input unit 15 is input to a downstream image processing unit 23 to apply mask processing, and is also input to an area designation position detection unit 16 to determine an area to which the mask processing is applied. The image input unit 15 constitutes an image input means in this embodiment.

[0064] The area designation position detection unit 16 is a means for detecting the position of a predetermined pattern in an input image, and includes a detection unit 17 for detecting a predetermined pattern shown in an input captured image, and a calculation unit 18 for calculating the position of the detected predetermined pattern, as described with reference to FIG. 6. The area designation position detection unit 16 outputs an array of coordinate values ​​indicating the area designation position in the detected image to the area setting unit 22. As described above, the area designation position detection unit 16 can determine the group to which the detected area designation position belongs, in addition to the coordinate values ​​of the area designation position. The area designation position detection unit 16 constitutes the detection means in this embodiment.

[0065] The region setting unit 22 sets a mask processing region (region to be processed) for the target image based on the detected multiple region designation positions. When a group is determined, the region setting unit 22 can set multiple mask processing regions based on the identified group configuration. The region designation position detection unit 16 can also set one mask processing region for each of the identified one or more groups.

[0066] The image processing unit 23 applies image processing, more specifically, mask processing, to a mask processing region in the input image. The mask processing is processing for irretrievably deleting or degrading image information in the mask processing region in the captured image. For example, the image processing unit 23 applies image processing to an image portion of the omnidirectional image corresponding to the mask processing region on the omnidirectional image format. The image processing unit 23 is an image processing unit in this embodiment.

[0067] The image output unit 24 outputs the image to which the mask processing has been applied by the image processing unit 23. The image output unit 24 constitutes an image output means in this embodiment. In Fig. 8, a storage / readout unit 29, a display control unit 13, and a communication unit 11 are further shown as output destinations of the image output unit 24.

[0068] The masked image is stored in the storage / readout unit 29. In response to pressing of the shutter button, a still image or a moving image is captured, and when the captured image is to be stored, the image output unit 24 causes the storage / readout unit 29 to store the generated still image or moving image in the storage unit 1000.

[0069] In response to pressing the shutter button, a still image or video is captured, and when the captured image is to be transmitted to an external device, the image output unit 24 transmits the generated still image or video to the external device via the communication unit 11.

[0070] As described above, the display control unit 13 can display the automatically detected mask processing area superimposed on the display of the input image on the live view screen. Note that the live view may be displayed on a display provided in an information terminal on which a camera control application runs, connected via a network using the communication unit 11.

[0071] Fig. 8 further shows the reception unit 12 as a functional block. In Fig. 8, the reception unit 12 for receiving an area correction instruction is shown as an area correction instruction reception unit 25. The area correction instruction reception unit 25 receives correction instructions such as addition, deletion, movement and grouping of points (area designation points or area designation positions) for defining the mask processing area detected automatically as described above. Here, the correction of grouping refers to an operation of moving a group to which a point for defining the mask processing area belongs from the current group to another group (including the case of defining a new group) or outside the group. The area correction instruction reception unit 25 constitutes a correction reception means in this embodiment.

[0072] The above modification instruction can be preferably accepted during the above-mentioned live view. The modification instruction is given via a user interface for controlling the imaging device 10 (hereinafter, a user interface for camera control). In an embodiment where the live view is displayed on a display unit included in the imaging device 10 itself, the display unit and operation unit 615 (including a touch panel, etc.) included in the imaging device 10 provide the user interface for camera control. Alternatively, in a specific embodiment where the live view is performed on an external display connected via a video interface, etc., the user interface is provided by an external display and a device (input device) for performing various operations on the external display. Alternatively, in another embodiment where the live view is displayed on a display included in an information terminal connected via a network on which a camera control application is executed, the user interface for camera control is provided by a display included in the information terminal and an input device such as a touch screen, mouse, or keyboard. For example, the mask processing area can be manually modified by designating the screen of the display included in the information terminal with a finger or the like. The above modification instruction is not limited to being accepted during the above-mentioned live view, and can also be accepted during video shooting.

[0073] 8 further illustrates a position information DB 1001 as a functional block. The position information DB 1001 stores the positions (area designation positions) of a plurality of predetermined patterns detected in a previously input image. The area setting unit 22 uses the information stored in the position information DB 1001 to estimate vertices that complement a specific shape formed based on the positions of the plurality of predetermined patterns detected by the area designation position detection unit 16.

[0074] During the above-mentioned live view or video shooting, the means for generating the above-mentioned predetermined pattern may be hidden by another subject. In such a case, there is a possibility that the mask processing area may be missing, and a part of the area to be protected may be removed from the mask target and become visible in the image. The position information DB1001 stores the coordinate values ​​of the past area designation positions, and the area setting unit 22 complements the missing range designation positions based on at least a part of the area designation positions stored in the position information DB1001. This makes it possible to more stably determine the area to be protected. The position information DB1001 constitutes the storage means in this embodiment. In the embodiment described, the area setting unit 22 complements the missing range designation positions based on at least a part of the area designation positions stored in the position information DB1001. However, in other embodiments, the estimation unit 21, which is solely responsible for the function of complementing, may complement the missing area designation positions based on at least a part of the area designation positions stored in the position information DB1001, and provide them to the area setting unit 22. In this case, the area setting unit 22 receives the missing area designation position and sets the area to be processed based on at least a part of the stored past position.

[0075] 8, the reception unit 12 for receiving a mask type designation is further shown as a mask type designation reception unit 26. The mask type designation reception unit 26 receives a designation of a type of mask processing to be applied to the above-mentioned mask processing region from the photographer. The mask type designation reception unit 26 receives the designation of the type of mask processing via the above-mentioned user interface for camera control.

[0076] Processing or operation of the embodiment The automatic mask processing according to the embodiment will be described in more detail below with reference to Fig. 9 to Fig. 15. Fig. 9 is a flowchart showing the automatic mask processing executed by the imaging device 10 according to the embodiment. Note that the processing shown in Fig. 9 is a processing flow when the processing target is a moving image, and a predetermined process is repeated for each frame from the first frame to the last frame.

[0077] The process shown in FIG. 9 starts from step S100 in response to an instruction to start shooting by a photographer pressing a shutter button, for example.

[0078] In step S101, the imaging device 10 determines the type of mask to be applied. As described above, the mask type is specified by the photographer to the mask type designation receiving unit 26 (receiving unit 12) via an appropriate camera control user interface prior to the start of imaging, and is stored as a setting value in, for example, a register. In step S101, the setting value is read out, and the type of mask to be applied is determined.

[0079] In step S102, the imaging device 10 receives an image of the current frame input to the image input unit 15 and acquires an image to be processed.

[0080] In step S103, the imaging device 10 causes the area designation position detection unit 16 to detect the position of a predetermined pattern (area designation position) in the image of the frame to be processed.

[0081] As described above, in this embodiment, a photographer who intends to capture a spherical image or a manager of a subject to be captured performs an operation of physically and three-dimensionally disposing a pattern generating means that generates a predetermined pattern around a subject that the photographer considers should be protected.

[0082] Fig. 10 is a diagram for explaining how a photographer or an administrator specifies a mask processing area and an image to which the mask processing is applied in that case. Fig. 10(A) shows a schematic diagram of a celestial sphere image 200 before the mask is applied, which is captured when a photographer or an administrator specifies a mask processing area.

[0083] In the omnidirectional image 200, a certain indoor space 202 is captured, and in the indoor space 202, there is a subject (such as a desk and paper documents on it in the example of FIG. 10(A)) 204 that the photographer does not want to appear in the image. In this case, as illustrated in FIG. 10(A), the photographer places, for example, an LED device (light-emitting device) 206 three-dimensionally around the subject 204 to be protected. In the example shown in FIG. 10(A), six LED devices 206a to 206f are included in the capture range.

[0084] The LED device 206 is an example of a light-emitting device that generates bright spots of a predetermined color, and is a device that is placed by a photographer or the like who intends to use this automatic mask processing function to specify a mask processing area in advance. For the sake of convenience of explanation, it is assumed that bright spots of a predetermined color alone form a pattern that specifies a mask range, but it is desirable for the predetermined pattern to be a pattern that can be distinguished from single-color lamps such as red and green that are used in various electronic devices, and as described above, it may be composed of a light intensity (blinking) pattern, a color of a bright spot, an arrangement of multiple bright spots, or a combination of these.

[0085] The LED device 206 is an example of a pattern generating means that is arranged within the field of view of the image capturing processor 14 and generates a predetermined pattern, but the pattern generating means is not particularly limited. The pattern generating means may be selected from the group consisting of a medium on which a one-dimensional or two-dimensional code is printed, a display device that displays a one-dimensional or two-dimensional code, a light emitting device that emits one or more bright spots, and a laser pointer that projects one or more bright spots. The pattern generating means may be a stand type, a ceiling type, or a batch type.

[0086] Examples of the light emitting device include the LED device 206 that emits one or more bright spots as described above, as well as wire lights and mesh lights in which light emitting elements such as LEDs are attached to wires or meshes. The medium on which the one-dimensional code or two-dimensional code is printed may be a panel or cube having a surface on which the one-dimensional code or two-dimensional code is printed. Since it is expected that the imaging device 10 will move or change its orientation during shooting, it is desirable to have multiple surfaces on which the code is printed so that it can be recognized from multiple viewpoints.

[0087] As shown in FIG. 10(A), the region-designated position detection unit 16 detects the coordinate values ​​of the center positions of bright spots 208a to 208f that correspond to the light-emitting portions of the plurality of LED devices 206a to 206f in the spherical image 200.

[0088] In addition, if multiple frames are required for detection, such as a temporal pattern (e.g., a light intensity (blinking) pattern of a bright spot), and sufficient frames have not yet been input, the area designation position may not be detected in the processing of step S103.

[0089] In step S104, the imaging device 10 stores the detected area designation position in the position information DB 1001. In step S105, the imaging device 10 determines whether or not a previously detected area designation position has disappeared. Note that in the initial frame, since no previously detected position is stored, it is determined that no area designation position has disappeared. If it is determined in step S105 that the area designation position has not disappeared (NO), the process proceeds to step S107.

[0090] In step S107, the imaging device 10 sets a mask processing area based on the detected area designation position using the area setting unit 22. In step S108, the imaging device 10 applies a designated type of mask processing to an image portion of the set mask processing area using the image processing unit 23. In step S109, the imaging device 10 outputs an image in which mask processing has been applied to the mask processing area from the image output unit 24. When shooting a moving image, the storage unit 1000 via the storage / readout unit 29 is the output destination. When checking an image during moving image shooting, the display control unit 122 is also the output destination.

[0091] FIG. 10B is a schematic diagram showing a celestial sphere image 210 after a mask is applied, which is captured with a mask processing region designated by a photographer or an administrator. The example of FIG. 10B illustrates a case where a black-painted mask processing is applied to a single polygonal region connecting the coordinate points of the region designation positions. As shown in FIG. 10B, in the celestial sphere image 210 after the mask is applied, a polygonal region 214 defined by connecting the center positions of the bright spots 208a to 208f corresponding to the light-emitting parts of the above-mentioned LED devices 206a to 206f is painted black, and image information of the subject 204 that would normally be in the region 214 is removed. In this way, an image in which a region defined by connecting the positions of a plurality of predetermined patterns present in an input image is masked is output.

[0092] FIG. 11A is a schematic diagram showing a celestial sphere image 220 before application of a mask, which is captured when two mask processing regions are designated in advance by a photographer or an administrator.

[0093] 10, a certain indoor space 222 is captured in the omnidirectional image 220, and two subjects 224 and 234 that the photographer does not want to be captured in the image are present in the indoor space 222. In the example of Fig. 11(A), the first subject 224 to be protected is a desk and paper documents on it, and the second subject 234 to be protected is the interior of an adjacent room.

[0094] In this case, the photographer places a first group of LED devices 226 that generate a first pattern around a first subject 224 to be protected, as illustrated in Fig. 11(A). In the example shown in Fig. 11(A), six first LED devices 226a to 226f are included in the shooting range. A second LED device 236 that generates a second pattern is placed around a second subject 234 to be protected. In the example shown in Fig. 11(A), four second LED devices 236a to 236d are included in the shooting range.

[0095] The first LED devices 226a-226f and the second LED devices 236a-236d emit bright spots of different patterns, and are devices that are placed by a photographer or the like who intends to use this automatic mask processing function to specify a mask processing area before shooting. For the sake of convenience of explanation, it is assumed that the mask range is specified by bright spots of different colors that are individually configured as different patterns, as in FIG. 10.

[0096] 11(A), the region-designated position detection unit 16 detects coordinate values ​​of the center positions of the bright spots 228a-228f, 238a-238d corresponding to the light-emitting portions of the LED devices 226a-226f, 236a-236d in the spherical image 200. The first bright spots 228a-228f and the second bright spots 238a-238d have different colors and therefore represent different identifiers, and are grouped into two groups.

[0097] In this case, in step S107, the area setting unit 22 sets different mask processing areas for multiple groups that include area designation positions having identifiers that each satisfy a predetermined condition (in this example, the condition is that the identifier is the same, but identifiers within a predetermined numerical range may also be grouped together).

[0098] FIG. 11(B) is a schematic diagram of the omnidirectional image 240 after the mask is applied. Note that, like FIG. 10, the example of FIG. 11(B) also illustrates a case where a black-painted mask process is applied to each of single polygonal regions connecting the coordinate points of the region designation positions of the same group. As shown in FIG. 11(B), in the omnidirectional image 240 after the mask is applied, a polygonal region 244 defined by the center positions of the bright spots 228a to 228f corresponding to the light-emitting portions of the first LED devices 226a to 226f described above and a polygonal region 246 defined by the center positions of the bright spots 238a to 238d corresponding to the light-emitting portions of the second LED devices 236a to 236f described above are painted black. As a result, image information of the subjects 224 and 234 that would otherwise be in these regions 244 and 246 is removed.

[0099] Fig. 12(A) and Fig. 12(B) are diagrams for explaining different types of mask processing. Note that S1 in Fig. 12(A) and S2 in Fig. 12(B) are the rectangular area S shown in Fig. 11(B) cut out and enlarged.

[0100] Fig. 12(A) shows a state in which a defined polygonal region 252 (corresponding to region 244 in Fig. 11(B)) in the omnidirectional image 240 after the application of the mask is blurred instead of painted black. As shown in Fig. 12(A), the image information of the subject 224 in this region 244 is not completely removed by the blurring process, but the original detailed image information is removed in a form that is unreadable and unrecoverable. Fig. 12(B) shows a state in which a defined polygonal region 254 (corresponding to region 244 in Fig. 11(B)) in the omnidirectional image 240 after the application of the mask is whitened instead of painted black.

[0101] FIG. 12(C) illustrates a portion of the omnidirectional image before masking is applied when a two-dimensional code panel (signboard or stand type) 260 is used instead of an LED device that generates a bright spot of a predetermined color. S3 in FIG. 12(C) is an enlarged view of a range corresponding to a rectangular region S of a omnidirectional image captured in a state where a panel 260 is placed instead of an LED device 226 in the same indoor space as that shown in FIG. 11(B). As shown in FIG. 12(C), a two-dimensional code panel 260 may be physically placed instead of an LED device. In that case, the mask target area is specified by the center position of the two-dimensional code on the panel 260 instead of the center position of the bright spot of the LED device. In the example shown in FIG. 12(C), six panels 260a to 260f are included in the captured range.

[0102] FIG. 13(A) illustrates a portion of a celestial sphere image before a mask is applied in the case where depth information is encoded using LED devices that generate bright spots of multiple colors. In the celestial sphere image illustrated in FIG. 13(A), an indoor space is captured as in FIG. 10, and a subject 274 that the photographer does not want to capture in the image is present in the indoor space. In the example illustrated in FIG. 13(A), the photographer places LED devices 276 around a subject 274 to be protected, and for the same subject, a first group of LED devices 276a-276d that emit bright spots of a first color and a second group of LED devices 276e-276h that emit bright spots of a second color different from the first color are placed. The colors of the bright spots of the LED devices 276 in each group are the same within the group and different between the groups, and for convenience of explanation, the groups are identified by the different colors of the bright spots as in FIG. 10.

[0103] In this case, in step S103, the region designation position detection unit 16 detects, as separate groups, the coordinate values ​​of the center positions of the bright spots 278a-208d corresponding to the light-emitting parts of the LED devices 276a-276d in the omnidirectional image and the coordinate values ​​of the center positions of the bright spots 278e-208h corresponding to the light-emitting parts of the LED devices 276e-276h, as shown in Fig. 13(A). Furthermore, it is determined in advance that each group represents a surface with a different depth, and the center positions of the bright spots 278a-278d of the first group are recognized as region designation positions that define the front surface 280 indicated by the dashed line, and the center positions of the bright spots 278e-278h of the second group are recognized as region designation positions that define the back surface 282 indicated by the dashed line. In step S107, the region setting unit 22 sets a mask processing region based on the depth information. As shown in Fig. 13(A), the area designation positions can be divided into a plurality of groups, and a single mask processing area can be defined by the plurality of groups. In this case, the front surface 280 and the back surface 282 can be treated as the surfaces of a predetermined solid (e.g., a cube) and can complement the other surfaces of the solid.

[0104] In the description, each depth information is identified by color, but by expressing the depth information by graphically representing the arrangement of the LED devices, polygonization can be favorably assisted. For example, it is possible to define that an area surrounded by a group of LED devices arranged in a "circle" shape represents the near side, and an area surrounded by a group of LED devices arranged in a "cross" shape represents the far side. Similarly, polygonization may be assisted by encoding the depth information by a two-dimensional code instead of a pattern of bright spots. For example, it is possible to define that an area surrounded by a group of two-dimensional codes representing a first identifier represents the near side, and an area surrounded by a group of two-dimensional codes representing a second identifier represents the far side. Alternatively, each vertex of a cube (for example, a cube or a rectangular parallelepiped) may be represented by a different identifier.

[0105] By encoding the depth information in a predetermined pattern, it becomes possible to appropriately respond to a movement or change in orientation of the imaging device 10 during imaging.

[0106] 9 again, in step S110, the imaging device 10 determines whether or not the current frame is the final frame. If it is determined in step S110 that the current frame is not yet the final frame (NO), control returns to step S102 to continue processing the next frame and onward. On the other hand, if, for example, an instruction to end shooting has been given in step S110 and it is determined that the current frame is the final frame (YES), the process proceeds to step S111 and ends this process.

[0107] Referring back to step S105, if it is determined in step S105 that at least one area designation position has disappeared (YES), the process proceeds to step S106. In step S106, the imaging device 10 uses past values ​​to complement the area designation position whose disappearance has been detected by the area setting unit 22 or the estimation unit 21, and proceeds to step S107.

[0108] FIG. 13(B) shows a portion of the celestial sphere image before the mask is applied when a region designation position detected in a previous frame has disappeared. FIG. 7 shows a data structure of a position information management table stored in the position information DB 1001. First, as in FIG. 13(A), it is assumed that the position information management table shown in FIG. 7 is stored when multiple region designation positions have been detected in the previous frame. FIG. 15 shows another example of the data structure of a position information management table stored in a storage unit according to an embodiment. The position information management table includes a region designation position number, an identifier, and coordinate values ​​on the celestial sphere image, and is provided for each designated position recognized as the same by tracking. The example shown in FIG. 13(B) shows a state in which a bright spot 278d of a front LED device 276d previously detected in the frame has been blocked by an obstruction 290, and one region designation position has disappeared.

[0109] In this case, if the amount of change in the center positions of the bright spots 278a-278c of the other LED devices 276a-276c that have not disappeared is less than a predetermined threshold, it is determined that the position of the disappeared bright spot 278d has not changed significantly from its position in the past frame, so the past position is taken as the estimated position of the area designation position, and the front surface 280 is specified. On the other hand, if the amount of change in the center positions of the bright spots 278a-278c of the other LED devices 276a-276c that have not disappeared is equal to or greater than a predetermined threshold, it is determined that the shooting viewpoint or direction has moved, so a change in the direction of the shooting field of view is estimated based on the amount of movement of the coordinates of each area designation position that has not disappeared between frames, and the disappeared area designation position is estimated after taking into account a predetermined amount of movement relative to the position in the past frame, and the front surface 280 is specified.

[0110] In this way, there are cases where the two-dimensional code or LED device cannot be recognized temporarily within the field of view when the imaging device 10 moves, when an object moves and blocks the camera's field of view, etc. In response to this, it is possible to deal with the temporary disappearance of the specified pattern by retaining the position information once recognized as it is, detecting the movement in the direction of the shooting field of view using past information on other area designation positions, and continuing to estimate the blocked area designation position by calculation.

[0111] Fig. 14 is a schematic diagram showing an example of a detection position of a predetermined pattern according to an embodiment. In the example of Fig. 14, compared with the position information shown in Fig. 7, the area designation position number "8" is missing, so the area setting unit 22 or the estimation unit 21 estimates the area designation position number "8" as a vertex to be complemented.

[0112] The processing flow during video shooting has been described above with reference to Figs. 9 to 15. In this case, as described above, during video shooting, the processing from steps S102 to S109 is repeated for each frame from the first frame to the last frame. In the case of live view during still image shooting, the output destination of step S109 shown in Fig. 9 may be changed to the display control unit 122. During still image shooting, the processing from steps S102 to S109 shown in Fig. 9 may be changed so as to be performed for a single frame, and the output destination of step S109 may be set to the storage unit 1000. Furthermore, when applying mask processing to a stored still image or video, rather than during shooting or live view, an input of an image read from a recording medium may be received in step S102.

[0113] The above describes an embodiment in which an automatic mask processing function is provided on an image capture device 10 having an image capture function. However, as described above, the present invention is not limited to the above embodiment. In another embodiment, the automatic mask processing function may be implemented on an image processing device having a communication means for communicating with the image capture device 10 as an external image capture device or a reading means for reading data from a removable medium in which the image capture device 10 stores images.

[0114] Modifications of the embodiment Next, an image processing system according to a modified example of the embodiment will be described. The same configurations and functions as those of the above embodiment will be denoted by the same reference numerals, and the description thereof will be omitted. The image processing system according to the modified example of the embodiment is a system in which a mask process for a captured image acquired by an imaging device 10a is executed by a PC 50 connected to the imaging device 10a.

[0115] System Configuration Fig. 16 is a diagram for explaining an example of an outline of an image processing system according to a modified example of the embodiment. As shown in Fig. 16, the image processing system according to the modified example of the embodiment includes a PC 50 connected to an imaging device 10a.

[0116] The PC 50 is an image processing device used by a user to perform image processing on a captured image. An editing application program for editing an image to be processed, such as a spherical image, or a control application program for controlling the imaging device 10a is installed in the PC 50. The PC 50 is connected to the imaging device 10 via a cable such as a USB or an HDMI (High-Definition Multimedia Interface, HDMI is a registered trademark).

[0117] The imaging device 10a and the PC 50 may communicate wirelessly using a short-range wireless communication technology such as Bluetooth (registered trademark), NFC, or Wi-Fi (registered trademark) without using the above-mentioned cable. The imaging device 10a and the PC 50 may communicate via a communication network such as the Internet, a mobile communication network, or a LAN (Local Area Network), and the PC 50 may be located at a different location from the imaging device 10a. In this case, the communication network may include not only wired communication but also wireless communication such as 4G (4th Generation), 5G (5th Generation), LTE (Long Term Evolution), Wi-Fi (registered trademark), or WiMAX (registered trademark) (Worldwide Interoperability for Microwave Access). The communication network may also include a network built by a blockchain.

[0118] The PC 50 may be a smartphone or a tablet terminal, etc. The PC 50 may be constructed by a single computer, or may be constructed by multiple computers to which each unit (function, means, or storage unit) is divided and arbitrarily assigned.

[0119] ●Hardware configuration Fig. 17 is a diagram showing an example of a hardware configuration of a PC according to a modified embodiment. The PC 50 is constructed by a computer, and includes a CPU 501, a ROM 502, a RAM (Random Access Memory) 503, a HD (Hard Disk) 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516, as shown in Fig. 17.

[0120] Among these, the CPU 501 controls the operation of the entire PC 50. The ROM 502 stores programs such as an IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data from the HD 504 under the control of the CPU 501. The display 506 displays various information such as a cursor, a menu, a window, a character, or an image. The display 506 is an example of a display unit. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external device is, for example, the imaging device 10. The network I / F 509 is an interface for data communication using a communication network. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG. 17.

[0121] Moreover, the keyboard 511 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513 as an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R or a Blu-ray Disc, etc. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515 such as a flash memory.

[0122] Each of the above programs may be recorded in a computer-readable recording medium as an installable or executable file and distributed. Examples of the recording medium include CD-R, DVD, Blu-ray Disc, SD card, USB memory, etc. The recording medium may be provided domestically or internationally as a program product. For example, the PC 50 realizes the image processing method according to the present invention by executing the program according to the present invention.

[0123] ●Function configuration 18 is a diagram showing an example of a functional configuration of an image processing system according to a modified example of the embodiment. The imaging device 10a has a communication unit 11, a reception unit 12, a display control unit 13, an imaging processing unit 14, and a storage / readout unit 29. Each of these units has the same function as each unit of the imaging device 10 shown in FIG.

[0124] The PC 50 has a communication unit 51, a reception unit 52, a display control unit 53, a detection unit 55, a calculation unit 56, an estimation unit 58, a region setting unit 59, an image processing unit 61, and a storage / reading unit 69. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 17 operates according to an instruction from the CPU 501 in accordance with a PC program expanded on the RAM 503. The PC 50 also has a storage unit 5000 constructed by the ROM 502, HD 504, or recording medium 515 shown in Fig. 17. A location information management DB 5001 constructed by a location information management table as shown in Fig. 7 is constructed in the storage unit 5000.

[0125] The communication unit 51 is mainly realized by the processing of the CPU 501, and communicates various data or information with the imaging device 10a. The communication unit 51 performs data communication with the imaging device 10a via a communication network, for example, using a network I / F 509. The communication unit 51 also performs data communication with the imaging device 10a via various cables, for example, using an external device connection I / F 508.

[0126] Reception unit 52 is mainly realized by processing of CPU 501 on keyboard 511 or pointing device 512, and receives various selections or inputs from a user. Display control unit 53 is mainly realized by processing of CPU 501, and displays various images on display 506. Reception unit 52 and display control unit 53 have the same functions as reception unit 12 and display control unit 13 of imaging device 10 shown in FIG. 7, respectively.

[0127] The detection unit 55, the calculation unit 56, the estimation unit 58, the region setting unit 59, and the image processing unit 61 are each realized by the processing of the CPU 501, and have the same functions as the detection unit 17, the calculation unit 18, the estimation unit 21, the region setting unit 22, and the image processing unit 23 of the imaging device 10 shown in FIG. 6. The storage / reading unit 69 is mainly realized by the processing of the CPU 501, and stores various data (or information) in the storage unit 5000, and reads various data (or information) from the storage unit 5000. The storage unit 5000 also stores captured image data acquired from the imaging device 10a or processed image data processed by the image processing unit 61. Note that the image data stored in the storage unit 5000 may be deleted when a predetermined time has passed since it was stored, or the data transmitted to another device or terminal may be deleted.

[0128] Processing or operation of modified embodiments Next, the processing or operation of the image processing system according to the modified example of the embodiment will be described with reference to Fig. 19. Fig. 19 is a sequence diagram showing an example of mask processing for an imaging device in the image processing system according to the modified example of the embodiment.

[0129] First, the imaging processing unit 14 of the imaging device 10a starts imaging processing for a predetermined area (step S51). Specifically, the photographer presses an input means such as a shutter button, and the reception unit 12 receives a request to start imaging. Then, in response to the request to start imaging received by the reception unit 12, the imaging processing unit 14 starts imaging processing. Next, the communication unit 11 of the imaging device 10a transmits the captured image data acquired in the imaging processing of step S51 to the PC 50 (step S52). As a result, the communication unit 51 of the PC 50 receives the captured image data transmitted from the imaging device 10a and accepts input of the image to be processed.

[0130] The PC 50 performs image processing on the image input by the communication unit 51 (step S53). The image processing in step S53 is a mask processing similar to the processing shown in FIG. 9. In particular, the display control unit 53 of the PC 50 may display the input captured image on the display 506 during the image processing in step S53, thereby allowing the user of the PC 50 to check the state of the captured image. The reception unit 52 of the PC 50 may also receive a selection of processing content for the captured image or processed image displayed on the display 506. In this case, the reception unit 52 may receive, for example, a selection of a mask type by the user, or a selection of a correction of the mask type for the processed image that has been subjected to mask processing.

[0131] In this way, the image processing system according to the modified embodiment uses the PC 50 to perform image processing on the image acquired by the imaging device 10a, making it possible to check the captured image in real time and to perform corrections to the mask processing while checking the captured image or the processed image.

[0132] Although the images obtained by the imaging elements 103a and 103b are partial images, the imaging processing unit 14 on the imaging device 10 can generate a celestial sphere image through the process described in FIG. 5 and input the celestial sphere image to the PC 50. In another embodiment, the imaging processing unit 14 on the imaging device 10 can input the two partial images to the PC 50 while leaving the two partial images as they are before the celestial sphere image is generated through the process described in FIG. 5. When the two partial images before the celestial sphere image is generated are input, the PC 50 generates the celestial sphere image by performing a process equivalent to the process described in FIG. 7. The process after the celestial sphere image is generated is the same as that described with reference to the above drawings.

[0133] ●Summary● As described above, according to the above-mentioned embodiments, it is possible to provide an image processing device, an image processing system, an image processing method, and a program that enable a user to recognize a pre-specified area of ​​a captured image to be processed and apply image processing to the captured image. An image processing device (e.g., the imaging device 10 or the PC 50) according to one embodiment of the present invention receives an input of a captured image and detects the position of a predetermined pattern shown in the input captured image. Then, the image processing device sets an area to be processed based on the detected positions of the multiple predetermined patterns, and performs image processing on the set area. This makes it possible to recognize the application range of the mask processing pre-specified in the image and apply the mask processing to the image.

[0134] In the above description, it has been assumed that the imaging processor 14 captures a celestial sphere image and inputs the celestial sphere image to the image input unit 15. The automatic mask processing function is particularly suitable for wide-angle images (celestial sphere image, hemispherical image, panoramic image) captured by the imaging device 10 or the like, which are different from normal cameras in which the photographer can easily remove undesirable objects from the shooting range by adjusting the shooting direction. This is because, in particular, when shooting with the imaging device 10, a wide range is the shooting range, and it is not easy to remove a subject to be protected from the shooting range, and the protected subject may also be dispersed over a wide range. By applying this automatic mask processing function to the celestial sphere image, the photographer can shoot the celestial sphere image with peace of mind. However, the automatic mask processing function itself may be similarly applied to images captured by a camera with a normal angle of view.

[0135] In addition, from the viewpoint of the target of the mask processing, the automatic mask processing function according to the present embodiment can be particularly suitably applied to a target whose characteristics are difficult to define in advance. When the characteristics of the target to be protected are clear in advance, such as a human face or body, they can be recognized by face recognition, body recognition or other object recognition technology. However, it may be difficult to define the characteristics of the target to be protected in advance. For example, in the case of content that guides the inside of a factory by a tour using a 360-degree image, a wide variety of subjects may exist in the factory, and it is generally difficult to make such subjects recognizable in advance by an object recognizer.

[0136] Furthermore, in the case of a fixed camera (including cases where the camera orientation is variable) with a fixed installation location such as a surveillance camera, it is easy to specify a mask for a specified coordinate range on the image, and the automatic mask processing function of this embodiment is particularly suitable for images captured by a photographing device with an indefinite installation location such as a handheld camera or a camera fixed on a tripod.

[0137] The automatic mask processing function according to this embodiment can also be suitably applied to real-time distribution using the imaging device 10. In particular, when distributing, there is a possibility that the imaging device 10 will be moved, and by using the automatic mask processing function, it becomes possible to hide three-dimensional areas, making it possible to perform real-time distribution with peace of mind.

[0138] ●Additional Information● Each function of the above-described embodiment can be realized by one or more processing circuits. Here, the "processing circuit" in the present embodiment includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and devices such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), a SOC (System on a chip), a GPU, and a conventional circuit module designed to execute each function described above.

[0139] In addition, the above functions can be realized by a computer-executable program written in legacy programming languages ​​such as assembler, C, C++, C#, Java (registered trademark), or object-oriented programming languages, and can be stored on a device-readable recording medium such as ROM, EEPROM, EPROM, flash memory, flexible disk, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, Blu-ray Disc, SD card, or MO, or distributed via telecommunications lines.

[0140] In addition, the various tables in the above-described embodiment may be generated by the learning effect of machine learning, and the data of each associated item may be classified by machine learning, making it unnecessary to use tables. Here, machine learning is a technology for making a computer acquire human-like learning capabilities, and refers to a technology in which a computer autonomously generates an algorithm required for judgment such as data identification from learning data previously acquired, and applies this to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods, and any learning method for machine learning may be used.

[0141] So far, an image processing device, an image processing system, an image processing method, and a program according to one embodiment of the present invention have been described. However, the present invention is not limited to the above-described embodiment, and other modifications, changes, or deletions, etc., may be made within the scope of what a person skilled in the art can conceive. In any aspect, as long as the functions and effects of the present invention are achieved, it is within the scope of the present invention. [Explanation of symbols]

[0142] 1...imaging system, 10...imaging device, 50...image processing device (PC), 11,51...communication unit, 12,52...reception unit, 13,53...display control unit, 14...imaging processing unit, 15...image input unit, 16...area designation position detection unit, 17,55...detection unit, 18,56...calculation unit, 21,58...estimation unit, 22,59...area setting unit, 23,61...image processing unit, 24...image output unit, 25...area correction instruction reception unit, 26...mask type designation reception unit, 29,69...storage / readout unit, 101...imaging unit, 104...image processing unit, 105...imaging control unit, 108...microphone, 109...sound processing unit, 111...CPU, 112...ROM, 113...SRAM , 114...DRAM, 115...operation unit, 116...input / output I / F, 117...short-range communication circuit, 117a...antenna, 118...electronic compass, 119...gyro sensor, 120...acceleration sensor, 121...network I / F, 501...CPU, 502...ROM, 503...RAM, 504...HD, 505...HDD controller, 506...display, 508...external device connection I / F, 509...network I / F, 510...data bus, 511...keyboard, 512...pointing device, 514...DVD-RW drive, 516...media I / F, 1000,5000...storage unit, 1001,5001...location information management DB [Prior art documents] [Patent documents]

[0143] [Patent Document 1] JP 2016-133908 A

Claims

1. an image input means for receiving an input of a captured image; a detection means for detecting a position of a predetermined pattern shown in a captured image input to the image input means; a setting means for setting a processing target area based on the positions of the plurality of predetermined patterns detected by the detection means; an image processing means for performing image processing on the area set by the setting means; Including, An image processing device, wherein the region includes a region that includes a polygon defined by connecting at least the positions of the plurality of predetermined patterns, or a region that includes at least a convex hull of the positions of the plurality of predetermined patterns.

2. The image processing device according to claim 1 , wherein the plurality of predetermined patterns are each a one-dimensional code in a captured image, a two-dimensional code in a captured image, a pattern of bright spots in a captured image, or a temporal pattern of bright spots across images captured at multiple points in time.

3. The image processing device according to claim 1 or 2, wherein the image processing includes at least one image effect selected from the group consisting of blurring, filling with a predetermined color or image pattern, mosaic processing, and overwriting with a graphic object.

4. The image processing device according to any one of claims 1 to 3, characterized in that the plurality of specified patterns include different patterns, the detection means detects an identifier that identifies the pattern along with the position, and the setting means sets different processing target areas for a plurality of groups each composed of positions having an identifier that satisfies a specified condition.

5. The image processing device according to any one of claims 1 to 3, characterized in that the plurality of specified patterns include different patterns, the detection means detects depth information corresponding to the pattern along with the position, and the setting means sets the area based on the depth information.

6. 6. An image processing device according to claim 1, further comprising: a storage means for storing the positions of a plurality of predetermined patterns detected in a captured image input in the past; and a means for setting the area based on at least a portion of the stored past positions.

7. a display control means for displaying an input captured image and the set area; a modification receiving means for receiving instructions for adding, deleting, moving, and grouping points for defining the area; The image processing device according to any one of claims 1 to 6, further comprising:

8. a type receiving means for receiving a designation of the type of the image processing to be applied to the area; The image processing device according to any one of claims 1 to 7, further comprising:

9. The image processing device includes: A communication means for communicating with an external image capture device is provided, and the communication means is 9. The image processing device according to claim 1, which inputs a captured image.

10. 10. The image processing device according to claim 1, wherein the region to be processed is a mask processing region, and the image processing is mask processing.

11. The image processing device according to any one of claims 1 to 10, wherein the captured image is a celestial sphere image, a hemispherical image, or a horizontal 360-degree image.

12. An image processing device, an image input means for receiving an input of a captured image; an image output means for outputting an image in which a region including a polygon defined by connecting the positions of a plurality of predetermined patterns shown in the captured image input to the image input means, or a region including at least a convex hull of the positions of the plurality of predetermined patterns has been image-processed; 23. An image processing device comprising:

13. An imaging means for capturing an image; a detection means for detecting a position of a predetermined pattern shown in an image captured by the imaging means; a setting means for setting an area to be processed based on the positions of the plurality of predetermined patterns detected by the detection means; an image processing means for performing image processing on the area set by the setting means; Including, An image processing system, wherein the region includes a region that encompasses a polygon defined by connecting at least the positions of the plurality of predetermined patterns, or a region that encompasses at least the convex hull of the positions of the plurality of predetermined patterns.

14. The image processing system further comprises: A plurality of pattern generating means arranged within the field of view of the imaging means, each of which generates the predetermined pattern. The image processing system of claim 13 .

15. The image processing system according to claim 14, wherein each of the pattern generating means is selected from the group consisting of a medium on which a one-dimensional code or a two-dimensional code is printed, a display device that displays a one-dimensional code or a two-dimensional code, a light-emitting device that emits one or more bright spots, and a laser pointer that projects one or more bright spots.

16. The computer Receiving an input of a captured image; detecting a position of a predetermined pattern shown in an input captured image; setting a processing target area based on the positions of the detected plurality of predetermined patterns; performing image processing on the set area; Including, An image processing method, wherein the region includes a region that includes a polygon defined by connecting at least the positions of the plurality of predetermined patterns, or a region that includes at least a convex hull of the positions of the plurality of predetermined patterns.

17. Computer, A detection means for detecting a position of a predetermined pattern shown in an input captured image; a setting means for setting a processing target area based on the positions of the plurality of predetermined patterns detected by the detection means; image processing means for performing image processing on the area set by the setting means; Functioning as a The program, wherein the region includes a region that includes a polygon defined by connecting at least the positions of the plurality of predetermined patterns, or a region that includes at least a convex hull of the positions of the plurality of predetermined patterns.

Citation Information

Patent Citations

  • Image input device provided with image processing function and recording medium recording its image processing program

    JP2000278584A

  • Image photographing system

    JP2006217161A

  • Image processing device

    JP2007243509A

  • Image processing apparatus, and method and program therefor

    JP2015179984A

  • Image forming device and image forming method

    JP2016133908A