Image processing device, image processing system, image processing method, image processing program, and recording medium

Abstract

One embodiment according to the technology of the present disclosure provides an image processing device, an image processing system, an image processing method, and an image processing program that cause an output device to output a warning on the basis of an index regarding joining of moving images. An image processing device according to one aspect of the present invention comprises a processor, wherein the processor performs image acquisition processing for acquiring a plurality of moving images that have an overlapping image capture range, information acquisition processing for acquiring information pertaining to a stitching region used for joining the plurality of moving images, index calculation processing for calculating an index based on the distance between a subject and a stitching line that is set for the stitching region, and warning processing for outputting a warning from the output device on the basis of the index.

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing system, an image processing method, and an image processing program for processing moving images. [Background technology]

[0002] Regarding technology for capturing moving images, for example, Patent Document 1 describes a technique for issuing a warning when a panoramic image is poorly composed. Also, Patent Document 2 describes processing when a foreground and background image appear at the boundary of a composite image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-004340 [Patent Document 2] Japanese Patent Application Publication No. 2018-142963 Summary of the Invention

[0004] One embodiment of the technique of the present disclosure provides an image processing device, an image processing system, an image processing method, and an image processing program that are related to joining moving images and cause an output device to output a warning based on an index. [Means for solving the problem]

[0005] An image processing device according to a first aspect of the present invention is an image processing device equipped with a processor, which performs an image acquisition process for acquiring multiple moving images with overlapping imaging ranges, an information acquisition process for acquiring information on a stitching area used to stitch together the multiple moving images, an index calculation process for calculating an index based on the distance between the subject and a stitching line included in the stitching area, and a warning process for outputting a warning from an output device based on the index.

[0006] In the image processing device according to the second aspect of the present invention, in the first aspect, the processor performs a determination process of determining whether or not the subject is within the stitching region based on the index.

[0007] An image processing device according to a third aspect is the image processing device of the second aspect, wherein the processor causes the output device to output a warning based on the determination processing in the warning processing.

[0008] In the image processing device of the fourth aspect, in any one of the first to third aspects, the processor calculates the time that the subject remains in the stitching area during the index calculation process, and calculates an index based on the distance and the calculated time.

[0009] An image processing device according to a fifth aspect is any one of the first to fourth aspects, wherein the processor calculates an index based on the size of a subject in a moving image in the index calculation process.

[0010] An image processing device according to a sixth aspect is the fifth aspect, wherein the processor determines the size of the subject based on the size of an area showing at least a part of a person.

[0011] The image processing device according to the seventh aspect is any one of the first to sixth aspects, in which the processor determines whether the subject is a moving object in the index calculation process, and calculates an index if the subject is determined to be a moving object.

[0012] An image processing device according to an eighth aspect is the seventh aspect, wherein the processor determines that the object is a moving object when a person is recognized.

[0013] An image processing device according to a ninth aspect is the seventh or eighth aspect, wherein the processor outputs information indicating a direction in which the subject should move in the warning process.

[0014] The image processing device of the 10th aspect is any one of the 1st to 9th aspects, in which the processor, in the information acquisition process, acquires common areas among multiple moving images as information of a stitching area, and sets a stitching line in the stitching area based on user operation.

[0015] In the image processing device of the 11th aspect, in any one of the 1st to 10th aspects, the processor assigns stitching area information to multiple moving images displayed in live view during the information acquisition process, and starts recording the multiple moving images after assigning the stitching area information.

[0016] The image processing device according to the twelfth aspect is any one of the first to eleventh aspects, in which the processor performs a synthesis process of connecting multiple moving images at stitching regions or stitching lines to generate a composite moving image.

[0017] An image processing device according to a thirteenth aspect is the twelfth aspect, wherein the processor generates a panoramic moving image in the synthesis process.

[0018] An image processing device according to a fourteenth aspect is the thirteenth aspect, wherein the processor causes the display device to display the composite moving image.

[0019] An image processing system according to a fifteenth aspect of the present invention includes the image processing device according to any one of the first to fourteenth aspects, an imaging device that captures a plurality of moving images, and an output device.

[0020] The image processing system according to a sixteenth aspect is the fifteenth aspect, wherein the output device outputs a warning by at least one of a screen display, a sound output, and a vibration.

[0021] An image processing method according to a 17th aspect of the present invention is an image processing method executed by an image processing device having a processor, wherein the processor executes an image acquisition step of acquiring multiple moving images with overlapping imaging ranges, an information acquisition step of acquiring information on a stitching area used to stitch together the multiple moving images, an index calculation step of calculating an index based on the distance between the subject and a stitching line included in the stitching area, and a warning output step of outputting a warning from an output device based on the index.

[0022] An image processing program according to an 18th aspect of the present invention is an image processing program that causes an image processing device having a processor to execute an image processing method, and the image processing method includes an image acquisition step of acquiring multiple moving images with overlapping imaging ranges, an information acquisition step of acquiring information on a stitching area used to stitch together the multiple moving images, an index calculation step of calculating an index based on the distance between a subject and a stitching line included in the stitching area, and a warning output step of outputting a warning to the subject from an output device. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 is a diagram showing the overall configuration of an image processing system according to the first embodiment. [Figure 2] FIG. 2 is a diagram showing the configuration of the system controller and the functional configuration of the control device. [Figure 3] FIG. 3 is a perspective view of the camera. [Figure 4] FIG. 4 is a rear view of the camera. [Figure 5] FIG. 5 is a diagram showing the imaging ranges and stitching areas of moving images captured by the three camera units. [Figure 6] FIG. 6 shows moving images captured by three cameras. [Figure 7] FIG. 7 is a flowchart showing the procedure of the image processing method according to the first embodiment. [Figure 8] FIG. 8 is a diagram showing an example of a warning message. [Figure 9] FIG. 9 is a flowchart showing details of the information acquisition process and the information acquisition step. [Figure 10] FIG. 10 is a diagram showing how stitching lines are set. [Figure 11] FIG. 11 is a conceptual diagram showing an example of synthetic moving image generation. [Figure 12] FIG. 12 is a conceptual diagram showing another example of synthetic moving image generation. [Figure 13] FIG. 13 is a flowchart showing the procedure for calculating the index for a moving object. [Figure 14] FIG. 14 is a flowchart showing the procedure for outputting a warning based on an index. [Figure 15] FIG. 15 is a flowchart showing the procedure for calculating an index based on distance and time. [Figure 16] FIG. 16 is a diagram showing a state in which a person, a subject, stays near the stitching line. [Figure 17] FIG. 17 is a flowchart showing the procedure for calculating an index based on the distance and the size of the subject. DETAILED DESCRIPTION OF THE INVENTION

[0024] [Removing unnaturalness at video joins] Users of imaging devices sometimes wish to capture moving images over a wide range. For example, when capturing 360-degree moving images (moving images captured continuously and seamlessly around the entire horizontal circumference (360 degrees); also known as "360-degree video"), a single wide-angle camera or multiple cameras may be used. When capturing high-quality video for professional or commercial use, using multiple cameras is generally advantageous. However, when using multiple cameras, stitching is required, which is the process of joining multiple images together (combining moving images). In this case, with conventional technologies such as those disclosed in Patent Documents 1 and 2, if a subject (person P in this figure) overlaps a seam as shown in Figure 6 (described later), the subject will be distorted or misaligned near the seam in the combined video (moving image), resulting in an unnatural image.

[0025] Under these circumstances, the inventors of the present application conducted extensive research and came up with the idea that "by calculating an index based on the distance between the subject and the stitching line and outputting a warning from an output device based on this index, it is possible to prevent the seams in the video from looking unnatural." One embodiment of the invention based on this idea will be described in detail below.

[0026] An embodiment of an image processing device, an image processing system, an image processing method, and an image processing program according to the present invention is as follows: In the description, the accompanying drawings will be referred to as necessary.

[0027] [First embodiment] [Overall configuration of image processing system] Fig. 1 is a diagram showing the overall configuration of an image processing system according to a first embodiment of the present invention. The image processing system 10 shown in Fig. 1 includes a system controller 100 (image processing device) and multiple camera units 200 (imaging devices), and performs operations such as capturing moving images, calculating indices, issuing warnings, and generating and displaying composite moving images, as will be described in detail later.

[0028] [System controller / controller configuration] 2 is a diagram showing the configuration of the system controller 100 and the functional configuration of the control device 101. The system controller 100 has a control device 101 (processor, image processing device), a memory 118, a display device 120 (output device), a recording device 122, and an operation unit 124. The control device 101 has, as its functions, an image acquisition unit 102, an information acquisition unit 104, an index calculation unit 106, a warning unit 108, a determination unit 110, a synthesis unit 112, a display control unit 114, and a recording control unit 116. As will be described in detail below, the control device 101 performs various processes such as index calculation processing and warning processing, and controls the camera unit 200.

[0029] The functions of the control device 101 described above can be realized using various processors. The various processors include, for example, a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to realize various functions. The various processors described above also include a GPU (Graphics Processing Unit), which is a processor specialized for image processing. The various processors described above also include a PLD (Programmable Logic Device), which is a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array). Furthermore, the various processors described above also include dedicated electrical circuits, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically to execute specific processing.

[0030] Each function of the control device 101 may be realized by one processor or by multiple processors. Also, one processor may correspond to multiple functions. Furthermore, each function of the control device 101 may be realized by a circuit, or some of the functions may be realized by a circuit and the rest may be realized by a processor.

[0031] When the processor or electrical circuit described above executes software (programs), the processor-readable code of the software to be executed and the data required to execute the software are stored in a non-transitory recording medium such as a flash memory, and the processor references the software and data. The software stored in the non-transitory recording medium includes an image processing program for executing each step of the image processing method according to this embodiment. Instead of flash memory, the code and data may be recorded in a non-transitory recording medium using various magneto-optical recording devices, semiconductor memory, or the like. Here, "semiconductor memory" includes flash memory, ROM (Read Only Memory), and EEPROM (Electronically Erasable and Programmable ROM). During software processing, for example, RAM (Random Access Memory) is used as a temporary storage area. Memory 118 (memory) can be composed of flash memory, ROM, RAM, EEPROM, or the like, and is used as the non-transitory storage medium and temporary storage area described above.

[0032] [Display device, recording device, and operation unit] The display device 120 is composed of a display device such as a liquid crystal display, and displays the acquired moving images and composite moving images, outputs warnings, etc. The recording device 122 is composed of various types of magneto-optical recording devices and semiconductor recording devices and their control devices, and records the acquired moving images and composite moving images, etc. The operation unit 124 is composed of devices such as a mouse and keyboard, and the user can perform operations necessary to execute the image processing method and image processing program of the present invention via these devices.

[0033] [Camera section configuration] The image processing system 10 comprises three camera units 200 (imaging devices, output devices). The camera units 200 are connected to the system controller 100, and perform functions such as capturing moving images, transmitting the captured moving images to the system controller 100, and outputting warnings based on indicators. Each camera unit 200 comprises a camera 210 (imaging device) and a display device 230 (output device), and performs functions such as acquiring moving images and outputting warnings. The display device 230 can be used to display warnings, and in this case, it is preferable to install it in a direction that allows the subject to see the warning.

[0034] 1 illustrates a case where image processing system 10 includes three camera units 200, but the image processing system of the present invention may include two, four, or more camera units as long as it can acquire multiple moving images with overlapping imaging ranges. Also, while FIG. 1 illustrates a case where each camera unit 200 includes a camera 210 and a display device 230, multiple camera units 200 may share one display device 230. In the first embodiment, the three camera units 200 (cameras 210) can have the same configuration, but when it is necessary to distinguish between the camera units 200, they will be referred to as "camera units 200A, 200B, 200C" or "cameras 210A, 210B, 210C."

[0035] [Camera configuration] FIG. 3 is an external perspective view of a camera 210 (imaging device). As shown in FIG. 3, the camera 210 is composed of a camera body 212 and an interchangeable lens 218 (optical axis L1). The top surface of the camera body 212 is provided with a release button 214, a dial 216 for setting the imaging mode and shutter speed, and the like. Meanwhile, FIG. 4 is a rear view of the camera 210. The rear surface is provided with a monitor 220 (display device, output device), a finder eyepiece 225, and various buttons and switches. The monitor 220 is a touch-panel display device using a liquid crystal display element or the like, and can be used to display moving images, accept settings for stitching lines, or output (display) warnings. The monitor 220 is not limited to a type fixed to the camera body 212; a vari-angle or tilt-type monitor whose orientation can be changed may be used so that the subject can view the display on the monitor 220.

[0036] The camera body 212 is provided with an image sensor (not shown). The image sensor may be a CMOS (Complementary Metal-Oxide Semiconductor) type, a CCD (Charge-Coupled Device) type, or other type. Furthermore, in the camera 210, each function (or part thereof) of the image processing program of the present invention may be stored in a memory (not shown), and a processor (not shown) may perform processing in accordance with the image processing program. The processing in the camera 210 may be performed in response to a user operation (such as pressing down the release button 214, operating the monitor 220, or operating various buttons and switches), or may be performed by a processor (not shown) without a user operation. Furthermore, the camera 210 may receive commands from the system controller 100, and the system controller 100 may control each unit of the camera 210 to perform various processes.

[0037] The camera 210 can be mounted on a tripod or a pan head (not shown). These tripods or pan heads may be of a type that allows control (such as a remote release function) of the camera 210. Alternatively, an electric tripod or pan head may be employed, and the system controller 100 may control the tripod or pan head (such as changing the imaging direction, starting and ending imaging, etc.).

[0038] 3 and 4 is an example of an imaging device that constitutes an image processing system, and imaging devices with different configurations may be used. For example, instead of a compact camera, a professional imaging device used for television broadcasting or the like may be used. Alternatively, a smartphone may be used as the imaging device.

[0039] Display device 230 (output device) is configured with a device such as a liquid crystal display, and is connected to camera 210 to display moving images captured by camera 210 and output (display) warnings based on indicators. Camera unit 200 may record the captured moving images in a recording device (not shown).

[0040] [Video capture range and stitching area] 5 is a diagram showing the imaging ranges and stitching areas of moving images captured by three camera units 200A, 200B, and 200C, and shows the state of camera units 200A, 200B, and 200C as viewed vertically from above. In FIG. 5, the imaging ranges of camera units 200A, 200B, and 200C are imaging ranges 300A, 300B, and 300C, respectively, and these imaging ranges overlap at stitching areas 310A, 310B, and 310C (shaded areas in the figure). With camera units 200A, 200B, and 200C arranged in this manner, multiple moving images with overlapping imaging ranges can be acquired.

[0041] 6 is a diagram showing moving images 400A, 400B, and 400C that are moving images captured by three cameras 210A, 210B, and 210C. As shown in the figure, these moving images have stitching areas 310A, 310B, and 310C at both ends, and a person P (subject, moving object) appears in stitching area 310A.

[0042] [Basic image processing methods] 7 is a flowchart showing the steps of the image processing method according to the first embodiment. This flowchart shows basic processing, and each step will be described in detail later. The image acquisition unit 102 (processor) acquires multiple moving images with overlapping imaging ranges captured by the camera units 200A, 200B, and 200C (cameras 210A, 210B, and 210C) (step S100: image acquisition processing, image acquisition step). The image acquisition unit 102 may acquire these moving images in real time, or may acquire moving images that have already been captured.

[0043] The information acquisition unit 104 (processor) acquires information on the stitching area used to stitch together multiple videos (step S110: information acquisition processing, information acquisition step). This processing can be performed based on common areas among the multiple videos. The synthesis unit 112 (processor) stitches the multiple videos together in the stitching area to generate a synthesized video (step S120: synthesis processing, synthesis step). When the camera unit 200 (camera 210) is arranged as shown in FIG. 5, the synthesis unit 112 can generate a panoramic video in step S120. Here, a "panoramic video" refers to a video that is continuous (acquired without interruption) around the entire horizontal circumference (360 degrees around the vertical axis).

[0044] Furthermore, in step S120, the display control unit 114 (processor) displays the composite moving image on the display device 120 (display device, output device), and the recording control unit 116 (processor) records the composite moving image in the recording device 122. Note that in the first embodiment, the system controller 100 (processor) may display and record the composite moving image in real time, or may record the original moving images (the three moving images captured by the cameras 210A, 210B, and 210C) in real time and then display and record the composite moving image afterwards.

[0045] The index calculation unit 106 (processor) calculates an index based on the distance between the subject and the stitching line (step S130: index calculation processing, index calculation step), and the warning unit 108 (processor) outputs a warning based on the index (step S140: warning output processing, warning output step). The index calculation unit 106 may calculate the distance between the subject and the stitching line (e.g., the number of pixels) itself as the index, or, as will be described later, may calculate another quantity based on the distance as the index. The warning unit 108 may display a warning message on the display device 230 (display device, output device) of the camera units 200A, 200B, and 200C, or may display a message on the monitor 220 if the monitor 220 (display device, output device) of the cameras 210A, 210B, and 210C is a variable-angle or tilt type whose display direction can be changed and the monitor 220 can be directed toward the subject. The index calculation unit 106 and the warning unit 108 can calculate these indexes and issue warnings in real time.

[0046] Fig. 8 is a diagram showing an example of a warning message. In the example shown in Fig. 8, in addition to the message "Approaching the stitching line!", an arrow is displayed as an example of "information indicating the direction in which the subject should move." With this display (warning), if the subject is a moving object such as a person, as will be described later, the subject can easily move away from the stitching area in an appropriate direction.

[0047] The calculation of the index and the output of the warning may be performed for all frames of the video image, or may be performed (intermittently) for some frames.

[0048] The control device 101 (processor) repeats the above-described process until imaging is completed (YES in step S150). The control device 101 can end imaging when an imaging termination operation (for example, an operation on a predetermined button or switch) is performed on the camera 210 or the control device 101. Note that when repeating the process, if information on the stitching area has already been acquired (if the stitching line has already been set), the process of step S110 may be skipped.

[0049] [Details of information acquisition process] Fig. 9 is a flowchart showing details of the information acquisition process, the information acquisition step (step S110 in Fig. 7). For a plurality of moving images displayed in live view on the monitor 220 or the display device 120 (step S112), the information acquisition unit 104 (processor) acquires a common area among the plurality of moving images as information on the stitching area (step S114). This "information" is, for example, the position and size of the stitching area in the moving images, as shown in Fig. 6.

[0050] The information acquiring unit 104 sets a stitching line in the stitching region based on the user's operation (step S116). For example, as shown in FIG. 10, the user can move a stitching line 510 (shown by a dotted line in the figure) in a stitching region 500 of the moving image displayed on the monitor 220 left and right (in the direction of the arrow in the figure) within the stitching region 500 (including the boundary line) by operating a touch panel constituting the monitor 220 (to move the stitching line included in the stitching region). Instead of moving left and right, the user may also perform an operation of tracing the monitor 220 from top to bottom (or bottom to top) with a finger, a touch pen, or the like. The information acquiring unit 104 can set a stitching line in the moving image (or determine a stitching line included in the stitching region) based on such an operation on the camera 210 (cameras 210A, 210B, 210C). In addition, the information acquisition unit 104 may output a warning from the display device 120, etc., when information about the stitching area cannot be properly acquired (for example, when the fields of view of the cameras 210 do not overlap) or when the stitching line cannot be properly set (when the user attempts to set the stitching line outside the stitching area).

[0051] The stitching line may be set in response to an operation on the system controller 100 side, rather than an operation on the camera 210 as described above. For example, the display control unit 114 (processor) may cause the display device 120 (display device) to display a live view of the moving images acquired by the camera 210 (step S112), and the user may operate the operation unit 124 (keyboard, mouse, touch panel, etc.) to set the stitching line in the moving images, as in the example of Fig. 10. The information acquisition unit 104 can set the stitching line in the moving images based on such a user operation (step S116).

[0052] [Moving image synthesis] The synthesis unit 112 (processor) synthesizes the moving images (generates a synthesized moving image) by stitching multiple moving images together using stitching regions or stitching lines, and the display control unit 114 can display the synthesized moving image on a display device as a live view (step S118). The display control unit 114 may display the synthesized moving image on one of the display device 120, the monitor 220, and the display device 230, or may display the synthesized moving image on multiple devices. Such a live view display allows the user to check the synthesized moving image on the spot (before recording starts).

[0053] 11 is a conceptual diagram showing an example of generating a composite video (an example of stitching a plurality of videos together at a stitching region), and shows how the videos 400A and 400B of the three videos shown in FIG. 6 are synthesized. Part (a) of FIG. 11 shows a stitching region 310A at the end of the videos 400A and 400B. but 11(b), the stitching area 310 of the moving image is set in the stitching area 312B. A same The two videos 400A and 400B can be overlaid so that the stitching line 312B coincides to generate a composite video 410. At this time, the composition unit 112 can perform weighted addition of the two videos 400A and 400B in the stitching region.

[0054] Fig. 12 is a conceptual diagram showing another example of composite moving image generation (an example in which multiple moving images are joined together at a stitching line). In this example, the synthesis unit 112 cuts off end regions 314 (the boundary is shown by a dotted line) of moving images 400A and 400B beyond the stitching line 312B as shown in part (a) of Fig. 12, and then butts the stitching lines 312B together (aligns the stitching lines 312B with each other), thereby generating a composite moving image 420 shown in part (b) of the same figure. In other words, it can also be said that "the weight of image synthesis is 1 / 0 at the stitching line 312 as the boundary."

[0055] The synthesis unit 112 can generate a composite video by performing the processing illustrated in Figures 11 and 12 on multiple videos with overlapping imaging ranges. In the situation shown in Figures 5 and 6, a panoramic video can be generated as a composite video by performing processing on videos 400A, 400B, and 400C.

[0056] The user can check the composite video image displayed in live view and, if the settings of the stitching lines, etc. are OK, can operate the operation unit 124, monitor 220, etc., and when an operation indicating that the settings are OK is performed, such as pressing a predetermined button or switch (YES in step S119), the recording control unit 116 (processor) starts recording the multiple videos and composite video image in response to this operation after information about the stitching area has been added (step S120 in Figure 7).

[0057] [Calculation of indicators for moving objects] In the first embodiment, in the index calculation process (step S130 in FIG. 7), an index may be calculated based on whether the subject is a moving object. For example, as shown in the flowchart of FIG. 13, the index calculation unit 106 (processor) may determine whether the subject is a moving object (step S133), and if it is a moving object (YES in step S133), calculate the distance and the index (steps S134 and S136). Here, the index calculation unit 106 may identify (detect and classify) the subject by, for example, image processing of the acquired video, and determine that "the subject is a moving object" if it is a predetermined type of subject. For such subject identification, a classifier configured by machine learning such as deep learning, for example, a hierarchical neural network, may be used.

[0058] A moving subject can move in response to a warning, and by making such a subject the target of index calculation and warnings, it is possible to efficiently prevent the subject from remaining near the stitching area (or near the stitching line), thereby preventing the seams in the video from appearing unnatural.

[0059] An example of a "moving subject" is a person. The index calculation unit 106 may detect people using a known face detection method. People can easily recognize warnings and move on, and it is effective to target people as the target of index calculation and warnings in order to prevent unnatural seams in the video.

[0060] It should be noted that in the present invention, the "moving subject" is not limited to a person. The subject may be a vehicle, a drone or other flying object, a mobile robot, or machinery (for example, a crane that can move or transform). These subjects may be objects that can move autonomously or objects that can be moved by remote control or the like.

[0061] [Warning output based on indicators] FIG. 14 is a flowchart showing the procedure for outputting a warning based on an index (step S140 in FIG. 7). The warning unit 108 (processor) determines whether the index calculated in step S130 satisfies a criterion (step S142: determination process, determination step), and if the criterion is satisfied (if the determination result is YES), a warning can be output from the display device 230 or the like (output device) (step S144). In step S142, the warning unit 108 may determine whether "the subject is within a stitching area" based on the index, or may determine whether "the index is equal to or greater than a threshold value." In making these determinations, the warning unit 108 may use statistics of the index (for example, the minimum, maximum, or average value of the index over a predetermined period).

[0062] As described above, according to the first embodiment, a warning based on an index can prevent a subject from remaining near a stitching area, making it possible to prevent the seams of the video from looking unnatural.

[0063] [Variations] Variations of the invention are described below.

[0064] [Calculation of indicators based on distance and time] In the first embodiment, an aspect of calculating an index based on the distance between the subject and the stitching area has been described (see FIG. 13), but in the present invention, the index may be calculated by further considering the time that the subject stays in the stitching area. FIG. 15 is a flowchart showing the procedure for calculating an index based on distance and time. Note that the processing up to step S134 is the same as in FIG. 13, so the same step numbers are used and detailed description will be omitted.

[0065] The index calculation unit 106 (processor) calculates the time (residence time) that the subject stays in the stitching area (step S135: index calculation processing, index calculation step), and can calculate an index based on this time and the distance calculated in step S134 (step S137: index calculation processing, index calculation step). Note that here, as shown in Fig. 16, it is assumed that the distance between person P (subject, moving object) and stitching line 510 is h (pixels) (in the figure, the size of the subject is represented by a face detection frame F; described later).

[0066] The index calculation unit 106 can calculate the time that person P stays in the stitching area (dwell time; time t) by multiplying, for example, the number of frames in which person P appears in the stitching area among the frames constituting the moving image by the frame interval (the inverse of the frame rate), and can calculate an index based on this time t and the distance h using the following formula (1). [Number 1] Index=t×(1 / h) ···(1)

[0067] As is clear from equation (1), the longer the residence time and the shorter the distance, the larger the index.

[0068] The index calculation unit 106 may weight "t" and / or "1 / h" by multiplying them by a coefficient as shown in the following equation (2). [Number 2] Index=(α×t)×{β×(1 / h)} ···(2) (where α, β>0)

[0069] The warning unit 108 (processor) can output a warning from the output device based on the index calculated by the formulas (1) and (2) (step S140 in FIG. 7; warning processing, warning step).

[0070] [Calculation of index based on distance and subject size] In the present invention, an index may be calculated based on the size of the subject in the video in addition to the distance between the subject and the stitching region. For example, as shown in the flowchart in Fig. 17, the index calculation unit 106 (processor) calculates the size of the subject in the video (step S138: index calculation processing, index calculation step), and calculates an index based on the distance between the subject and the stitching line and the size of the subject in the video (step S139: index calculation processing, index calculation step).

[0071] In step S138, if the subject is a person, the index calculation unit 106 may determine the size of the subject based on the size of an area showing the person's body parts in the video. As the "size of the area showing the person's body parts in the video," the size of the face detection frame F of person P can be used, for example, as shown in FIG. 16. In a situation like that of FIG. 16, a "large face detection frame F" means that the camera 210 and person P are close to each other, and even a slight movement of person P could cause person P to cross the stitching line. For this reason, such a situation can be prevented by calculating an index based on the "size of the area showing the person's body parts in the video," such as the face detection frame F, and outputting a warning. Note that, instead of such a face detection frame, the index calculation unit 106 may determine the size of the subject based on the size of a bounding box that surrounds the entire person (entire subject) in the video.

[0072] The index calculation unit 106 may calculate the index based on three points: the distance between the subject and the stitching region, the time the subject remains in the stitching region, and the size of the subject in the moving image.

[0073] [others] [About the video images to be captured / composite] In the first embodiment, a case has been described in which moving images are acquired (captured) all around and a full-circle moving image is generated (combined), but the acquisition and generation of moving images in the present invention are not limited to this mode. In the present invention, it is sufficient to acquire multiple moving images with overlapping imaging ranges. For example, two moving images may be captured by two camera units, or moving images may be captured by four or more camera units and then combined. Furthermore, the generated composite moving image may not be a full-circle moving image, but may be a moving image of only a portion of the circumference (e.g., 120 degrees, 180 degrees, etc.). According to the present invention, even in such a mode, it is possible to prevent unnatural images from appearing at the seams of the moving images.

[0074] [Other warning output modes] In the first embodiment, a warning message or the like is displayed on the monitor 220 or the display device 230 (see the example in FIG. 8 ), but the output of a warning is not limited to this mode in the present invention. Instead of or in addition to outputting a warning message, the warning unit 108 (processor) can output a warning by turning on or off a light source (output device) (not shown). In this case, the warning unit 108 can change the blinking speed based on the index (for example, the larger the index, the faster the blinking) or change the color of the light source (for example, the light will emit red light if the index is large and blue light if the index is small).

[0075] Furthermore, instead of or in addition to such a screen display, the warning unit 108 can output a warning by audio output or vibration. For example, a subject, which is a moving object such as a person, can carry or wear a device (output device) such as a smartphone, earphone-type device, goggle-type device, or wristwatch-type device, and the warning unit 108 (system controller 100, processor) can control these devices to output a warning by at least one of a screen display, audio output, and vibration. In this case, audio can be output from earphones or speakers (output devices), or vibration can be output by the operation of electronic or mechanical components (output devices). As in the case of the above-mentioned screen display, the warning unit 108 can change the volume or pitch of the sound based on the indicator (for example, increasing the volume and / or pitch when the indicator is large, and decreasing the volume and / or pitch when the indicator is small). When outputting a warning by these devices, the warning unit 108 can also output the direction in which the subject should move.

[0076] Even when a warning is output in this manner, as in the first embodiment, the subject, which is a moving object, can recognize the warning and move, thereby preventing the image from appearing unnatural at the seams of moving images.

[0077] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described aspects, and various modifications are possible without departing from the spirit of the present invention. [Explanation of symbols]

[0078] 10 Image Processing System 100 System Controller 101 Control device 102 Image acquisition unit 104 Information Acquisition Department 106 Indicator calculation section 108 Warning section 110 Judgment Department 112 Synthesis section 114 Display control unit 116 Recording control section 118 memory 120 Display device 122 Recording Device 124 Operation section 200 Camera Department 200A Camera unit 200B camera unit 200C camera unit 210 Camera 210A Camera 210B Camera 210C Camera 212 Camera body 214 release button Dial 216 218 Interchangeable Lenses 220 monitors 225 Finder eyepiece 230 Display device 300A imaging range 300B imaging range 300C imaging range 310A Stitching Area 310B stitching area 310C stitching area 312 Stitching Line 312B Stitching Line 314 areas 400A Video 400B Video 400C Video 410 Synthetic Video 420 Synthetic Video 500 stitching area 510 Stitching Line F Face detection frame h distance L1 optical axis P person S100~S150 Image processing steps

Claims

1. An image processing device including a processor, The processor: an image acquisition process for acquiring a plurality of moving images having overlapping imaging ranges; an information acquisition process for acquiring information on a stitching region used to stitch together the plurality of moving images; an index calculation process for calculating an index based on the distance between the subject and the stitching line included in the stitching area; a warning process for outputting a warning from an output device based on the indicator; and an image processing device that calculates a time during which the subject remains in the stitching area, and calculates the index based on the distance and the calculated time, in the index calculation process.

2. An image processing device comprising a processor, The processor: an image acquisition process for acquiring a plurality of moving images having overlapping imaging ranges; an information acquisition process for acquiring information on a stitching region used to stitch together the plurality of moving images; an index calculation process for calculating an index based on the distance between the subject and the stitching line included in the stitching area; a warning process for outputting a warning from an output device based on the indicator; and In the index calculation process, the image processing device calculates the index based on the size of the subject in the moving image.

3. An image processing device comprising a processor, The processor: an image acquisition process for acquiring a plurality of moving images having overlapping imaging ranges; an information acquisition process for acquiring information on a stitching region used to stitch together the plurality of moving images; an index calculation process for calculating an index based on the distance between the subject and the stitching line included in the stitching area; a warning process for outputting a warning from an output device based on the indicator; and In the index calculation process, the image processing device determines whether the subject is a moving object, and calculates the index when the subject is determined to be a moving object.

4. An image processing device comprising a processor, The processor: an image acquisition process for acquiring a plurality of moving images having overlapping imaging ranges; an information acquisition process for acquiring information on a stitching region used to stitch together the plurality of moving images; an index calculation process for calculating an index based on the distance between the subject and the stitching line included in the stitching area; a warning process for outputting a warning from an output device based on the indicator; and In the information acquisition process, acquiring a common area among the plurality of moving images as information on the stitching area; An image processing device that sets the stitching line in the stitching area based on a user's operation.

5. An image processing device described in any one of claims 1 to 4, wherein the processor performs a judgment process to determine whether the subject is within the stitching area based on the index.

6. An image processing device as described in Claim 5, wherein the processor, in the warning processing, outputs the warning from the output device based on the judgment processing.

7. An image processing device as described in Claim 2, wherein the processor determines the size of the subject based on the size of an area that indicates at least a part of a person's body.

8. An image processing device as described in Claim 3, wherein the processor determines that the object is a moving object when it recognizes a person.

9. An image processing device as described in claim 3 or 8, wherein the processor outputs information indicating the direction in which the subject should move during the warning process.

10. The processor assigns information about the stitching region to the plurality of moving images displayed in a live view state in the information acquisition process; The image processing device according to claim 1 , wherein recording of the plurality of moving images is started after the stitching area information is added.

11. An image processing device described in any one of claims 1 to 4, wherein the processor performs a synthesis process to connect the multiple moving images in the stitching area or the stitching line to generate a composite moving image.

12. An image processing device as described in Claim 11, wherein the processor generates a panoramic moving image in the synthesis process.

13. An image processing device as described in Claim 12, wherein the processor displays the composite moving image on a display device.

14. An image processing device according to any one of claims 1 to 4; an imaging device that captures the plurality of moving images; the output device; An image processing system comprising:

15. The image processing system described in Claim 14, wherein the output device outputs the warning by at least one of a screen display, an audio output, and a vibration.

16. An image processing method executed by an image processing device having a processor, comprising: The processor: an image acquisition step of acquiring a plurality of moving images having overlapping imaging ranges; an information acquisition step of acquiring information on a stitching region used to stitch together the plurality of moving images; an index calculation step of calculating an index based on the distance between the subject and the stitching line included in the stitching area; a warning output step of outputting a warning from an output device based on the indicator; Run In the index calculation step, a time during which the subject remains in the stitching area is calculated, and the index is calculated based on the distance and the calculated time.

17. An image processing method executed by an image processing device having a processor, comprising: The processor: an image acquisition step of acquiring a plurality of moving images having overlapping imaging ranges; an information acquisition step of acquiring information on a stitching region used to stitch together the plurality of moving images; an index calculation step of calculating an index based on the distance between the subject and the stitching line included in the stitching area; a warning output step of outputting a warning from an output device based on the indicator; and The image processing method, in the index calculation step, calculates the index based on the size of the subject in the moving image.

18. An image processing method executed by an image processing device having a processor, comprising: The processor: an image acquisition step of acquiring a plurality of moving images having overlapping imaging ranges; an information acquisition step of acquiring information on a stitching region used to stitch together the plurality of moving images; an index calculation step of calculating an index based on the distance between the subject and the stitching line included in the stitching area; a warning output step of outputting a warning from an output device based on the indicator; and In the index calculation step, it is determined whether the subject is a moving object, and if it is determined that the subject is a moving object, the index is calculated.

19. An image processing method executed by an image processing device having a processor, comprising: The processor: an image acquisition step of acquiring a plurality of moving images having overlapping imaging ranges; an information acquisition step of acquiring information on a stitching region used to stitch together the plurality of moving images; an index calculation step of calculating an index based on the distance between the subject and the stitching line included in the stitching area; a warning output step of outputting a warning from an output device based on the indicator; and In the information acquisition step, acquiring a common area among the plurality of moving images as information on the stitching area; An image processing method for setting the stitching line in the stitching area based on a user's operation.

20. An image processing program for causing an image processing device having a processor to execute an image processing method, The image processing method includes: an image acquisition step of acquiring a plurality of moving images having overlapping imaging ranges; an information acquisition step of acquiring information on a stitching region used to stitch together the plurality of moving images; an index calculation step of calculating an index based on the distance between the subject and the stitching line included in the stitching area; a warning output step of outputting a warning to the subject from an output device; Including, An image processing program in which the processor calculates the time that the subject remains in the stitching area in the index calculation step, and calculates the index based on the distance and the calculated time.

21. An image processing program for causing an image processing device having a processor to execute an image processing method, The image processing method includes: an image acquisition step of acquiring a plurality of moving images having overlapping imaging ranges; an information acquisition step of acquiring information on a stitching region used to stitch together the plurality of moving images; an index calculation step of calculating an index based on the distance between the subject and the stitching line included in the stitching area; a warning output step of outputting a warning from an output device based on the indicator; Including, The image processing program is configured to cause the processor to calculate the index based on the size of the subject in the moving image in the index calculation step.

22. An image processing program for causing an image processing device having a processor to execute an image processing method, The image processing method includes: an image acquisition step of acquiring a plurality of moving images having overlapping imaging ranges; an information acquisition step of acquiring information on a stitching region used to stitch together the plurality of moving images; an index calculation step of calculating an index based on the distance between the subject and the stitching line included in the stitching area; a warning output step of outputting a warning from an output device based on the indicator; Including, The processor determines whether the subject is a moving object or not in the index calculation step, and calculates the index when the subject is determined to be a moving object.

23. An image processing program for causing an image processing device having a processor to execute an image processing method, The image processing method includes: an image acquisition step of acquiring a plurality of moving images having overlapping imaging ranges; an information acquisition step of acquiring information on a stitching region used to stitch together the plurality of moving images; an index calculation step of calculating an index based on the distance between the subject and the stitching line included in the stitching area; a warning output step of outputting a warning from an output device based on the indicator; Including, In the information acquisition step, the processor acquiring a common area among the plurality of moving images as information on the stitching area; an image processing program that sets the stitching line in the stitching area based on a user's operation;

24. A non-transitory computer-readable recording medium on which a program described in any one of claims 20 to 23 is recorded.

Images