Representation method

Abstract

To provide a more useful method for displaying a video in which a portion of a captured video is cut out and the cut-out video can be switched.SOLUTION: The display method for displaying a video captured by a photographing apparatus includes a setting step of automatically setting a plurality of setting areas or candidates for setting areas within a reference area, which is an area in which a reference video is captured, a selection step of selecting a recording area, which is an area of recorded video to be recorded, from among the plurality of setting areas, and a switching step of switching the recording area by reselecting a recording area from among the plurality of setting areas after implementation of the selection step.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] One embodiment of the present invention relates to a method for displaying an image. [Background technology]

[0002] A technology has already been developed that extracts and stores an image of one of multiple subjects from an image captured by a single camera. One example of such a technology is described in Patent Document 1. The technology described in Patent Document 1 sets multiple different image areas within the shooting range of a camera, and records an image of one of those areas. The technology described in Patent Document 1 also makes it possible to change (switch) the image area to be recorded among the multiple areas. This makes it possible, for example, to obtain zoomed-in images of each of multiple subjects in the shot image while keeping the camera fixed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-92467 Summary of the Invention [Problem to be solved by the invention]

[0004] One embodiment of the present invention has been made in consideration of the above circumstances, and aims to provide a more useful method for displaying images that can cut out a portion of a captured image and switch between the cut-out images. [Means for solving the problem]

[0005] In order to achieve the above object, one embodiment of the present invention is a display method for displaying video captured by a shooting device, comprising: a setting process for automatically setting multiple setting areas or candidate setting areas within a reference area, which is the shooting area of the reference video; a selection process for selecting a recording area, which is the area of the recorded video to be recorded, from the multiple setting areas; and a switching process for reselecting a recording area from the multiple setting areas and switching the recording area after the selection process is performed.

[0006] The setting process may also include a first setting process for automatically setting a candidate within the reference area, and a second setting process for setting a setting area at the position where the candidate is set in accordance with a user's operation on the candidate.

[0007] In addition, in the setting step, characteristic locations where visual changes occur in the reference video may be detected, and the setting region or candidate may be automatically set at the position of the detected characteristic location in the reference region.

[0008] In the setting step, a photographic scene of the reference video may be recognized based on the reference video, and the setting area or candidate may be automatically set in a position in the reference area corresponding to the recognized photographic scene.

[0009] In the setting step, the setting area may be automatically set in the reference image at a position where the characteristics for determining the shooting conditions of the shooting device satisfy the criteria. Here, the shooting conditions may be at least one of a focus position, an exposure amount, and a white balance in the reference image.

[0010] In addition, in the setting step, the set area or the candidate may be automatically set based on position information of the set area stored in advance. Here, the position information may include at least one of position information of a setting area set in a setting process performed in the past and position information of a setting area determined based on a user operation.

[0011] The method may further include an adjustment step of adjusting at least one of the position and size of the set area when the size or position of the reference area changes.

[0012] Furthermore, one embodiment of the present invention is a display method for displaying video captured by a shooting device, comprising: a setting step for setting a plurality of setting areas within a reference area, which is the shooting area of the reference video; a selection step for selecting a recording area, which is the area of the recorded video to be recorded, from the plurality of setting areas; a switching step for reselecting a recording area from the plurality of setting areas after the selection step is performed and switching the recording area; a recording step for recording a moving image of the recorded video; a display step for displaying the setting video, which is video of the setting area; and a change step for executing a change process for changing the number of setting videos to be displayed while the recording step is being performed.

[0013] The change process may also include a process of increasing the number of display areas by adding set areas in response to an increase in the number of subjects in the reference video. Furthermore, the change process may include a process of reducing the number of display by stopping the display of the setting video of the setting area that satisfies the stop condition.

[0014] Furthermore, the setting area where the display of the setting video has been stopped may be stored while the recording step is being performed. In this case, the change process may include a process of redisplaying the setting video of the stored stopping area based on a redisplay instruction from the user.

[0015] The change process may also include a process of outputting proposal information regarding a change in the number of items to be displayed.

[0016] Furthermore, when the number of subjects in the reference video exceeds the upper limit of the number of set regions, the change process may execute a change process for reducing the number of displayed regions. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a perspective view showing an example of the appearance of an imaging device according to an embodiment of the present invention. [Figure 2]FIG. 2 is a rear view showing the rear side of the imaging device according to the embodiment of the present invention. [Figure 3] 1 is a block diagram showing the configuration of an imaging device according to an embodiment of the present invention; [Figure 4] FIG. 2 is an explanatory diagram of electronic image stabilization. [Figure 5] FIG. 10 is a diagram illustrating the relationship between a reference area and a set area. [Figure 6] FIG. 10 is a diagram showing the positional relationship between a reference video and a recorded video. [Figure 7] 10A and 10B are diagrams showing recorded video before and after switching, among moving images showing recorded video. [Figure 8] FIG. 10 is a diagram showing a reference image when a new subject enters the shooting area. [Figure 9] FIG. 10 is a diagram showing how a moving region follows a moving subject. [Figure 10] FIG. 10 is a diagram showing a video display flow. [Figure 11] 10A to 10C are diagrams illustrating transitions of UI screens in the setting process. [Figure 12] FIG. 10 is a diagram showing how a setting area is set when a previously used pattern is selected. [Figure 13] FIG. 10 is a diagram showing a selection screen for a preset pattern. [Figure 14] FIG. 10 is an explanatory diagram of a user operation regarding the position of a setting area. [Figure 15] FIG. 10 is a diagram showing a setting area that is set based on an AF position specifying pattern. [Figure 16] FIG. 10 is a diagram showing characteristic points in a reference video. [Figure 17] FIG. 10 is a diagram showing a state in which a set region is set in a characteristic location. [Figure 18] FIG. 10 is a diagram showing an example of a display screen when a display step is performed. [Figure 19] FIG. 10 is a diagram showing an example of a display screen when a second display step is performed. [Figure 20] FIG. 10 is a diagram showing a modified example of the display screen when the second display step is performed. [Figure 21]FIG. 10 is a diagram showing an example of a screen when the number of standby areas reaches the maximum number of standby videos to be displayed (part 1). [Figure 22] 10 is a diagram showing the flow of a display process when the number of standby areas reaches the maximum number of standby videos to be displayed. FIG. [Figure 23] FIG. 10 is a diagram showing an example of a screen when the number of standby areas reaches the maximum number of standby videos to be displayed (part 2). [Figure 24] 10A and 10B are diagrams showing modified screens when the number of standby areas reaches the maximum number of standby videos that can be displayed. [Figure 25] FIG. 10 is a diagram showing a process flow during a recording step. [Figure 26] FIG. 10 is a diagram showing an example of a screen before a change process is performed. [Figure 27] 10A and 10B are diagrams illustrating an example of a screen when the number of displaying setting images is increased by a change process. [Figure 28] FIG. 10 is a diagram showing an example of a screen when a moving area overlaps with another setting area. [Figure 29] FIG. 10 is a diagram showing an example of a screen when an overlapping moving area and another set area are integrated into one set area. [Figure 30] FIG. 10 is a diagram showing an example of a screen when the display of a setting image in a certain setting area is stopped. [Figure 31] FIG. 10 is a diagram showing an example of a screen on which suggestion information prompting a user to change the reference region is displayed. DETAILED DESCRIPTION OF THE INVENTION

[0018] A preferred embodiment of the present invention (hereinafter referred to as the present embodiment) will be described in detail with reference to the accompanying drawings. This embodiment relates to a method for displaying an image using the image capturing device 10 shown in Figures 1 to 3. However, the embodiment described below is merely an example given to facilitate understanding of the present invention, and is not intended to limit the present invention. In other words, the present invention may be modified or improved from the embodiment described below without departing from the spirit of the present invention. Furthermore, the present invention includes equivalents thereof.

[0019] [Basic configuration of imaging device] The image capturing device 10 is, for example, a digital camera that captures video. Here, unless otherwise specified, "video" refers to live video (live view image), i.e., video captured in real time. Furthermore, the video in this embodiment is primarily moving images. That is, the image capturing device 10 captures moving images at a predetermined frame rate.

[0020] The photographing device 10 is, for example, a lens-interchangeable model, and includes a photographing device body 12 and a photographing lens 14, as shown in Figures 1 and 2. The photographing lens 14 is interchangeably attached to a mount 13 of the photographing device body 12. However, the present invention is not limited to this, and the photographing device 10 may also be a model with an integrated lens.

[0021] (photographic lens) As shown in FIG. 3, the photographing lens 14 includes an optical unit 17, an aperture 20, a zoom driver 21, a focus driver 22, and an aperture driver 23. The optical unit 17 includes an optical device 18 for zooming (e.g., a zoom lens). By moving the optical device 18 in the direction of the optical axis L1, the shooting area of the photographing device 10 expands or contracts, thereby changing the zoom. In other words, the photographing device 10 has an optical zoom function using the optical device 18. The image capturing device 10 of this embodiment has an optical zoom function as well as an electronic zoom (digital zoom) function that enlarges a subject in an image by electronic processing.

[0022] The optical unit 17 also includes a focusing optical device 19 (for example, a focus lens). By moving the optical device 19 in the direction of the optical axis L1, the in-focus position of the photographing lens 14 changes and the focus is adjusted. The photographing device 10 of this embodiment has an autofocus (AF) function. That is, when a user performs a predetermined operation (for example, an instruction to start recording video) while photographing video, the focus driver 22 operates to move the optical device 19. As a result, the focus position is automatically adjusted so that a predetermined part of the video is in focus.

[0023] The optical unit 17 also includes a wide-angle lens, an ultra-wide-angle lens, a 360-degree lens, an anamorphic lens, or the like. As a result, the image capturing device 10 can capture images with a wide angle of view in the horizontal direction (width direction) of the image capturing device 10. The angle of view of the image capturing device 10 changes as optical zoom is performed. The maximum angle of view (full angle of view) of the image capturing device 10 is determined by the design specifications of the optical unit 17 and the image sensor 40 (described later). Incidentally, the optical unit 17 provided in the photographing device 10 is not limited to one, and a plurality of optical units 17 each having a different angle of view may be provided.

[0024] The diaphragm 20 has a variable aperture shape, which is adjusted by a diaphragm driver 23. By adjusting the aperture shape, the amount of aperture for light incident on the photographing lens 14 changes, and the amount of exposure is adjusted.

[0025] (imaging device body) 1 and 2, the photographing device body 12 has an operation unit including an operation button 30. In addition, a release button 26, which is one of the operation units, is disposed on the top surface of the photographing device body 12. When the release button 26 is pressed, recording of the video captured by the photographing device 10 or video extracted from that video (strictly speaking, recorded video, which will be described later) begins.

[0026] Here, the shooting area of the image capturing device 10, i.e., the range included in the angle of view of the image capturing device 10, is determined at the time when a user's instruction to start recording is received. The shooting area also changes depending on, for example, whether electronic image stabilization is on or off. When electronic image stabilization is off at the time when a recording start instruction is received, the shooting area is generally a full-size (full angle of view) shooting area. The user's operation to instruct the start of video recording is not limited to pressing the release button 26, but may be an operation of touching a predetermined position on the rear display 28, which will be described later.

[0027] Hereinafter, the image captured by the image capturing device 10 at the angle of view at that time will be referred to as the "reference image." Also, the capture area of the reference image will be referred to as the "reference area."

[0028] 2 (hereinafter also referred to as rear display 28) is provided on the rear surface of the imaging device body 12. The rear display 28 is configured, for example, with an LCD (Liquid Crystal Display), an organic EL (Organic Electroluminescence) display, an LED (Light Emitting Diode) display, electronic paper, or the like.

[0029] The display 28 on the rear surface displays the reference image, the recorded image (described later), etc. In this way, the image capturing device 10 having a display has an image display function and is therefore used as an image display device. The rear display 28 can also display information other than images, such as advice or suggested information regarding photography (see FIG. 18).

[0030] As shown in Figures 2 and 3, the image capturing device body 12 of this embodiment is provided with an electronic viewfinder (referred to as EVF in Figure 3) 29 as a sub-display. The rear display 28 and the electronic viewfinder 29 can display the same image, or different images. Incidentally, the image capturing device does not have to be provided with the electronic viewfinder 29. Also, it is not necessary to display an image on the electronic viewfinder 29 and not on the rear display 28.

[0031] Furthermore, a touch panel 36 that detects user operations is provided inside the rear display 28 or on the exposed surface of the display 28. For example, when a user touches a predetermined position on the rear display 28, the touch panel 36 detects the touch position and outputs the detection result to a control unit 46, which will be described later.

[0032] As shown in FIG. 3, the photographing device body 12 further includes a shutter 38, an image sensor 40, an analog signal processing circuit 44, a control unit 46, an internal memory 50, a card slot 52, and a buffer memory 56.

[0033] The imaging element 40 is an image sensor that receives light that has passed through the photographing lens 14, converts the received light image into an electrical signal (video signal), and outputs it. As the imaging element 40, a solid-state imaging element such as a CCD (Charged Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor Image Sensor) can be used.

[0034] The control unit 46 controls each unit of the image capturing device 10 and executes various processes related to image capturing, display, recording, etc. The control unit 46 includes a controller 47 and an image processing unit 48 as shown in FIG.

[0035] The control unit 46 has one or more processors, and the control unit 46 is configured by cooperation between the processors and a control program. The processor may be configured, for example, by a central processing unit (CPU), a field programmable gate array (FPGA), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or other integrated circuits (ICs). Alternatively, the processor may be configured by a combination of these. The processor may also be configured such that the entire functions of the control unit 46 including the controller 47 and the video processing unit 48 are implemented in a single IC (Integrated Circuit) chip, as typified by an SoC (System on Chip). The hardware configuration of each of the processors listed above may be realized by an electric circuit that combines circuit elements such as semiconductor elements.

[0036] The controller 47 controls each part of the image capturing device 10 based on a user operation or in accordance with a predetermined control program. For example, the controller 47 controls the image capturing element 40 and the analog signal processing circuit 44 so as to capture video at a predetermined frame rate.

[0037] The controller 47 controls the driving units 21-23, the image sensor 40, the analog signal processing circuit 44, etc. so that the photographing conditions are the desired conditions. The photographing conditions include focus, exposure, white balance, etc. That is, the controller 47 realizes the autofocus (AF) function, autoexposure (AE) function, and autowhite balance (AWB) function, which are the automatic adjustment functions of the photographing device 10.

[0038] Furthermore, when the controller 47 receives a recording start instruction from the user, it controls the video processing unit 48 to start recording the video.

[0039] The video processing unit 48 converts the analog video signal sent from the analog signal processing circuit 44 into digital video data, and performs gamma correction, white balance correction, etc. on the digital video data. The processed digital video data is compressed in a compression format that complies with a predetermined standard.

[0040] During shooting, the video processor 48 generates compressed digital image data at a predetermined frame rate and acquires video (strictly speaking, frame images) from the data. The video acquired at this time corresponds to the reference video captured in the reference area.

[0041] Furthermore, the video processing unit 48 cuts out the recording video from the reference video under the control of the controller 47. The recording video is an image with a smaller angle of view than that of the reference video, and is an image of a partial area of the reference region. The recording video is generated at the same frame rate as the reference video, and is displayed on the rear display 28 or electronic viewfinder 29, for example.

[0042] The recorded video is then recorded on a recording medium. As a result, a video file of the recorded video is created. At this time, the resolution (number of pixels) and aspect ratio of the recorded video are determined according to the recording format (video format) of the recorded video. The recording format is not particularly limited and can be freely determined.

[0043] Furthermore, while electronic image stabilization is enabled (on), image processing unit 48 extracts, from the pre-correction image captured at the full angle of view, an extracted image that is slightly smaller in size than the pre-correction image, as shown in Fig. 4. If the extracted image is displaced due to vibration or the like, image processing unit 48 performs electronic image stabilization and shifts the position of the extracted area in the pre-correction image (for example, from the position indicated by the solid line to the position indicated by the dashed line in Fig. 4). While the electronic image stabilization is on, the extracted image becomes the reference image.

[0044] Hereinafter, unless otherwise specified, the operations and processes of the controller 47 and the video processing unit 48 will be described as the operations and processes of the control unit 46.

[0045] The internal memory 50 built into the image capturing device main body 12 and the memory card 54 that is detachable from the card slot 52 are recording media, and for example, record video is recorded thereon. The internal memory 50 and the memory card 54 may be provided in a device (external) separate from the image capturing device main body 12. In this case, the control unit 46 may record the recorded video onto the external recording medium via a wired or wireless connection. The buffer memory 56 functions as a work memory for the control unit 46 .

[0046] [Regarding recording and switching of recorded video] The image capturing device 10 records a recording image cut out from a base image on a recording medium. The image capturing device 10 can also switch between recording images by changing the cut-out area in the base image. This function will be described with reference to FIG. 5. For ease of explanation, the subject in the video is not shown in FIG.

[0047] When the image capturing device 10 starts capturing an image, a reference image is displayed on a display (for example, the rear display 28). At this time, the full-angle image normally becomes the reference image, and the area surrounded by the outermost edge in Fig. 5 becomes the reference area As. Also, while electronic image stabilization is on, an extracted image extracted from the full-angle image becomes the reference image, and the area surrounded by the dashed frame in Fig. 5 becomes the reference area At.

[0048] When the user performs a predetermined operation before or during the start of shooting, the control unit 46 sets a setting area Ar within the reference areas As and At. The setting area Ar is a part of the reference areas As and At, and, for example, a subject in the reference video is reflected within the setting area Ar (see FIG. 6).

[0049] The position and size of the set area Ar can be adjusted automatically or based on a user operation. By performing a predetermined operation on the screen, the user can change the position and size of the set area Ar in which the subject is displayed, for example, according to the position and size of the subject in the reference video.

[0050] In this embodiment, the aspect ratio of the setting area Ar is determined to a value according to the recording format (movie format) of the recorded video. Therefore, when changing the size of the setting area Ar, the size is changed while the aspect ratio of the setting area Ar remains constant. However, the aspect ratio of the setting area Ar may be determined independently of the recording format of the recorded video, in which case the size of the setting area Ar may be changed while changing the aspect ratio.

[0051] When the setting area Ar is set, as shown in FIG. 5, a marker f is displayed in the reference image that is currently being displayed on the display. The marker f indicates the position of the set area Ar in the reference image and includes the boundary of the set area Ar. The boundary of the set area Ar is set to distinguish (distinguish) the set area Ar from its surroundings.

[0052] Examples of markers f that include boundaries include a frame that surrounds the set area Ar and a graphic object such as a pointer that indicates a representative position of the set area Ar. Differences in brightness, chromaticity, and shading between the set area Ar and its surroundings also fall under the category of marker f. Anything that can distinguish the set area Ar from its surroundings can also be included in the category of marker f. In the following, a sign f consisting of a colored rectangular frame will be used as an example.

[0053] A plurality of set areas Ar can be set in the reference image as shown in Fig. 5. In this case, a sign f is displayed for each set area Ar. Also, identification information (a number displayed in the upper right corner of the sign f in Fig. 5) is displayed for each sign f. The identification information is information that is individually assigned to each marker f in accordance with a predetermined assignment rule by the control unit 46. By displaying the identification information together with the marker f, the user can distinguish between the markers f. The identification information is not particularly limited and may be a number, a letter, a symbol, or a display color of each sign f. When numbers are used as the identification information, it is desirable that they be consecutive numbers.

[0054] Once the setting of the setting area Ar is complete, the user selects a recording area Ap, which is an area for recording video, from the multiple setting areas Ar that have been set. The control unit 46 selects a recording area Ap from the multiple setting areas Ar in accordance with the user's selection operation. This determines the setting area Ar in which the video within the area will be recorded, i.e., the recording area Ap.

[0055] When the recording area Ap is determined, the control unit 46 displays the markers f indicating the position of the recording area Ap and the markers f indicating the positions of the other setting areas Ar, among the markers f displayed in the reference image, in different modes. For example, the display mode of the markers f may include the color, brightness, density (degree of shading), shape, thickness and type of the border, and whether or not to flash the marker f. In the case shown in FIG. 5, the display color and thickness of the marker f indicating the position of the recording area Ap are different from those of the other markers f.

[0056] When the user issues a recording start instruction after the recording area Ap has been determined, recording of the image in the recording area Ap (hereinafter referred to as the recorded image) begins. At that point, the reference area (i.e., the orientation and angle of view of the image capturing device 10) is determined. Note that "the reference area is determined" means that the reference area is determined once at the time when the recording start instruction is received. Furthermore, even after the reference area is determined, the user can freely change the reference area. Incidentally, when the user changes the reference area, it is preferable to change the position of the set area in accordance with the position of the subject, as will be described later.

[0057] By recording a part of the base video as a recording video as described above, it is possible to record, for example, a zoomed video of a part of the base video that is of interest. Although the recorded video has a smaller angle of view than the reference video, if the reference video has high image quality (for example, if the number of pixels is 10 million or more), the recorded video will have a sufficiently high image quality. The number of pixels of the reference image is not particularly limited, but the lower limit is preferably 10 million or more, and more preferably 60 million or more. The upper limit is preferably 1 billion or less, and more preferably 500 million or less. If the number of pixels is greater than the above lower limit, the visibility of the recorded image is ensured. If the number of pixels is less than the above upper limit, the amount of data of the reference image is reduced, and processing by the control unit 46 is accelerated.

[0058] Furthermore, in this embodiment, after one set area Ar is selected from the multiple set areas Ar to be determined as the recording area Ap, the recording area Ap can be switched to another set area Ar during video recording. As a result, it is possible to record video in the recording area Ap before switching and video in the recording area Ap after switching. This will be explained in detail with reference to FIG. 6.

[0059] Assume that the image capturing device 10 captures multiple subjects (people) in a certain scene (location) and sets a set area Ar for each subject as shown in Fig. 6. In this case, one set area Ar selected by the user from the multiple set areas Ar becomes the recording area Ap. In the case shown in Fig. 6, four set areas Ar are set, and the set area Ar surrounded by the marker f with identification information "1" is the recording area Ap.

[0060] When the user issues a recording start instruction under the above circumstances, the video of the set area Ar, which is the recording area Ap at that time, is recorded. Then, if the user instructs to change the recording area Ap to another set area Ar (for example, the set area Ar surrounded by the marker f with the identification information "2") during video recording, the recording area Ap is reselected and the recorded video is switched. In this manner, in this embodiment, the recorded video can be switched by reselecting the recording area Ap during video recording, which makes it possible to capture zoomed video of each subject without the need to install individual cameras for each of the multiple subjects.

[0061] Furthermore, when the recorded video is switched, the control unit 46 records the recorded video before the switch and the recorded video after the switch, and combines these videos into one moving image. As a result, a moving image file (video file) is obtained in which the subject in the video changes as the recorded video is switched, as shown in Fig. 7. However, the present invention is not limited to this, and the recorded video before the switching and the recorded video after the switching may be recorded as separate file data without being combined. Also, the base video and the standby area described below may be recorded as separate file data.

[0062] In this embodiment, as shown in Figures 5 and 6, a marker f indicating the position in the base video is displayed for each of the multiple setting areas Ar including the recording area Ap. This allows the user to grasp the position of each setting area Ar during shooting. In other words, the display method of this embodiment improves convenience (ease of use) from the perspective of making it easier to grasp the position of the setting area Ar. This effect is particularly effective in a configuration in which the recording area Ap can be switched among the multiple setting areas Ar. In other words, by displaying a marker f for each setting area Ar, it is possible to easily grasp which part of the base video the recording video corresponds to after switching.

[0063] To further explain the set areas Ar, the number of set areas Ar (the set number) can be changed automatically in response to a change in the number of subjects in the reference video, for example, or can be changed based on a user operation. For example, as shown in Fig. 8, if a new subject enters the reference video, a new set area Ar can be added to the position of that subject. Conversely, if a subject in the reference video moves outside the base video, the set area Ar that was the video area of that subject can be automatically deleted.

[0064] Furthermore, when a subject moves within a certain set area Ar (for example, in FIG. 8, the set area Ar surrounded by a sign f with identification information "5"), the set area Ar can be displaced to follow the movement of the subject. This makes it possible to track a subject moving in the reference video. Here, subject tracking can be achieved by known subject detection technology using video analysis. Furthermore, if the set area Ar for tracking a moving subject is the recording area Ap, it is possible to record video of the moving subject.

[0065] Hereinafter, the set area Ar that tracks a moving subject will be referred to as a moving area Am. Note that the moving area Am is not limited to an area that moves to track a moving subject, but may be an area that moves in a predetermined direction at a constant or irregular speed.

[0066] [Video display flow] The processing flow (hereinafter referred to as the video display flow) executed by the control unit 46 of the image capturing device 10 will be described with reference to FIG.

[0067] When the user turns on the power of the image capturing device 10, the video display flow starts. At the start of the video display flow, the user performs initial adjustments to the image capturing device 10. In the initial adjustments, for example, the user adjusts the resolution (number of pixels) and aspect ratio of the recorded video by specifying a recording format, etc. Such adjustments are performed, for example, when the image capturing device 10 is first started up, and the adjustments are thereafter maintained. The above adjustments may be performed by the device manufacturer before shipping the photographing device 10. Also, the user can readjust the resolution, aspect ratio, etc. after starting up the photographing device 10 in order to review the resolution, aspect ratio, etc. that have already been adjusted.

[0068] In the video display flow, first, the control unit 46 starts the shooting process (S001). This shooting process is a process of shooting a reference video in the reference area at that time. As shooting starts, it is preferable that a real-time reference video be displayed as a live view video on the rear display 28 or the like. The shooting process continues until the user instructs the end of shooting or until the imaging device 10 is powered off.

[0069] Thereafter, the control unit 46 performs a setting process (S002). The setting process is a process of automatically setting a plurality of setting areas Ar in the shooting area of the reference video. In the setting process, the initial position and initial size of each setting area Ar are set, and it is possible to subsequently change the position and size of each setting area Ar based on a user operation or automatically in accordance with a predetermined rule.

[0070] After the setting step is completed, the control unit 46 executes the selection step (S003). The selection step is a step of selecting a recording area Ap from the plurality of setting areas Ar set in the setting step based on a selection operation by the user. When performing the selection process, the control unit 46 may display an image of each of the plurality of setting areas Ar (hereinafter referred to as "setting images") on the display. In this case, the user can check the setting image for each of the plurality of setting areas Ar.

[0071] Thereafter, when the user issues a command to start recording (S004), the control unit 46 starts the recording process (S005). The recording process is a process of recording the video of the selected recording area Ap onto the recording medium. The video is recorded in accordance with a predetermined recording format (moving image format). As described above, the reference area is determined at the time of receiving a recording start instruction, which is the trigger for starting the recording process.

[0072] During the recording step, the control unit 46 also performs a display step (S006). The display step is a step of displaying a plurality of setting images including the recording image, the reference image at that time, and the marker f in the reference image on the display. By viewing these images, the user can confirm the position of each setting area Ar in the reference image and determine which setting area Ar is the recording area Ap.

[0073] During the recording step, the control unit 46 also performs a control step (S007). The control step is a step of executing a control process related to a change of the reference area determined at the time of receiving the recording start instruction. In this embodiment, the control process includes a process for suppressing the change of the reference area, and is, for example, a control process related to zooming (optical zooming) by the optical device 19.

[0074] Furthermore, if the user instructs reselection of the recording area Ap during the recording step (S008), the control unit 46 performs a switching step (S009). The switching step is a step of switching the recording area Ap by reselecting the recording area Ap from among the multiple set areas Ar in accordance with the reselection instruction from the user after the selection step has been performed. When the switching step has been performed, the video of the recording area after the switch (recorded video) will be recorded in the subsequent recording step.

[0075] Then, the series of steps from the recording step (i.e., S005 to S009) are repeatedly performed until the user instructs the end of video recording (S010). On the other hand, when the user instructs the end of video recording, the recording step ends at that point. Thereafter, when an instruction to end the photographing is given or the power of the photographing device 10 is turned off (S011), the video display flow ends at that point.

[0076] [About the setup process] The setting process in the video display flow will be described. In this embodiment, when the setting process is performed, a first UI (User Interface) screen 101 shown in Fig. 11 is displayed on the rear display 28. The user selects either "recording pattern" or "automatic setting pattern" as the setting pattern for the setting area Ar, and presses one of buttons Bt1 and Bt2 on the UI screen 101 that indicates the selected pattern.

[0077] In the setting step when a recording pattern is selected, the control unit 46 reads out position information of the set area Ar stored in advance in the image capturing device 10. Then, the set area Ar is automatically set at the position indicated by the read-out position information. At this time, it is more preferable to highlight the image within the sign f in the reference image in order to make the sign f easier to see for the user. For example, the brightness of the image in the reference image other than the image within the sign f (i.e., within the boundary of the set area Ar) may be reduced or grayed out (see FIG. 18).

[0078] In this embodiment, when a recording pattern is selected, a second UI screen 102 is displayed on the rear display 28 as shown in Fig. 11. The user selects one of a plurality of recording patterns prepared in advance and presses one of buttons Bt3, Bt4, or Bt5 on the second UI screen 102 that indicates the selected pattern. As the recording pattern, for example, a "last used pattern," a "preset pattern," or a "user input pattern" can be selected.

[0079] On the other hand, in the setting step when the automatic setting pattern is selected, the control unit 46 automatically sets the setting area Ar at a predetermined position in the reference image based on the reference image being shot. In this embodiment, when an automatic setting pattern is selected, the third UI screen 103 is displayed as shown in FIG. 11. The user selects one of a plurality of pre-prepared automatic setting patterns and presses one of buttons Bt6, Bt7, or Bt8 on the third UI screen 103 that indicates the selected pattern. Examples of selectable automatic setting patterns include a "scene recognition pattern," a "subject detection pattern," and an "AF position identification pattern." In the case of an automatic setting pattern, it is preferable to display the markers f so that the setting areas Ar do not overlap each other at the start of the setting process. This makes it easier to distinguish between the multiple markers f and facilitates setting the setting areas Ar in the setting process.

[0080] The plurality of recording patterns and the plurality of automatic setting patterns will be described below.

[0081] (Last used pattern) When the previously used pattern is selected, the control unit 46 reads out position information of the setting area Ar set in the setting step of the previous (immediately preceding) video display flow. This position information indicates the position of each of the multiple setting areas Ar set in the setting step of the previous video display flow, and is stored, for example, in the internal memory 50 of the image capturing device 10. It is preferable that the stored position information be retained even after the power of the image capturing device 10 is turned off.

[0082] It should be noted that size information of the setting area Ar may be stored together with the position information of the setting area Ar. In this case, when the previously used pattern is selected, the position information and size information may be read out together.

[0083] Then, the control unit 46 automatically sets the setting area Ar to the position indicated by the read position information, i.e., the same position as the set position in the previous video display flow, as shown in Fig. 12. Such a previously used pattern is effective when shooting video with the same composition as the previous video display flow (for example, when shooting video at the same location and angle).

[0084] In the above embodiment, the read position information indicates the position of the setting area Ar set in the previous (immediately preceding) video display flow, but this is not limited to this. For example, the position information of the setting area Ar set in several setting steps performed in the past may be stored. In this case, the current setting area Ar may be set at the position of the setting area Ar set in the setting step specified by the user from the stored position information.

[0085] (Preset patterns) When a preset pattern is selected, the control unit 46 reads out the position information of the setting area Ar that is arranged in a predetermined standard arrangement pattern. The standard arrangement pattern is pre-registered in the photographing device 10, or is newly registered when the program is updated after purchase, for example.

[0086] When the position information relating to a standard arrangement pattern (preset pattern) is read, a fourth UI screen 104 shown in FIG. 13 is displayed on the rear display 28. A plurality of selectable patterns (patterns A to D in FIG. 13) are displayed on the fourth UI screen 104, and the user selects one of the preset patterns. The control unit 46 automatically sets the setting area Ar at a position corresponding to the preset pattern selected by the user. Such a preset pattern is effective when shooting video with a predetermined composition (for example, when shooting video at a specific location) and there is a pattern corresponding to the composition. The number of types of preset patterns, and the number and arrangement positions of the setting areas Ar in each preset pattern are not particularly limited and can be determined arbitrarily.

[0087] (user input pattern) When a user input pattern is selected, the control unit 46 reads out position information of the setting area Ar determined based on the user's operation. The user's operation here is an input operation for registering the position of the setting area Ar, and is performed in a stage prior to the setting process (for example, the stage of initial registration of the pattern). The input operation is performed, for example, through the input screen 105 for pattern registration shown in FIG. 14. The user touches and drags the inner area of the marker fx for area setting displayed on the input screen 105. As a result, the display position of the marker fx after dragging is registered as the position of the setting area Ar.

[0088] The control unit 46 automatically sets the setting area Ar at the position indicated by the read position information, i.e., at the setting position determined based on the user's input operation. Since such a user input pattern cannot be handled by the above-mentioned preset patterns, this is effective when preparing a pattern based on the user's input.

[0089] When the position of the setting area Ar is registered, the size of the setting area Ar may also be registered. That is, information indicating the position and size of the setting area Ar determined based on the user's input operation may be stored. In this case, each of the multiple setting areas Ar is set at the position and size registered as the user input pattern.

[0090] (Scene Recognition Pattern) In the setting process when a scene recognition pattern is selected, the control unit 46 recognizes the shooting scene of the reference video based on the reference video. The shooting scene of the reference video is, for example, the scenery, event, type of subject, scene (e.g., daytime view or night view), and situation (e.g., weather, etc.) to be shot as the reference video.

[0091] Note that known scene recognition techniques can be used to recognize the captured scene based on the reference video. For example, a technique for determining whether the brightness or color of a specific area of the video is within a preset range can be used. Alternatively, machine learning can be performed using a ground truth label indicating the captured scene of the video and the video data as a training dataset, and the captured scene of the reference video can be recognized using a scene recognition model obtained as a result of the training.

[0092] (Subject detection pattern) In the setting process when the subject detection pattern is selected, the control unit 46 detects a subject in the reference video and automatically sets a setting area Ar at the position of the detected subject. The subject in the reference video is a subject other than a landscape in the reference video that can be detected by the control unit 46, and includes, for example, a person, an animal, a vehicle, etc. When detecting an object, a known template matching technique may be applied. For example, an image of the object may be stored in advance as a template image, and the template image may be matched with each part of the reference video to detect an image similar to the template image as the object. Alternatively, a model that recognizes objects in a video may be constructed by machine learning, and the object in the reference video may be detected using the model. Known algorithms such as YOLO (You Only Look Once) or R-CNN (Region with Convolution Neural Network) may be used as the object recognition model algorithm.

[0093] In the subject detection pattern, the control unit 46 can set one set area Ar for each detected subject, as shown in Fig. 6. At this time, each set area Ar may be set so that the entire corresponding subject fits within it, or so that only a part of the corresponding subject (for example, the upper body including the face) fits within it.

[0094] According to the subject detection pattern described above, it is possible to automatically set a setting area Ar for a subject in a reference video without requiring any effort from the user. In this embodiment, an image of a recording area Ap selected from multiple setting areas Ar is recorded, and therefore the subject detection pattern is effective when recording an image of a subject in a reference video.

[0095] (AF position identification pattern) In the setting step when an AF position identification pattern is selected, the control unit 46 identifies a position in the reference image where a feature for determining the focus of the image capturing device 10 satisfies a criterion, and automatically sets the setting area Ar at the identified position. The feature for determining the focus is, for example, a feature for determining the in-focus position within the reference image. Specifically, if the autofocus is a phase difference AF (auto focus) method, the phase difference corresponds to the feature. Furthermore, if the autofocus is a contrast AF method, the contrast between multiple pixels corresponds to the feature. The AF method may be an AF method using directional light, or an AF method using DFD (depth from defocus).

[0096] Furthermore, the position that satisfies the criteria is a position where the above-mentioned characteristics (for example, phase difference or contrast) become characteristics when in focus. To put it simply, the control unit 46 identifies a position in the reference image that is in focus using the AF function as a position that satisfies the criteria. Then, as shown in FIG. 15, a setting area Ar is automatically set for the identified position (the in-focus position, which is the hatched position in FIG. 15). At this time, it is preferable that the size of the setting area Ar is also automatically set according to, for example, the area of the in-focus portion in the reference image.

[0097] The portion of the image that is in focus using the AF function is generally the subject of recording. In light of this, by setting the setting area Ar at the in-focus position, the in-focus image can be easily recorded, improving convenience for the user.

[0098] In the AF position identification pattern, when the control unit 46 sets a setting area Ar at a focused position, it automatically sets another setting area Ar accordingly. For example, as shown in FIG. 15, another setting area Ar is set at a symmetrical position of the focused position (for example, a position point-symmetrical when viewed from the center of the reference image). In this way, when an AF position identification pattern is selected, multiple setting areas Ar can be automatically set within the reference area. However, the position at which another setting area Ar is set is not limited to the above position.

[0099] Furthermore, when the entire reference image is in focus in the AF position identification pattern, the entire reference area may be set as one set area Ar. Furthermore, if a subject is not detected in the reference video and an appropriate setting position for the set area Ar cannot be found in the reference video, the set area Ar may be set near the center of the reference video.

[0100] In this embodiment, the in-focus position in the reference image is taken as an example of a shooting condition, and the set area Ar is set at a position in the reference image where the characteristics for determining the in-focus position satisfy the criteria (i.e., the in-focus position). Here, the shooting condition may be other than the in-focus position, and examples include exposure and white balance. Note that the shooting conditions referred to here exclude shutter speed and ISO sensitivity.

[0101] Taking exposure, which is one of the shooting conditions, as an example, a setting area Ar can be set at a position in the reference image where the characteristics for determining the exposure (e.g., pixel value or brightness) satisfy a standard. For example, it is advisable to set the setting area Ar at a position in the reference image where the exposure has been adjusted to an appropriate value by an AE (Automatic Exposure) function. Alternatively, the setting area Ar may be set at a position in the reference image where the exposure is relatively low and a position in the reference image where the exposure is relatively high.

[0102] Taking white balance, which is one of the shooting conditions, as an example, a setting area Ar can be set at a position in the reference image where the characteristics for determining white balance (e.g., chromaticity) meet the criteria. For example, it is advisable to set the setting area Ar at a position in the reference image where the white balance has been adjusted to an appropriate value using an AWB (Auto White Balance) function. As a specific example, if the reference image includes areas illuminated by light from multiple light sources, such as a lamp and the sun, a setting area Ar may be set for each illuminated area for each light source.

[0103] The user selects any pattern from the multiple arrangement patterns described above. As a result, in the setting process, the setting area Ar is set to reflect the user's will or preference. Furthermore, in the setting process, the setting area Ar is set automatically, so the user does not need to take any effort (setting work). In other words, the display method of this embodiment improves convenience (ease of use) from the perspective of saving the user effort and setting the setting area Ar in the reference area to reflect the user's will, etc. Note that the arrangement pattern may be an arrangement pattern that combines multiple arrangement patterns shown in FIG. 11. For example, multiple setting areas Ar may be set by simultaneously employing a scene recognition pattern and an AF position identification pattern.

[0104] It should be noted that the above-described patterns are merely examples, and arrangement patterns other than the above-described patterns may be added, or conversely, some of the above-described patterns may be omitted.

[0105] Furthermore, in the setting step of this embodiment, regardless of the type of selected pattern, if a characteristic location, which will be described later, exists in the reference video, a candidate Ac for the setting area is automatically set at the position of the characteristic location in the reference area. Then, when the user performs a predetermined operation on the set candidate Ac, a setting area Ar is set at the position of the set candidate Ac. The procedure for setting the candidates Ac in the setting step will be described below.

[0106] (About setting candidates) In the setting step, the control unit 46 determines whether or not a characteristic portion exists in the reference image based on the characteristics (e.g., pixel value) of each pixel in the reference image. A characteristic portion is a portion in the reference image where a visual change occurs (excluding those detected as subjects), such as a portion where there is movement in the background image and a portion where color, brightness, etc. change. Specific examples include a portion where there is a waterfall-like flow, a portion where lights are turned on or flashing like illuminations, and a portion where displayed characters, images, etc. change like on an electronic bulletin board. In the reference video shown in FIG. 16, for example, the decorative illumination appearing in the upper left part of the figure corresponds to a characteristic portion.

[0107] If a characteristic portion exists in the reference video, the control unit 46 performs a first setting step. The first setting step is a step of automatically setting a candidate Ac for the setting region at the position of the characteristic portion in the reference video.

[0108] When the first setting step is performed, in the display step performed thereafter, a mark (hereinafter referred to as a candidate mark K) indicating the position of the candidate Ac in the reference video is displayed as shown in FIG. 16. The candidate mark K is displayed in a manner different from the mark f indicating the position of the setting area Ar. For example, the candidate mark K in FIG. 16 is a substantially L-shaped graphic object representing the square of a rectangular area surrounding the characteristic location. The type and shape of the candidate mark K are not particularly limited, and may be, for example, a pointer in the shape of an arrow or a finger, or a graphic object consisting of a symbol such as a circle or a star. Alternatively, the candidate mark K may be the difference in shading (difference in brightness) between the candidate Ac in the reference video and its surroundings.

[0109] When the user performs an operation such as tapping on the candidate marker K during the display step, this triggers the execution of a second setting step. The second setting step is a step in which the control unit 46 sets a setting area Ar at the setting position of the candidate Ac where the user performed an operation (for example, a tap operation) on the candidate Ac. By performing the second setting step, a new setting area Ar is set in the area of the video of the characteristic location, as shown in FIG.

[0110] As described above, in this embodiment, a candidate setting area Ac can be automatically set for a characteristic portion in the base video that is not recognized as a subject but has undergone a visual change. Furthermore, by displaying a candidate indicator K indicating the position of the set candidate Ac in the base video, the user can easily confirm the candidate Ac. Furthermore, by the user performing a predetermined operation on the candidate Ac, a setting area Ar is set at the position of the candidate Ac. In this way, a setting area Ar can be set for a characteristic portion in the base video that interests the user. Furthermore, by selecting the setting area Ar set in this way as a recording area, it becomes possible to record the video of the characteristic portion.

[0111] [About the display process] The display process in the video display flow will be described in detail below. When the set area Ar is set in the setting step, the control unit 46 starts the display step. After that, the control unit 46 continues the display step until the recording step is completed (strictly speaking, until the shooting is completed).

[0112] The display step is a step of displaying on a display the recorded video, the reference video, and markers f indicating the positions of each of the plurality of setting areas Ar in the reference video. In the display step, the markers f are displayed for each setting area Ar in the reference video, as shown in Fig. 18. This allows the user to accurately grasp the position of each setting area Ar in the base video.

[0113] In the display step, as shown in FIG. 18, it is possible to display the images (hereinafter referred to as standby images) of the setting areas Ar other than the recording area Ap among the plurality of setting areas Ar. Furthermore, in the display step when the aforementioned characteristic location exists in the reference video, a candidate marker K indicating the position of the characteristic location is displayed together with the marker f in the reference video.

[0114] To explain an example of the display process, the recorded video, the reference video, and the standby video can be simultaneously displayed on the screen of the rear display 28 (hereinafter referred to as the display screen). For example, as shown in Fig. 18, the display screen is divided into a main screen G1, a sub-screen G2, and an information display screen G3. In this case, the standby video can be displayed on the sub-screen G2.

[0115] The main screen G1 occupies a relatively wide area of the display screen (for example, more than half of the display screen) and displays the reference video. The aspect ratio of the main screen G1 is preferably set to the same value as the aspect ratio of the reference video. The sub-screens G2 are smaller than the main screen G1 and are arranged in a row along the vertical or horizontal direction of the display screen. Each sub-screen G2 displays a recorded video or one standby video. It is preferable that the aspect ratio of each sub-screen G2 be set to the same value as the aspect ratio of the recorded video and the standby video. The information display screen G3 is a screen that displays text information other than video, and displays advice regarding video recording, for example, as shown in FIG.

[0116] The layout of the display screen (e.g., the size and position of each of the main screen G1 and sub-screen G2) is not limited to the form shown in FIG. 18, and can be freely designed according to the specifications of the device or the user's preferences, etc.

[0117] Furthermore, on the main screen G1, together with the reference image, a marker f is displayed for each of the plurality of setting areas Ar set in the setting step. The number of markers f (display number) displayed on the main screen G1 in the display step is variable, and the maximum number of markers f that can be displayed (upper limit number of display) can be displayed simultaneously. Furthermore, in order to make the sign f more visible to the user, it is more preferable to highlight the image within the sign f in the reference image. For example, as shown in Fig. 18, the brightness of the image in the reference image other than the image within the sign f (i.e., within the boundary of the set area Ar) may be reduced or grayed out.

[0118] In the display step, the marker f indicating the position of the recording area Ap and the marker f indicating the position of the other setting area Ar (hereinafter also referred to as a waiting area) are displayed in different modes. At this time, the marker f indicating the position of the recording area Ap is preferably displayed in an emphasized manner, for example, by making the border color a relatively conspicuous color such as red or fluorescent color, making the border thicker, or flashing. Also, the image within the recording area Ap (i.e., the recorded image) itself may be displayed in an emphasized manner.

[0119] Furthermore, if a moving subject is present in the reference video, the moving region Am set for that subject moves to follow the subject. In the display process, the marker f indicating the position of the moving region Am is displayed in a manner different from the marker f indicating the position of the set region Ar other than the moving region Am, as shown in Fig. 9. This makes it easy to distinguish between the moving region Am and the other set regions Ar, and to easily find the position of the moving region Am in the reference video.

[0120] When the subject within the moving area Am moves in the depth direction (i.e., toward or away from the image capturing device 10), the size of the subject in the reference image changes. Accordingly, the ratio of the size of the subject to the size of the moving area Am (hereinafter referred to as the subject ratio) changes. In this case, the control unit 46 uses the electronic zoom function to change the size of the moving area Am in accordance with the change in the subject ratio, adjusting it so that the subject ratio remains constant.

[0121] Furthermore, as the subject moves, the moving area Am may move to the vicinity of the edge of the reference area (i.e., the outer edge of the angle of view) or may protrude from the reference area. In such cases, it is preferable that the control unit 46 notify the user of the above situation by displaying a warning message on the information display screen G3 or outputting a warning sound.

[0122] Furthermore, when the reference image (i.e., the extracted image extracted from the image with the full angle of view) changes due to electronic image stabilization, the relative position of each set area Ar with respect to the reference area changes (see FIG. 4). In this case, control unit 46 calculates the amount of shift of the reference image due to electronic image stabilization, and adjusts the position of each set area Ar based on the calculated amount of shift. This allows the position of each set area Ar with respect to the reference area to be maintained before and after electronic image stabilization is performed. It is preferable that the function of adjusting the position of the setting area Ar in conjunction with electronic image stabilization can be freely switched on and off in response to a user operation.

[0123] Furthermore, in the display process, as shown in Fig. 18, the identification information of each marker f is displayed for each marker f. The identification information of each marker f is fixed during the display process, and remains unchanged even if the number of displayed markers f changes during the display process. For example, in Fig. 18, if the display of marker f with identification number "3" is stopped, the identification numbers of the other markers f (i.e., "1," "2," "4") do not change. By fixing the identification number of each marker f during the display process in this way, it is possible to easily grasp where each setting area Ar is located, even if, for example, the number of displayed markers f changes.

[0124] The sub-screens G2 display a plurality of setting images including recorded images and standby images. Specifically, each of the sub-screens G2 displays a corresponding setting image. If the aspect ratio of each sub-screen G2 is the same as the aspect ratio of the corresponding setting image, the setting image can be displayed so that it fits snugly on the screen.

[0125] Furthermore, it is preferable that the recorded video be highlighted among the multiple setting videos displayed. For example, it is preferable that the sub-screen G2 on which the recorded video is displayed and the sub-screen G2 on which the other setting videos are displayed differ in the color of the screen frame, the thickness or type of the frame line, or whether or not it is lit. This allows a user viewing multiple setting videos on the display to easily distinguish which video is the recorded video.

[0126] Also, as shown in Figure 18, on the sub-screen G2 (the second sub-screen from the top in Figure 18) that displays the image of the moving area Am, it is advisable to display information indicating that the image is of the moving area Am (for example, text information such as "moving").

[0127] In this embodiment, a plurality of modes for performing the display process are prepared, and the image display method in the display process changes for each mode. That is, the display process is performed according to the mode specified by the user. For example, when the user specifies the first mode, a first display step is performed as the display step. In the first display step, as shown in Fig. 18, the reference video and the indicator f for each setting area Ar are displayed on the main screen G1, and the recorded video and standby video are displayed on the sub-screen G2. On the other hand, when the user specifies the second mode, a second display step is performed as the display step. In the second display step, as shown in Fig. 19, the recorded video and standby video are displayed, and the reference video and the indicator f for each setting area Ar are not displayed.

[0128] In the first display step, the recorded image, the standby image, the reference image, and the marker f for each setting area Ar do not necessarily have to be displayed on the same display (for example, the rear display 28). That is, as shown in Fig. 20, the image displayed on the main screen G1 and the image displayed on the sub-screen G2 may be displayed on different displays. To give a specific example, the image displayed on the main screen G1 may be displayed on the rear display 28, and the image displayed on the sub-screen G2 may be displayed on the electronic viewfinder 29 or an external monitor (not shown) connected to the image capturing device 10.

[0129] As described above, in this embodiment, the display process includes a first display process and a second display process, and one of these two display processes designated by the user is executed. This allows the displayed image to be switched according to the user's needs. For example, if the user wishes to check the reference image and marker f, the first display process may be executed to display the reference image and marker f. On the other hand, if the user wishes to check only the recorded image and standby image (i.e., the setting image), the second display process may be executed to display only the recorded image and standby image.

[0130] Furthermore, it is preferable that the implementation mode of the display process, i.e., which of the first display process and the second display process is to be implemented, can be freely switched in response to a switching operation by the user during the implementation of the display process. For example, it is preferable that the user can switch which display process to adopt via the touch panel 36 or the operation button 30 shown in FIG. 2.

[0131] In the second display step, the recorded video and other setting video may be displayed on the same display. For example, as shown in Fig. 19, the recorded video may be displayed on a main screen G1 of the display screen, and the standby video may be displayed on a sub-screen G2. Alternatively, in the second display step, the recorded image and the other setting image may be displayed on different displays. For example, as shown in Fig. 20, the recorded image may be displayed on the rear display 28, and the other setting image may be displayed on the electronic viewfinder 29 or an external monitor (not shown) connected to the image capturing device 10.

[0132] The information display screen G3 displays, for example, suggested information, which will be described later. The information display screen G3 can also display advice for the user about photography, warning messages, etc. For example, if adjustments to the exposure or ISO sensitivity are required during the recording process, a message to that effect may be displayed. Furthermore, the information display screen G3 can display information other than the above, such as the currently used shooting mode, current values related to shooting conditions such as resolution and ISO sensitivity, and the on / off status of various functions such as electronic zoom.

[0133] (Regarding display of video settings based on priority) In the display process, the number of sub-screens G2 is determined according to at least one of the resolution and aspect ratio of the recorded video (hereinafter simply referred to as "at least one item"). In other words, the number of setting videos to be displayed on the sub-screens G2 is determined according to at least one of the items. Similarly, the size of each sub-screen G2 is determined according to at least one of the items. That is, the display size of the setting video is determined according to at least one of the items.

[0134] Here, at least one of the items is adjusted according to the recording format (video format) of the recorded video. Therefore, the number of display (strictly speaking, the maximum number of display) and the display size of the setting video in the display process are determined according to the recording format of the recorded video.

[0135] In this embodiment, the number of setting images to be displayed and the display size are determined according to at least one of the items, so that each of the setting images, including the recorded image, can be displayed well. For example, each setting image can be displayed at an appropriate resolution so that it fits comfortably on the sub-screen G2.

[0136] In this embodiment, the aspect ratio of the recorded video is determined according to the recording format of the recorded video, but the aspect ratio of the recorded video may be set arbitrarily regardless of the recording format. In this case, as the aspect ratio of the recorded video is adjusted, the number and size of the sub-screens G2 are changed according to the aspect ratio after the adjustment.

[0137] In addition, in this embodiment, both the number and size of the setting images displayed in the display step (i.e., the number and size of the sub-screens G2) are determined according to at least one of the items. However, this is not limiting, and either the number or size of the setting images may be determined according to at least one of the items.

[0138] The number of setting images to be displayed in the display step does not have to be determined based on at least one of the items. For example, the user may set or change the number of setting images to be displayed.

[0139] In this embodiment, the number of setting videos to be displayed in the display step is determined according to the resolution and aspect ratio of the recorded video, as described above. Here, in the first display step, one of the setting videos to be displayed is the recorded video, and the rest are standby videos. Therefore, the setting value for the number of standby videos to be displayed, i.e., the maximum number to be displayed, is determined according to the resolution and aspect ratio of the recorded video, and specifically, is one less than the number of sub-screens G2. In the case shown in FIG. 21, the maximum number of standby videos that can be displayed is three.

[0140] On the other hand, the number of set areas Ar in the reference area changes depending on the subject and the like in the reference video, and accordingly the number of set areas Ar other than the recording area Ap, that is, the number of waiting areas, changes. In the case shown in FIG. 21, the number of waiting areas is six.

[0141] When the number of the plurality of waiting areas reaches (strictly speaking, exceeds) the maximum number of displayable standby videos, the control unit 46 performs a display step according to the flow shown in Fig. 22. To explain the flow of the display step shown in Fig. 22, the control unit 46 performs a determination step when the number of the plurality of waiting areas reaches the maximum number of displayable standby videos (S021).

[0142] In the determination step, the control unit 46 determines a priority for each of the plurality of standby areas. Explaining the procedure, the control unit 46 first detects a subject in each of the plurality of standby images (S101). Next, the control unit 46 acquires information about the detected subject in the standby image (S102). Note that at this time, the control unit 46 may acquire performance information as a recording area, which will be described later, instead of the information about the subject in the standby image.

[0143] The information about the subject includes, for example, the time or number of times the subject appears in the reference video during the shooting period, and the size (display size) of the subject in the reference video. This information can be obtained by the control unit 46 analyzing and measuring or counting the reference video during the shooting period. Furthermore, if the subject is a person, the person's face may be recognized using a technique such as pattern matching, the importance of the person whose face is recognized may be determined, and the result of the determination may be used as information about the subject. The importance of a person may be determined, for example, by determining the attributes of the person whose face is recognized (e.g., family, relatives, acquaintance, or stranger) and then based on the determination result. Furthermore, if there are multiple detected subjects, the priority may be determined based on the category of the subject (e.g., person, animal, object, landscape, etc.). Furthermore, the priority may be determined based on the subject that was detected earliest. Naturally, other information about the subject may be considered in addition to the above. It is desirable that the user be able to freely select what information to use as information about the subject in the determination step.

[0144] Thereafter, in a determination step, the control unit 46 determines the priority of each of the plurality of waiting areas according to the information on the detected subject (S103).

[0145] After the determination step is performed, the control unit 46 selects N standby areas from the plurality of standby areas based on the priorities (S022). Here, the number N of selected standby areas is the maximum number of standby videos that can be displayed, and is a number determined according to the number of sub-screens G2 (for example, a number that is one less than the number of sub-screens G2). In step S022, the control unit 46 selects N standby areas in descending order of priority.

[0146] Then, the control unit 46 displays the images (standby images) of the selected N standby areas on the sub-screen G2 (S023). Here, when the sub-screens G2 are arranged vertically on the display screen as shown in Fig. 21, it is desirable to display the standby images of standby areas with higher priorities on the upper sub-screens G2.

[0147] As described above, in this embodiment, when the number of standby areas reaches (exceeds) the maximum number of standby videos to be displayed, N standby videos selected according to priority are displayed. This allows standby videos with high priority to be displayed appropriately while complying with screen display restrictions.

[0148] The method of determining the priority is not limited to a method based on information about the subject in the standby video. For example, the priority may be determined according to performance information as a recording area. To explain in more detail, for each of the multiple standby areas, the number of times it has been selected as a recording area from the start of shooting to the present time, the length of time it has been a recording area, or the elapsed time since it was last selected as a recording area is identified. This information corresponds to information about the performance of being selected as a recording area in the past. The priority of each of the plurality of waiting areas may be determined based on the information on the performance. For example, if the information on the performance is the number of times or the time for which the waiting area has been selected as a recording area, the higher the number of times or the time for which the waiting area has been selected, the higher the priority may be assigned.

[0149] In addition, in the flow of the display process shown in Fig. 22, N waiting areas are selected based on priority and only the images of the selected waiting areas are displayed, but this is not limited to this. For example, the top N waiting areas and lower waiting areas may be selected in order of priority. In this case, the waiting images of the top N waiting areas may be displayed first, and the waiting images of the lower waiting areas may be displayed by scrolling or switching the display screen as shown in Fig. 23.

[0150] Furthermore, each of the multiple standby videos may be displayed in a size according to the priority. For example, as shown in Fig. 24, the standby video of a standby area with a high priority may be displayed on a normal-sized sub-screen G2, and the standby video of a standby area with a low priority may be displayed on a reduced-sized sub-screen G2x. In this way, each of the multiple standby videos can be displayed, and the standby area with a high priority can be displayed more easily.

[0151] [Regarding the control process] The control process in the video display flow will be described. The control process is a process of executing a control process to suppress changes to the reference area, and is performed at least after the reference area is determined. In other words, the control process is a process to suppress changes to the reference area after a user's instruction to start recording is received.

[0152] In the control process, the control unit 46 locks the zoom driver 21 to stop the movement of the zoom optical device 18. This limits the optical zoom function during the recording process, fixes the reference area, and maintains the angle of view at that time. By executing the above control process, it is possible to prevent the relative position of each set area Ar with respect to the reference area from changing during the recording process, thereby avoiding a situation in which each set area Ar deviates from the reference area and each set image, such as a recording image, is not captured properly.

[0153] Note that the control process performed to suppress the change of the reference area is not limited to the above process, and may be, for example, a process for notifying suggested information regarding the change of the reference area. To give a specific example, a message prohibiting optical zoom may be displayed on the information display screen G3, or may be played as audio. Alternatively, as shown in FIG. 18, a message prohibiting the operation of moving the image capture device 10 (for example, panning) or a message recommending that the image capture device 10 be fixed using a platform such as a tripod may be displayed.

[0154] The control step may also be performed before the reference area is determined (i.e., before a recording start instruction is received), or may be performed during the setting step, in which case it is possible to prevent the reference area from changing while multiple setting areas Ar are set within the reference area.

[0155] [About the change process] When a predetermined condition is satisfied during the recording step, the control unit 46 executes a change step. The change step is a step of executing a change process related to changing the number of images in the setting area Ar (i.e., the setting images) to be displayed. By executing the change process, the number of setting images to be displayed can be dynamically increased or decreased during the recording step. The processing flow during the recording step will be described below with reference to the flow shown in Fig. 25. Incidentally, in Figs. 26 to 31 referred to in the following description, the grayed-out display of images other than the image within the sign f is omitted for convenience of illustration.

[0156] During the recording step, the control unit 46 monitors the reference video and determines whether the number of subjects in the reference video has increased (S031). If the number of subjects in the reference video has increased, the control unit 46 executes the modification step (S032). In this modification step, a modification process is executed to increase the number of displaying setting videos.

[0157] In the change process in step S032, a setting area Ar is added in accordance with the increase in the number of subjects in the base video, and the setting video of the added setting area Ar is newly displayed on the display. As a result, the number of setting videos displayed increases. For example, as shown in FIGS. 26 and 27, if the number of subjects in the base video increases from three to four, the number of setting videos displayed (i.e., the number of videos displayed on the sub-screen G2) increases from three to four.

[0158] In addition, if a setting area Ar is added as the number of subjects in the reference image increases, the number of markers f in the reference image displayed on the main screen G1 will increase accordingly by the amount of the added setting area Ar (see Figure 27).

[0159] Furthermore, in the above embodiment, the setting area Ar is set at the position of a new subject in the base video, but this is not limited to this. For example, a setting area candidate Ac may be set instead of the setting area Ar. In this case, a sign indicating the position of the candidate Ac (more specifically, the aforementioned candidate sign K) is displayed in the base video. Then, when the user performs an operation such as tapping on the candidate sign K, this triggers the addition of the setting area Ar to the position of the candidate Ac. As a result, a setting video for the added setting area Ar is newly displayed, and the number of displayed setting videos increases.

[0160] Furthermore, instead of setting a setting area Ar or its candidate Ac at the position of a new subject in the reference video, suggestion information regarding the addition of a setting area Ar (i.e., a change in the number of setting videos to be displayed) may be output. For example, a message suggesting adding a setting area Ar to the position of the new subject to increase the number of setting videos to be displayed may be displayed on the information display screen G3 or played as audio, etc. In this way, a process of outputting suggestion information regarding a change in the number of setting videos to be displayed may be executed as a change process.

[0161] In the flow shown in Fig. 25, the control unit 46 monitors the reference video during the recording step and determines whether the moving area Am in the reference area overlaps with another set area Ar (S033). If the moving area Am overlaps with another set area Ar as shown in Fig. 28 or is about to overlap with another set area Ar, the control unit 46 outputs a warning message or a warning sound (S034). This is because, when the moving area Am overlaps with another set area Ar, a subject in one area may be behind a subject in the other area and not be captured.

[0162] By outputting a warning message or the like, the user can be notified of a situation in which the moving area Am overlaps or is about to overlap with another setting area Ar. The user who receives the warning can take appropriate measures (for example, resetting the setting area Ar, etc.). As one measure, when the user makes a change request through a screen operation or the like (S035), the control unit 46 executes a change step (S036). In this change step, a change process is executed to reduce the number of displaying setting images.

[0163] In the change process in step S036, the overlapping moving area Am and other set areas Ar are updated into a single set area Ar, and the set image of the updated set area Ar is displayed on the display. For example, the image of the moving area Am and the image of the other set area Ar, which are displayed separately as in Fig. 28, are integrated into a single image including the subjects of each of these two images, as in Fig. 29. As a result, the number of set areas Ar is reduced by one, and the number of displayed set images is also reduced by one.

[0164] It is preferable that the size and position of the updated set area Ar be set so that the subjects of the two images are accommodated within it. Furthermore, the identification information of the marker f indicating the updated set area Ar may take over the identification information of the marker f indicating either of the two areas before the update (i.e., the moving area Am or the other set area Ar). Alternatively, a new identification number may be assigned to the updated set area Ar.

[0165] 25, the control unit 46 periodically determines whether each of the plurality of setting areas Ar satisfies a stop condition during the recording process (S037). The stop condition is a criterion for determining whether to stop displaying the image (setting image) of the setting area Ar, and includes, for example, the following conditions (1) to (3): Cancellation condition (1): At least a part of the set area Ar is outside the reference area due to optical zoom or the like. Cancellation condition (2): There is no change in the setting image of the setting area Ar for a certain period of time or more, or there is no subject in the setting image. Cancellation condition (3): The set area Ar has not been selected as a recording area for a certain period of time or more. The stop conditions are not limited to the above conditions (1) to (3), and conditions other than those mentioned above may also be included in the stop conditions.

[0166] Then, if there is a setting area Ar that satisfies any of the above-mentioned cancellation conditions (1) to (3), the control unit 46 carries out a change step (S038). In this change step, a change process is executed to reduce the number of displaying setting videos.

[0167] In the change process in step S038, the display of the setting video of the setting area Ar that satisfies the cancellation condition is cancelled. For example, in Fig. 26, if the setting area Ar in which the identification information of the marker f is "2" satisfies the cancellation condition, the display of the setting video of that setting area Ar is cancelled (disappears from the sub-screen G2), as shown in Fig. 30. As a result, the number of displayed setting videos is reduced by one. As for the setting area Ar where the display of the setting video is stopped (hereinafter referred to as the stopped area), the display of the indicator f indicating the position of the stopped area is also stopped on the main screen G1 as shown in FIG.

[0168] Incidentally, instead of immediately ceasing the display of the setting video in the setting area Ar that satisfies the cancellation condition, suggestion information regarding ceasing the display of the setting video (in other words, changing the number of setting videos displayed) may be output. For example, a message recommending ceasing the display of the setting video in the setting area Ar that satisfies the cancellation condition may be displayed on the information display screen G3 or played as audio, etc. Meanwhile, a different video (video different from the setting video) may be displayed on the sub-screen G2 instead of the setting video in the setting area Ar that satisfies the cancellation condition.

[0169] When the display of the setting image of the stopped image is stopped, the stopped area is stored in the internal memory 50 or the like in the image capturing device 10 while the recording process is being performed. The stopped area may be stored until the end of image capturing, or until the power of the image capturing device 10 is turned off.

[0170] Furthermore, when the display of the setting video of the stop area is stopped, an operation icon such as a back button Bt9 is displayed on the main screen G1 as shown in Fig. 30. The user can issue an instruction to redisplay the setting video of the stop area by, for example, tapping the back button Bt9. When the user issues an instruction to redisplay (S039), the control unit 46 executes a change step (S040). In this change step, a change process is executed to redisplay the stored setting video of the stop area based on the user's instruction to redisplay.

[0171] By the change process in step S040, the video in the stop area is displayed again on the sub-screen G2, and the number of displayed setting videos returns to the number before the display of the setting video in the stop area was stopped. Note that the setting image of the stop area that can be redisplayed is not limited to the image of the immediately previous stop area (the setting area where the display of the setting image was most recently stopped). For example, it may be possible to redisplay images of the past several stop areas during the current shooting period. In this case, the shooting device 10 stores the past several stop areas, and the user specifies which setting image of the stop area to redisplay and instructs the redisplay of the image.

[0172] 25, the control unit 46 monitors the reference video during the recording step, and determines whether the number of subjects in the reference area exceeds the upper limit of the number of set areas Ar (S041). If the number of subjects exceeds the upper limit of the number of set areas Ar, the control unit 46 executes a change step (S042). In this change step, a change process is executed to reduce the number of displaying set videos. In the change process in step S042, a message is displayed on the information display screen G3 or played as audio, informing the user that the number of subjects has exceeded the upper limit of the number of set areas Ar and recommending that the number of set videos be reduced. This prompts the user to review the settings of the set areas Ar and delete unnecessary set areas Ar, such as the set areas Ar that have not been selected as recording areas for the longest period of time. As a result, the number of set videos to be displayed is reduced by the number of set areas Ar deleted by the user.

[0173] The series of steps S031 to S042 described above are repeatedly performed during the recording process, and end when the recording process ends (S043). As described above, in this embodiment, the change process is performed in response to a change in the reference image (for example, the addition of a subject) or the establishment of a cancellation condition, and the number of displayed setting images is changed appropriately. This allows the user to easily check the number of setting images appropriate for the current situation, and also allows the user to omit checking images such as the setting area Ar that are unlikely to become a recording area. As a result, convenience for the user is improved.

[0174] <<Other embodiments>> The embodiment described above is a specific example given to clearly explain the display method of the present invention, and is merely an example, and other embodiments may be considered. In the above-described embodiment, the image capturing device 10 functions as a display device to display various images, but this is not limiting. For example, the display device may be configured with the image capturing device 10, a camera controller connected to the image capturing device 10 via a wired or wireless connection, and an external display. In this case, the camera controller functions as the control unit 46, causes the image capturing device 10 to capture a reference image and display various images on the external display. The camera controller may also set the setting area Ar, select and reselect (switch) the recording area, and record the recording image.

[0175] In the above embodiment, a control process for suppressing changes to the reference area is executed in the control step in the video display flow. However, this is not limited to this, and a process for prompting changes to the reference area may be executed as a control process related to changes to the reference area. In this case, the reference area can be adjusted in accordance with changes in the reference video (for example, movement of a subject in the reference video).

[0176] To explain in more detail, suppose that the recording image is zoomed in using electronic zoom during the recording process. If the zoom magnification exceeds a predetermined value, the resolution of the recorded image may fall below a certain value, making it difficult to obtain the desired image quality. In this case, as shown in FIG. 31, optical zoom may be performed instead of electronic zoom, and information prompting the user to change the reference area may be displayed on the information display screen G3. Alternatively, the control unit 46 may operate the zoom driver to forcibly perform optical zoom and automatically change the reference area so that the resolution is equal to or greater than a predetermined value. When the reference area changes due to optical zoom, the relative position of each set area Ar with respect to the reference area also changes. In this case, it is preferable that the control unit 46 calculates the amount of change in the reference area and adjusts the position and size of each set area Ar based on the calculated amount of change. This allows the relative position of each set area Ar with respect to the reference area to be maintained before and after zooming.

[0177] During the recording process, the moving area Am may follow the movement of the subject and reach the edge of the reference area, or at least a part of the moving area may extend beyond the reference area. In such cases, information may be displayed on the information display screen G3 to prompt the user to move the image capturing device 10 so that the moved moving area Am fits within the reference area (i.e., to change the reference image). Here, if the user changes the reference area by moving the photographing device 10 during the recording process, a known detection sensor such as a gyro sensor provided in the photographing device 10 may detect the movement (e.g., the amount of displacement) of the photographing device 10. Then, it is preferable that the control unit 46 adjusts the position and size of each set area Ar based on the detected movement of the photographing device 10. This allows the relative position of each set area Ar with respect to the reference area to be adjusted according to the changed reference area. At this time, any set area Ar that moves outside the reference area due to the change of the reference area may be automatically deleted.

[0178] In the above embodiment, the recorded video is a moving image, that is, a collection of multiple frame images captured continuously at a constant frame rate. However, the recorded video is not limited to a moving image and may be a still image. For example, the control unit 46 displays a reference image on the main screen G1 as a live view image and sets multiple setting areas Ar within the reference area. The user selects a recording area Ap from the multiple setting areas Ar, and the control unit 46 displays the multiple setting images including the recording image on the sub-screen G2. Then, when the user issues a recording instruction by operating the release button 26, the control unit 46 may record the recording image as a still image on a recording medium.

[0179] In the above embodiment, when a subject is detected in the reference video, a setting area Ar is automatically set at the position of the detected subject. However, the setting area Ar is not limited to being set immediately at the position of the detected subject, and a setting area candidate Ac may be first set at the position of the detected subject. In this case, a candidate marker K may be displayed at the position of the candidate Ac, and when the user performs a predetermined operation on the candidate marker K, the setting area Ar may be set at the position of the candidate Ac.

[0180] In the above embodiment, for a setting area Ar that satisfies the cancellation condition, the display of the setting image is stopped, but the image is stored in the imaging device 10 during the recording process. However, this is not limited to this, and the setting area Ar that satisfies the cancellation condition may be deleted, and the image of that area may not be stored. In this case, it is preferable to display suggested information regarding the deletion of the setting area Ar on the information display screen G3 at a stage before deleting the setting area Ar that satisfies the cancellation condition. Then, it is preferable to separately display a screen (not shown) for selecting whether or not to delete the setting area Ar, and delete the setting area Ar that satisfies the cancellation condition after confirming the user's intention regarding whether or not to delete it.

[0181] In the above-described embodiment, the photographing device is a digital camera, but it may be a video camera, a mobile phone with an imaging optical system, a smartphone, a tablet terminal, etc. The imaging lens may be a lens unit that is externally attached to the imaging optical system of the mobile terminal. [Explanation of symbols]

[0182] 10 Imaging equipment 12 Imaging device body 13 Mount 14. Photographic lenses 17 Optical unit 18,19 Optical equipment 20 apertures 21 Zoom drive unit 22 Focus drive unit 23 Aperture drive unit 26 Release button 28 Display 29 Electronic viewfinder 30 Operation buttons 36 Touch Panel 38 Shutter 40 Image sensor 42 pixels 44 Analog signal processing circuit 46 Control Unit 47 Controller 48 Video Processing Unit 50 internal memory 52 card slots 54 Memory Card 56 Buffer Memory 61 Camera Controller 62 External Display 101 First UI screen 102 Second UI Screen 103 Third UI Screen 104 Fourth UI Screen 105 Input screen Ac candidate Am moving area Ap Recording Area Ar setting area As, At imaging area Bt1~Bt8 buttons Bt9 Back button f,fx sign G1 Main Screen G2, G2x subscreen G3 information display screen K candidate indicator L1 optical axis T Table Data

Claims

1. A display method for displaying an image captured by a photographing device, comprising: a setting step of automatically setting a plurality of area candidates within a reference area, which is a shooting area of a reference video; a selection step of selecting a recording area, which is an area of a recording video to be recorded, from the plurality of area candidates based on a selection operation by a user; a display step of displaying a setting image which is an image of an area located at the position of the candidate; a switching step of reselecting the recording area from among the plurality of area candidates and switching the recording area after the selection step is performed, the display step includes a first display step of displaying the reference video, the recorded video, and the setting video, and a second display step of displaying the recorded video and the setting video, and is switchable between the first display step and the second display step; Display method.

2. The display method according to claim 1 , wherein the setting step detects a characteristic location where a visual change occurs in the reference image, and automatically sets the candidate at the position of the detected characteristic location in the reference area.

3. The display method according to claim 1 or 2, wherein the setting step recognizes a shooting scene of the reference video based on the reference video, and automatically sets the candidate at a position in the reference area corresponding to the recognized shooting scene.

4. 4. The display method according to claim 1, wherein the setting step automatically sets the candidate at a position in the reference image where a feature for determining the shooting conditions of the shooting device satisfies a criterion.

5. The display method according to claim 4 , wherein the shooting conditions are at least one of a focus position, an exposure amount, and a white balance in the reference image.

6. 6. The display method according to claim 1, wherein in the setting step, the candidate is automatically set based on pre-stored position information.

7. The display method according to claim 6, wherein the location information includes at least one of the location information of the candidate set in the setting process previously performed and the location information of the candidate determined based on a user operation.

8. The display method according to claim 1 , further comprising an adjustment step of adjusting at least one of the position and size of the candidate when the size or position of the reference area has changed.

9. a recording step of recording a moving image of the recorded video; The display method according to claim 1 , further comprising: a change step of executing a change process for changing the number of displaying said setting video images during said recording step.

10. The display method according to claim 9 , wherein the change process includes a process of increasing the number of displayed images in accordance with an increase in the number of subjects in the reference video.

11. The display method according to claim 9 or 10, wherein the change process includes a process of reducing the number of displayed images by canceling the display of the setting image in an area that satisfies a cancel condition.

12. a stop area where the display of the setting image is stopped is stored while the recording step is being performed; The display method according to claim 9 , wherein the change process includes a process of redisplaying the setting image of the stored suspension area based on a redisplay instruction from a user.

13. The display method according to claim 9 , wherein the change process includes a process of outputting proposal information regarding a change in the number of items to be displayed.

14. The display method according to claim 9 , wherein, when the number of subjects in the reference video exceeds a predetermined value, the change step executes the change process related to a decrease in the number of displayed images.

15. A program for causing a computer to execute each step included in the display method according to claim 1.

Images