Display method

Abstract

To improve convenience to a user by a display method for video capable of cutting a part of a photographic video and changing over the cut video.SOLUTION: A display method for displaying a video photographed by an imaging device comprises: a setting process of setting a plurality of set regions in a reference region as a photographic region of a reference video; a selecting process of selecting a recording region as a region of a recorded video out of the plurality of set regions; a changing-over process of selecting the recording region out of the plurality of set regions and changing over the recording region after the selecting process; a display process of displaying a plurality of set videos as videos of the plurality of set regions; and a control process of executing control processing related to change of the reference region at least after determining the reference region. The control processing includes processing to maintain a state in which an optical zooming function of the imaging device is restricted throughout the recording process.SELECTED DRAWING: Figure 18

Description

Technical Field

[0001] One embodiment of the present invention relates to a method for displaying video.

Background Art

[0002] A technique for extracting and storing the video of any one of a plurality of subjects in a video captured by a single imaging device has already been developed. As an example, for instance, the technique described in Patent Document 1 can be cited. In the technique described in Patent Document 1, a plurality of different video regions are set within the imaging range of the imaging device, and the video of one of the regions is recorded. Further, in the technique described in Patent Document 1, the region of the video to be recorded can be changed (switched) among the plurality of regions. Thereby, for example, while keeping the imaging device fixed, it is possible to obtain a zoomed-in video for each of a plurality of subjects in the captured video.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] One embodiment of the present invention has been made in view of the above circumstances, and aims to improve the convenience for the user in a video display method of extracting a part of a captured video and being able to switch the extracted video.

Means for Solving the Problems

[0005] In order to achieve the above object, one embodiment of the present invention is a display method for displaying an image captured by a photographing device, the display method including: a setting step of setting a plurality of setting areas in a reference area which is a photographing area of a reference image; a selection step of selecting a recording area which is an area of a recorded image from among the plurality of setting areas; a switching step of reselecting the recording area from among the plurality of setting areas after the selection step is performed and switching the recording area; and a display step of displaying the recorded image, the reference image, and a mark indicating the position of each of the plurality of setting areas in the reference image.

[0006] Further, the mark includes a boundary of the setting area, and in the display step, an image within the boundary in the reference image may be highlighted.

[0007] Further, at the start of the setting step, the mark may be displayed in a state where the respective setting areas do not overlap each other.

[0008] Further, in the display step, a mark indicating the position of the recording area in the reference image and a mark indicating the position of a standby area which is a setting area other than the recording area may be displayed in different manners.

[0009] Further, in the display step, a mark indicating the position of a moving area which is a setting area that moves following a moving subject may be displayed in a manner different from a mark indicating the position of a setting area other than the moving area.

[0010] Further, in the display step, the recorded image and the reference image may be displayed on different displays.

[0011] Further, the display step includes a first display step of displaying the recorded image, the reference image, and the mark, and a second display step of displaying the recorded image and not displaying the reference image and the mark, and one of the first display step and the second display step designated by the user may be performed.

[0012] Further, in the second display step, the recorded image and a standby image which is an image of a setting area other than the recording area may be displayed on different displays.

[0013] Also, the number of signs displayed in the display step may be variable, and in the display step, the identification information of the signs may be displayed for each sign. In that case, when the number of signs displayed is changed, it is preferable that the identification information set for each sign is maintained before and after the change in the number of signs displayed.

[0014] Also, one embodiment of the present invention is a display method for displaying an image captured by a photographing device, including a setting step of setting a plurality of setting areas in a reference area that is a photographing area of a reference image, a selection step of selecting a recording area, which is an area of a recorded image to be recorded, from among the plurality of setting areas, a switching step of reselecting the recording area from among the plurality of setting areas and switching the recording area after the selection step is performed, a determination step of determining priorities for a plurality of standby areas when the setting areas other than the recording area are used as standby areas, and a display step of displaying a plurality of standby images that are images of the plurality of standby areas. When the number of the plurality of standby areas set in the reference area reaches a set value related to the number of standby images to be displayed, in the display step, a standby image of a standby area selected based on the priority is displayed among the plurality of standby areas, or the plurality of standby images are displayed in sizes corresponding to the priorities.

[0015] Also, in the determination step, a subject in the plurality of standby images may be detected, and the priority of each of the plurality of standby areas may be determined according to information regarding the detected subject.

[0016] Also, in the determination step, the record of a standby area having been selected as a recording area in the past may be specified, and the priority of each of the plurality of standby areas may be determined based on information regarding the specified record.

[0017] Also, one embodiment of the present invention is a display method for displaying an image captured by a photographing device, including: a setting step of setting a plurality of setting areas within a reference area that is the photographing area of a reference image; a selection step of selecting, from among the plurality of setting areas, a recording area that is the area of a recorded image to be recorded; a switching step of reselecting, from among the plurality of setting areas, the recording area and switching the recording area after the selection step; a display step of displaying a plurality of setting images that are images of the plurality of setting areas; and a control step of executing control processing related to a change of the reference area at least after the determination of the reference area.

[0018] Further, the control processing may include at least one of processing for suppressing a change of the reference area, processing for prompting a change of the reference area, and processing for notifying proposal information related to a change of the reference area. Also, when the photographing device includes an optical device for zooming, in the control step, control processing related to zooming by the optical device may be executed.

[0019] Also, when a recording start instruction from the user is received, the reference area is determined, and the control processing may be processing related to a change of the reference area after the time point of receiving the recording start instruction.

[0020] Also, one embodiment of the present invention is a display method for displaying an image captured by a photographing device, including: a setting step of setting a plurality of setting areas within a reference area that is the photographing area of a reference image; a selection step of selecting, from among the plurality of setting areas, a recording area that is the area of a recorded image to be recorded; a switching step of reselecting, from among the plurality of setting areas, the recording area and switching the recording area after the selection step; and a display step of displaying a setting image that is an image of the setting area, wherein at least one of the number of displayed setting images and the display size of the setting images displayed in the display step is determined according to at least one item of the resolution and aspect ratio of the recorded image.

[0021] Further, the number of setting videos displayed in the display step may be determined according to at least one item adjusted according to the recording format of the recorded video.

Brief Description of Drawings

[0022]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Embodiments for Carrying Out the Invention

[0023] A preferred embodiment of the present invention (hereinafter referred to as this embodiment) will be described in detail with reference to the accompanying drawings. This embodiment relates to a method of displaying video using the imaging device 10 shown in FIGS. 1 to 3. However, the embodiments described below are merely examples given for ease of understanding of the present invention and do not limit the present invention. That is, the present invention can be changed or improved from the embodiments described below without departing from the spirit thereof. Also, equivalents thereof are included in the present invention.

[0024] [Basic Configuration of Imaging Device] The imaging device 10 is, for example, a digital camera that captures video. Here, unless otherwise specified, the term "video" means a live video (live view image), that is, a video captured in real time. Also, the video in this embodiment is mainly a moving image. That is, the imaging device 10 captures a moving image at a predetermined frame rate.

[0025] The imaging device 10 is, for example, a model with an interchangeable lens, and includes an imaging device body 12 and an imaging lens 14 as shown in FIGS. 1 and 2. The imaging lens 14 is detachably attached to the mount 13 of the imaging device body 12. However, it is not limited thereto, and the imaging device 10 may be a model with an integrated lens.

[0026] (Imaging Lens) As shown in FIG. 3, the imaging lens 14 includes an optical unit 17, a diaphragm 20, a zoom drive unit 21, a focus drive unit 22, and a diaphragm drive unit 23. The optical unit 17 includes optical equipment 18 for zooming (for example, a zoom lens). By moving the optical equipment 18 in the direction of the optical axis L1, the imaging area of the imaging device 10 expands or contracts, and the zoom changes. That is, the imaging device 10 has an optical zoom function by the optical equipment 18. Note that the imaging device 10 of this embodiment has an electronic zoom (digital zoom) function that enlarges a subject in the video by electronic processing in addition to the optical zoom function.

[0027] In addition, the optical unit 17 includes an optical device 19 for focusing (e.g., a focus lens). By moving the optical device 19 in the direction of the optical axis L1, the focus position of the photographing lens 14 changes and the focus is adjusted. Note that the photographing apparatus 10 of the present embodiment has an autofocus (AF) function. That is, during video shooting, when the user performs a predetermined operation (e.g., an instruction operation to start video recording), the focus driving unit 22 operates to move the optical device 19. As a result, the focus position is automatically adjusted so as to be in focus on a predetermined portion in the video.

[0028] In addition, the optical unit 17 includes a wide-angle lens, an ultra-wide-angle lens, a 360-degree lens, an anamorphic lens, or the like. As a result, the photographing apparatus 10 can shoot a video with a wide angle of view in the horizontal direction (width direction) of the photographing apparatus 10. The angle of view of the photographing apparatus 10 changes with the execution of optical zoom or the like. Also, the maximum angle of view (full angle of view) of the photographing apparatus 10 is determined according to the design specifications of the optical unit 17 and the imaging element 40 described later. Incidentally, the optical unit 17 provided in the photographing apparatus 10 is not limited to one, and a plurality of optical units 17 having different angles of view may be provided.

[0029] The aperture 20 has a variable aperture shape, and the aperture shape is adjusted by the aperture driving unit 23. By adjusting the aperture shape, the aperture amount with respect to the incident light to the photographing lens 14 changes and the exposure amount is adjusted.

[0030] (Photographing apparatus main body) As shown in FIGS. 1 and 2, the photographing apparatus main body 12 includes an operation unit including an operation button 30. Further, a release button 26, which is one of the operation units, is disposed on the upper surface of the photographing apparatus main body 12. When the release button 26 is pressed, recording of the video shot by the photographing apparatus 10 or a video cut out from the video (strictly speaking, a recorded video described later) is started.

[0031] Here, the shooting area of the imaging device 10, that is, the range within the angle of view of the imaging device 10, is determined at the time when the user's recording start instruction is received. Further, the shooting area changes, for example, according to the on / off of electronic shake correction. When the electronic shake correction is off at the time of receiving the recording start instruction, the shooting area generally becomes a full-size (full angle of view) shooting area. Note that the user operation related to the video recording start instruction is not limited to pressing the release button 26, and an operation of touching a predetermined position on the rear display 28 described later may also be used.

[0032] Hereinafter, the video captured by the imaging device 10 at the angle of view at that time will be referred to as the "reference video". Also, the shooting area of the reference video will be referred to as the "reference area".

[0033] A display (hereinafter, also referred to as the rear display 28) shown in FIG. 2 is provided on the rear surface of the imaging device main body 12. The rear display 28 is composed of, for example, an LCD (Liquid Crystal Display), an organic EL (Organic Electroluminescence) display, an LED (Light Emitting Diode) display, or electronic paper.

[0034] The rear display 28 displays the reference video and the recorded video described later. In this way, the imaging device 10 having a display has a video display function and is thus used as a video display device. In addition, the rear display 28 can display information other than video, for example, advice or proposal information related to shooting (see FIG. 18).

[0035] Note that as shown in FIGS. 2 and 3, the imaging device main body 12 of the present embodiment includes an electronic viewfinder (denoted as EVF in FIG. 3) 29 as a sub-display. The same image can be displayed on the rear display 28 and the electronic viewfinder 29, or different images can be displayed on them. Incidentally, the imaging device may not include the electronic viewfinder 29. Also, it is not necessary to display an image on the electronic viewfinder 29 and not display an image on the rear display 28.

[0036] Also, inside the rear display 28 or on the exposed surface of the display 28, a touch panel 36 for detecting user operations is provided. For example, when the user touches a predetermined position on the rear display 28, the touch panel 36 detects the touch position and outputs the detection result to a control unit 46 described later.

[0037] As shown in FIG. 3, the imaging device main body 12 is further provided with a shutter 38, an imaging element 40, an analog signal processing circuit 44, a control unit 46, an internal memory 50, a card slot 52, and a buffer memory 56.

[0038] The imaging element 40 is an image sensor that receives light that has passed through the photographing lens 14 and converts the received image into an electrical signal (video signal) and outputs it. As the imaging element 40, a solid-state imaging element such as a CCD (Charged Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor Image Sensor) can be used.

[0039] The control unit 46 controls each part of the imaging device 10 and executes various processes related to photographing, displaying, and recording of images. The control unit 46 includes a controller 47 and an image processing unit 48 as shown in FIG. 3.

[0040] The control unit 46 has one or more processors, and the control unit 46 is configured by the cooperation of the processors and a control program. The processor may be constituted by, for example, a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a GPU (Graphics Processing Unit), or other IC (Integrated Circuit). Alternatively, the processor may be constituted by combining these. Also, as represented by an SoC (System on Chip) or the like, the processor may configure the functions of the entire control unit 46 including the controller 47 and the video processing unit 48 with one IC (Integrated Circuit) chip. The hardware configuration of each of the processors listed above may be realized by an electric circuit (Circuitry) combining circuit elements such as semiconductor elements.

[0041] The controller 47 controls each part of the imaging device 10 based on a user operation or according to a predetermined control program. For example, the controller 47 controls the imaging element 40 and the analog signal processing circuit 44 to capture an image at a predetermined frame rate.

[0042] The controller 47 controls the drive units 21 to 23, the imaging element 40, the analog signal processing circuit 44, etc. so that the imaging conditions become desired conditions. The imaging conditions include focus, exposure amount, white balance, etc. That is, the controller 47 realizes the autofocus (AF) function, the autoexposure (AE) function, and the auto white balance (AWB) function, which are the automatic adjustment functions of the imaging device 10.

[0043] Furthermore, when the controller 47 receives a user's recording start instruction, the controller 47 controls the video processing unit 48 to start recording the video.

[0044] The video processing unit 48 converts the analog video signal sent from the analog signal processing circuit 44 into digital video data, and performs gamma correction, white balance correction, etc. on the digital video data. The processed digital video data is compressed in a compression format conforming to a predetermined standard.

[0045] During shooting, the video processing unit 48 generates compressed digital image data at a predetermined frame rate, and obtains a video (strictly speaking, a frame image) from the data. The video obtained at this time corresponds to the reference video shot in the reference area.

[0046] Also, the video processing unit 48 cuts out a recording video from the reference video under the control of the controller 47. The recording video is a video with an angle of view smaller than that of the reference video, and is a video of a partial area in the reference area. The recording video is generated at the same frame rate as the reference video, and is displayed on, for example, the rear display 28 or the electronic viewfinder 29.

[0047] Also, the recording video is recorded on a recording medium. As a result, a video file of the recording video is created. At this time, the resolution (number of pixels) and aspect ratio of the recording video are determined according to the recording format (video format) of the recording video. Note that the recording format is not particularly limited and can be freely determined.

[0048] Furthermore, while the electronic shake correction is effective (on), as shown in FIG. 4, the video processing unit 48 extracts an extracted video that is slightly smaller in size than the pre-correction video from the pre-correction video shot at the full angle of view. When the extracted video is shifted due to vibration or the like, the video processing unit 48 executes electronic shake correction and shifts the position of the extraction area in the pre-correction video (for example, shifts from the solid line position to the broken line position in FIG. 4). Note that while the electronic shake correction is on, the extracted video becomes the reference video.

[0049] Hereinafter, unless otherwise specified, the operations and processes of each of the controller 47 and the video processing unit 48 will be described as the operations and processes of the control unit 46.

[0050] The internal memory 50 built into the imaging device main body 12 and the memory card 54 detachable from the card slot 52 are recording media, and for example, recorded video is recorded thereon. Note that the internal memory 50 and the memory card 54 may be provided in a device (external) separate from the imaging device main body 12. In that case, the control unit 46 may record the recorded video on the external recording medium by wire or wirelessly. The buffer memory 56 functions as a work memory of the control unit 46.

[0051] [Regarding Recording and Switching of Recorded Video] The imaging device 10 records the recorded video cut out from the reference video on the recording medium. Also, the imaging device 10 can switch (switch) the recorded video by changing the cut-out area in the reference video. Such a function will be described with reference to FIG. 5. Note that in FIG. 5, for convenience of explanation, illustration of the subject in the video is omitted.

[0052] When the imaging device 10 starts shooting, the reference video is displayed on the display (for example, the rear display 28). At this time, normally, the video of the entire angle of view becomes the reference video, and the area surrounded by the outermost edge in FIG. 5 becomes the reference area As. Also, while the electronic shake correction is on, the extracted video extracted from the video of the entire angle of view becomes the reference video, and the area surrounded by the broken line frame in FIG. 5 becomes the reference area At.

[0053] When the user performs a predetermined operation before or during the start of shooting, the control unit 46 sets a setting area Ar in the reference areas As and At. The setting area Ar is a part of the reference areas As and At, and for example, the subject in the reference video is reflected in the setting area Ar (see FIG. 6).

[0054] In addition, the position and size of the setting area Ar can be adjusted automatically or based on user operations. The user can change the position and size of the setting area Ar where the subject is projected, for example, according to the position and size of the subject in the reference video, by performing a predetermined operation or the like on the screen.

[0055] Here, in the present embodiment, the aspect ratio of the setting area Ar is determined to be a value corresponding to the recording format (video format) of the recorded video. Therefore, when changing the size of the setting area Ar, the size is changed while the aspect ratio of the setting area Ar remains constant. However, the aspect ratio of the setting area Ar may be determined regardless of the recording format of the recorded video, and in that case, the size of the setting area Ar may be changed while changing the aspect ratio.

[0056] When the setting area Ar is set, as shown in FIG. 5, a marker f is displayed in the reference video displayed on the display at that time. The marker f indicates the position of the setting area Ar in the reference video and includes the boundary of the setting area Ar. The boundary of the setting area Ar is set to distinguish (differentiate) the setting area Ar from its surroundings.

[0057] Examples of the marker f including the boundary include graphic objects such as a frame surrounding the setting area Ar and a pointer indicating the representative position of the setting area Ar. In addition, differences in brightness, chromaticity, and shading between the setting area Ar and its surroundings also correspond to the marker f. In addition, anything that can distinguish the setting area Ar from its surroundings can be included in the category of the marker f. Hereinafter, the marker f composed of a colored rectangular frame will be described as an example.

[0058] As shown in FIG. 5, a plurality of setting areas Ar can be set in the reference video. In that case, a marker f is displayed for each setting area Ar. In addition, identification information (the number displayed in the upper right of the marker f in FIG. 5) is displayed for each marker f. The identification information is information that is individually assigned by the control unit 46 to each individual label f according to a predetermined assignment rule. When the identification information is displayed together with the label f, the user can distinguish each label f. Note that the identification information is not particularly limited and may be a number, a character, or a symbol, or may be the display color of the label f divided for each label f. When using a number as the identification information, it is desirable that it be a consecutive number.

[0059] When the setting of the setting area Ar is completed, the user selects a recording area Ap, which is the area of the recorded video, from among the plurality of set setting areas Ar. The control unit 46 selects the recording area Ap from the plurality of setting areas Ar according to the user's selection operation. As a result, the setting area Ar in which the video within the area is recorded, that is, the recording area Ap, is determined.

[0060] When the recording area Ap is determined, the control unit 46 displays, in different manners, the label f indicating the position of the recording area Ap and the label f indicating the positions of the other setting areas Ar among the labels f for each setting area Ar displayed in the reference video. For example, as the display manner of the label f, there are the color, brightness, density (shading degree), shape, thickness and line type of the border line, and the presence or absence of blinking display of the label f. In the case shown in FIG. 5, the display color and thickness of the label f indicating the position of the recording area Ap are different from those of the other labels f.

[0061] When the user gives a recording start instruction after the determination of the recording area Ap, the recording of the video of the recording area Ap (hereinafter referred to as the recorded video) is started. At that time, the reference area (that is, the orientation and the angle of view of the imaging device 10) is determined. Note that the determination of the reference area means that the reference area is determined once at the time of receiving the recording start instruction. Also, even after the reference area is determined, the user can freely change the reference area. Incidentally, when the user changes the reference area, as will be described later, it is preferable to change the position of the setting area according to the position of the subject.

[0062] By recording a part of the reference video as the recorded video as described above, for example, a zoom video of a portion of interest in the reference video can be recorded. Note that, although the recorded video has an angle of view smaller than that of the reference video, if the reference video has high image quality (for example, if the number of pixels is 10 million or more), the recorded video will be of sufficiently high image quality. The number of pixels of the reference video is not particularly limited, but its lower limit is preferably 10 million or more, and more preferably 60 million or more. Also, the upper limit of the number of pixels is preferably 1 billion or less, and more preferably 500 million or less. When the number of pixels is greater than the above lower limit, the visibility of the recorded video is ensured. Also, when the number of pixels is less than the above upper limit, the data amount of the reference video is reduced, and the processing by the control unit 46 can be speeded up.

[0063] Also, in the present embodiment, after selecting one setting area Ar determined as the recording area Ap from among a plurality of setting areas Ar, the recording area Ap can be switched to another setting area Ar during video recording. As a result, the video of the recording area Ap before switching and the video of the recording area Ap after switching can be recorded respectively. This will be described in detail with reference to FIG. 6.

[0064] Suppose the imaging device 10 captures a plurality of subjects (persons) at a certain scene (location) and sets setting areas Ar for each subject as shown in FIG. 6. In this case, one setting area Ar selected by the user from among the plurality of setting areas Ar becomes the recording area Ap. In the case shown in FIG. 6, four setting areas Ar are set, and the setting area Ar surrounded by the label f with identification information "1" is the recording area Ap.

[0065] When the user gives a recording start instruction in the above situation, the video of the setting area Ar that is the recording area Ap at that time is recorded. Then, when the user gives an instruction to change the recording area Ap to another setting area Ar (for example, the setting area Ar surrounded by the label f with identification information "2") during video recording, the recording area Ap is reselected and the recorded video is switched. Thus, in this embodiment, during video recording, the recorded video can be switched by reselecting the recording area Ap. As a result, it is possible to capture a zoomed video of each subject without the need to install individual cameras for each of the multiple existing subjects.

[0066] Further, when the recorded video is switched, the control unit 46 records the recorded video before the switch and the recorded video after the switch respectively, and combines these videos into one moving image. As a result, as shown in FIG. 7, a moving image file (video file) in which the subject in the video changes as the recorded video is switched is obtained. However, it is not limited to this, and the recorded video before the switch and the recorded video after the switch may be recorded as separate file data without being combined. Also, the reference video and the standby area described later may be recorded as separate file data.

[0067] In this embodiment, as shown in FIGS. 5 and 6, for each of the plurality of setting areas Ar including the recording area Ap, a marker f indicating the position in the reference video is displayed. As a result, the user can grasp the position of each setting area Ar during shooting. That is, the display method of this embodiment improves the convenience (ease of use) from the viewpoint of making it easier to grasp the position of the setting area Ar. Such an effect is particularly effective in a configuration in which the recording area Ap can be switched among the plurality of setting areas Ar. That is, by displaying the marker f for each setting area Ar, it is possible to easily grasp which video in the reference video the recorded video after the switch is.

[0068] Supplementing the setting area Ar, the number of setting areas Ar (the number of settings) can be changed automatically according to the change, for example, when the number of subjects in the reference video changes, or based on a user operation. For example, as shown in FIG. 8, when a new subject enters the reference video, a new setting area Ar can be added at the position of the subject. Conversely, when a subject in the reference video moves outside the reference video, the setting area Ar that was the video area of the subject can be automatically deleted.

[0069] Also, when a subject within a certain setting area Ar (for example, the setting area Ar surrounded by the label f of the identification information "5" in FIG. 8) moves, the setting area Ar can be displaced following the movement of the subject. Thereby, it is possible to track a moving subject in the reference video. Here, tracking of the subject can be realized by a known subject detection technique based on video analysis. Also, when the setting area Ar that tracks the moving subject is the recording area Ap, it is possible to record the video of the moving subject.

[0070] Hereinafter, the setting area Ar that tracks a moving subject will be referred to as a moving area Am. Note that the moving area Am is not limited to an area that moves following the moving subject, and may be an area that moves at a constant speed or an irregular speed in a predetermined direction.

[0071] [Video display flow] A processing flow (hereinafter, video display flow) executed by the control unit 46 of the imaging device 10 will be described with reference to FIG. 10.

[0072] When the user turns on the power of the imaging device 10, the video display flow is started. At the start of the video display flow, the user performs initial adjustment of the imaging device 10. In the initial adjustment, for example, the resolution (number of pixels) and aspect ratio of the recorded video are adjusted by specifying a recording method or the like. Such adjustment is performed, for example, at the first startup of the imaging device 10, and thereafter, the adjustment content is retained. Note that the above adjustment may be performed by the device manufacturer side before the imaging device 10 is shipped. Also, the user can readjust these values after the imaging device 10 is started up in order to review the resolution and aspect ratio that have been adjusted once.

[0073] In the image display process, first, the control unit 46 starts the shooting process (S001). This shooting process is a process of shooting a reference image in the reference area at that time. When the shooting starts, it is preferable that the real-time reference image is displayed on the rear display 28 or the like as a live view image. The shooting process continues until the user instructs the end of shooting or turns off the power of the imaging device 10.

[0074] After that, the control unit 46 performs a setting process (S002). The setting process is a process of automatically setting a plurality of setting areas Ar in the shooting area of the reference image. In the setting process, the initial position and initial size of each setting area Ar are set, and thereafter, based on user operations or automatically according to a predetermined rule, it is possible to change the position and size of each setting area Ar.

[0075] After the end of the setting process, the control unit 46 performs a selection process (S003). The selection process is a process of selecting a recording area Ap from among the plurality of setting areas Ar set in the setting process based on the user's selection operation. When performing the selection process, the control unit 46 may display the image of each of the plurality of setting areas Ar (hereinafter referred to as "setting image") on the display. In this case, the user can check the setting images for each of the plurality of setting areas Ar.

[0076] Thereafter, when the user gives an instruction to start recording (S004), the control unit 46 starts the recording process (S005). The recording process is a process of recording a recording image, which is the image of the selected recording area Ap, on a recording medium. The recording image is recorded according to a predetermined recording format (video format). Also, as described above, the reference area is determined at the time when the recording start instruction, which triggers the start of the recording process, is received.

[0077] Also, during the recording process, the control unit 46 performs a display process (S006). The display process is a process of displaying a plurality of setting images including the recorded video, the reference image at that time, and the marker f in the reference image on the display. The user can view these images, confirm the positions of the respective setting areas Ar in the reference image, and grasp which setting area Ar is the recording area Ap.

[0078] Also, during the recording process, the control unit 46 performs a control process (S007). The control process is a process of executing control processing related to the change of the reference area determined at the time of receiving the recording start instruction. In the present embodiment, the control processing includes processing for suppressing the change of the reference area, and for example, is control processing related to zooming (optical zoom) by the optical device 19.

[0079] Also, during the recording process, when the user instructs re-selection of the recording area Ap (S008), the control unit 46 performs a switching process (S009). The switching process is a process of re-selecting the recording area Ap from among the plurality of setting areas Ar and switching the recording area Ap according to the re-selection instruction by the user after the selection process is performed. When the switching process is performed, in the subsequent recording process, the video (recorded video) of the switched recording area is recorded.

[0080] Then, a series of processes from the recording process (that is, S005 to S009) are repeatedly performed until the user instructs the end of video recording (S010). On the other hand, when there is an instruction to end video recording, the recording process ends at that time. After that, when there is an instruction to end shooting or the power of the imaging device 10 is turned off (S011), the video display flow ends at that time.

[0081] [Regarding the setting process] The setting process in the video display flow will be described. In this embodiment, when the setting process is executed, the first UI (User Interface) screen 101 shown in FIG. 11 is displayed on the rear display 28. The user selects either the "recording pattern" or the "automatic setting pattern" as the setting pattern for the setting area Ar, and presses the button among the buttons Bt1 and Bt2 on the UI screen 101 that indicates the selected pattern.

[0082] In the setting process when the recording pattern is selected, the control unit 46 reads out the position information of the setting area Ar pre-stored in the imaging device 10. Then, the setting area Ar is automatically set at the position indicated by the read position information. At this time, for the purpose of making the label f more visible to the user, it is more preferable to highlight the image within the label f in the reference image. For example, in the reference image, the brightness of the image other than the image within the label f (that is, within the boundary of the setting area Ar) may be lowered or grayed out (see FIG. 18).

[0083] In this embodiment, when the recording pattern is selected, as shown in FIG. 11, the second UI screen 102 is displayed on the rear display 28. The user selects one of a plurality of pre-prepared recording patterns and presses the button among the buttons Bt3, Bt4, and Bt5 on the second UI screen 102 that indicates the selected pattern. As the recording pattern, for example, the "previous use pattern", the "preset pattern", and the "user input pattern" can be selected.

[0084] On the other hand, in the setting process when the automatic setting pattern is selected, the control unit 46 automatically sets the setting area Ar at a predetermined position in the reference image based on the reference image during shooting. In this embodiment, when an automatic setting pattern is selected, as shown in FIG. 11, a third UI screen 103 is displayed. The user selects one of a plurality of pre-prepared automatic setting patterns and presses the button among buttons Bt6, Bt7, and Bt8 on the third UI screen 103 that indicates the selected pattern. As the automatic setting pattern, for example, a "scene recognition pattern", a "subject detection pattern", and an "AF position specification pattern" can be selected. In the case of an automatic setting pattern, at the start of the setting process, it is preferable to display the label f in a state where each setting area Ar does not overlap with each other and does not overlap. This makes it easier to distinguish a plurality of labels f and facilitates the setting of the setting area Ar in the setting process.

[0085] Hereinafter, each of a plurality of recording patterns and a plurality of automatic setting patterns will be described.

[0086] (Previous use pattern) When the previous use pattern is selected, the control unit 46 reads out the position information of the setting area Ar set in the setting process in the previous (immediately preceding) video display flow. This position information indicates the positions of the plurality of setting areas Ar set in the setting process of the previous video display flow and is stored, for example, in the internal memory 50 of the imaging device 10. It is preferable that the stored position information is retained even after the power of the imaging device 10 is turned off.

[0087] Note that the size information of the setting area Ar may be stored together with the position information of the setting area Ar. In that case, when the previous use pattern is selected, the position information and the size information may be read out together.

[0088] Then, as shown in FIG. 12, the control unit 46 automatically sets the setting area Ar at the position indicated by the read position information, that is, the same position as the setting position in the previous video display flow. Such a previous use pattern is effective when shooting a video with the same composition as the previous video display flow (for example, when shooting a video at the same location and at the same angle).

[0089] In the above-described embodiment, the read position information indicates the position of the setting area Ar set in the previous (immediately preceding) video display flow, but it is not limited thereto. For example, the position information of the setting area Ar set in the setting steps for several times performed in the past may be stored respectively. In that case, among the stored position information, the current setting area Ar may be set at the position of the setting area Ar set in the setting step of the number specified by the user.

[0090] (Preset pattern) When a preset pattern is selected, the control unit 46 reads out the position information of the setting area Ar arranged in a predetermined standard arrangement pattern. The standard arrangement pattern is pre-registered in the imaging device 10 or newly registered at the time of program update after purchase or the like.

[0091] When the position information related to the standard arrangement pattern (preset pattern) is read out, the fourth UI screen 104 shown in FIG. 13 is displayed on the rear display 28. On the fourth UI screen 104, a plurality of types of patterns (patterns A to D in FIG. 13) are displayed so as to be selectable, and the user selects any type of preset pattern. The control unit 46 automatically sets the setting area Ar at the position corresponding to the preset pattern selected by the user. Such a preset pattern is effective when there is a pattern corresponding to the composition when shooting a video with a determined composition (for example, when shooting a video at a specific location). Note that the number of types of preset patterns, and the number and arrangement positions of the setting areas Ar in each preset pattern are not particularly limited and can be arbitrarily determined.

[0092] (User input pattern) When a user input pattern is selected, the control unit 46 reads the position information of the setting area Ar determined based on the user's operation. The user's operation here is an input operation for registering the position of the setting area Ar and is performed in a previous stage of the setting process (for example, at the stage of initial registration regarding the pattern). The above input operation is performed, for example, through the input screen 105 for pattern registration shown in FIG. 14. The user touches and drags the inner area of the area setting label fx displayed on the input screen 105. As a result, the display position of the label fx after dragging is registered as the position of the setting area Ar.

[0093] The control unit 46 automatically sets the setting area Ar at the position indicated by the read position information, that is, the setting position determined based on the user's input operation. Such a user input pattern is effective when preparing a pattern based on the user's input in a situation where it cannot be handled by the aforementioned preset patterns.

[0094] Note that when the position of the setting area Ar is registered, the size of the setting area Ar may be registered together. That is, information indicating the position and size of the setting area Ar determined based on the user's input operation may be stored. In that case, each of the plurality of setting areas Ar will be set at the position and size registered as the user input pattern.

[0095] (Scene recognition pattern) In the setting process when the scene recognition pattern is selected, the control unit 46 recognizes the shooting scene of the reference video based on the reference video. The shooting scene of the reference video is, for example, the scenery, event, type of subject, scene (for example, whether it is a daytime scene or a night scene), and situation (for example, weather, etc.) photographed as the reference video.

[0096] As a method for recognizing a shooting scene based on a reference video, known scene recognition technologies can be used. For example, a method of determining whether the luminance or color tone of the video in a predetermined area is within a preset range can be mentioned. Further, machine learning may be performed using the correct label indicating the shooting scene of the video and the video data as a learning dataset, and the shooting scene of the reference video may be recognized by a scene recognition model obtained as a result of the learning.

[0097] (Subject detection pattern) In the setting step when the subject detection pattern is selected, the control unit 46 detects the subject in the reference video, and automatically sets the setting area Ar at the position of the detected subject. The subject in the reference video is a subject other than the scenery in the reference video and can be detected by the control unit 46, and includes, for example, people, animals, vehicles, and the like. When detecting a subject, a known template matching technique may be applied. For example, an image of the subject may be stored in advance as a template image, the template image may be collated with each part of the reference video, and a video similar to the template image may be detected as the subject. Alternatively, a model for recognizing the subject in the video by machine learning may be constructed, and the model may be used to detect the subject in the reference video. As an algorithm of the subject recognition model, known algorithms such as YOLO (You Only Look Once) or R-CNN (Region with Convolution Neural Network) can be used.

[0098] In the subject detection pattern, as shown in FIG. 6, the control unit 46 can set the setting area Ar one by one for each of the detected subjects. At this time, the entire corresponding subject may be accommodated within each setting area Ar, or a part of the corresponding subject (for example, the upper body part including the face) may be accommodated.

[0099] According to the subject detection pattern as described above, without requiring the user's effort, the setting area Ar can be automatically set for the subject in the reference video. In the present embodiment, since the video of the recording area Ap selected from among the plurality of setting areas Ar is recorded, the subject detection pattern is effective when recording the video of the subject in the reference video.

[0100] (AF position specifying pattern) In the setting step when the AF position specifying pattern is selected, the control unit 46 specifies the position in the reference video where the feature for determining the focus of the imaging device 10 satisfies the standard, and automatically sets the setting area Ar at the specified position. The feature for determining the focus is, for example, a feature for determining the in-focus position within the reference video. As a specific example, when the autofocus is the phase difference AF (Auto Focus) method, the phase difference corresponds to the above feature. Also, when the autofocus is the contrast AF method, the contrast between a plurality of pixels corresponds to the above feature. Note that the AF method may be an AF method using directional light or an AF method using DFD (Depth from defocus).

[0101] Also, the position that satisfies the standard is the position where the above feature (for example, phase difference or contrast, etc.) becomes the feature at the time of focusing. To put it simply, the control unit 46 specifies, in the reference video, the position where the focus is achieved by the AF function as the position that satisfies the standard. Then, as shown in FIG. 15, the setting area Ar is automatically set for the specified position (the position where the focus is achieved, the position hatched in FIG. 15). At this time, for example, the size of the setting area Ar may be automatically set according to the area of the in-focus portion in the reference video or the like.

[0102] Generally, the video of the portion where the focus is achieved by the AF function is often the recording target. Taking this point into consideration, by setting the setting area Ar at the position where the focus is achieved, the video where the focus is achieved can be easily recorded, improving convenience for the user.

[0103] In the AF position specifying pattern, when the control unit 46 sets the setting area Ar at the focused position, another setting area Ar is automatically set accordingly. For example, as shown in FIG. 15, another setting area Ar is set at the symmetric position of the focused position (for example, the point-symmetric position when viewed from the center of the reference image). Thereby, when the AF position specifying pattern is selected, a plurality of setting areas Ar can be automatically set in the reference area. However, the position where another setting area Ar is set is not limited to the above position.

[0104] Also, when the entire reference image is in focus in the AF position specifying pattern, the entire reference area may be set as one setting area Ar. Also, when no subject is detected in the reference image and no setting position of the appropriate setting area Ar can be found within the reference image, the setting area Ar may be set near the center of the reference image.

[0105] In this embodiment, as an example of the shooting conditions, the in-focus position in the reference image is cited, and the setting area Ar is set at the position where the feature for determining the in-focus position in the reference image satisfies the standard (that is, the focused position). Here, the shooting conditions may be other than the in-focus position, and for example, the exposure amount and the white balance are also cited. Note that the shutter speed and the ISO sensitivity are excluded from the shooting conditions mentioned here.

[0106] Here, taking the exposure amount, which is one of the shooting conditions, as an example, the setting area Ar can be set at the position where the feature (for example, the pixel value or the luminance) for determining the exposure amount in the reference image satisfies the standard. For example, it is advisable to set the setting area Ar at the position where the exposure amount is adjusted to an appropriate value by the AE (Automatic Exposure) function in the reference image. Alternatively, the setting area Ar may be set at each of the position where the exposure amount is relatively low and the position where the exposure amount is relatively high in the reference image.

[0107] Also, taking white balance, which is one of the shooting conditions, as an example, in the reference video, a setting area Ar can be set at a position where features (for example, chromaticity) for determining the white balance satisfy the criteria. For example, in the reference video, it is advisable to set the setting area Ar at the position where the white balance is adjusted to an appropriate value by the AWB (Auto White Balance) function. Specifically, when there are areas in the reference video irradiated with light from each of a plurality of light sources such as lighting and the sun, setting areas Ar may be set for the irradiation areas of each light source respectively.

[0108] The user selects an arbitrary pattern from among the plurality of arrangement patterns described above. As a result, in the setting process, the setting area Ar is set reflecting the user's intention or preference. Also, in the setting process, since the setting area Ar is automatically set, it does not require the user's effort (setting work). That is, the display method of the present embodiment improves convenience (ease of use) from the viewpoint of saving the user's effort and setting the setting area Ar in the reference area reflecting the user's intention and the like. Note that the arrangement pattern may be an arrangement pattern in which a plurality of the arrangement patterns shown in FIG. 11 are combined. For example, a scene recognition pattern and an AF position specifying pattern may be adopted simultaneously to set a plurality of setting areas Ar.

[0109] Note that the above patterns are merely examples, and arrangement patterns other than the above patterns may be added. Conversely, some of the above patterns may be omitted.

[0110] Also, in the setting process of the present embodiment, regardless of the type of the selected pattern, if there are characteristic portions described later in the reference video, candidate setting areas Ac are automatically set at the positions of the characteristic portions in the reference area. Then, when the user performs a predetermined operation on the set candidate Ac, the setting area Ar is set at the position of the set candidate Ac. Hereinafter, the setting procedure of the candidate Ac in the setting process will be described.

[0111] (Regarding the setting of candidates) In the setting process, the control unit 46 determines whether or not there is a feature location in the reference video based on the features (e.g., pixel values) of each pixel of the reference video. A feature location is a location where a visual change occurs in the reference video (excluding what is detected as a subject), for example, a location with movement in the background video, and a location where color, brightness, etc. change. Specific examples include a location with a flow like a waterfall, a location where light such as illumination lights up or blinks, and a location where characters, images, etc. displayed like an electronic display board are switched, etc. In the reference video shown in FIG. 16, for example, the lighting decoration appearing in the upper left part of the figure corresponds to the feature location.

[0112] When there is a feature location in the reference video, the control unit 46 performs the first setting process. The first setting process is a process of automatically setting a candidate Ac of the setting area at the position of the feature location in the reference video.

[0113] When the first setting process is performed, in the subsequent display process, as shown in FIG. 16, a mark (hereinafter referred to as a candidate mark K) indicating the position of the candidate Ac within the reference video is displayed. The candidate mark K is displayed in a manner different from the mark f indicating the position of the setting area Ar. For example, the candidate mark K in FIG. 16 is a substantially L-shaped graphic object representing the four corners of a rectangular area surrounding the feature location. The type, shape, etc. of the candidate mark K are not particularly limited, and for example, it may be a pointer in the shape of an arrow or a finger, or a graphic object composed of a symbol such as a circle or a star. Alternatively, the shading difference (brightness difference) between the candidate Ac in the reference video and its periphery may be used as the candidate mark K.

[0114] When the user performs an operation such as tapping the candidate marker K during the execution of the project, the second setting process is carried out with this as a trigger. The second setting process is a process in which the control unit 46 sets a setting area Ar at the setting position of the candidate Ac on which the operation was performed in response to an operation (for example, a tap operation, etc.) by the user on the candidate Ac. By carrying out the second setting process, as shown in FIG. 17, a new setting area Ar is set in the area of the video of the feature location.

[0115] As described above, in this embodiment, in the reference video, it is possible to automatically set a candidate Ac for the setting area for a feature location that is not recognized as a subject but has a visual change. Further, by displaying a candidate marker K indicating the position of the set candidate Ac in the reference video, the user can easily confirm the candidate Ac. Furthermore, when the user performs a predetermined operation on the candidate Ac, a setting area Ar is set at the position of the candidate Ac. Thereby, in the reference video, the setting area Ar can be set for the feature location that the user shows interest in. Moreover, if the thus-set setting area Ar is selected as the recording area, it becomes possible to record the video of the feature location.

[0116] [Regarding the display process] The display process in the video display flow will be described in detail below. When the setting area Ar is set in the setting process, the control unit 46 starts the display process. After that, the control unit 46 continues the display process until the recording process ends (strictly speaking, until the shooting ends).

[0117] The display process is a process of displaying the recorded video, the reference video, and a marker f indicating the position of each of the plurality of setting areas Ar in the reference video on the display. In the display process, as shown in FIG. 18, the marker f is displayed for each setting area Ar in the reference video. Thereby, the user can accurately grasp the position of each setting area Ar in the reference video.

[0118] In the display process, as shown in FIG. 18, among the plurality of setting areas Ar, images of each of the setting areas Ar other than the recording area Ap (hereinafter referred to as standby images) can be displayed. Furthermore, in the display process when the above-mentioned characteristic part exists in the reference image, a candidate marker K indicating the position of the characteristic part is displayed together with the marker f in the reference image.

[0119] An example of the display process will be described. The recorded image, the reference image, and the standby image can be simultaneously displayed on the screen of the rear display 28 (hereinafter referred to as the display screen). For example, as shown in FIG. 18, assume that the display screen is divided into a main screen G1, a sub-screen G2, and an information display screen G3. In this case, the standby image can be displayed on the sub-screen G2.

[0120] The main screen G1 occupies a relatively large area (for example, more than half of the display screen) on the display screen and displays the reference image. The aspect ratio of the main screen G1 is preferably set to the same value as the aspect ratio of the reference image. The sub-screen G2 is a screen smaller than the main screen G1 and is arranged in a column along the vertical or horizontal direction of the display screen. Each sub-screen G2 displays a recorded image or one standby image. The aspect ratio of each sub-screen G2 is preferably set to the same value as the aspect ratio of each of the recorded image and the standby image. The information display screen G3 is a screen for displaying character information other than images. For example, as shown in FIG. 18, it displays advice regarding video recording.

[0121] Note that the layout of the display screen (for example, the size and arrangement position of each of the main screen G1 and the sub-screen G2, etc.) is not limited to the form shown in FIG. 18 and can be freely designed according to the specifications of the device or the preferences of the user, etc.

[0122] In addition, on the main screen G1, for each of a plurality of set areas Ar set in the setting process, a label f is displayed together with the reference video. The number of labels f (display number) displayed on the main screen G1 in the display process is variable, and the maximum number of displayable (display upper limit number) labels f can be displayed simultaneously. Also, for the purpose of making the label f more visible to the user, it is more preferable to highlight the video within the label f in the reference video. For example, as shown in FIG. 18, in the reference video, the brightness of the video outside the label f (that is, within the boundary of the set area Ar) may be lowered or grayed out.

[0123] In addition, in the display process, the label f indicating the position of the recording area Ap and the label f indicating the positions of the other set areas Ar (hereinafter also referred to as standby areas) are displayed in different manners. At this time, the label f indicating the position of the recording area Ap is preferably highlighted. For example, the color of the frame line may be set to a relatively conspicuous color such as red or fluorescent color, the frame line may be made thicker, or it may be made to blink. Also, the video itself within the recording area Ap (that is, the recorded video) may be highlighted.

[0124] In addition, when there is a subject moving in the reference video, the moving area Am set for the subject moves following the subject. In the display process, as shown in FIG. 9, the label f indicating the position of the moving area Am is displayed in a different manner from the label f indicating the positions of the set areas Ar other than the moving area Am. Thereby, the moving area Am and the other set areas Ar can be easily distinguished, and the position of the moving area Am within the reference video can be easily found.

[0125] Note that when the subject within the moving area Am moves in the depth direction (that is, the direction approaching and separating from the imaging device 10), the size of the subject in the reference video changes. Along with this, the ratio of the size of the subject to the size of the moving area Am (hereinafter referred to as the subject ratio) changes. In that case, the control unit 46 uses the electronic zoom function to change the size of the moving area Am according to the change in the subject ratio and adjusts to keep the subject ratio constant.

[0126] In addition, as the subject moves, the movement area Am may move to near the edge of the reference area (i.e., the outer edge of the angle of view) or may protrude from the reference area. In such a case, it is preferable that the control unit 46 displays a warning message on the information display screen G3 or outputs a warning sound to notify the user of the above situation.

[0127] Further, when the reference video (i.e., the extracted video extracted from the video of the entire angle of view) changes due to electronic shake correction, the relative positions of the respective setting areas Ar with respect to the reference area change (see FIG. 4). In that case, the control unit 46 calculates the deviation amount of the reference video due to electronic shake correction, and adjusts the positions of the respective setting areas Ar based on the calculated deviation amount. Thereby, the positions of the respective setting areas Ar with respect to the reference area are maintained before and after the implementation of electronic shake correction. Note that it is preferable that the function of adjusting the position of the setting area Ar in accordance with electronic shake correction can be freely switched on and off according to the user's operation.

[0128] In the display step, as shown in FIG. 18, the identification information of the label f is displayed for each label f. The identification information of each label f is fixed during the execution of the display step, and even if the number of displayed labels f changes during the display step, it is maintained without changing before and after. For example, in FIG. 18, when the display of the label f with the identification number "3" is stopped, the identification numbers of the other labels f (i.e., "1", "2", "4") do not change. By fixing the identification number of each label f during the execution of the display step in this way, for example, even if the number of displayed labels f changes, it is possible to easily grasp which setting area Ar exists where.

[0129] A plurality of setting videos including the recorded video and the standby video are displayed on the plurality of sub - screens G2. Specifically, one corresponding setting video is displayed on each of the plurality of sub - screens G2. Here, if the aspect ratio of each sub - screen G2 is the same as the aspect ratio of the corresponding setting video, the setting video can be displayed on the screen neatly.

[0130] Also, among the plurality of setting videos to be displayed, the recorded video is preferably highlighted. For example, it is preferable that there are differences in the color of the screen frame, the thickness or type of the frame line, or the presence or absence of lighting, etc. between the sub-screen G2 on which the recorded video is displayed and the sub-screen G2 on which other setting videos are displayed. Thereby, a user who views a plurality of setting videos on the display can easily distinguish which video is the recorded video.

[0131] Also, as shown in FIG. 18, for the sub-screen G2 (the second sub-screen from the top in FIG. 18) that displays the video of the moving area Am, information indicating that it is the video of the moving area Am (for example, character information such as "moving") may be displayed.

[0132] In this embodiment, a plurality of implementation modes of the display process are prepared, and the display method of the video in the display process changes for each mode. That is, the display process is implemented according to the mode specified by the user. For example, when the user designates the first mode, the first display process is implemented as the display process. In the first display process, as shown in FIG. 18, the reference video and the label f for each setting area Ar are displayed on the main screen G1, and the recorded video and the standby video are displayed on the sub-screen G2. On the other hand, when the user designates the second mode, the second display process is implemented as the display process. In the second display process, as shown in FIG. 19, the recorded video and the standby video are displayed, and the reference video and the label f for each setting area Ar are not displayed.

[0133] Note that in the first display process, the recorded video, the standby video, the reference video, and the label f for each setting area Ar do not necessarily have to be displayed on the same display (for example, the rear display 28). That is, as shown in FIG. 20, the video displayed on the main screen G1 and the video displayed on the sub-screen G2 may be displayed on different displays from each other. To give a specific example, the video to be displayed on the main screen G1 may be displayed on the rear display 28, and the video to be displayed on the sub-screen G2 may be displayed on the electronic viewfinder 29 or an external monitor (not shown) connected to the imaging device 10.

[0134] As described above, in this embodiment, the display process includes a first display process and a second display process, and one of these two display processes designated by the user is executed. Thereby, the display video can be switched according to the user's needs. For example, when the user wants to view the reference video and the label f, it is preferable to execute the first display process to display the reference video and the label f. On the other hand, when the user wants to view only the recorded video and the standby video (i.e., the set video), it is preferable to execute the second display process to display only the recorded video and the standby video.

[0135] In addition, the execution mode of the display process, that is, which of the first display process and the second display process is executed, is preferably freely switchable according to a switching operation by the user during the execution of the display process. For example, it is preferable that the user can switch which display process to adopt through the touch panel 36 or the operation button 30 shown in FIG. 2.

[0136] In the second display process, the recorded video and other set videos may be displayed on the same display. For example, as shown in FIG. 19, the recorded video may be displayed on the main screen G1 in the display screen, and the standby video may be displayed on the sub-screen G2. Alternatively, in the second display process, the recorded video and other set videos may be displayed on different displays. For example, as shown in FIG. 20, the recorded video may be displayed on the rear display 28, and other set videos may be displayed on the electronic viewfinder 29 or an external monitor (not shown) connected to the imaging device 10.

[0137] On the information display screen G3, for example, proposed information described later is displayed. In addition, on the information display screen G3, it is possible to display advice for shooting for the user, warning messages, etc. For example, when adjustment of the exposure amount or ISO sensitivity, etc. is required during the recording process, that fact may be displayed. Furthermore, information other than the above can be displayed on the information display screen G3. For example, the current values of shooting conditions such as the shooting mode, resolution, and ISO sensitivity currently adopted, as well as the on / off states of various functions such as electronic zoom, may be displayed.

[0138] (Regarding the display of set videos based on priority) In the display process, the number of sub - screens G2 is determined according to at least one of the items of the resolution and aspect ratio of the recorded video (hereinafter, simply referred to as "at least one item"). That is, the number of set videos displayed on the sub - screen G2 is determined according to at least one item. Similarly, the size of each sub - screen G2 is also determined according to at least one item. That is, the display size of the set video is the size according to at least one item.

[0139] Here, at least one item is adjusted according to the recording format (video format) of the recorded video. Therefore, the number of set videos displayed (strictly speaking, the maximum number of displays) and the display size in the display process are the number and size according to the recording format of the recorded video.

[0140] In this embodiment, by determining the number of set videos to be displayed and the display size according to at least one item, each of a plurality of set videos including the recorded video can be displayed well. For example, each set video can be displayed neatly within the sub - screen G2 at an appropriate resolution.

[0141] Note that in this embodiment, the aspect ratio of the recorded video is determined according to the recording format of the recorded video, but the aspect ratio of the recorded video may be arbitrarily set without depending on the recording format. In that case, with the adjustment of the aspect ratio of the recorded video, the number and size of the sub - screens G2 will be changed according to the adjusted aspect ratio.

[0142] In the present embodiment, both the number of setting videos displayed in the display process and the display size (i.e., the number and size of the sub - screens G2) are determined according to at least one of the items. However, it is not limited to this, and either the number of setting videos or the display size may be determined according to at least one of the items.

[0143] Note that the number of setting videos displayed in the display process does not necessarily have to be determined according to at least one of the items. For example, the user may set or change the number of displays.

[0144] In the present embodiment, the number of setting videos displayed in the display process is determined according to the resolution and aspect ratio of the recorded video as described above. Here, in the case of the first display process, out of the setting videos to be displayed, one is the recorded video and the rest are standby videos. Therefore, the set value regarding the number of standby videos to be displayed, that is, the maximum number of displays, is determined according to the resolution and aspect ratio of the recorded video, and specifically, it is a number one less than the number of sub - screens G2. Note that in the case shown in FIG. 21, the maximum number of standby videos is three.

[0145] On the other hand, the number of setting areas Ar in the reference area changes according to the subject in the reference video and the like, and accordingly, the number of setting areas Ar other than the recording area Ap, that is, the number of a plurality of standby areas, changes. Note that in the case shown in FIG. 21, the number of standby areas is six.

[0146] When the number of a plurality of standby areas reaches (strictly speaking, exceeds) the maximum number of standby videos, the control unit 46 performs the display process in the flow shown in FIG. 22. Regarding the flow of the display process shown in FIG. 22, when the number of a plurality of standby areas reaches the maximum number of standby videos, the control unit 46 performs a determination process (S021).

[0147] In the determination process, the control unit 46 determines the priority for each of the plurality of standby areas. To explain the procedure, first, for each of the plurality of standby videos, the control unit 46 detects the subject in the standby video (S101). Next, the control unit 46 acquires information regarding the detected subject in the standby video (S102). At this time, instead of the information regarding the subject in the standby video, the control unit 46 may acquire performance information as a recording area described later.

[0148] Examples of the information regarding the subject include the time or number of times the subject appears in the reference video during the shooting period, and the size (display size) of the subject in the reference video. These pieces of information can be acquired by the control unit 46 analyzing the reference video during the shooting period for measurement or counting. Also, when the subject is a person, the face of the person may be recognized by a technique such as pattern matching, the importance of the person whose face is recognized may be specified, and the specified result may be used as the information regarding the subject. Regarding the importance of the person, for example, the attribute of the person whose face is recognized (such as family, relative, acquaintance, or other person) can be determined and obtained from the determination result. Also, when there are a plurality of detected subjects, the priority may be determined according to the category of the subject (such as person, animal, object, or scenery, etc.). Furthermore, the priority may be determined preferentially starting from the subject with an earlier detected timing. Regarding the information regarding the subject, of course, other possibilities can be considered in addition to the above. In the determination process, it is desirable that the user can freely select what information to use as the information regarding the subject.

[0149] Thereafter, in the determination process, the control unit 46 determines the priority for each of the plurality of standby areas according to the information regarding the detected subject (S103).

[0150] After the decision process is executed, the control unit 46 selects N standby areas out of the plurality of standby areas based on the priority (S022). Here, the number N of the selected standby areas is the maximum number of displayable standby images and is a number determined according to the number of sub - screens G2 (for example, a number one less than the number of sub - screens G2). The control unit 46 selects N standby areas in descending order of priority in step S022.

[0151] Then, the control unit 46 displays the images (standby images) of the selected N standby areas on the sub - screen G2 (S023). Here, when the sub - screens G2 are arranged in the vertical direction of the display screen as shown in FIG. 21, it is desirable to display the standby images of the standby areas with higher priority on the upper sub - screens G2.

[0152] As described above, in this embodiment, when the number of the plurality of standby areas reaches (exceeds) the maximum number of displayable standby images, N standby images selected according to the priority are displayed. Thereby, while responding to the constraints in the screen display, the standby images with high priority can be appropriately displayed.

[0153] Note that the method of determining the priority is not limited to the method of determining based on the information about the subject in the standby image. For example, the priority may be determined according to the performance information as the recording area. More specifically, for each of the plurality of standby areas, identify the number of times selected as the recording area, the time as the recording area, or the elapsed time from the time immediately before being selected as the recording area, etc. between the start of shooting and the current time. These information correspond to the performance information of being selected as the recording area in the past. Then, based on the above - mentioned performance information, the priority of each of the plurality of standby areas may be determined. For example, when the performance information is the number of times or the time selected as the recording area, the higher the number or time, the higher the priority should be given.

[0154] Also, in the display process flow shown in FIG. 22, N standby areas are selected based on priority, and only the images of the selected standby areas are displayed, but it is not limited to this. For example, the top N standby areas in descending order of priority and the standby areas below them may be selected respectively. In that case, the standby images of the top N standby areas may be displayed first, and the standby images of the standby areas below them may be displayed by scrolling or switching the display screen as shown in FIG. 23.

[0155] Also, each of the plurality of standby images may be displayed in a size corresponding to the priority. For example, as shown in FIG. 24, the standby image of the standby area with high priority may be displayed on the normal-sized sub-screen G2, and the standby image of the standby area with low priority may be displayed on the reduced-sized sub-screen G2x. In this way, each of the plurality of standby images can be displayed, and the standby area with high priority can be displayed more clearly.

[0156] [Regarding the control process] The control process in the video display flow will be described. The control process is a process that executes control processing to suppress changes in the reference area, and is performed at least after the determination of the reference area. That is, the control processing is a process for suppressing changes in the reference area after receiving the user's recording start instruction.

[0157] In the control processing, the control unit 46 locks the zoom drive unit 21 to stop the movement of the zoom optical device 18. As a result, during the recording process, the optical zoom function is restricted, the reference area is fixed, and the angle of view at that time is maintained. By executing the above control processing, it is possible to prevent the relative position of each set area Ar with respect to the reference area from changing during the recording process. As a result, it is possible to avoid a situation where each set area Ar deviates from the reference area and each set image such as a recorded image is not properly captured.

[0158] Note that the control process implemented to suppress changes to the reference area is not limited to the above process. For example, it may be a process for notifying proposal information regarding changes to the reference area. To give a specific example, a message prohibiting optical zoom may be displayed on the information display screen G3, or it may be reproduced by voice or the like. Additionally, as shown in FIG. 18, a message prohibiting an operation of moving the imaging device 10 (for example, a panning operation), or a message recommending fixing the imaging device 10 using a tripod head or the like may be displayed.

[0159] Also, the control step may be performed at a point in time before the determination of the reference area (that is, before receiving the recording start instruction), or for example, it may be performed during the setting step. In this case, it is possible to suppress the reference area from changing while a plurality of setting areas Ar are set in the reference area.

[0160] [Regarding the change step] During the execution of the recording step, when a predetermined condition is satisfied, the control unit 46 performs the change step. The change step is a step of executing a change process regarding the number of displays of the video of the setting area Ar (that is, the set video). By executing the change process, it is possible to dynamically increase or decrease the number of displays of the set video during the execution of the recording step. Hereinafter, the flow of processing during the execution of the recording step will be described with reference to the flow shown in FIG. 25. Incidentally, in FIGS. 26 to 31 referred to in the following description, for the sake of illustration, the gray-out display of videos other than the video within the label f is omitted.

[0161] During the execution of the recording step, the control unit 46 monitors the reference video and determines whether the subject in the reference video has increased (S031). If the subject in the reference video has increased, the control unit 46 performs the change step (S032). In such a change step, a change process for increasing the number of displays of the set video is executed.

[0162] In the change process in step S032, the setting area Ar is added according to the increase of the subject in the reference video, and the setting video of the added setting area Ar is newly displayed on the display. As a result, the number of displayed setting videos increases. For example, as shown in FIGS. 26 and 27, when the number of subjects in the reference video increases from three to four, the number of displayed setting videos (that is, the number of videos displayed on the sub-screen G2) increases from three to four.

[0163] In addition, when the setting area Ar is added along with the increase of the subject in the reference video, in conjunction with this, the number of signs f in the reference video displayed on the main screen G1 increases by the amount of the added setting area Ar (see FIG. 27).

[0164] Also, in the above embodiment, the setting area Ar is set at the position of the new subject in the reference video, but it is not limited to this. For example, a candidate Ac for the setting area may be set instead of the setting area Ar. In that case, a sign (specifically, the aforementioned candidate sign K) indicating the position of the candidate Ac is displayed in the reference video. Then, when the user performs an operation such as tapping on the candidate sign K, the setting area Ar is added at the position of the candidate Ac using this as a trigger. As a result, the setting video of the added setting area Ar is newly displayed, and the number of displayed setting videos increases.

[0165] Also, instead of setting the setting area Ar or its candidate Ac at the position of the new subject in the reference video, proposal information regarding the addition of the setting area Ar (that is, the change in the number of displayed setting videos) may be output. For example, a message proposing to add the setting area Ar at the position of the new subject to increase the number of displayed setting videos may be displayed on the information display screen G3 or reproduced by voice or the like. In this way, as the change process, a process of outputting proposal information regarding the change in the number of displayed setting videos may be executed.

[0166] In the flow shown in FIG. 25, during the execution of the recording process, the control unit 46 monitors the reference video and determines whether the moving area Am in the reference area overlaps with another set area Ar (S033). Then, when the moving area Am overlaps with another set area Ar as shown in FIG. 28, or when it is just about to overlap, the control unit 46 outputs a warning message or a warning sound (S034). This is because in a situation where the moving area Am overlaps with another set area Ar, there may be a case where the subject in one area comes behind the subject in the other area and does not appear in the image.

[0167] By outputting a warning message or the like, it is possible to notify the user of the situation where the moving area Am overlaps with another set area Ar or is just about to overlap. The user who receives the warning can take appropriate measures (for example, re-setting the set area Ar, etc.). As one of the measures, when the user makes a change request through a screen operation or the like (S035), the control unit 46 performs a change process (S036). In such a change process, a change process for reducing the number of displayed set videos is executed.

[0168] In the change process in step S036, the overlapping moving area Am and the other set area Ar are updated to one set area Ar, and the set video of the updated set area Ar is displayed on the display. For example, as shown in FIG. 28, the video of the moving area Am and the video of the other set area Ar that are displayed separately are integrated into one video including the subjects of these two videos as shown in FIG. 29. As a result, the number of set areas Ar decreases by one, and the number of displayed set videos also decreases by one.

[0169] Note that it is preferable that the size and position of the updated set area Ar are set so that the subjects of the above two videos can be accommodated. Also, the identification information of the label f indicating the updated set area Ar may inherit the identification information of the label f indicating either one of the two areas before the update (that is, the moving area Am and the other set area Ar). Alternatively, a new identification number may be assigned to the updated set area Ar.

[0170] In the flow shown in FIG. 25, during the execution of the recording process, the control unit 46 periodically determines whether each of the plurality of setting areas Ar satisfies the termination condition (S037). The termination condition is a criterion for determining whether to terminate the display of the video (set video) of the setting area Ar. For example, the following conditions (1) to (3) can be cited. Termination condition (1): At least a part of the setting area Ar protrudes from the reference area due to optical zoom or the like. Termination condition (2): There is no change in the set video of the setting area Ar for a certain period of time or more, or there is no subject in the set video. Termination condition (3): The setting area Ar that has not been selected as the recording area for a certain period of time or more. Note that the termination condition is not limited to the above conditions (1) to (3), and conditions other than the above may be included in the termination condition.

[0171] When there is a setting area Ar that satisfies any of the above termination conditions (1) to (3), the control unit 46 performs a change process (S038). In such a change process, a change process for reducing the number of displays of the set video is executed.

[0172] In the change process in step S038, the display of the set video of the setting area Ar that satisfies the termination condition is terminated. For example, in FIG. 26, when the setting area Ar whose identification information of the label f is "2" satisfies the termination condition, as shown in FIG. 30, the display of the set video of the setting area Ar is terminated (disappears from the sub-screen G2). As a result, the number of displays of the set video decreases by one. Regarding the setting area Ar (hereinafter referred to as the termination area) where the display of the set video has been terminated, as shown in FIG. 30, the display of the label f indicating the position of the termination area is also terminated on the main screen G1.

[0173] Incidentally, instead of immediately stopping the display of the setting video in the setting area Ar that meets the stop condition, proposal information regarding the stop of the display of the setting video (in other words, the change in the number of displayed setting videos) may be output. For example, a message recommending the stop of the display of the setting video in the setting area Ar that meets the stop condition may be displayed on the information display screen G3, or reproduced by voice or the like. During that time, on the sub-screen G2, instead of the setting video in the setting area Ar that meets the stop condition, another video (a video different from the setting video) may be displayed.

[0174] When the display of the setting video of the stop video is stopped, the stop area is stored in the internal memory 50 or the like in the imaging device 10 while the recording process is being carried out. The period during which the stop area is stored may be until the end of shooting, or until the power of the imaging device 10 is turned off.

[0175] Also, when the display of the setting video in the stop area is stopped, as shown in FIG. 30, operation icons such as a return button Bt9 are displayed on the main screen G1. The user can, for example, tap the return button Bt9 or the like to give an instruction to redisplay the setting video in the stop area. When there is a user's redisplay instruction (S039), the control unit 46 performs a change process (S040). In such a change process, based on the user's redisplay instruction, a change process for redisplaying the stored setting video in the stop area is executed.

[0176] By the change process in step S040, the video in the stop area is again displayed on the sub-screen G2, and the number of displayed setting videos returns to the number before the display of the setting video in the stop area was stopped. Note that the setting video of the stop area that can be redisplayed is not limited to the video of the immediately preceding stop area (the setting area in which the display of the setting video was stopped most recently). For example, the videos of the stop areas for the past several times during the current shooting period may be made redisplayable. In that case, the imaging device 10 stores the stop areas for the past several times, and the user specifies which setting video of the stop area to redisplay to give an instruction to redisplay the video.

[0177] In the flow shown in FIG. 25, during the execution of the recording process, the control unit 46 monitors the reference video and determines whether the number of subjects in the reference area exceeds the upper limit value of the number of the set areas Ar (S041). Then, when the number of subjects exceeds the upper limit value of the number of the set areas Ar, the control unit 46 executes a change process (S042). In such a change process, a change process regarding the decrease in the display number of the set videos is executed. In the change process in step S042, a message indicating that the number of subjects exceeds the upper limit value of the number of the set areas Ar and recommending to reduce the display number of the set videos is displayed on the information display screen G3 or reproduced by voice or the like. Triggered by this, the user reviews the setting of the set area Ar and deletes unnecessary set areas Ar, such as the set area Ar with the longest period not selected as the recording area. As a result, the display number of the set videos decreases by the number of the set areas Ar deleted by the user.

[0178] The series of steps S031 to S042 described above are repeatedly executed during the execution of the recording process and end at the end of the recording process (S043). As described above, in this embodiment, the change process is executed triggered by a change in the reference video (for example, addition of a subject, etc.) and the establishment of the stop condition, and the display number of the set videos is appropriately changed. Thereby, the user can easily check the set videos in a number according to the situation at that time and omit the check of videos of the set areas Ar with a low possibility of becoming the recording area. As a result, the convenience for the user is improved.

[0179] <<Other Embodiments>> The embodiments described above are specific examples given to explain the display method of the present invention in an easy-to-understand manner, and are merely examples, and other embodiments can also be considered. In the above-described embodiment, the imaging device 10 was configured to display various images as a display device, but the present invention is not limited thereto. For example, the display device may be configured by the imaging device 10, a camera controller connected to the imaging device 10 by wire or wirelessly, and an external display of an external type. In that case, the camera controller functions as the control unit 46, causes the imaging device 10 to capture a reference image, and causes the external display to display various images. Further, the camera controller may perform setting of the setting area Ar, selection and re-selection (switching) of the recording area, and recording of the recorded image, etc.

[0180] Also, in the above embodiment, in the control process during the video display flow, control processing for suppressing the change of the reference area was executed. However, the present invention is not limited thereto, and as control processing regarding the change of the reference area, processing for promoting the change of the reference area may be executed. In this case, the reference area can be adjusted according to the change in the reference image (for example, movement of a subject in the reference image, etc.).

[0181] More specifically, for example, it is assumed that the recorded image is zoomed up by electronic zoom in the recording process. At that time, if the zoom magnification exceeds a predetermined value, there is a possibility that the resolution of the recorded image becomes equal to or lower than a certain value and it becomes difficult to obtain a desired image quality. In that case, as shown in FIG. 31, instead of electronic zoom, optical zoom may be performed and information for promoting the change of the reference area may be displayed on the information display screen G3. Alternatively, for example, so that the resolution becomes equal to or higher than a predetermined value, the control unit 46 may move the zoom driving unit to forcibly perform optical zoom and automatically change the reference area. When the reference area changes due to optical zoom, the relative position of each setting area Ar with respect to the reference area changes. In that case, it is preferable that the control unit 46 calculates the amount of change of the reference area and adjusts the position and size of each setting area Ar based on the calculated amount of change. Thereby, the relative position of each setting area Ar with respect to the reference area is maintained before and after the implementation of the zoom.

[0182] Also, during the recording process, the movement area Am may follow the movement of the subject and reach near the edge of the reference area, or at least a part of the movement area may protrude from the reference area. In that case, information prompting the moving of the imaging device 10 (i.e., changing the reference video) so that the moved movement area Am fits within the reference area may be displayed on the information display screen G3. Here, when the user changes the reference area by moving the imaging device 10 during the recording process, a known detection sensor such as a gyro sensor provided in the imaging device 10 may detect the movement (e.g., displacement amount) of the imaging device 10. Then, it is preferable that the control unit 46 adjusts the position and size of each setting area Ar based on the detected movement of the imaging device 10. Thereby, the relative position of each setting area Ar with respect to the reference area is adjusted according to the changed reference area. At this time, the setting area Ar that has gone outside the reference area due to the change of the reference area may be automatically deleted.

[0183] Also, in the above embodiment, it is assumed that the recorded video is a moving image, that is, a set of a plurality of frame images continuously captured at a certain frame rate. However, the recorded video is not limited to a moving image and may be a still image. For example, the control unit 46 displays the reference video as a live view image on the main screen G1 and sets a plurality of setting areas Ar within the reference area. The user selects the recording area Ap from among the plurality of setting areas Ar, and the control unit 46 displays a plurality of setting videos including the recording video on the sub-screen G2. Then, when the user gives a recording instruction by operating the release button 26 or the like, the control unit 46 may record the recording video as a still image on the recording medium.

[0184] Also, in the above embodiment, when a subject in the reference video is detected, the setting area Ar is automatically set at the position of the detected subject. However, it is not limited to the case where the setting area Ar is immediately set at the position of the detected subject, and a candidate Ac for the setting area may first be set at the position of the detected subject. In that case, a candidate mark K is displayed at the position of the candidate Ac, and when the user performs a predetermined operation on the candidate mark K, the setting area Ar may be set at the position of the candidate Ac.

[0185] In the above-described embodiment, for the setting area Ar that satisfies the suspension condition, the display of the setting video is suspended, while it is stored in the imaging device 10 during the recording process. However, the present invention is not limited to this, and the setting area Ar that satisfies the suspension condition may be deleted, and the video of that area may not be stored. In that case, it is preferable to display proposal information regarding the deletion of the setting area Ar on the information display screen G3 at a stage before deleting the setting area Ar that satisfies the suspension condition. Then, a screen (not shown) for selecting whether or not to delete the setting area Ar is separately displayed, and after confirming the user's intention regarding whether or not to delete, the setting area Ar that satisfies the suspension condition may be deleted.

[0186] In the above-described embodiment, the imaging device is a digital camera, but it may also be a video camera, a mobile phone with an imaging optical system, a smartphone, a tablet terminal, or other mobile terminals. Further, the imaging lens may be a lens unit that is externally attached to the imaging optical system of the mobile terminal.

Explanation of Reference Numerals

[0187] 10 Imaging device 12 Imaging device body 13 Mount 14 Imaging lens 17 Optical unit 18, 19 Optical devices 20 Diaphragm 21 Zoom drive unit 22 Focus drive unit 23 Diaphragm drive unit 26 Release button 28 Display 29 Electronic viewfinder 30 Operation button 36 Touch panel 38 Shutter 40 Image sensor 42 Pixel 44 Analog signal processing circuit 46 Control unit 47 Controller 48 Image processing unit 50 Internal memory 52 Card slot 54 Memory card 56 Buffer memory 61 Camera controller 62 External display 101 First UI screen 102 Second UI screen 103 Third UI screen 104 Fourth UI screen 105 Input screen Ac candidate Am movement area Ap recording area Ar setting area As, At shooting area Bt1~Bt8 Buttons Bt9 Rewind button f, fx markers G1 Main screen G2, G2x Sub screens G3 Information display screen K candidate marker L1 Optical axis T Table data

Claims

1. A display method for displaying an image captured by a photographing device, comprising: a setting step of setting a plurality of setting areas within a reference area which is a photographing area of a reference image; a selection step of selecting, from among the plurality of setting areas, a recording area which is an area of a recording image to be recorded; a switching step of, after the selection step is performed, reselecting the recording area from among the plurality of setting areas to switch the recording area; a recording step of recording the recording image on a recording medium; a display step of displaying a plurality of setting images which are images of the plurality of setting areas; a control step of executing control processing related to changing the reference area, at least after the reference area is determined, the display method comprising: the control processing includes processing for maintaining a state in which an optical zoom function of the photographing device is restricted while the recording step is being performed.

2. The display method according to claim 1, wherein, in the processing for maintaining the state in which the optical zoom function is restricted, a drive unit for zooming corresponding to the optical zoom function is locked.

Images