Imaging system

Abstract

To provide an imaging system capable of assisting uniform photographing of respective subjects by confirming the number of respective subjects photographed so far by a photographer in real time.SOLUTION: A head-mounted display device 100 includes: a display unit 103; a first imaging unit 105; a first communication unit 106 for communicating with an imaging device 200; and a first subject recognition unit 108 for recognizing a subject of an image photographed through the first imaging unit 105 and acquiring first subject information. The imaging device 200 includes: a second imaging unit 22; a second communication unit 54 for communicating with the head-mounted display device 100; a second subject recognition unit 250 for recognizing the subject of the image photographed through the second imaging unit 22 and acquiring second subject information; an image number counting unit 252 for counting the number of images including respective subjects from the image stored in the imaging device 200; and a collation unit 251 for collating the first subject information with the second subject information.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an imaging system, and more particularly to an imaging system that counts the number of shots of a subject. [Background technology]

[0002] When shooting a large number of people at the same time, such as school events such as athletic meets and field trips, the number of shots taken of each subject can vary. Also, when there are a large number of shots taken, it can take time to check the number of shots taken of each subject.

[0003] For example, Patent Document 1 discloses a method for counting the number of shots of a subject from images stored in a camera and displaying the counted number on a display.

[0004] Patent Document 2 discloses a method of using face recognition information to determine whether a subject is photographed and updating a count of the total number of shots. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2012-15912 A [Patent Document 2] JP 2012-129659 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the conventional technology disclosed in the above-mentioned patent document, the number of shots is checked on the camera display, so it is not possible to check the number of shots in real time while shooting, which may result in missing a photo opportunity. Furthermore, the number of shots of a subject that has never been photographed is not displayed, so there is a possibility that a shot will be missed.

[0007] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to provide an imaging system that allows a photographer to check in real time the number of images that have been taken of each subject, and that can assist in photographing each subject evenly. [Means for solving the problem]

[0008] An imaging system according to one aspect of the present invention is an imaging system having a head-mounted display device and an imaging device, wherein the display device has a display unit, a first imaging unit, a first communication unit that communicates with the imaging device, and a first subject recognition unit that recognizes a subject of an image captured via the first imaging unit and acquires first subject information, and the imaging device has a second imaging unit, a second communication unit that communicates with the display device, a second subject recognition unit that recognizes a subject of an image captured via the second imaging unit and acquires second subject information, a number of shots counting unit that counts the number of images in which each subject appears from images stored in the imaging device, and a comparison unit that compares the first subject information with the second subject information, and the imaging device transmits the number of shots corresponding to the subject compared by the comparison unit to the display device, and displays the number of shots of each subject on the display unit.

[0009] Other aspects of the present invention will become apparent from the embodiments described below. Effect of the Invention

[0010] According to the present invention, it is possible to provide an imaging system that allows a photographer to check in real time the number of images that have been taken of each subject, and that can assist in photographing each subject evenly. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is an external view of a head-mounted display device 100 according to each embodiment of the present invention. [Diagram 2] FIG. 1 is an external view of an imaging device 200 according to each embodiment of the present invention. [Diagram 3]FIG. 1 is a configuration diagram of a head-mounted display device 100 and an imaging device 200 according to each embodiment of the present invention. [Figure 4] 2 is a flowchart according to the first embodiment of the present invention. [Diagram 5] FIG. 1 is an explanatory diagram according to a first embodiment of the present invention. [Figure 6] 11 is a flowchart according to a second embodiment of the present invention. [Figure 7] FIG. 11 is a first explanatory diagram according to the second embodiment of the present invention. [Figure 8] FIG. 11 is a second explanatory diagram according to the second embodiment of the present invention. [Figure 9] 11 is a flowchart according to a third embodiment of the present invention. [Figure 10] FIG. 11 is an explanatory diagram according to a third embodiment of the present invention. [Figure 11] 11 is a flowchart according to a fourth embodiment of the present invention. [Figure 12] FIG. 11 is an explanatory diagram according to a fourth embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Fig. 1 shows an external view of a head-mounted display device 100 as an example of a device to which the present invention can be applied. Fig. 3(a) is a block diagram showing an example of the configuration of the head-mounted display device 100 according to this embodiment.

[0013] The control unit 101 is a control unit that is composed of at least one processor or circuit, and controls the entire head-mounted display device 100.

[0014] The first power switch 102 is an operating member for switching the power of the head-mounted display device 100 between ON and OFF.

[0015] The first power supply unit 107 is composed of a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like.

[0016] The display unit 103 is a transmissive display unit, and is disposed so as to cover the field of view of the photographer. The photographer visually recognizes the image displayed on the display unit 103 as being superimposed on the actual image.

[0017] The gaze detection unit 104 has a function of detecting the gaze position of the photographer on the display unit 103. The gaze detection unit 104 detects the gaze position of the photographer from the movement of the photographer's eyeball 161, and outputs the detection information to the control unit 101.

[0018] The first imaging unit 105 is an imaging element configured with a CCD, CMOS element, or the like that converts an optical image into an electrical signal.

[0019] The first subject recognition unit 108 detects a subject within the field of view of the head-mounted display device 100 and performs face recognition, etc. The first subject recognition unit 108 outputs first subject information (recognition result) obtained by subject recognition to the control unit 101. Here, an example of the first subject information will be given. - Image data itself - Feature point data of eyes, nose, ears, mouth, etc. obtained through face recognition - Coordinate information of the subject's location in the image Facial expression information such as smiling and closing eyes -ID assigned to each subject

[0020] The first communication unit 106 is connected wirelessly or via a wired cable, and transmits and receives video signals and audio signals. The first communication unit 106 can also connect to a wireless LAN (Local Area Network) or the Internet. The first communication unit 106 can also communicate with an external device via Bluetooth (registered trademark) or Bluetooth Low Energy. The first communication unit 106 can transmit first subject information and images (including LV images) captured by the first imaging unit 105, and can also receive images and various other information from external devices. The number of shots and main subject ratio information from the imaging device 200, which will be described later, are also received via the first communication unit 106.

[0021] 2(a) and (b) show external views of an imaging device 200 as an example of an apparatus to which the present invention can be applied. Fig. 2(a) is a front perspective view of the imaging device 200, and Fig. 2(b) is a rear perspective view of the imaging device 200. Fig. 3(b) is a block diagram showing an example of the configuration of the imaging device 200 according to this embodiment.

[0022] The eyepiece 16 is the eyepiece of an eyepiece finder (a peer-type finder), and the photographer can view the image displayed on the finder display unit 28 through the eyepiece 16. The rear display unit 29 is a display unit provided on the rear of the camera for displaying images and various information.

[0023] The shutter button 61 is an operation unit for issuing a shooting instruction. The second power switch 72 is an operation member for switching ON and OFF the power of the imaging device 200. The lens 153 is usually composed of multiple lenses, but is shown here as only one lens for simplicity.

[0024] The terminal cover 202 is a lid for a slot in which a recording medium 249 is stored.

[0025] The shutter 151 is a focal plane shutter that can freely control the exposure time of the second imaging unit 22 under the control of the system control unit 50.

[0026] The second imaging unit 22 is an imaging element that converts an optical image into an electrical signal and is composed of a CCD, a CMOS element, etc. The A / D converter 23 is used to convert an analog signal output from the second imaging unit 22 into a digital signal.

[0027] The image processing unit 24 performs predetermined pixel interpolation, resizing such as reduction, and color conversion processing on the data from the A / D converter 23 or data from the memory control unit 15 described later. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data. The system control unit 50 performs exposure control and distance measurement control based on the arithmetic results obtained by the image processing unit 24. This allows TTL (through-the-lens) type AF (autofocus) processing, AE (autoexposure) processing, and EF (flash pre-flash) processing to be performed. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data, and performs TTL type AWB (auto white balance) processing based on the arithmetic results obtained.

[0028] The memory control unit 15 controls data transmission and reception between the A / D converter 23, the image processing unit 24, and the memory 32. Output data from the A / D converter 23 is written into the memory 32 via the image processing unit 24 and the memory control unit 15, or directly via the memory control unit 15. The memory 32 stores image data obtained by the imaging unit 22 and converted into digital data by the A / D converter 23, as well as image data to be displayed on the viewfinder display unit 28 and the rear display unit 29. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and sound.

[0029] The memory 32 also serves as a memory (video memory) for displaying images. The image data for display written in the memory 32 is displayed on the viewfinder display unit 28 and rear display unit 29 via the memory control unit 15. The viewfinder display unit 28 and rear display unit 29 perform display on a display device such as an LCD or an organic EL in response to a signal from the memory control unit 15. A live view display (LV display) can be performed by sequentially transferring and displaying data that has been A / D converted by the A / D converter 23 and stored in the memory 32 to the viewfinder display unit 28 or rear display unit 29. Hereinafter, an image displayed in live view will be referred to as a live view image (LV image).

[0030] The non-volatile memory 56 is an electrically erasable and recordable memory, and may be, for example, a Flash-ROM. Constants, programs, and the like for the operation of the system control unit 50 are stored in the non-volatile memory 56. The programs referred to here are programs for executing various flowcharts described later in this embodiment.

[0031] The system control unit 50 is a control unit composed of at least one processor or circuit, and controls the entire image capture device 200. Each process of this embodiment, which will be described later, is realized by executing a program recorded in the nonvolatile memory 56 described above. The system memory 52 may be a RAM, for example, in which constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 56, and the like are loaded. The system control unit 50 also performs display control by controlling the memory 32, the D / A converter 19, the viewfinder display unit 28, the rear display unit 29, and the like.

[0032] The first shutter switch 62 and the second shutter switch 64 are operating means for inputting various operational instructions to the system control unit 50. The first shutter switch 62 is turned ON by half-pressing (instruction to prepare for shooting) during the operation of the shutter button 61 provided on the imaging device 200, and generates a first shutter switch signal SW1. The first shutter switch signal SW1 starts shooting preparation operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing.

[0033] The second shutter switch 64 is turned on when the shutter button 61 is fully pressed (photographing instruction) and generates a second shutter switch signal SW2. The system control unit 50 starts a series of photographing processing operations, from reading out a signal from the imaging unit 22 to writing the captured image to the recording medium 249 as an image file, in response to the second shutter switch signal SW2.

[0034] The power supply control unit 80 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to be energized, and the like, and detects whether a battery is attached, the type of battery, and the remaining battery power. The power supply control unit 80 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the necessary voltage for the necessary period to each unit including the recording medium 249. The second power supply unit 30 is composed of primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, and Li batteries, an AC adapter, and the like.

[0035] The recording medium I / F 18 is an interface with a recording medium 249 such as a memory card or a hard disk. The recording medium 249 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like.

[0036] The second communication unit 54 is connected wirelessly or by a wired cable, and transmits and receives video signals and audio signals. The second communication unit 54 can also be connected to a wireless LAN (Local Area Network) or the Internet. The second communication unit 54 can also communicate with an external device via Bluetooth (registered trademark) or Bluetooth Low Energy. The second communication unit 54 can transmit images (including LV images) captured by the second imaging unit 22 and images recorded in the recording medium 249, and can also receive images and various other information from an external device. In this embodiment, the second communication unit 54 receives first subject information and transmits the number of shots and main subject ratio information to the head-mounted display device 100.

[0037] The second subject recognition unit 250 detects subjects captured by the second imaging unit 22 and subjects appearing in images recorded in the recording medium 249, and recognizes the faces of the detected subjects. The second subject recognition unit 250 outputs second subject information (recognition result) obtained by subject recognition to the matching unit 251, the number of shots counting unit 252, the calculation unit 253, and the determination unit 254. Here, examples of the second subject information include the following, similar to the first subject information. - Image data itself - Feature point data of eyes, nose, ears, mouth, etc. obtained through face recognition - Coordinate information of the subject's location in the image Facial expression information such as smiling and closing eyes -ID assigned to each subject

[0038] The photographed number counting section 252 uses the second subject information to count the number of images in which each subject appears from among the images stored in the recording medium 249 .

[0039] The calculation unit 253 calculates the ratio of images in which each subject is captured as a main subject based on the second subject information. A specific calculation method will be described later.

[0040] The determination unit 254 determines whether each subject is a person to be photographed or a person not to be photographed based on the first subject information or the second subject information. Here, a person not to be photographed refers to a subject that the photographer has no intention of photographing.

[0041] The collation unit 251 collates the first subject information with the second subject information to identify the same subject. Based on the collation result, the collation unit 251 outputs to the system control unit 50 the number of shots of each subject in the images stored in the recording medium 249, the main subject ratio, and the non-photographed subject determination result. EXAMPLES

[0042] Hereinafter, control of the head-mounted display device 100 and the image capture device 200 according to the first embodiment of the present invention will be described with reference to FIGS.

[0043] 4 is a flowchart according to the first embodiment of the present invention. In this flowchart, the processes of steps S401 to S406 are executed by the system control unit 50 included in the imaging device 200. Also, the processes of steps S411 to S415 are executed by the control unit 101 included in the head-mounted display device 100.

[0044] In step S401, the system control unit 50 determines whether or not the user has performed a shooting operation. If the user has not performed a shooting operation, the process proceeds to step S404, and if the user has performed a shooting operation, the process proceeds to step S402.

[0045] In step S402, the second subject recognition unit 250 analyzes the subject appearing in the captured image. The second subject recognition unit 250 can extract feature points such as the person's eyes, nose, ears, and mouth by analyzing the image data to acquire subject information. Examples of subject information are given below. The acquired subject information may be any one of the following, or any combination thereof. - Image data itself - Feature point data of eyes, nose, ears, mouth, etc. obtained through face recognition - Coordinate information of the subject's location in the image Facial expression information such as smiling and closing eyes -ID assigned to each subject

[0046] After acquiring the second object information, the process proceeds to step S403. In step S403, the shot number counting unit 252 updates the number of shots of each object based on the second object information.

[0047] In step S411, the first object recognition unit 108 determines whether or not a human object is within the field of view of the head-mounted display device 100. If the object is within the field of view, the process proceeds to step S412, and if the object is not within the field of view, step S411 is repeated.

[0048] In step S412, the first subject recognition unit 108 performs face recognition of the subject in the field of view and acquires first subject information. Like the second subject recognition unit 250 described above, the first subject recognition unit 108 can acquire subject information by extracting feature points such as the person's eyes, nose, ears, and mouth by analyzing image data. Here, examples of subject information are given. The acquired subject information may be any one of the following, or any combination thereof. - Image data itself - Feature point data of eyes, nose, ears, mouth, etc. obtained through face recognition - Coordinate information of the subject's location in the image Facial expression information such as smiling and closing eyes -ID assigned to each subject

[0049] In step S413, the head-mounted display device 100 transmits the first object information to the imaging device 200 via the first communication unit .

[0050] In step S404, the imaging device 200 determines whether or not the first object information has been received from the head-mounted display device 100. If the first object information has been received, the process proceeds to step S405, and if the first object information has not been received, step S404 is repeated.

[0051] In step S405, the matching unit 251 matches the first subject information with the second subject information. Here, an example of the matching method will be described. The matching unit 251 compares the positions and ratios of feature points such as the eyes, nose, ears, and mouth contained in the first subject information and the second subject information, respectively, and identifies subjects having the same features as the same person. At this time, a common ID may be assigned to each subject by the head-mounted display device 100 and the imaging device 200. The system control unit 50 obtains the number of shots of the target subject from the shot number counting unit 252 based on the matching result.

[0052] In step S406, the imaging device 200 transmits the number of shots of the subject to the head-mounted display device 100 via the second communication unit .

[0053] In step S414, the head-mounted display device 100 determines whether or not the number of captured images has been received from the imaging device 200. If the number has been received, the process proceeds to step S415; if not, step S414 is repeated.

[0054] In step S 415 , the control unit 101 displays the number of shots of each subject on the display unit 103 based on the number of shots received from the imaging device 200 .

[0055] FIG. 5 is an explanatory diagram according to the first embodiment of the present invention.

[0056] Reference numeral 500 denotes a frame line indicating the field of view of the head-mounted display device 100, and reference numeral 501 denotes a frame line indicating the angle of view of the imaging device 200. Note that in this embodiment, the frame lines 500 and 501 are displayed for the sake of convenience of explanation, but they do not need to be displayed in actual operation.

[0057] Reference numerals 502 to 505 indicate subjects within the visual field of the head-mounted display device 100. Of these, subjects 503 and 504 are also shown to be within the angle of view 501 of the imaging device 200.

[0058] Reference numeral 506 denotes an image of the number of shots of each subject, acquired in step S405 of the above-mentioned flow chart.

[0059] Reference numerals 507 to 510 indicate the display of the number of shots corresponding to the subjects 502 to 505, respectively. In this manner, the control unit 101 displays the number of shots of each subject on the display unit 103 based on the information 506 on the number of shots.

[0060] In this embodiment, the imaging device 200 has the matching unit 151. However, the head-mounted display device 100 may have the matching unit 151.

[0061] Note that a subject may be registered in advance in the imaging device 200, and the number of shots of the registered subject may be constantly displayed on the display unit 103. Also, a first threshold may be set for the number of shots, and the number of shots of a subject that exceeds the first threshold may be hidden.

[0062] Furthermore, a second threshold value may be set for the number of captured images, and an alert may be displayed on the display unit 103 if the number of captured images of a subject registered in the image capture device 200 does not reach the second threshold value.

[0063] Also, the line of sight detection unit 104 may detect the line of sight of the photographer, and the number of shots of the subject in the line of sight may be displayed in an emphasized manner.

[0064] As described above, according to the first embodiment of the present invention, it is possible to provide an imaging system that allows a photographer to check in real time the number of images that have been taken of each subject and can assist in photographing each subject evenly. EXAMPLES

[0065] Hereinafter, control of the head-mounted display device 100 and the image capture device 200 according to the second embodiment of the present invention will be described with reference to FIGS.

[0066] 6 is a flowchart according to the first embodiment of the present invention. In this flowchart, the processes of steps S601 to S606 are executed by the system control unit 50 included in the imaging device 200. Also, the processes of steps S611 to S615 are executed by the control unit 101 included in the head-mounted display device 100.

[0067] In step S601, the system control unit 50 determines whether or not the user has performed a shooting operation. If the user has not performed a shooting operation, the process proceeds to step S604, and if the user has performed a shooting operation, the process proceeds to step S602.

[0068] In step S602, the second subject recognition unit 250 analyzes the subject appearing in the captured image. The second subject recognition unit 250 can extract feature points such as the person's eyes, nose, ears, and mouth by analyzing the image data to acquire subject information. Examples of subject information are given below. The acquired subject information may be any one of the following, or any combination thereof. - Image data itself - Feature point data of eyes, nose, ears, mouth, etc. obtained through face recognition - Coordinate information of the subject's location in the image Facial expression information such as smiling and closing eyes -ID assigned to each subject

[0069] After acquiring the second object information, the process proceeds to step S603. In step S603, the calculation unit 253 calculates the main object ratio of each object in the image stored in the recording medium 249 based on the second object information. An example of a method for calculating the main object ratio will be described later.

[0070] In step S611, the first object recognition unit 108 determines whether or not a human object is within the field of view of the head-mounted display device 100. If the object is within the field of view, the process proceeds to step S612, and if the object is not within the field of view, step S611 is repeated.

[0071] In step S612, the first subject recognition unit 108 performs face recognition of the subject in the field of view and acquires first subject information. Like the second subject recognition unit 250 described above, the first subject recognition unit 108 can acquire subject information by extracting feature points such as the person's eyes, nose, ears, and mouth by analyzing image data. Here, examples of subject information are given. The acquired subject information may be any one of the following, or any combination thereof. - Image data itself - Feature point data of eyes, nose, ears, mouth, etc. obtained through face recognition - Coordinate information of the subject's location in the image Facial expression information such as smiling and closing eyes -ID assigned to each subject

[0072] In step S613, the head-mounted display device 100 transmits the first object information to the imaging device 200 via the first communication unit .

[0073] In step S604, the imaging device 200 determines whether or not the first object information has been received from the head-mounted display device 100. If the first object information has been received, the process proceeds to step S605, and if the first object information has not been received, the process repeats step S604.

[0074] In step S605, the matching unit 251 matches the first subject information with the second subject information. Here, an example of the matching method will be described. The matching unit 251 compares the positions and ratios of feature points such as the eyes, nose, ears, and mouth contained in the first subject information and the second subject information, respectively, and identifies subjects having the same features as the same person. At this time, a common ID may be assigned to each subject by the head-mounted display device 100 and the imaging device 200. The system control unit 50 acquires the main subject rate of the target subject from the calculation unit 253 based on the matching result.

[0075] In step S606, the imaging device 200 transmits the main subject ratio of the subject to the head-mounted display device 100 via the second communication unit .

[0076] In step S614, the head-mounted display device 100 determines whether or not the main subject ratio has been received from the imaging device 200. If it has been received, the process proceeds to step S615; if it has not been received, step S614 is repeated.

[0077] In step S615, the control unit 101 displays the main subject ratio of each subject on the display unit 103 based on the main subject ratio received from the imaging device 200.

[0078] 7 is a first explanatory diagram according to the second embodiment of the present invention. 701 to 704 indicate images stored in the recording medium 249. 751 to 754 are tables showing the results of calculating the main subject ratio from the images 701 to 704, respectively.

[0079] A method for calculating the main subject ratio from the ratio of face area in an image will be described with reference to Fig. 7(a). 711 and 712 are subjects recognized by second subject recognition unit 250. 713 is a frame representing the face area of ​​subject 711, and 714 is a frame representing the face area of ​​subject 712. In this case, since face area 713 has a larger area compared to face area 714, subject 711 is weighted more heavily when calculating the main subject ratio (subject 711 is determined to have a larger proportion of being captured as the main subject).

[0080] A method of calculating the main subject ratio depending on whether or not the subject is in focus will be described with reference to FIG. 7(b). 721 and 722 are subjects recognized by the second subject recognition unit 250. 723 is a frame showing the point (focus position) where the focus was achieved when the image 702 was captured. In this embodiment, the left eye of the subject 721 is in focus, and the subject 722 is not in focus. At this time, the main subject ratio is calculated according to the degree of focus, that is, so that the subject 721, which is in focus, is weighted more heavily. (The subject 721 is determined to have a higher proportion of being captured as the main subject).

[0081] A method of calculating the main subject ratio based on which area in the image the face is in will be described with reference to FIG. 7(c). 731 and 732 are subjects recognized by the second subject recognition unit 250. The shaded area of ​​733 indicates the central area in the image 703. In this embodiment, the subject 731 is shown to be in the central area 733, and the subject 732 is shown to be in a range outside the central area 733. At this time, the main subject ratio is calculated so that the subject 731 in the central area 733 is weighted more heavily. (The subject 731 is determined to have a higher proportion of being the main subject.) The range of the central area 731 may be wider or narrower than the example shown in this embodiment.

[0082] A method for calculating the main subject rate from the facial expression of the subject will be described with reference to FIG. 7(d). 741 and 742 are subjects recognized by the second subject recognition unit 250. In this embodiment, subject 741 is photographed with a smile, and subject 742 is photographed with his eyes closed. In this case, the main subject rate is calculated so that the smiling subject 741 is weighted more heavily. (Subject 741 is determined to have a higher proportion of being photographed as the main subject.) The four main subject ratio calculation methods have been described above. Each of the four methods may be implemented independently, or may be implemented in any combination.

[0083] In this embodiment, for convenience of explanation, the frame lines 713, 714, 723, and central area 733 are displayed, but they do not need to be displayed in actual operation.

[0084] 8 is a second explanatory diagram according to the second embodiment of the present invention. Reference numeral 800 denotes a frame line indicating the field of view of the head-mounted display device 100, and reference numeral 801 denotes a frame line indicating the angle of view of the imaging device 200. Note that in this embodiment, the frame lines 800 and 801 are displayed for the sake of convenience of explanation, but they do not need to be displayed in actual operation.

[0085] Reference numerals 802 to 805 denote subjects within the visual field of the head-mounted display device 100. Of these, subjects 803 and 804 are also shown to be within the angle of view 801 of the imaging device 200.

[0086] Reference numeral 806 denotes an image of the main subject ratio of each subject calculated in step S603 of the above-mentioned flowchart. In this embodiment, the main subject ratio of subject 803 is the highest, the main subject ratios of subjects 802 and 805 are the second highest, and the main subject ratio of subject 804 is the lowest.

[0087] Reference numerals 807 to 810 denote indices based on the main subject ratio corresponding to the subjects 802 to 805, respectively. The indices 807 to 810 may change color or shape according to the main subject ratio. Also, a configuration may be adopted in which no indices are displayed for subjects whose main subject ratio exceeds a predetermined value.

[0088] As described above, according to the second embodiment of the present invention, it is possible to provide an imaging system that allows the photographer to check the main subject ratio of each subject in real time and can assist in photographing each subject evenly. EXAMPLES

[0089] Hereinafter, control of the head-mounted display device 100 and the image capture device 200 according to the third embodiment of the present invention will be described with reference to FIGS.

[0090] 9 is a flowchart according to the third embodiment of the present invention. In this flowchart, the processes of steps S901 to S907 are executed by the system control unit 50 included in the imaging device 200. Also, the processes of steps S911 to S915 are executed by the control unit 101 included in the head-mounted display device 100.

[0091] In step S901, the system control unit 50 determines whether or not the user has performed a shooting operation. If the user has not performed a shooting operation, the process proceeds to step S905, and if the user has performed a shooting operation, the process proceeds to step S902.

[0092] In step S902, the second subject recognition unit 250 analyzes the subject appearing in the captured image. The second subject recognition unit 250 can extract feature points such as the person's eyes, nose, ears, and mouth by analyzing the image data to acquire subject information. Examples of subject information are given below. The acquired subject information may be any one of the following, or any combination thereof. - Image data itself - Feature point data of eyes, nose, ears, mouth, etc. obtained through face recognition - Coordinate information of the subject's location in the image Facial expression information such as smiling and closing eyes -ID assigned to each subject

[0093] After acquiring the second subject information, the process proceeds to step S903. In step S903, the determination unit 254 determines whether each subject in the captured image is a person to be photographed or a person not to be photographed based on the second subject information. The determination result is added to the second subject information. A person to be photographed refers to a subject that the photographer has the intention of photographing, and a person not to be photographed refers to a subject that the photographer has no intention of photographing. Here, a method of distinguishing between a person to be photographed and a person not to be photographed will be described. Any age or clothing may be registered in the system control unit 50 in advance, and a person who does not correspond may be determined as a person not to be photographed, or face data of a person to be photographed may be registered in the system control unit 50 in advance to determine a person not to be photographed. In addition, a subject that was not in focus at the time of photographing may be determined as a person not to be photographed. A configuration may be adopted in which a subject that the photographer does not gaze at is determined as a person not to be photographed based on the detection result of the gaze detection unit 104, and the imaging device 200 is notified of this.

[0094] In step S904, the photograph number counting unit 252 updates the number of photographs of the subject based on the second subject information.

[0095] In step S911, the first object recognition unit 108 determines whether or not a human object is within the field of view of the head-mounted display device 100. If the object is within the field of view, the process proceeds to step S912, and if the object is not within the field of view, step S911 is repeated.

[0096] In step S912, the first subject recognition unit 108 performs face recognition of the subject in the field of view and acquires first subject information. Like the second subject recognition unit 250 described above, the first subject recognition unit 108 can acquire subject information by extracting feature points such as the person's eyes, nose, ears, and mouth by analyzing image data. Examples of subject information are given below. The acquired subject information may be any one of the following, or any combination. - Image data itself - Feature point data of eyes, nose, ears, mouth, etc. obtained through face recognition - Coordinate information of the subject's location in the image Facial expression information such as smiling and closing eyes -ID assigned to each subject

[0097] In step S913, the head-mounted display device 100 transmits the first subject information to the imaging device 200 via the first communication unit 106. In step S905, the imaging device 200 determines whether or not the first subject information has been received. If the first subject information has been received, the process proceeds to step S906, and if the first subject information has not been received, step S905 is repeated.

[0098] In step S906, the matching unit 251 matches the first subject information with the second subject information. Here, an example of the matching method will be described. The matching unit 251 compares the positions and ratios of feature points such as eyes, nose, ears, and mouth contained in the first subject information and the second subject information, respectively, and identifies subjects having the same features as the same person. At this time, a common ID may be assigned to each subject by the head-mounted display device 100 and the imaging device 200. Based on the matching result, the system control unit 50 obtains the number of shots from the shot number count unit 252 for the person to be photographed, and does nothing for non-subjects.

[0099] In step S907, the matching unit 251 determines whether or not the first subject information includes a person to be photographed based on the matching result in step S906. If the person to be photographed is included, the process proceeds to step S908; if the person to be photographed is not included, the process proceeds to step S909.

[0100] In step S908, the imaging device 200 transmits information about the number of images captured of the subject to the head-mounted display device 100 via the second communication unit .

[0101] In step S909, the imaging device 200 notifies the head-mounted display device 100 via the second communication unit 54 that there is no subject within the field of view.

[0102] In step S914, the head-mounted display device 100 determines whether or not information has been received from the image capture device 200. If information has been received, the process proceeds to step S915, and if information has not been received, step S914 is repeated.

[0103] In step S915, the control unit 101 determines whether the information received in step S914 is information on the number of shots. If information on the number of shots has been received, the process proceeds to S916, and if a notification has been received that there is no subject within the field of view, the process ends.

[0104] In step S916, the control unit 101 displays the number of shots of the subject on the display unit 103 based on the number of shots received from the imaging device 200. Here, the number of shots is not displayed for persons who are not subjects.

[0105] 10 is an explanatory diagram according to a third embodiment of the present invention. Reference numeral 1000 denotes a frame line indicating the field of view of the head-mounted display device 100, and reference numeral 1001 denotes a frame line indicating the angle of view of the imaging device 200. Note that in this embodiment, the frame lines 1000 and 1001 are displayed for the sake of convenience of explanation, but they do not need to be displayed in actual operation.

[0106] Reference numerals 1002 to 1005, 1011, and 1012 represent subjects within the field of view of head-mounted display device 100. Among these, subjects 1003 and 1004 are also within angle of view 1001 of imaging device 200. Subjects 1002 to 1005 are subjects determined by determination unit 254 to be subjects to be photographed, and subjects 1011 and 1012 are subjects determined by determination unit 254 to be subjects not to be photographed.

[0107] Reference numeral 1006 denotes an image of information on the number of shots of each subject. The number of shots is counted for subjects 1002 to 1005, who are the subjects of photography, but the number of shots is not counted for subjects 1011 and 1012, who are not subjects of photography, and information that they are not subjects of photography is stored.

[0108] Reference numerals 1007 to 1010 indicate the display of the number of shots corresponding to the subjects 1002 to 1005, respectively. In this manner, the control unit 101 displays the number of shots of the subject on the display unit 103 based on the number of shots 1006.

[0109] Moreover, 1011 and 1012 denote non-target people who are within the field of view of the head-mounted display device 100. In this way, the number of shots taken is not displayed for non-target people.

[0110] In this embodiment, the display of the number of shots has been described, but the contents of this embodiment may be applied to a case where an index based on the main subject ratio is displayed as in the second embodiment.

[0111] As described above, according to the third embodiment of the present invention, it is possible to provide an imaging system that displays the number of shots taken to non-subjects, prevents the photographer's field of vision from being obstructed, and assists in photographing each subject evenly. EXAMPLES

[0112] Hereinafter, control of the head-mounted display device 100 and the image capture device 200 according to the fourth embodiment of the present invention will be described with reference to FIGS.

[0113] 11 is a flowchart according to the fourth embodiment of the present invention. In this flowchart, the processes of steps S1101 to S1103 and S1107 to S1112 are executed by the control unit 101 included in the head-mounted display device 100. Also, the processes of steps S1104 to S1106 and S1113 to S1115 are executed by the system control unit 50 included in the imaging device 200.

[0114] In step S1101, the first object recognition unit 108 determines whether or not a human object is within the field of view of the head-mounted display device 100. If the object is within the field of view, the process proceeds to step S1102, and if the object is not within the field of view, step S1101 is repeated.

[0115] In step S1102, the first subject recognition unit 108 performs face recognition of a subject within the field of view to acquire first subject information. The first subject recognition unit 108 can acquire subject information by extracting feature points such as the person's eyes, nose, ears, and mouth by analyzing image data. Examples of subject information are given below. The acquired subject information may be any one of the following, or any combination thereof. - Image data itself - Feature point data of eyes, nose, ears, mouth, etc. obtained through face recognition - Coordinate information of the subject's location in the image Facial expression information such as smiling and closing eyes -ID assigned to each subject

[0116] In step S1103, the head-mounted display device 100 transmits the first object information to the imaging device 200 via the first communication unit .

[0117] In step S1104, the imaging device 200 determines whether or not the first object information has been received from the head-mounted display device 100. If the first object information has been received, the process proceeds to step S1105, and if the first object information has not been received, the process repeats step S1104.

[0118] In step S1105, the matching unit 251 matches the first subject information with the second subject information held by the second subject recognition unit 250. Here, an example of a matching method will be described. The matching unit 251 compares the positions and ratios of feature points such as eyes, nose, ears, and mouth contained in the first subject information and the second subject information, respectively, and identifies subjects having the same features as the same person. At this time, a common ID may be assigned to each subject by the head-mounted display device 100 and the imaging device 200. The system control unit 50 obtains the number of shots of the target subject from the shot number counting unit 252 based on the matching result.

[0119] In step S1106, the imaging device 200 transmits the number of shots of the subject to the head-mounted display device 100 via the second communication unit .

[0120] In step S1107, the head-mounted display device 100 determines whether or not the number of captured images has been received from the imaging device 200. If the number has been received, the process proceeds to step S1108; if not, step S1107 is repeated.

[0121] In step S 1108 , the control unit 101 displays the number of shots of each subject on the display unit 103 based on the number of shots received from the imaging device 200 .

[0122] In step S1109, the line-of-sight detection unit 104 detects the line-of-sight direction of the photographer.

[0123] In step S1110, the control unit 101 determines whether or not a subject is present in the line of sight detected in step S1110. If it is determined that a subject is present, the process proceeds to step S1111, and if it is determined that a subject is not present, S1110 is repeated.

[0124] In step S1111, the control unit 101 determines whether the number of shots of the subject is equal to or less than a predetermined value based on the number of shots of the subject and the information on the subject that is determined to be in the line of sight of the photographer. If it is determined to be equal to or less than the predetermined value, the process proceeds to step S1112, and if it is determined to be greater than the predetermined value, the process returns to step S1110.

[0125] In step S1112, the control unit 101 transmits a shooting instruction to the imaging device 200 via the first communication unit 106 to perform a shooting operation.

[0126] In step S1113, the system control unit 50 receives a shooting instruction, and the second object recognition unit 250 determines whether or not there is an object for which the number of shots was a predetermined value or less within the angle of view of the imaging device 200. If it is determined that the object is present, the process proceeds to step S1114, and if it is determined that the object is not present, the flow ends.

[0127] In step S1114, the system control unit 50 performs automatic shooting. In step S1115, the shot number counter 252 updates the number of shots.

[0128] 12 is an explanatory diagram according to a fourth embodiment of the present invention. Reference numeral 1200 denotes a frame line indicating the field of view of the head-mounted display device 100, and reference numeral 1201 denotes a frame line indicating the angle of view of the imaging device 200. Note that in this embodiment, the frame lines 1200 and 1201 are displayed for the sake of convenience of explanation, but they do not need to be displayed in actual operation.

[0129] Reference numerals 1202 to 1205 denote subjects within the field of view of the head-mounted display device 100. Among these, subjects 1203 and 1204 are also within the angle of view 1201 of the imaging device 200.

[0130] Reference numeral 1206 denotes an image of the number of shots of each subject, obtained as a result of analysis by the second subject recognition unit 250 of the image capture device 200 .

[0131] Reference numerals 1207 to 1210 indicate the display of the number of shots corresponding to the subjects 1202 to 1205, respectively. In this manner, the control unit 101 displays the number of shots of each subject on the display unit 103 based on the information 1206 on the number of shots.

[0132] Reference numeral 1211 denotes a threshold value set in the head-mounted display device 100 as the minimum number of images to be captured.

[0133] Reference numeral 1212 denotes the gaze direction of the photographer calculated in step S1109. Here, the number of images of the subject 1203 in the direction of 1212 does not meet the minimum number of images, so the head-mounted display device 100 sends a shooting instruction to the imaging device 200, and the imaging device 200 automatically shoots the subject 1203 when it enters the angle of view 1201.

[0134] In this embodiment, an example of performing automatic shooting based on a threshold value for the number of shots has been described. However, the contents of this embodiment may be applied to a case where the main subject ratio is calculated as in the second embodiment.

[0135] As described above, according to the fourth embodiment of the present invention, it is possible to provide an imaging system that allows a photographer to check in real time the number of images that have been taken of each subject, and that can detect when a subject enters the angle of view of the imaging device 200 and perform automatic shooting.

[0136] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0137] 100 display device 103 Display section 105 First imaging unit 106 First Communication Department 108 First object recognition unit 200 Imaging device 22 Second imaging unit 54 Second Communication Department 250 Second Object Recognition Unit 252 Number of shots counter 251 Collation Department

Claims

1. An imaging system having a head-mounted display device and an imaging device, The display device includes a display unit, a first imaging unit, a first communication unit that communicates with the imaging device, and a first object recognition unit that recognizes an object in an image captured via the first imaging unit and acquires first object information; The imaging device has a second imaging unit, a second communication unit that communicates with the display device, a second subject recognition unit that recognizes subjects in images captured via the second imaging unit and acquires second subject information, a number-of-shots counting unit that counts the number of images in which each subject appears from among images stored in the imaging device, and a comparison unit that compares the first subject information with the second subject information, the imaging device transmits to the display device the number of shots corresponding to the subject collated by the collation unit; The number of shots of each subject is displayed on the display unit.

1. An imaging system comprising:

2. An imaging system having a head-mounted display device and an imaging device, The display device includes a display unit, a first imaging unit, a first communication unit that communicates with the imaging device, and a first object recognition unit that recognizes an object in an image captured via the first imaging unit and acquires first object information; The imaging device has a second imaging unit, a second communication unit that communicates with the display device, a second subject recognition unit that recognizes subjects in an image captured via the second imaging unit and acquires second subject information, a calculation unit that calculates a ratio of images in which each subject is captured as a main subject from among images stored in the imaging device, and a comparison unit that compares the first subject information with the second subject information, the imaging device transmits to the display device a main subject rate corresponding to the subject collated by the collation unit; An index based on the main subject ratio of each subject is displayed on the display unit.

1. An imaging system comprising:

3. An imaging system having a head-mounted display device and an imaging device, The display device includes a display unit, a first imaging unit, a first communication unit that communicates with the imaging device, and a first object recognition unit that recognizes an object in an image captured via the first imaging unit and acquires first object information; The imaging device has a second imaging unit, a second communication unit that communicates with the display device, a second subject recognition unit that recognizes subjects in images captured via the second imaging unit and acquires second subject information, a number of images counting unit that counts the number of images in which each subject appears from images stored in the imaging device, a comparison unit that compares the first subject information with the second subject information, and a determination unit that determines whether each subject is a person to be photographed or not based on a recognition result of the second subject recognition unit, For those who are not the subject of photography, the number of photographs taken is not displayed on the display unit.

1. An imaging system comprising:

4. The first subject information is at least one of image data captured by the first imaging unit, feature point data of the subject's face, coordinate data of the position of the face in the image, and an ID associated with each subject.

4. The imaging system according to claim 1, wherein the imaging device is a digital camera.

5. The imaging device has a means for notifying the display device of the angle of view of the second imaging unit, and displays a frame indicating the angle of view on the display device.

4. The imaging system according to claim 1, wherein the imaging device is a digital camera.

6. The imaging device includes a means for registering a subject and a means for transmitting information about the registered subject to a head-mounted display device, and the display unit displays the number of images of the registered subject.

4. The imaging system according to claim 1, wherein the imaging device is a digital camera.

7. The imaging device includes a means for registering a subject in the imaging device and a means for setting a first threshold, and the number of shots of the subject that exceeds the first threshold is not displayed on the display unit.

4. The imaging system according to claim 1, wherein the imaging device is a digital camera.

8. The imaging device includes a means for registering a subject in the imaging device and a means for setting a second threshold, and when the number of images of the registered subject does not reach the second threshold, a notification is displayed on the display unit.

4. The imaging system according to claim 1, wherein the imaging device is a digital camera.

9. A gaze detection unit is provided to detect the gaze position of the photographer relative to the display unit, and the number of shots of a subject in the gaze direction is highlighted and displayed.

4. The imaging system according to claim 1, wherein the imaging device is a digital camera.

10. The calculation unit has a means for determining the ratio of the face area of ​​each subject in the image, and performs weighting according to the ratio of the face area when calculating the main subject ratio.

3. The imaging system according to claim 2.

11. The calculation unit has a means for identifying a focus position in an image, and when calculating the main subject ratio, weights each subject according to the degree of focus.

3. The imaging system according to claim 2.

12. The calculation unit has a means for detecting the position of the face of each subject in the image, and performs weighting according to the position of the face when calculating the main subject ratio.

3. The imaging system according to claim 2.

13. The calculation unit has a means for determining the facial expression of each subject, and when calculating the main subject ratio, weights are assigned according to the facial expression of each subject.

3. The imaging system according to claim 2.

14. The imaging device has a means for notifying the display device of the angle of view of the second imaging unit, and the display device notifies a photographer when it detects that a subject displaying an index based on a main subject ratio on the display device has entered the angle of view.

3. The imaging system according to claim 2.

15. The imaging system according to claim 3, further comprising a means for registering the age of the person to be photographed in advance in the imaging device, and the determination unit determines that a subject that does not correspond to the age of the person to be photographed from the second subject information is a non-subject.

16. The imaging device has a means for registering clothing of the subject in advance, and the determination unit determines a subject that does not match the clothing of the subject from the second subject information as a non-subject.

4. The imaging system according to claim 3.

17. The imaging device has a means for registering face information of a person to be photographed in advance, and the determination unit determines a subject that does not correspond to the face information of the person to be photographed from the second subject information as a non-photographed subject.

4. The imaging system according to claim 3.

18. The determination unit has a means for identifying a focus position in an image, and determines that an out-of-focus subject is not a person to be photographed.

4. The imaging system according to claim 3.

19. The display device has a gaze detection unit that detects the gaze position of the photographer relative to the display unit, and the first subject recognition unit determines a subject with whom the photographer does not make gaze contact as a non-photographable subject and notifies the imaging device of information about the non-photographable subject.

4. The imaging system according to claim 3.

20. An imaging system having a head-mounted display device and an imaging device, The display device includes a display unit, a first imaging unit, a first communication unit that communicates with the imaging device, a gaze detection unit that detects a gaze position of a photographer relative to the display unit, and a first subject recognition unit that recognizes a subject in an image captured via the first imaging unit and acquires first subject information; The imaging device has a second imaging section, a second communication section which communicates with the display device, a means for notifying the display device of an angle of view of the second imaging section, a second subject recognition section which recognizes subjects in images captured via the second imaging section and acquires second subject information, a number-of-shots counting section which counts the number of images in which each subject appears from among images stored in the imaging device, and a comparison section which compares the first subject information with the second subject information, the display device transmits a photographing instruction to the imaging device to perform a photographing operation when the number of photographs of the subject in the line of sight of the photographer is equal to or less than a predetermined value; After receiving the shooting instruction, the imaging device performs automatic shooting when the subject enters the angle of view of the second imaging unit.

1. An imaging system comprising:

21. An imaging system having a head-mounted display device and an imaging device, The display device includes a display unit, a first imaging unit, a first communication unit that communicates with an imaging device, a gaze detection unit that detects a gaze position of a photographer relative to the display unit, and a first subject recognition unit that recognizes a subject in an image captured via the first imaging unit and acquires first subject information; The imaging device includes a second imaging unit, a second communication unit that communicates with the display device, a means for notifying the display device of an angle of view of the second imaging unit, a second subject recognition unit that recognizes subjects in an image captured via the second imaging unit and acquires second subject information, a calculation unit that calculates a ratio of images in which each subject is captured as a main subject from among images stored in the imaging device, and a comparison unit that compares the first subject information with the second subject information, the display device transmits a photographing instruction to the imaging device to perform a photographing operation when a main subject ratio of subjects in a line of sight of a photographer is equal to or smaller than a predetermined value; After receiving the shooting instruction, the imaging device performs automatic shooting when the subject enters the angle of view of the second imaging unit.

1. An imaging system comprising:

Images