Video processing device and method, imaging device, video distribution system, program, and storage medium

Abstract

To enhance the possibility that a subject to which focus and exposure are adjusted is included in a distributed video, while distributing the video of a video area desired to be distributed by a user.SOLUTION: A video processing device includes detection means for detecting a predetermined subject from the video obtained from imaging means for repeatedly photographing and outputting the video, acquisition means for acquiring the information of at least one partial area set to the video obtained from the imaging means, and decision means for deciding a main subject on the basis of the priority of the subject in the partial area among the subjects detected by the detection means.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a video processing device and method, an imaging device, a program, and a storage medium. [Background technology]

[0002] In video distribution systems that distribute video to remote locations, a crop function is commonly known that distributes images based on signals read from pixels in a portion of the multiple pixels that make up an image sensor, rather than all of the effective pixels. This crop function makes it possible to distribute images captured by a camera that omit areas that are unnecessary for the user.

[0003] On the other hand, many of the cameras used in the above-mentioned video distribution system are equipped with a function that detects a person's face from the captured video and adjusts the focus and exposure to the detected person's face (subject). When using this function in a video distribution system equipped with a crop function, it is important that the subject on which the focus and exposure are being adjusted is present in the video being distributed.

[0004] Patent document 1 proposes a method for recording an image of an area corresponding to a subject determined based on the results of measurement operations such as photometry from among subjects detected based on an entire range of video, thereby ensuring that the subject for which focus and exposure are being adjusted is included in the recorded area of ​​the video. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2006-33291 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the case of the method described in Patent Document 1, if the range of the image to be recorded or distributed is determined based on the area corresponding to the subject determined based on the results of measurement operations such as photometry, the image may differ from the image that the user wishes to distribute.

[0007] The present invention has been made in consideration of the above problems, and aims to improve the likelihood that the subject being focused on and exposed will be included in the distributed image while distributing an image of the image area that the user wants to distribute. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, the image processing device of the present invention has a detection means for detecting a predetermined subject from an image obtained from an imaging means that repeatedly captures and outputs an image, an acquisition means for acquiring information on at least one partial area set for the image obtained from the imaging means, and a determination means for determining a main subject based on the priority of subjects within the partial area from among the subjects detected by the detection means. Effect of the Invention

[0009] According to the present invention, it is possible to distribute video of an image area that a user wishes to distribute, while increasing the likelihood that a subject on which focus and exposure are adjusted will be included in the distributed video. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a configuration of a video distribution system according to an embodiment of the present invention. [Diagram 2] 1A to 1C are diagrams for explaining a problem in a video distribution system and a method for setting a main face in a first embodiment. [Diagram 3] 13A and 13B are graphs illustrating a method of calculating a priority in main face selection. [Figure 4] 5 is a flowchart of a main face selection process according to the first embodiment. [Diagram 5]6 is a flowchart of a process for calculating a priority level of each face in the first embodiment. [Figure 6] 13A to 13C are diagrams for explaining a method of setting a main face in the second embodiment. [Figure 7] 13 is a flowchart of a main face selection process according to the second embodiment. [Figure 8] 13 is a flowchart of a process for calculating a priority level of each face according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] ●Video distribution system configuration 1 is a block diagram showing a configuration of a video distribution system that distributes video according to an embodiment of the present invention. The video distribution system includes an imaging device 10 and a distribution device 12. Note that in this embodiment, a case will be described in which the distribution device 12 is a device configured separately from a camera, such as a PC, but the imaging device 10 and the distribution device 12 may be configured as an integrated unit.

[0013] [Configuration of imaging device] In this embodiment, a description will be given of a case where the imaging device 10 is a video camera. Note that the present invention is not limited to video cameras, and can also be applied to other imaging devices such as digital still cameras.

[0014] 1, the imaging optical system is composed of a first fixed lens 101, a variable magnification lens 102, an aperture 103, a second fixed lens 104, and a focus lens 105. The variable magnification lens 102 is a variable magnification lens that can change magnification by moving in the optical axis direction. The focus lens 105 has both a general focus adjustment function and a function of correcting the movement of the focal plane associated with magnification change, and can adjust the illumination state of the light incident on the incidence surface of the image sensor 106 that is incident on the imaging optical system.

[0015] The image sensor 106 is composed of a CCD sensor, a CMOS sensor, etc., and performs photoelectric conversion on the optical image formed on the incident surface via the imaging optical system to output an analog signal. The CDS / AGC / AD converter 107 performs sampling, gain adjustment, and analog-to-digital conversion processing on the analog signal output from the image sensor 106, and outputs the resulting digital signal. The camera signal processing unit 108 performs various image processing on the digital signal output from the CDS / AGC / AD converter 107 to generate a video signal.

[0016] Display unit 109 displays the video signal from camera signal processing unit 108. Recording unit 110 records the video signal from camera signal processing unit 108 on a recording medium such as a magnetic tape, an optical disk, or a semiconductor memory. In addition, the video signal generated by camera signal processing unit 108 is also output to video receiving unit 123 of distribution device 12.

[0017] The AF gate 111 is used to pass only the digital signals of an area used for focus detection among the digital signals of all pixels output from the CDS / AGC / AD converter 107. The AF gate 111 outputs the digital signals of an area (within an AF frame) set by the camera microcomputer 114 to the focus signal processing unit 112 as described later.

[0018] The focus signal processor 112 generates a focus detection signal from the digital signal that has passed through the AF gate 111, and sends it to the camera microcomputer 114. Note that the focus detection signal may be a value that represents the sharpness (contrast state) of an image, or, if the digital signal is a pair of signals having parallax, may be a value that represents the distance to a subject or the amount of defocus calculated from the phase difference.

[0019] The first subject detection unit 113 performs a known subject detection process on the video signal of all pixels output from the camera signal processing unit 108, and detects the position, size, and angle (roll, pitch, yaw) of the subject in the screen. Note that various known methods can be used to detect a person's face by subject detection process. For example, a method of extracting a skin color area from the gradation color of each pixel represented by image data and detecting a face based on the degree of matching with a face outline plate prepared in advance, or a method of performing pattern recognition from extracted facial feature points such as eyes, nose, and mouth, etc. Also, not only a person's face but also pupils, torsos, and even subjects other than people such as animals and vehicles may be detected using parameters adjusted in advance by a method such as machine learning. The first subject detection unit 113 transmits the detection result to the camera microcomputer 114 and the second subject detection unit 117 described later.

[0020] The second subject detection unit 117 receives the detection result, represented by the detection position and size of the subject detected by the first subject detection unit 113, and performs subject detection processing by known subject tracking based on the received detection result and the video signal from the camera signal processing unit 108. Then, the obtained subject detection result is transmitted to the camera microcomputer 114. As the subject detection processing performed by the second subject detection unit 117, for example, a known method can be used in which an image of the subject area based on the detection result of a past frame is used as a template, pattern matching or histogram matching is performed on the current frame, and an area with a high degree of matching is determined to be the subject area. In addition to the template image, the subject area may be determined taking into account the defocus amount.

[0021] Camera microcomputer 114 determines the subject in the current frame based on the results received from first subject detection section 113 and second subject detection section 117. Here, a method by which camera microcomputer 114 determines the subject in the current frame from the results of first subject detection section 113 and second subject detection section 117 will be described.

[0022] First subject detection section 113 detects a subject from a video signal, and therefore has higher accuracy in determining the position and size of the subject compared to second subject detection section 117. On the other hand, detection accuracy tends to be lower when the subject faces left or right or up or down, when there is an obstacle, or when the shape of the subject changes significantly.

[0023] On the other hand, second subject detection unit 117 detects a subject by performing pattern matching or histogram matching based on a template image, and therefore can specify the position even when the subject's shape changes or there is an obstacle, compared to first subject detection unit 113. On the other hand, the accuracy of the subject's position and size is low, since it may be affected by the color and brightness of the surroundings of the subject.

[0024] As a result of the above, when the same subject is detected by first subject detection section 113 and second subject detection section 117, camera microcomputer 114 adopts the detection result of first subject detection section 113 if the reliability of the detection result by first subject detection section 113 is equal to or higher than a threshold. On the other hand, when the reliability of the detection result is less than the threshold, camera microcomputer 114 adopts the detection result by second subject detection section 117. However, the above-mentioned method of adopting the results is only one example, and the subject area may be identified based on the detection results of both first subject detection section 113 and second subject detection section 117 depending on the situation.

[0025] Then, the camera microcomputer 114 sets a main subject from among the subjects detected as described above using a method described below, sets an AF frame in an area including the subject area of ​​the main subject, and transmits the information to the AF gate 111. Then, the focus signal processing unit 112 performs focus adjustment control based on a focus detection signal generated from the digital signal that has passed through the AF gate 111. Also, exposure control may be performed based on the subject area set here.

[0026] Furthermore, camera microcomputer 114 controls focus lens driving source 116 to drive focus lens 105 based on the focus detection signal output from focus signal processing unit 112. Furthermore, camera microcomputer 114 outputs an image recording command to recording unit 110. Note that these processes are repeatedly executed by camera microcomputer 114 at predetermined intervals (for example, the generation period of a vertical synchronization signal of a video signal). In addition, camera microcomputer 114 receives information regarding the area of ​​the video being distributed by distribution device 12 from video distribution control unit 122, which will be described later.

[0027] The variable magnification lens driving source 115 includes an actuator and a driving circuit for moving the variable magnification lens 102, and the focus lens driving source 116 includes an actuator and a driving circuit for moving the focus lens 105. The actuators of the variable magnification lens driving source 115 and the focus lens driving source 116 are composed of a stepping motor, a DC motor, a vibration type motor, a voice coil motor, or the like.

[0028] The operation unit 118 is an operation member such as a touch panel, a joystick, a switch, or a button, and receives operations from a user and transmits the contents of the operations to the camera microcomputer 114 .

[0029] [Distribution device configuration] Next, the distribution device 12 in this embodiment will be described. As shown in FIG. 1, distribution device 12 includes video distribution device operation unit 121, video distribution control unit 122, video receiving unit 123, and distribution video transmitting unit .

[0030] Video distribution device operation unit 121 is a component that allows a user to operate distribution device 12, and can change the operation of distribution device 12. Video distribution device operation unit 121 may be a hardware component or a software component. Also, it may be a component shared with operation unit 118. In the case of sharing, video distribution control unit 122 receives operation information from camera microcomputer 114.

[0031] Video distribution control unit 122 changes the operation of distribution device 12 in response to an operation from video distribution device operation unit 121, and controls the video transmitted by distribution video transmission unit 124. Furthermore, video distribution control unit 122 transmits information to camera microcomputer 114 regarding which area of ​​the received video is being distributed (hereinafter referred to as "distribution information").

[0032] The video receiving unit 123 receives video from the camera signal processing unit 108 and passes it to the video distribution control unit 122 . Based on the distribution information set by the video distribution control unit 122, the distribution video transmission unit 124 receives from the video distribution control unit 122 a video of a partial area or the entire area of ​​the video received from the video receiving unit 123, and distributes it to the outside.

[0033] Issues with video distribution systems In the above-mentioned video distribution system, problems that may occur when processing is performed using the crop function and the function of adjusting focus and exposure for a detected subject will be described with reference to Figures 2 and 3. Note that the subject is assumed to be a human face. FIG. 2 is a diagram illustrating a video image captured by a video distribution system and a partial area of ​​the video image (hereinafter, referred to as a "crop area").

[0034] 2(a) shows an image of a full-angle region 200 captured by the imaging device 10, and includes human faces 210, 211, 212, and 213 as subjects. Regions 201, 202, 203, and 204 show cropped regions (partial regions) cut out from the full-angle region 200.

[0035] Fig. 2(b) shows an image of cropped region 201, Fig. 2(c) shows an image of cropped region 202, Fig. 2(d) shows an image of cropped region 203, and Fig. 2(e) shows an image of the aforementioned cropped region 204. Also, mark 220 in Fig. 2(d) indicates the center of cropped region 203.

[0036] When selecting a main subject from an image of the full angle of view region 200 shown in Fig. 2(a), there is a method for calculating a priority based on the distance of the detected subject from the center of the screen and the size of the subject, and then determining the main subject. As an example of a priority calculation method, the method shown in Fig. 3 will be described here.

[0037] In Figure 3(a), the horizontal axis indicates the distance to the center of the subject, with the center of the screen as the origin, and the vertical axis indicates the priority according to the distance. The priority remains constant until the distance of the subject from the center of the screen reaches Dmin, but decreases proportionally from Dmin to Dmax, and then becomes a constant value again when the distance exceeds Dmax. In other words, the closer the subject is to the center of the screen, the higher the priority is calculated to be so that it can become the main subject.

[0038] In Figure 3(b), the horizontal axis shows the size of the detected subject, and the vertical axis shows the priority according to the subject size. If the minimum detectable size is Smin, the priority increases proportionally from Smin to Smax, which is the maximum detectable size. In other words, the larger the subject, the higher the priority is calculated to be as the main subject.

[0039] The priority of each subject is determined by adding up the above-mentioned position and size priorities, and the subject having the highest priority is determined as the main subject.

[0040] However, the shape of the graph in Fig. 3 and Dmin, Dmax, Smin, and Smax are not limited to this, and a method of adding weights to the priorities of position and size may also be used. Furthermore, information other than position and size may also be taken into account.

[0041] 2(a), assuming that all faces are the same size, when the priority of the main subject (main face) is calculated based on the above method, it is clear that the priority of the person's face 212 will be the highest and will be the main face. Therefore, when the video distribution system distributes video of the full angle of view region 200 or video of the cropped region 203, the person's face 212, which is the main face, is present within the angle of view being distributed, and the focus and exposure are adjusted to the person's face 212, so there is no problem.

[0042] On the other hand, when the video distributed by the video distribution system is video in crop area 201 or crop area 202, the person's face 212, which is the main face, does not exist within the distributed angle of view, so a situation may occur in which the focus and exposure are not correct for any of the subjects within the angle of view. Also, when the video is in crop area 204, a situation may occur in which the focus and exposure are correct for the person's face 212, which is the main face and does not fit within the angle of view, and the focus and exposure are not correct for the faces 210 and 211 of the people that fit within the angle of view.

[0043] In the following embodiment, a main subject selection process for solving such a problem will be described.

[0044] <First embodiment> Hereinafter, a first embodiment of the present invention will be described. In the first embodiment, a case will be described in which the video distribution control unit 122 can notify the camera microcomputer 114 of distribution information.

[0045] 4 is a flowchart showing the main subject selection process in the first embodiment, which is executed by camera microcomputer 114. In the following description, the subject is a person's face. However, the subject in the present invention is not limited to a person's face, and may be a predetermined subject other than a person's face, such as a vehicle or an animal.

[0046] When the process starts, in S401, the positions and sizes of all faces present in the captured image are detected based on the detection results obtained by the first subject detection unit 113 and the second subject detection unit 117. At this time, in the example shown in Fig. 2, the image of the entire angle of view region 200 is displayed on the display unit 109, and if there is a face that was previously set as the main face among the detected human faces 210-213, at least a frame showing the main face is displayed.

[0047] Next, in S402, the video distribution control unit 122 acquires information on the cropping area to which the video is being distributed. Note that the video distribution control unit 122 transmits information on one or more cropping areas set by the user via the video distribution device operation unit 121 and information on the cropping area during video distribution to the camera microcomputer 114. At this time, a frame indicating at least the acquired cropping area during video distribution among the cropping areas 201-204 may be displayed on the display unit 109 by superimposing it on the video of the full angle of view area 200.

[0048] Next, in S403, it is determined whether the previously set main face is included among the faces detected in S401. If it is determined that the main face is included, the process proceeds to S405, and if it is determined that the main face is not included, the face designation flag is turned OFF in S404 and then the process proceeds to S405.

[0049] In S403, if it is determined that the previously set main face is not included among the faces detected in S401, the face designation flag is immediately turned OFF, but the face designation flag may be maintained in the ON state for a predetermined time after the main face is lost. For example, even if the main face is temporarily lost, the information of the main face may be maintained and the face designation flag may be maintained in the ON state for a predetermined time in the hope of finding it again. By not changing the focus position during this time, the focus state is prevented from switching frequently, and deterioration of the appearance of the moving image can be suppressed.

[0050] Furthermore, when the main face is lost, the above-mentioned predetermined time may be different between when the main face is lost within a specific range of the crop region (for example, within a horizontal / vertical 90% region of the crop region) and when the main face is lost outside the specific range of the crop region (for example, a region outside the horizontal / vertical 90% region of the crop region). In that case, for example, the predetermined time in the former case may be 5 seconds, whereas the predetermined time in the latter case may be 1 second.

[0051] Also, it may be possible to determine whether the user has cancelled the designation of the main face by operating the operation unit 118, and to control so as to turn off the face designation flag when the designation is cancelled.

[0052] Next, in S405, it is determined whether a face has been designated. Here, it is determined whether an operation to designate a specific face by the user has been accepted from operation unit 118 and / or whether the face designation flag is ON. If an operation to designate a face has been accepted from operation unit 118 and / or if the face designation flag is ON, it is determined that a face has been designated, and the process proceeds to S408. On the other hand, if an operation to designate a face has not been accepted from operation unit 118 and the face designation flag is OFF, the process proceeds to S406.

[0053] The user may specify a face by touching a touch panel on the display unit 109, or by specifying the face left, right, up, down, etc. using a cross key or the like. However, when selecting a face by specifying the face left, right, up, down, etc., if the camera microcomputer 114 accepts the operation, it is assumed that a face corresponding to the direction of the operation is specified. If there is no face to be specified in the direction of the operation, it is assumed that the face furthest away in the direction opposite to the direction of the operation is specified.

[0054] In S408, it is determined whether or not the face specified by the user is within the image of the crop area being distributed. If it is determined that the face specified by the user is within the image of the crop area being distributed, the process proceeds to S409, and if it is determined that the face is not within the image of the crop area being distributed, the process proceeds to S411.

[0055] In S411, it is determined whether the face designation method used by the user is a specific designation method. If it is the specific designation method, the process proceeds to S409, and if it is not the specific designation method, the process proceeds to S412.

[0056] The specific designation method refers to a state in which the user selects a face outside the crop area with a stronger intention than when selecting a face within the image of the crop area being distributed. For example, if the normal method of designating a face within the image of the crop area being distributed is a single click of the mouse of the operation unit 118, the specific designation method can be a double click or a long press operation. In addition, in the case of a touch panel, a difference can be a double tap or a long press operation compared to a single tap. Furthermore, dragging and dropping a specific mark or icon on the screen onto the face to be designated can also be used. In other words, any method can be used as long as it is clear that a face outside the image of the crop area being distributed was not designated by mistake but was designated on purpose.

[0057] In S409, the face designated by the user is set as the main face, and then in S410, the face designation flag is set to ON, and the process ends.

[0058] In S412, it is determined whether the face designation flag is ON. Here, when the face designation flag is ON, it means that the main face previously set by the user operation is in the image, and the face designated by the user in the normal designation method is not in the image of the crop area being distributed, or the user has not designated a face. In either case, the process is terminated without changing the main face being set, with the face designation flag kept ON. This can reduce the possibility that the main face will be changed to an unintended face due to an erroneous operation, for example, when the touch panel is touched unintentionally.

[0059] On the other hand, when the face designation flag is OFF in S412, it means that the main face previously set by the user's operation is not in the image, and the face designated by the user in the normal designation method is not in the image of the crop area being distributed. In this case, the process proceeds to S406, and a new main face is determined by the above-mentioned process.

[0060] In S406, a process of calculating the priority of each face is performed. Note that details of the process in S406 will be described later with reference to FIG. In the next step S407, the main face is determined based on the calculated priority of each face. Specifically, the face with the highest priority is determined as the main face.

[0061] Next, the process of calculating the priority level of each face in S406 will be described with reference to the flowchart in FIG. First, in S501, the center coordinates C of the crop area being distributed crop and get the size. Next, in S502, the center coordinate C crop A priority according to the distance from the image is calculated for all faces present in the image of the crop area being distributed.

[0062] By the above process, when the main face is not set by the user's operation, the main face can be determined from the faces in the crop area (partial area) being distributed according to the priority. Also, when the main face is set by the user's operation, the set main face can be maintained until the user specifies another main face or the set main face cannot be detected. Furthermore, when the main face set by the user's operation cannot be detected, the face specification flag is turned OFF to make it easier to change the target to be tracked, thereby increasing the probability of focusing on the subject in the crop area.

[0063] In S407, the main face is determined based on the priority calculated in S406, but when the face currently set as the main face is different from the newly determined main face, the previous main face may be maintained for a predetermined period of time instead of immediately switching the main face. This prevents the focus position from switching frequently, and suppresses deterioration of the appearance of the moving image.

[0064] Next, the above-mentioned processing will be specifically explained with reference to FIG. 2 by taking as an example a case where the main face is not specified by the user (the face specification flag is OFF) and the video distributed by the video distribution system is the video of the crop area 203.

[0065] First, the camera microcomputer 114 acquires information on the positions and sizes of the faces 210 to 213 of the subjects from the image of the full angle of view region 200 shown in FIG. 2(a) (S401). Next, the video distribution control unit 122 transmits position information and size information of the center (mark 220) of the crop area 203 to the camera microcomputer 114 (S402). The position information of the mark 220 obtained here is the center coordinate C cropと (S501).

[0066] Next, the camera microcomputer 114 calculates the center coordinate C cropBased on this, the priority of two faces, a person's face 212 and a person's face 213, that are present in the video of the cropping area 203 being distributed is calculated (S502). In the example of FIG. 2(d), the size of both faces is almost the same, but the person's face 212 is closer to the center of the cropping area 203 and has center coordinates C crop Since the face 212 is closer to the subject, it has a higher priority and is selected as the main face (S407).

[0067] By selecting the person's face 212 through the above-mentioned processing, it is possible to avoid a situation in which the subject in the distributed video is out of focus.

[0068] In the above description, in a similar scene in which the main face is a person's face 212 (face designation flag is ON), if the user operates the cross key to the left, the face 211 of the person to the left of the main face person's face 212 is outside the crop region 203. Therefore, in the case of a normal operation (NO in S411), the processing is terminated while maintaining the main face. However, the present invention is not limited to this, and in the processing in which, in response to the designation operation of the cross key described in S405, when there is no face to be designated in the direction of operation, the face farthest in the direction opposite to the direction of operation is designated, the face may be limited to within the crop region 203. In this case, the face 213 of the person to the right, which is within the crop region 203, is determined to be the designated face, so YES is obtained in S408 and the person's face 213 is set as the main face.

[0069] As described above, according to the first embodiment, in a video distribution system, it is possible to increase the likelihood that a subject on which focus and exposure are to be adjusted will be present in a video being distributed.

[0070] <Second embodiment> Next, a second embodiment of the present invention will be described. In the second embodiment, a case where the video distribution control unit 122 cannot notify the camera microcomputer 114 of distribution information will be described with reference to Figs. 6 to 8. However, it is assumed that the video distribution control unit 122 transmits information regarding the crop region designated by the user to the camera microcomputer 114 via the video distribution device operation unit 121. In Figs. 6 and 7, the same reference numbers are used for configurations and processes similar to those in Figs. 2 and 4, and descriptions thereof will be omitted as appropriate. The process shown in the flowchart of FIG.

[0071] In S401, when information on the positions and sizes of the faces of all people in the captured video is acquired, in S701, information on all cropping areas set by the video distribution control unit 122 is acquired, and the process proceeds to S403. As described above, in the second embodiment, information on the cropping area being distributed cannot be acquired.

[0072] Then, after the processes of S403 and S404, if it is determined in S405 that a face has been designated, the process proceeds to S703, and if it is determined that a face has not been designated, the process proceeds to S702.

[0073] In S703, it is determined whether the face specified by the user is within any of the crop areas acquired in S701. Note that in S703, since it is unclear which crop area the distribution device 12 is distributing, the camera microcomputer 114 determines only whether a face within the crop area is selected and then determines whether to change the main face.

[0074] If it is determined that the image is within any of the cropping regions, the process proceeds to S409, and if it is determined that the image is not within any of the cropping regions, the process proceeds to S411.

[0075] If it is determined in S411 that the face designation method used by the user is not a specific designation method, and if the face designation flag is OFF in S412, the process proceeds to S702, where a new main face is determined.

[0076] In S702, a process of calculating the priority of each face is performed, and the main face is determined based on the priority of each face calculated in S407.

[0077] Next, the process of calculating the priority level of each face in S702 will be described with reference to the flowchart in FIG.

[0078] First, in S801, the center coordinates and sizes of all the cropping areas that have been set are obtained. Then, the center coordinates of the obtained Nth cropping area are crop_N Let us assume that.

[0079] Next, in S802, the center coordinates and sizes of all the cropping areas obtained in S801 are used to calculate the priority of only the faces present in the cropping areas according to predetermined criteria.

[0080] The method of calculating the priority in S802 will be described below with reference to FIG. 6A shows an image of the full angle of view region 200 captured by the imaging device 10, and includes human faces 210, 211, 212, and 213 as subjects. Regions 201, 202, 204, and 601 are cropped regions that are cut out from the full angle of view region 200. Marks 620, 621, 622, and 623 in the cropped regions 201, 202, 204, and 601, respectively, indicate the center C of the angle of view of the cropped region. crop_N The coordinates of mark 620 are C crop_1 , the coordinates of mark 621 are C crop_2 , the coordinates of mark 622 are C crop_3 , the coordinates of mark 623 are C crop_4 Let us assume that.

[0081] 6(b) shows an image of cropped region 201, FIG. 6(c) shows an image of cropped region 202, FIG. 6(d) shows an image of cropped region 204, and FIG. 6(e) shows an image of cropped region 601.

[0082] [First method] The first method is based on the distance between each face in any of the crop regions and the center of the screen, with the shorter the distance being, the higher the priority is. 6, the face 212 of the person closest to the center of the screen has a higher priority among the faces in any of the cropped areas 201, 202, 204, and 601. As a result, the face 212 of the person is determined to be the main face in S407.

[0083] If crop region 601 is not set, then of the faces in crop regions 201, 202, and 204, the face 211 of the person that is closest to the center of the screen has the highest priority and is determined to be the main face. The faces 212 and 213 of the person that are not in any of crop regions 201, 202, and 204 cannot be the main face. Furthermore, if the face 211 does not exist, then the face 210 of the person has the highest priority and is determined to be the main face.

[0084] [Second Method] In the second method, a crop area C is cropped from the center of the full-angle area 200. crop Calculate the priority based on the distance to _N. Specifically, first, C crop_1 ~C crop_4 The center of gravity CM of the object is calculated, and the closer to the calculated center of gravity CM, the higher the priority is.

[0085] In addition, when calculating the center of gravity CM, by increasing the weight of the area where a narrower range is cropped, the priority of the face that exists in the area of ​​interest is increased, and it is easier to make it the main face. As an example, the inverse of the horizontal size of the cropped area is calculated by dividing the area by C from the center of the screen. crop_N 6, the cropped areas 204 and 601 are half the horizontal size of the full angle of view, so they are weighted twice as much. Also, the cropped areas 201 and 202 are 1 / 4 the horizontal size of the full angle of view, so they are weighted four times as much. In other words, if the central coordinate of the full angle of view area 200 is C, the center of gravity CM can be found by the following formula (1).

[0086] CM=(4(C-C crop_1 )+4(C-C crop_2 )+2(C-C crop_3 )+2(C-C crop_4 )) / (4+4+2+2) …(1)

[0087] By performing this weighted average, faces that are enlarged at a higher magnification are given a higher priority and are more likely to be selected as the main face, which is thought to increase the likelihood of reducing the sense of incongruity in the distributed video.

[0088] In the example shown in FIG. 6, among the faces in any of the cropping regions 201, 202, 204, and 601, the face of the person that is closest to the determined center of gravity CM is determined to be the main face.

[0089] [Third Method] The third method is a method in which, when a cropping area is specified along with the main face by the user, and the main face goes outside the cropping area (outside the partial area), the priority is determined based on the direction in which the main face goes.

[0090] 6(d), consider a case where the user has specified the person's face 211 in crop area 202 as the main face, and the person's face 211 frames out to the right of crop area 202 (toward the person's face 212). In this case, there is a high possibility that the person's face 211 is in a crop area to the right of the original crop area 202, or is heading toward the right crop area, so by selecting a main subject from within a crop area to the right of crop area 202, there is a high possibility that a natural-looking video can be delivered.

[0091] In this case, it can be achieved by setting the priority of the face in the direction in which the main face is out of the frame higher than the priority of the face in the opposite direction. Alternatively, in the second determination method described above, the weight of the cropped area in the direction in which the main face is out of the frame may be set higher than the weight of the cropped area in the opposite direction.

[0092] Even if the main face is framed out of the cropping area as described above, the method of calculating the priority may be changed depending on whether or not another face is present in the cropping area where the main face was originally located.

[0093] For example, consider a case where the face 212 of the person in the crop area 601 shown in Fig. 6(e) is the main face, and at a certain timing the face moves out of the crop area 601 to the left (toward the person's face 210). In this case, the priority of the face in the crop area where the main face was present is increased, and the priority of the face in the direction in which the main face started to move is increased. By setting the priority in this manner, the main face is not selected from the subjects in the crop areas 201 and 204, and even if a face is detected to the right of the person's face 212 in the same crop area 601, the person's face 211 to the left is selected as the main face.

[0094] This increases the likelihood of delivering the most natural image, given that although the area of ​​the image being delivered is unknown, it is highly likely that the area in which the main face was located will be delivered.

[0095] [Fourth Method] In the fourth method, the more cropping regions a face belongs to, the higher its priority is.

[0096] 6, for example, the priority of the person's face 211 is increased (e.g., tripled) because it belongs to three areas, the crop areas 202, 204, and 601. Also, the priority of the person's face 210 is increased, for example, by double because it belongs to two areas, the crop areas 201 and 204, and the priority of the person's face 212 is increased, for example, by single, because it belongs only to the crop area 601.

[0097] By performing the above process for all detected faces, for example, based on the priorities calculated using the first to third determination methods described above, the priority of faces that are included in more crop areas will be increased and they will be more likely to be selected as the main face. This increases the likelihood that the face will be included in the crop area being distributed, making it possible to increase the likelihood that natural video will be distributed.

[0098] Among the above-mentioned first and fourth methods, any determination method may be set in advance or may be selected by the user. Furthermore, the method of calculating the priority is not limited to the first and fourth methods, and any determination condition may be used as long as it can increase the possibility that the main face is present in the crop area being distributed.

[0099] <Other embodiments> The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0100] <Summary> The disclosure of this embodiment includes the following configuration.

[0101] (Item 1) a detection means for detecting a predetermined subject from an image obtained by an imaging means for repeatedly capturing and outputting an image; an acquisition means for acquiring information on at least one partial region set in the image acquired by the imaging means; a determining means for determining a main subject based on a priority of a subject located within the partial region among the subjects detected by the detecting means; 13. A video processing device comprising: (Item 2) The camera further includes an operation unit for designating a main subject among the subjects detected by the detection unit, 2. The image processing device according to item 1, wherein the determining means determines a main subject when a main subject is not specified by the operating means. (Item 3) 3. The image processing device according to item 2, further comprising a setting means for setting a subject as a main subject when the subject is designated by a predetermined operation by the operation means, regardless of whether the designated subject is within the partial area or not. (Item 4) 4. The image processing device according to item 3, characterized in that, when the designated main subject is not detected by the detection means, the setting means does not change the designated main subject for a predetermined time from when the designated main subject is no longer detected. (Item 5) The acquiring means acquires information on a partial region that is distributed to an external device, 5. The image processing device according to any one of items 1 to 4, wherein the determining means determines a main subject from subjects present within the partial area that is distributed to the outside. (Item 6) 6. The image processing device according to item 5, wherein the determining means sets a higher priority to a subject in the partial area being distributed externally, the closer the subject is to the center of the partial area. (Item 7) 7. The image processing device according to item 5 or 6, wherein the determining means sets a higher priority to an object in the partial area being distributed to the outside as the size of the object increases. (Item 8) 5. The image processing device according to any one of items 1 to 4, characterized in that the determining means sets a higher priority to an object in the partial region, the closer the object is to the center of the image captured by the imaging means. (Item 9) 5. The image processing device according to any one of items 1 to 4, wherein the determining unit sets a higher priority for an object in the partial region, the closer the object is to a center of gravity of the partial region. (Item 10) 10. The image processing device according to item 9, wherein the determining means, when determining the center of gravity of the center of the partial region, assigns a greater weight to a smaller partial region. (Item 11) 5. The image processing device according to any one of items 1 to 4, characterized in that, when the determined main subject moves outside the partial region, the determination means sets a higher priority to a subject in a partial region in a direction of movement of the main subject, and sets a higher priority to a subject in the partial region to which the main subject belonged before the main subject moved. (Item 12) 12. The image processing device according to any one of items 8 to 11, wherein the determining means assigns a higher priority to a subject that is included in a larger number of the partial regions. (Item 13) 13. The image processing device according to any one of items 7 to 12, wherein the determining unit sets a higher priority for an object in the partial region as the size of the object increases. (Item 14) A video processing device according to any one of items 1 to 13, An image sensor that repeatedly captures and outputs images An imaging device comprising: (Item 15) 15. The imaging apparatus according to item 14, further comprising a focus adjustment control means for adjusting the focus on the main subject. (Item 16) 16. The imaging apparatus according to item 14 or 15, further comprising an exposure control means for adjusting the exposure to the main subject determined by the determination means. (Item 17) A setting means for setting at least one partial region for an image captured by the imaging device according to any one of items 14 to 16; a distribution means for selecting and distributing either the image or the image of the partial region; A video distribution system comprising: (Item 18) a detection step in which a detection means detects a predetermined subject from an image obtained by an imaging means that repeatedly captures and outputs an image; an acquisition step in which an acquisition means acquires information of at least one partial region set in the image acquired from the imaging means; a determining step of determining a main subject based on a priority of a subject within the partial region among the subjects detected in the detecting step; A video processing method comprising: (Item 19) A program for causing a computer to function as each of the means of the video processing device according to any one of items 1 to 13. (Item 20) 20. A computer-readable storage medium storing the program according to item 19.

[0102] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0103] 10...imaging device, 102...variable magnification lens, 103...diaphragm, 105...focus lens, 106...imaging element, 111...AF gate, 112...focus signal processing circuit, 113...first subject detection unit, 114...camera microcomputer, 117...second subject detection unit, 118...operation unit, 12...distribution device, 121...video distribution device operation unit, 122...video distribution control unit, 123...video receiving unit, 124...distribution video transmission unit

Claims

1. a detection means for detecting a predetermined subject from an image obtained by an imaging means for repeatedly capturing and outputting an image; an acquisition means for acquiring information on at least one partial region set in the image acquired from the imaging means; a determining means for determining a main subject based on a priority of a subject located within the partial region among the subjects detected by the detecting means; 13. A video processing device comprising:

2. The camera further includes an operation unit for designating a main subject among the subjects detected by the detection unit, 2. The image processing device according to claim 1, wherein the determining means determines the main subject when the main subject is not specified by the operating means.

3. 3. The image processing device according to claim 2, further comprising a setting means for setting, when a subject is designated by a predetermined operation by the operating means, the designated subject as a main subject regardless of whether the designated subject is within the partial area or not.

4. 4. The image processing device according to claim 3, wherein, when the designated main subject is not detected by the detection means, the setting means does not change the designated main subject for a predetermined time from when the designated main subject is no longer detected.

5. The acquiring means acquires information on a partial region that is distributed to an external device, The image processing device according to claim 1 , wherein the determining unit determines a main subject from subjects present within the partial area that is distributed to the outside.

6. The image processing device according to claim 5 , wherein the determining unit sets a higher priority for an object in the partial area distributed to the outside, the closer the object is to a center of the partial area.

7. The image processing device according to claim 5 , wherein the determining unit sets a higher priority for an object in the partial area distributed to the outside as the object becomes larger in size.

8. The image processing device according to claim 1 , wherein the determining unit sets a higher priority for the object in the partial region, the closer the object is to the center of the image captured by the imaging unit.

9. The image processing device according to claim 1 , wherein the determining unit sets a higher priority for the object in the partial region, the closer the object is to a center of gravity of the partial region.

10. The image processing device according to claim 9 , wherein the determining means weights smaller partial regions more heavily when determining the center of gravity of the partial regions.

11. 2. The image processing device according to claim 1, wherein, when the determined main subject moves outside the partial region, the determining means sets a higher priority to a subject in a partial region in the direction of movement of the main subject, and sets a higher priority to a subject in the partial region to which the main subject belonged before the main subject moved.

12. The image processing device according to claim 8 , wherein the determining unit assigns a higher priority to a subject that is included in a larger number of the partial regions.

13. The image processing device according to claim 7 , wherein the determining unit sets a higher priority for an object in the partial region as the size of the object increases.

14. A video processing device according to any one of claims 1 to 13, An image sensor that repeatedly captures and outputs images An imaging device comprising:

15. 15. The imaging apparatus according to claim 14, further comprising a focus adjustment control means for adjusting the focus on the main subject.

16. 15. The imaging apparatus according to claim 14, further comprising an exposure control unit for adjusting exposure to the main subject determined by the determination unit.

17. A setting unit that sets at least one partial region for an image captured by the imaging device according to claim 14; a distribution means for selecting and distributing either the image or the image of the partial region; A video distribution system comprising:

18. a detection step in which a detection means detects a predetermined subject from an image obtained by an imaging means that repeatedly captures and outputs an image; an acquisition step in which an acquisition means acquires information of at least one partial region set in the image acquired from the imaging means; a determining step of determining a main subject based on a priority of a subject within the partial region among the subjects detected in the detecting step; A video processing method comprising:

19. A program for causing a computer to function as each of the means of the video processing device according to any one of claims 1 to 13.

20. A computer-readable storage medium storing the program according to claim 19.

Images