Video processing apparatus, video processing method, imaging apparatus, video distribution system, storage medium, and computer program product

Abstract

The invention provides a video processing apparatus, a video processing method, an imaging apparatus, a video distribution system, a storage medium, and a computer program product. The video processing apparatus includes: a detection section configured to detect one or more predetermined subjects from a video obtained by an imaging section for repeatedly capturing and outputting the video; an acquisition section configured to acquire information related to at least one partial region set in the video obtained by the imaging section; and a determination section configured to determine a main subject based on a priority of each subject within the partial region from among the one or more subjects detected by the detection section.

Description

Technical Field

[0001] The invention relates to a video processing device, a video processing method, a camera device, a video distribution system and a storage medium. Background Art

[0002] In a video distribution system for distributing video to a remote location, it is generally known to use a cropping function for distributing an image based on signals read out from pixels in a partial area rather than from all effective pixels among a plurality of pixels constituting an image sensor. This cropping function enables distribution of a video captured by a camera excluding an area not required by a user.

[0003] On the other hand, there are many cameras that can be used in the above-mentioned video distribution system, and these cameras are equipped with the following functions: detect the face of a person from the shot video, and adjust the focus and exposure of the detected face of the person (subject). When this function is used in a video distribution system equipped with a cropping function, it is important that the subject whose focus and exposure are adjusted exists in the video being distributed.

[0004] Japanese Patent Laid-Open No. 2006-33291 proposes a method in which, by recording a video of an area corresponding to a subject determined based on a result of a measuring operation such as photometry among subjects detected from the entire range of a video, the subject for which focus and exposure are adjusted is included in the video recording area.

[0005] However, in the case of the method described in Japanese Patent Laid-Open No. 2006-33291, if the area of ​​the video to be recorded or allocated is determined based on the area corresponding to the subject determined based on the result of a measurement operation such as photometry, there is a possibility that the obtained video will be different from the video the user wishes to allocate. Summary of the invention

[0006] The present invention has been made in consideration of the above circumstances, and allocates a region of a video that a user wishes to allocate while increasing the possibility that a subject whose focus and exposure are adjusted will be included in the allocated video.

[0007] According to the present invention, there is provided a video processing device, comprising: a detection component configured to detect a predetermined one or more subjects from a video obtained by a camera component for repeatedly shooting and outputting a video; an acquisition component configured to acquire information related to at least one partial area set in the video obtained by the camera component; and a determination component configured to determine a main subject from among the one or more subjects detected by the detection component based on the priority of each subject within the partial area.

[0008] In addition, according to the present invention, a camera device is provided, which includes: a video processing device, which includes: a detection component, which is configured to detect a predetermined one or more subjects from a video obtained by a camera component for repeatedly shooting and outputting videos, an acquisition component, which is configured to acquire information related to at least one partial area set in the video obtained by the camera component, and a determination component, which is configured to determine a main subject from the one or more subjects detected by the detection component based on the priority of each subject in the partial area; and the camera component is used to repeatedly shoot and output videos.

[0009] In addition, according to the present invention, a video distribution system is provided, which includes: a setting component, which is configured to set at least one partial area in the video captured by the camera device; and an allocation component, which is configured to select and allocate any video among the video and the video of the partial area, wherein the camera device includes: a video processing device, which includes: a detection component, which is configured to detect a predetermined one or more subjects from a video obtained by a camera component for repeatedly capturing and outputting videos, an acquisition component, which is configured to acquire information related to at least one partial area set in the video obtained by the camera component, and a determination component, which is configured to determine a main subject from among the one or more subjects detected by the detection component based on the priority of each subject in the partial area; and the camera component is used to repeatedly capture and output videos.

[0010] In addition, according to the present invention, a video processing method is provided, which includes: a detection step for detecting a predetermined one or more subjects from a video obtained by a camera component for repeatedly shooting and outputting videos; an acquisition step for acquiring information related to at least one partial area set in the video obtained by the camera component; and a determination step for determining a main subject from among the one or more subjects detected in the detection step based on the priority of each subject in the partial area.

[0011] In addition, according to the present invention, a computer-readable storage medium is provided, which stores a program for causing a computer to be used as various components of a video processing device, wherein the video processing device includes: a detection component, which is configured to detect a predetermined one or more subjects from a video obtained by a camera component for repeatedly shooting and outputting videos; an acquisition component, which is configured to acquire information related to at least one partial area set in the video obtained by the camera component; and a determination component, which is configured to determine a main subject from among the one or more subjects detected by the detection component based on the priority of each subject within the partial area.

[0012] Further features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the attached drawings). BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0014] Figure 1 is a block diagram showing a configuration of a video distribution system according to an embodiment of the present invention.

[0015] FIG. 2A to FIG. 2E is a diagram for explaining a problem in a video distribution system and a method for setting a main face according to the first embodiment.

[0016] Figure 3A and Figure 3B is a graph for explaining a method of calculating priorities in main face selection.

[0017] Figure 4 is a flowchart of the main face selection process in the first embodiment.

[0018] Figure 5 4 is a flowchart of the priority calculation process for each face in the first embodiment.

[0019] FIG. 6A to FIG. 6E is a diagram for explaining a method for setting a main face according to the second embodiment.

[0020] Figure 7 is a flowchart of the main face selection process in the second embodiment.

[0021] Figure 8 4 is a flowchart of the priority calculation process for each face in the second embodiment. DETAILED DESCRIPTION

[0022] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the invention claimed, and are not limited to the invention of the combination of all the features described in the embodiments. Two or more of the multiple features described in the embodiments may be appropriately combined. In addition, the same reference numerals are given to the same or similar configurations, and their redundant descriptions are omitted.

[0023] Configuration of video distribution system

[0024] Figure 11 is a block diagram showing the configuration of a video distribution system for distributing videos according to an embodiment of the present invention. The video distribution system includes a camera 10 and a distribution device 12. In this embodiment, a case where the distribution device 12 is a device (such as a PC, etc.) configured separately from a camera will be described, but the camera 10 and the distribution device 12 may be configured integrally.

[0025] [Configuration of the camera device]

[0026] In the present embodiment, a case where the image pickup device 10 is a video camera will be described. Note that the present invention is not limited to a video camera, but can also be applied to other image pickup devices such as a digital still camera.

[0027] exist Figure 1 , the imaging optical system includes a first fixed lens 101, a variable magnification lens 102, an aperture 103, a second fixed lens 104, and a focus lens 105. The variable magnification lens 102 is a variable magnification lens whose magnification can be changed by moving in the optical axis direction. The focus lens 105 has both a general focus adjustment function and a function of correcting a shift of a focal plane due to zooming, and adjusts the focus state of light incident on a light receiving surface of an image sensor 106 through the imaging optical system.

[0028] The image sensor 106 is formed of a CCD sensor or a CMOS sensor, etc., and outputs an analog signal by photoelectric conversion of an optical image formed on a light receiving surface via an imaging optical system. A CDS / AGC / AD converter (CDS AGC) 107 performs sampling, gain adjustment, and analog-to-digital conversion processing on the analog signal output from the image sensor 106, and outputs the obtained digital signal. A camera signal processing unit 108 performs various image processing on the digital signal output from the CDS / AGC / AD converter 107 to generate a video signal.

[0029] The display unit 109 displays the video signal from the camera signal processing unit 108. The recording unit 110 records the video signal from the camera signal processing unit 108 on a recording medium such as a magnetic tape, an optical disk, or a semiconductor memory. In addition, the video signal generated by the camera signal processing unit 108 is output to the video receiving unit 123 of the distribution device 12.

[0030] The AF gate 111 is for passing only the digital signal corresponding to the area for focus detection among the digital signals of all pixels output from the CDS / AGC / AD converter 107. Note that the AF gate 111 outputs the digital signal corresponding to the area (within the AF frame) set by the camera microcomputer 114 as described later to the focus signal processing unit 112.

[0031] The focus signal processing unit 112 generates a focus detection signal from the digital signal that has passed through the AF gate 111, and sends it to the camera microcomputer 114. Note that the focus detection signal may be a value indicating the sharpness (contrast state) of the video, or in the case where the digital signal is a pair of signals having parallax, the focus detection signal may be a distance to the subject determined from a phase difference or a value indicating a defocus amount.

[0032] The first subject detection unit 113 performs a known subject detection process on the video signals of all pixels output from the camera signal processing unit 108, and detects the position, size, and angle (roll, pitch, and yaw) of the subject in the screen. Note that in the subject detection process, various known methods can be used to detect the face of a person. For example, there is a method of extracting a skin color area based on the hue of each pixel represented by the image data, and detecting a face based on the degree of matching with a face contour template of a person prepared in advance, and a method of performing pattern recognition using detected facial feature points (such as eyes, nose, and mouth). In addition, by using methods such as machine learning, by using pre-adjusted parameters, not only the face of a person but also the eyes, torso, and subjects other than a person (such as animals and vehicles) can be detected. The first subject detection unit 113 transmits the detection result to the camera microcomputer 114 and the second subject detection unit 117, which will be described later.

[0033] The second subject detection unit 117 receives the detection result represented by the detection position and size of the subject detected by the first subject detection unit 113, and performs known subject detection processing accompanied by subject tracking based on the received detection result and the video signal from the camera signal processing unit 108. Then, the obtained subject detection result is transmitted to the camera microcomputer 114. For example, the subject detection processing performed by the second subject detection unit 117 includes a known method of performing pattern matching or histogram matching on the current frame by using an image of a subject area based on the detection result of a past frame as a template, and determining an area with a high degree of matching as a subject area. In addition, the subject area can be determined by considering not only the template image but also the defocus amount.

[0034] The camera microcomputer 114 determines the subject in the current frame based on the results received from the first and second subject detection units 113 and 117. Here, a method in which the camera microcomputer 114 determines the subject in the current frame according to the determination results of the first and second subject detection units 113 and 117 will be described.

[0035] Since the first subject detection unit 113 detects the subject using the video signal, the accuracy regarding the position and size of the subject is higher than that of the second subject detection unit 117. On the other hand, in the case where the shape of the subject changes significantly (such as when the subject faces left, right, up, or down, or when there is an obstacle, etc.), the detection accuracy tends to decrease.

[0036] On the other hand, the second subject detection unit 117 performs pattern matching and histogram matching based on the template image to detect the subject, and thus can determine the position of the subject even when the shape of the subject changes or there is an obstacle, compared to the first subject detection unit 113. On the other hand, since it may be affected by the color and brightness of the surrounding environment of the subject, the accuracy of the position and size of the subject is low.

[0037] In consideration of the above, in a case where the first subject detection unit 113 and the second subject detection unit 117 detect the same subject, if the reliability of the detection result of the first subject detection unit 113 is equal to or greater than the threshold, the detection result of the first subject detection unit 113 is adopted. On the other hand, if the reliability of the detection result is less than the threshold, the detection result of the second subject detection unit 117 is adopted. However, the above method of selecting the result is only an example, and the subject area may be specified based on the detection results of both the first subject detection unit 113 and the second subject detection unit 117 according to the situation.

[0038] Then, the camera microcomputer 114 sets the main subject among the subjects detected as described above using a method described later, sets an AF frame in an area including the subject area of ​​the main subject, and transmits the information to the AF gate 111. Then, the focus signal processing unit 112 performs focus adjustment control based on a focus detection signal generated using the digital signal that has passed through the AF gate 111. Furthermore, exposure can be controlled based on the subject area set here.

[0039] Furthermore, the camera microcomputer 114 controls the focus lens actuation source 116 to actuate the focus lens 105 based on the focus detection signal output from the focus signal processing unit 112. Furthermore, the camera microcomputer 114 outputs an image recording command to the recording unit 110. Note that the camera microcomputer 114 repeatedly executes these processes at predetermined intervals (for example, a generation cycle of a vertical synchronization signal of a video signal).

[0040] Furthermore, the camera microcomputer 114 receives information about the area to which the video is being distributed by the distribution device 12 from a video distribution control unit 122 to be described later.

[0041] The variable magnification lens actuation source 115 includes an actuator and a driver circuit thereof for moving the variable magnification lens 102, and the focus lens actuation source 116 includes an actuator and a driver circuit thereof for moving the focus lens 105. In addition, the actuators of the variable magnification lens actuation source 115 and the focus lens actuation source 116 each include a stepping motor, a DC motor, a vibration type motor, a voice coil motor, and the like.

[0042] The operation unit 118 includes operation members such as a touch panel, a lever, a switch, buttons, and the like, and receives a user operation and transmits the operation content to the camera microcomputer 114 .

[0043] [Configuration of distribution device]

[0044] Next, the dispensing device 12 in this embodiment will be described.

[0045] like Figure 1 As shown, the distribution device 12 includes a video distribution device operating unit 121 , a video distribution control unit 122 , a video receiving unit 123 , and a distribution video sending unit 124 .

[0046] The video distribution device operation unit 121 is a member for the user to operate the distribution device 12, and the operation of the distribution device 12 can be changed by using the video distribution device operation unit 121. The video distribution device operation unit 121 may be a hardware member or a software member. In addition, the video distribution device operation unit 121 may be a member shared with the operation unit 118, and in this case, the video distribution control unit 122 receives operation information from the camera microcomputer 114.

[0047] The video distribution control unit 122 changes the operation of the distribution device 12 according to the operation from the video distribution device operation unit 121, and controls the distribution of the video to be transmitted by the video transmission unit 124. In addition, the video distribution control unit 122 also transmits information about which area of ​​the received video is being distributed (hereinafter referred to as "distribution information") to the camera microcomputer 114.

[0048] The video receiving unit 123 receives the video from the camera signal processing unit 108 and passes it to the video distribution control unit 122 .

[0049] The distribution video transmission unit 124 receives a part or all of the video received from the video reception unit 123 from the video distribution control unit 122 based on the distribution information set by the video distribution control unit 122, and distributes it to the outside.

[0050] ·Problems in video distribution systems

[0051] Will use FIG. 2A to FIG. 2E as well as Figure 3A and Figure 3B The following is an explanation of the problems that may occur in the above video distribution system when using the cropping function and the function of adjusting the focus and exposure of the detected subject. Note that the description here assumes that the subject is a person's face.

[0052] FIG. 2A to FIG. 2E is a diagram illustrating a video being captured by a video distribution system and a partial area of ​​the video (hereinafter referred to as a "cropped area").

[0053] Figure 2A An image captured by the camera 10 is shown, which covers a full-angle area 200 and includes human faces 210 , 211 , 212 , and 213 as subjects. In addition, areas 201 , 202 , 203 , and 204 indicate cropped areas (partial areas) partially cut out from the full-angle area 200 .

[0054] Figure 2B A video showing a cropped region 201, Figure 2C A video showing a cropped region 202, Figure 2D The video of the cropped area 203 is shown, and Figure 2E The video of the cropped region 204 is shown. In addition, Figure 2D The mark 220 in represents the center of the cropping area 203 .

[0055] In order to Figure 2A In the video of the full-angle area 200 shown in the figure, there is a method of determining the main subject by calculating the priority based on the distance between each detected subject and the center of the screen and the size of the subject. Here, as an example, Figure 3A and Figure 3B The method shown is described as a priority calculation method.

[0056] exist Figure 3A In the figure, the horizontal axis represents the distance from the center of the screen to the center of the subject, and the vertical axis represents the priority according to the distance. The priority is constant until the distance from the center of the screen to the subject reaches Dmin, decreases proportionally from Dmin to Dmax, and then becomes constant again when the distance exceeds Dmax. In other words, the closer the subject is to the center of the screen, the higher the calculated priority is, so that the subject can be determined as the main subject.

[0057] exist Figure 3BIn the figure, the horizontal axis represents the detected subject size, and the vertical axis represents the priority according to the subject size. Assuming the minimum detectable size is Smin, the priority increases proportionally from Smin to Smax (i.e., the maximum detectable size). In other words, the larger the subject that appears in the video, the higher the calculated priority, so that the subject can be determined as the main subject.

[0058] Then, a value obtained by adding the above-mentioned priorities corresponding to the distance and the size is determined as the priority of each subject, and the subject with the highest priority is determined as the main subject.

[0059] However, Figure 3A and Figure 3B The shape of the graph in and Dmin, Dmax, Smin, and Smax are not limited thereto, and a method of adding weights to the priorities of distance and size, respectively, may be used. In addition, information other than distance and size may be considered.

[0060] exist Figure 2A In the video shown, assuming that all faces have the same size, when the priority of the main subject (main face) is calculated based on the above method, it will be understood that the human face 212 has the highest priority and will be determined as the main face. Therefore, in the case where the video distribution system is distributing the video of the full-angle area 200 or the video of the cropped area 203, the human face 212 as the main face exists within the perspective of the video being distributed, and there will be no problem because the focus and exposure are adjusted for the human face 212.

[0061] On the other hand, if the video distributed by the video distribution system is a video of the crop area 201 or the crop area 202, the human face 212 as the main face does not exist within the angle of view of the video being distributed, so the following situation may occur: the focus or exposure is not appropriately adjusted for any subject within the angle of view of the video being distributed. In addition, in the case of a video to which the crop area 204 is distributed, the following situation may occur: the focus and exposure are adjusted for the human face 212 as the main face that partially exists outside the angle of view of the crop area 204, while the focus and exposure are not adjusted for the human faces 210 and 211 that exist within the angle of view of the crop area 204.

[0062] In the following embodiments, main subject selection processing for solving these problems will be described.

[0063] <First Embodiment>

[0064] A first embodiment of the present invention will be described below. Note that in the first embodiment, a case will be described where the video distribution control unit 122 can notify the camera microcomputer 114 of distribution information.

[0065] Figure 4 1 is a flowchart showing the main subject selection processing in the first embodiment, which is performed by the camera microcomputer 114. Note that in the following description, the case where the subject is a human face will be described. However, the subject in the present invention is not limited to a human face, and may be a predetermined subject other than a human face (such as a vehicle or an animal, etc.).

[0066] When the process starts, in step S401, based on the detection results obtained by the first subject detection unit 113 and the second subject detection unit 117, the positions and sizes of all faces present in the captured video are determined. FIG. 2A to FIG. 2E In the example shown, the display unit 109 displays the video of the full-angle area 200, and if any of the detected human faces 210 to 213 has been determined as a main face last time, at least a frame showing the main face is displayed.

[0067] Next, in step S402, the camera microcomputer 114 acquires information about the cropping area to which the video is being allocated by the video allocation control unit 122. Note that the video allocation control unit 122 transmits information about one or more cropping areas set by the user through the video allocation device operation unit 121, and information about the cropping area of ​​the video currently being allocated, to the camera microcomputer 114. At this time, a frame indicating the cropping area of ​​the video currently being allocated among the cropping areas 201 to 204 may be displayed on the display unit 109 in a manner superimposed on the video of the full-angle area 200.

[0068] Next, in step S403, it is determined whether the faces detected in step S401 include the previously set main face. If it is determined that the main face is included, the process proceeds to step S405; if it is determined that the main face is not included, the face designation flag is turned OFF in step S404, and the process proceeds to step S405.

[0069] Note that in step S403, if it is determined that the faces detected in step S401 do not include the previously set main face, the face designation flag is immediately turned off; however, after the main face is lost, the face designation flag may be kept ON for a predetermined period of time. For example, even if the main face is temporarily lost, it is expected that the main face will be found again, and the information related to the main face is maintained and the face designation flag is kept ON for a predetermined period of time. By not changing the focus position during this period of time, it is possible to prevent the focus state from changing frequently and prevent the appearance of the moving image from being deteriorated.

[0070] In addition, in the case where the main face is lost, the above-mentioned predetermined time period may be changed according to the following situations: the case where the main face is lost within a specific range of the cropped area (for example, within 90% of the cropped area in the horizontal and vertical directions) and the case where the main face is lost outside a specific range of the cropped area (for example, outside 90% of the cropped area in the horizontal and vertical directions). In this case, for example, the predetermined time period in the former case may be set to 5 seconds, while the predetermined time period in the latter case may be set to 1 second.

[0071] Alternatively, it may be determined whether the user operates the operation unit 118 to release the designation of the main face, and if the designation is released, the face designation flag may be turned off.

[0072] Next, in step S405, it is determined whether a face is specified. Here, it is determined whether the user receives an operation for specifying a specific face from the operation unit 118 and / or whether the face specification flag is ON. If an operation for specifying a face is received from the operation unit 118 and / or if the face specification flag is ON, it is determined that a face is specified, and the process proceeds to step S408. On the other hand, if an operation for specifying a face is not received from the operation unit 118 and the face specification flag is OFF, the process proceeds to step S406.

[0073] Note that the user's designation may be, for example, a touch operation on the touch panel of the display unit 109, or a left / right / up / down designation operation using a cross key or the like. It should be noted that, regarding face selection by the left / right / up / down designation operation, when the camera microcomputer 114 accepts the operation, it is assumed that the face corresponding to the operation direction is designated. Furthermore, if there is no face to be designated in the operation direction, it is assumed that the farthest face in the direction opposite to the operation direction is designated.

[0074] In step S408, it is determined whether the face designated by the user is within the video of the cropping area being assigned. If it is determined that the face designated by the user is within the video of the cropping area being assigned, the process proceeds to step S409, and if it is determined that the face designated by the user is not within the video of the cropping area being assigned, the process proceeds to step S411.

[0075] In step S411, it is determined whether the method for the user to specify a face is a specific specifying method. If it is a specific specifying method, the process proceeds to step S409, and if it is not a specific specifying method, the process proceeds to step S412.

[0076] The specific designation method refers to a state in which the user's intention to select a face outside the area is stronger than a state in which a face in the video of the crop area being assigned is selected. As an example, if the usual method for designating a face in the video of the crop area being assigned is to click the mouse of the operation unit 118, the specific designation method includes double-clicking or long pressing, etc. In addition, if a touch panel or the like is used, a single tap can be used for the usual method, while a double tap or long pressing operation can be used for the specific designation method. Alternatively, a specific mark or icon on the screen can be dragged and dropped onto the desired face. In other words, any method can be used as long as it is obvious that the following operation is performed: the face outside the video of the crop area being assigned is not mistakenly designated, but is intentionally designated.

[0077] In step S409, the face designated by the user is set as the main face, and then in step S410, the face designation flag is turned on, and the processing ends.

[0078] In step S412, it is determined whether the face designation flag is ON. Here, the state where the face designation flag is ON indicates that the main face previously set by the user operation is in the video, the face designated by the user using the usual designation method is not in the video of the cropping area being assigned, or the user has not designated a face. In either state, the process ends with the face designation flag remaining in the ON state without changing the main face being set. This can reduce the possibility that the main face will be changed to an unexpected face due to a misoperation, for example, in the case where the user accidentally touches the touch panel.

[0079] On the other hand, the state in which the face designation flag is OFF in step S412 indicates that the main face previously set by the user operation is not in the video, and the face designated by the user using the usual designation method is not in the video of the cropping area being assigned. In this case, the process proceeds to step S406, and a new main face is determined by the above process.

[0080] In step S406, the priority of each face is calculated. Figure 5 The details of the processing in step S406 are described.

[0081] In the next step S407, the main face is determined and set based on the calculated priorities of the faces. Specifically, the face with the highest priority is determined as the main face.

[0082] Next, we will use Figure 5 The flowchart illustrates the priority calculation process for each face performed in step S406.

[0083] First, in step S501, the center coordinates C of the cropping area to which the video is being assigned are obtained.crop and size.

[0084] Next, in step S502, for all faces existing in the video of the cropping area being assigned, the face coordinates corresponding to the center coordinates C crop The priority of the distance.

[0085] Through the above processing, if the main face is not set by user operation, the main face can be determined from the faces within the cropping area (partial area) to which the video is being assigned according to the priority. In addition, if the main face is set by user operation, the currently set main face can be maintained until the user specifies a different main face or the set main face can no longer be detected. In addition, if the main face set by user operation cannot be detected, the face designation flag is turned off to make it easier to change the subject to be tracked, thereby increasing the probability that the subject within the cropping area will be focused.

[0086] Note that in step S407, the main face is determined based on the priority calculated in step S406, however, if the face currently set as the main face is different from the newly determined main face, the previous main face may be maintained for a predetermined period of time instead of switching the main face immediately. Thus, it is possible to prevent the focus position from changing frequently and suppress the deterioration of the appearance of the moving image.

[0087] Next, the case where the user has not designated the main face (the face designation flag is OFF) in the above process and the video being distributed by the video distribution system is the video of the cropped area 203 is taken as an example, and the FIG. 2A to FIG. 2E Describe the situation in detail.

[0088] First, the camera microcomputer 114 receives Figure 2A Information on the positions and sizes of faces 210 to 213 of people as subjects is acquired from the video of the full-angle area 200 shown (step S401).

[0089] Next, the video distribution control unit 122 transmits the position information and size information of the center (marker 220) of the cropping area 203 to the camera microcomputer 114 (step S402). The position information of the mark 220 obtained here is the center coordinate C in step S501. crop .

[0090] Next, the camera microcomputer 114 calculates the value of the center coordinate C based on the center coordinate C. crop The priorities of the two faces (the human face 212 and the human face 213) existing in the video of the cropping area 203 being assigned are calculated (step S502). Figure 2DIn the example of FIG. 1 , the sizes of the two faces are almost the same, but the face 212 of the person is closer to the center of the cropping area 203 and is closer to the center coordinate C crop , the distance between the human face 212 and the human face 213 is shorter, so the priority of the human face 212 is higher than the priority of the human face 213. As a result, the human face 212 is selected as the main face (step S407).

[0091] By selecting the human face 212 through the above-described process, a situation in which no subject is focused in the distributed video can be avoided.

[0092] In the above description, in a case where the main face is the human face 212 in a similar scene (the face designation flag is ON), if the user presses the left key of the cross key, the human face 211 on the left side of the main face of the human face 212 is located outside the cropping area 203. Therefore, in the case of a normal operation ("No" in step S411), it is described that the processing is ended while maintaining the main face. However, the present invention is not limited to this, and the following processing described in step S405 can be used while limiting the range to the cropping area 203: in the case where there is no face to be designated in the operation direction, the farthest face in the direction opposite to the operation direction is designated. In this case, since the human face 213 on the right side in the cropping area 203 is determined to be the designated face, it is judged as "Yes" in step S408, and the human face 213 is set as the main face.

[0093] As described above, according to the first embodiment, in the video distribution system, the possibility that a subject whose focus and exposure are adjusted exists in the video being distributed can be increased.

[0094] <Second Embodiment>

[0095] Next, a second embodiment of the present invention will be described. In the second embodiment, FIG. 6A to FIG. 8 A case is described in which the video distribution control unit 122 cannot notify the camera microcomputer 114 of distribution information. It should be noted that it is assumed that the video distribution control unit 122 transmits information about the cropping area specified by the user through the video distribution device operation unit 121 to the camera microcomputer 114. Note that in FIG. 6A to FIG. 6E and Figure 7 In, with FIG. 2A to FIG. 2E and Figure 4 The same configurations and processes as those shown are given the same reference numerals, and description thereof will be appropriately omitted.

[0096] Figure 7 The processing shown in the flowchart in is performed by the camera microcomputer 114.

[0097] After acquiring information about the positions and sizes of the faces of all persons in the captured video in step S401, in step S701, information about all cropping areas set in the video allocation control unit 122 is acquired, and the process proceeds to step S403. As described above, in the second embodiment, information about the cropping area to which the video is being allocated cannot be acquired.

[0098] After the processing in step S403 and step S404, if it is determined in step S405 that a face is specified, the processing proceeds to step S703, and if it is determined that a face is not specified, the processing proceeds to step S702.

[0099] In step S703, it is determined whether the face designated by the user is located in any of the cropping areas obtained in step S701. Note that in step S703, since it is unknown to which cropping area the allocating device 12 is allocating the video, the camera microcomputer 114 only determines whether a face in the cropping area is selected, and then determines whether to change the main face.

[0100] If it is determined that the specified face is within any cropping area, the process advances to step S409; if it is determined that the specified face is not within any cropping area, the process advances to step S411.

[0101] If it is determined in step S411 that the method by which the user specifies a face is not the specific specifying method, and if the face specifying flag is determined to be OFF in step S412, the process advances to step S702 and a new main face is determined.

[0102] In step S702, a process of calculating the priority of each face is performed, and in step S407, a main face is determined based on the calculated priority of each face and is set.

[0103] Next, we will use Figure 8 The flowchart illustrates the priority calculation process for each face performed in step S702.

[0104] First, in step S801, the center coordinates and sizes of all the cropping regions that are set are obtained. In the following, the center coordinates of the Nth cropping region obtained will be referred to as C crop_N .

[0105] Next, in step S802, using the center coordinates and sizes of all the cropped areas obtained in step S801, the priorities of only the faces present in the cropped areas are calculated according to a predetermined standard.

[0106] The following will refer to FIG. 6A to FIG. 6E The priority calculation method in step S802 is described.

[0107] Fig. 6A 2 is a video of the full-angle area 200 captured by the camera 10, and includes human faces 210, 211, 212, and 213 as subjects. In addition, areas 201, 202, 204, and 601 are cropped areas partially cut out from the full-angle area 200. In addition, marks 620, 621, 622, and 623 in the cropped areas 201, 202, 204, and 601 respectively represent the center C of the angle of view of the cropped areas. crop_N The coordinates of marker 620 are C crop_1 Indicates that the coordinates of marker 621 are C crop_2 Indicates that the coordinates of marker 622 are C crop_3 and the coordinates of marker 623 are represented by C crop_4 express.

[0108] Figure 6B A video showing a cropped region 201, Figure 6C A video showing a cropped region 202, Fig.6D The video of the cropped region 204 is shown, and Fig. 6E The video of the cropped region 601 is shown.

[0109] [First method]

[0110] In the first method, based on the distance between each face in any cropped area and the center of the picture, the shorter the distance, the higher the priority.

[0111] exist FIG. 6A to FIG. 6E In the example shown, the human face 212 closest to the center of the screen has the highest priority among the faces in any of the cropped areas 201, 202, 204, and 601. As a result, in step S407, the human face 212 is determined as the main face.

[0112] Note that if the cropping area 601 is not set, the person face 211 closest to the center of the screen has the highest priority among the faces in any of the cropping areas 201, 202, and 204, and is determined to be the main face. The person face 212 or the person face 213 that does not exist in any of the cropping areas 201, 202, and 204 cannot be the main face. In addition, if the person face 211 does not exist, the person face 210 has the highest priority and is determined to be the main face.

[0113] [Second method]

[0114] In the second method, based on the center of the full-angle area 200 to the crop area C crop_N The distance is used to calculate the priority.

[0115] Specifically, first, calculate the clipping area C crop_1 To Ccrop_4 The closer to the calculated centroid CM, the higher the priority.

[0116] In addition, when calculating the centroid CM, by giving a larger weight to the area with a narrower cropped range, the priority of the face existing in the more concerned area becomes higher, which increases the possibility of determining the face as the main face. As an example, the following method can be considered: crop_N The distance multiplied by the inverse of the horizontal size of the clipping area. That is, FIG. 6A to FIG. 6E In the figure, the cropped areas 204 and 601 have half the horizontal size of the full view, so the cropped areas 204 and 601 are 2 times the weight. In addition, since the size of the cropped areas 201 and 202 is 1 / 4 of the horizontal size of the full view, the cropped areas 201 and 202 are 4 times the weight. That is, if the center coordinate of the full-angle area 200 is C, the centroid CM can be obtained using the following formula (1).

[0117] CM=(4((-C crop_1 )+4((-C crop_2 )+2(CC crop_3 )+2(CC crop_4 ))(4+4+2+2)...(1)

[0118] By weighted averaging in this manner, a face that is enlarged at a higher magnification is given a higher priority and is more likely to be selected as the main face, thereby increasing the possibility of reducing the sense of incongruity in the video being distributed.

[0119] exist FIG. 6A to FIG. 6E In the example shown, among the faces existing in any of the cropped areas 201 , 202 , 204 , and 601 , the face of the person closest to the calculated centroid CM becomes the main face.

[0120] [Third Method]

[0121] In the third method, when the user has designated a cropping region together with the main face, and the main face is beyond the cropping region (outside a partial region), priority is set based on the direction in which the main face exits.

[0122] For example, Fig.6DAs shown, when the user specifies the human face 211 in the cropping area 202 as the main face, the following case is considered: the frame of the human face 211 disappears toward the right side of the cropping area 202 (toward the direction of the human face 212). At this time, the possibility that the human face 211 exists in the cropping area on the right side of the cropping area 202 or faces the cropping area on the right side is high. Therefore, by selecting the main subject existing in the cropping area on the right side of the cropping area 202, the possibility of allocating a video that does not give an awkward impression is increased.

[0123] In this case, the above purpose can be achieved by setting the priority of the face existing in the direction where the frame of the main face exits higher than the priority of the face existing in the opposite direction. Alternatively, in the above-mentioned second determination method, the weight of the cropped area existing in the direction where the frame of the main face exits can be set to be greater than the weight of the cropped area existing in the opposite direction.

[0124] Furthermore, even if the frame of the main face exits outside the cropping area as described above, the priority calculation method may be changed depending on whether or not other faces exist in the cropping area where the main face originally existed.

[0125] For example, consider the following situation: Fig. 6E The human face 212 in the cropping area 601 shown is the main face, and the frame of the human face 212 exits to the left (in the direction of the human face 210) outside the cropping area 601 at a certain timing. In this case, the priority of the face located in the cropping area where the main face is located is given high priority, and the priority of the face located in the direction in which the main face starts to move is also given high priority. By setting the priorities in this way, instead of selecting the main face from the subjects in the cropping areas 201 and 204, even if the face on the right side of the human face 212 is detected in the same cropping area 601, the human face 211 on the left side of the human face 212 is selected as the main face.

[0126] As a result, although the area to which the video is being assigned is unknown, assuming that the area where the main face exists has a high probability of being assigned, the possibility of assigning the video with the least offensive impression can be increased.

[0127] [Fourth method]

[0128] In the fourth method, the greater the number of cropping regions to which a face belongs, the higher the priority given to the face.

[0129] For example, in FIG. 6A to FIG. 6EIn the case of , the human face 211 belongs to three areas (i.e., the cropping areas 202, 204, and 601), so the priority is set to high (e.g., multiplied by 3). In addition, the human face 210 belongs to two areas (i.e., the cropping areas 201 and 204), so its priority is doubled, for example, while the human face 212 belongs only to the cropping area 601, so its priority is not doubled or multiplied by 1.

[0130] For example, by performing the above-described processing on the priorities calculated for all faces detected using the above-described first to third determination methods, the priority of a face belonging to more cropped areas is increased more, which increases the possibility that the face is selected as the main face. This increases the possibility that the main face exists in the cropped area to which the video is being assigned, thereby making it possible to increase the possibility of assigning a video that does not give an offensive impression.

[0131] Among the above-mentioned first method to fourth method, any determination method can be set in advance, or any determination method can be selected by the user. In addition, the priority calculation method is not limited to the first method to fourth method, and any determination condition can be adopted as long as it can increase the possibility that the main face exists in the cropping area to which the video is being assigned.

[0132] <Other embodiments>

[0133] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0134] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A video processing device, comprising: A detection component configured to detect a predetermined one or more subjects from a video obtained by a camera component for repeatedly capturing and outputting the video; an acquisition component configured to acquire information related to at least one partial area set in the video obtained by the camera component; as well as A determining component is configured to determine a main subject from among the one or more subjects detected by the detecting component based on the priority of each subject in the partial area.

2. The video processing apparatus according to claim 1, further comprising an operating unit configured to specify a main subject from among the one or more subjects detected by the detecting unit, in, The determining section determines the main subject when the operating section does not specify the main subject.

3. The video processing device according to claim 2 further includes a setting component, which is configured to set the designated subject as the main subject regardless of whether the designated subject is within the partial area when the subject is designated by a predetermined operation of the operating component.

4. The video processing device according to claim 3, wherein: In a case where the detection section does not detect the designated main subject, the setting section does not change the designated main subject within a predetermined time since the designated main subject was not detected.

5. The video processing device according to claim 1, wherein: The acquisition component acquires information related to the partial area where the video is allocated to the outside, and The determining means determines the main subject from one or more subjects in the partial area to which the video is allocated to the outside.

6. The video processing device according to claim 5, wherein: The determination section sets a higher priority for a subject, among the one or more subjects in the partial area, that is closer to the center of the partial area to which the video is allocated to the outside.

7. The video processing device according to claim 5, wherein: The determination section sets a higher priority for a larger-sized object in the partial area to which the video is allocated to the outside.

8. The video processing device according to claim 1, wherein: The determination means sets a higher priority to the subject which is in the partial area and is located closer to the center of the video captured by the imaging means.

9. The video processing device according to claim 1, wherein: The determination section sets a higher priority for a subject that is in the partial area and is located closer to the centroid of one or more partial areas.

10. The video processing device according to claim 5, wherein: When obtaining the centroid of the partial area, the determining component assigns a greater weight to a smaller partial area.

11. The video processing device according to claim 1, wherein: In the case where the determined main subject moves and exits from the partial area, the determining component sets a higher priority for the subjects in the partial area in the moving direction of the main subject, and also sets a higher priority for the subjects in the partial area to which the main subject previously belonged.

12. The video processing device according to claim 8, wherein: The determining component assigns a higher priority to a subject belonging to more partial areas.

13. The video processing device according to claim 7, wherein: The determination section sets a higher priority for a subject that is in the partial area and has a larger size.

14. A camera device, comprising: A video processing device, comprising: a detection component configured to detect a predetermined one or more subjects from a video obtained by a camera component for repeatedly capturing and outputting the video, an acquisition component configured to acquire information related to at least one partial area set in the video obtained by the camera component, and a determining component configured to determine a main subject from among the one or more subjects detected by the detecting component based on the priority of each subject in the partial area; and The camera component is used to repeatedly shoot and output videos. 15 . The imaging device according to claim 14 , further comprising a focus adjustment control section configured to focus on the main object. 16 . The imaging device according to claim 14 , further comprising an exposure control unit configured to adjust exposure of the main object.

17. A video distribution system comprising: A setting component configured to set at least one partial area in a video captured by the camera device according to any one of claims 14 to 16; as well as The allocating component is configured to select and allocate any video among the video and the video of the partial area.

18. A video processing method, comprising: A detection step for detecting a predetermined one or more subjects from a video obtained by a camera component for repeatedly capturing and outputting the video; An acquisition step for acquiring information related to at least one partial area set in the video obtained by the camera component; as well as The determining step is used to determine the main subject from the one or more subjects detected in the detecting step based on the priority of each subject in the partial area.

19. A computer-readable storage medium storing a program for causing a computer to execute the video processing method according to claim 18.

20. A computer program product comprising a program for causing a computer to execute the video processing method according to claim 18.

Images