Control device, control method, and program
The imaging device adjusts its direction and field of view based on subject positions to prevent image distortion and missed shots by extracting partial images of specific subjects, addressing the limitations of conventional PTZ control.
Patent Information
- Application Number
- JP2025078026
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-08
- Publication Date
- 2026-11-19
AI Technical Summary
Conventional PTZ control in imaging systems risks image distortion or missing important scenes when switching imaging targets, especially when subjects are outside the predetermined field of view.
An imaging device that acquires position information of multiple subjects, adjusts the imaging direction and field of view to include all subjects, and extracts partial images of specific subjects using wireless communication and digital PTZ processing.
Prevents image distortion and missed shots during subject switching by ensuring all subjects remain within the field of view and allowing seamless transitions without interruption.
Smart Images

Figure 2026182096000001_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to control technology for controlling an imaging apparatus.
Background Art
[0002] Imaging systems using imaging apparatuses include, for example, systems that automatically track and image a target subject in a space where a plurality of subjects are present. In this system, when switching the imaging target subject to another subject, the pan, tilt, and zoom (hereinafter referred to as PTZ) of the imaging apparatus is controlled so as to image the other subject.
[0003] Further, Patent Document 1 discloses a control technique suitable for switching an imaging target subject to another subject. The imaging apparatus described in Patent Document 1 outputs an image at a first angle of view corresponding to the subject before switching, and controls the imaging angle of view such that the first angle of view and a second angle of view corresponding to the other subject are both contained. Then, after the second angle of view falls within the imaging angle of view, the imaging apparatus cuts out and outputs an image of the second angle of view from the captured image.
Prior Art Documents
Patent Documents
[0004]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0005] In the case of the conventional PTZ control described above, image distortion may occur or important imaging scenes may be missed while pan-tilt control is being performed to adjust the imaging direction of the imaging device. On the other hand, according to the technology of Patent Document 1, the risk of image distortion or missing important imaging scenes is low. However, with the technology of Patent Document 1, the imaging field of view is predetermined, so it cannot handle cases where another subject located outside the imaging field of view is to be used as the imaging pair. In this case, pan-tilt control similar to the conventional method is still required, and there is a risk of image distortion or missing important imaging scenes. [Means for solving the problem]
[0006] The control device of this disclosure is a control device for controlling an imaging device, and is characterized by comprising: acquisition means for acquiring position information of each of a plurality of subjects; adjustment means for adjusting either or both the imaging direction and the field of view of the imaging device based on the position information of the plurality of subjects so that the plurality of subjects are within the field of view of the imaging device; and extraction means for extracting a partial image corresponding to at least one of the plurality of subjects from an image captured by the imaging device adjusted so that the plurality of subjects are within the field of view. [Effects of the Invention]
[0007] According to this disclosure, it is possible to prevent image distortion and missed shots when the subject being captured is switched. [Brief explanation of the drawing]
[0008] [Figure 1] This is a diagram showing an example of the overall configuration according to the embodiment. [Figure 2] This figure shows an example of the configuration of an imaging device. [Figure 3] This is a flowchart showing the control process flow according to the first embodiment. [Figure 4] This is an explanatory diagram illustrating an example of the operation of the control device in the first embodiment. [Figure 5]This is an explanatory diagram of the field of view and cropped image of the first embodiment. [Figure 6] This is a flowchart showing the control process flow according to the second embodiment. [Figure 7] This figure shows an example of a GUI screen for setting the composition of a subject. [Figure 8] This is an explanatory diagram of the field of view and cropped image of the second embodiment. [Figure 9] This is a flowchart showing the control process flow according to the third embodiment. [Figure 10] This is an explanatory diagram illustrating an example of the operation of the control device in the third embodiment. [Modes for carrying out the invention]
[0009] Embodiments relating to this disclosure will be described below with reference to the drawings. The embodiments described below are not limiting to this disclosure, and not all combinations of features described in these embodiments are essential to the solutions of this disclosure. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications of the apparatus to which this disclosure applies and various conditions (usage conditions, usage environment, etc.). In the following embodiments, redundant explanations of the same or similar configurations and processing steps will be omitted.
[0010] <First Embodiment> In this embodiment, we will describe a system that automatically tracks and captures images of multiple subjects in a space where multiple subjects exist, ensuring that these subjects are within the imaging field of view of the imaging device. In this embodiment, to distinguish it from tracking only a specific subject, tracking multiple subjects so that they are within the imaging field of view will be referred to as multiple subject tracking. In this embodiment, pan-tilt-zoom (PTZ) control is performed to keep all of the multiple subjects within the imaging field of view of the imaging device to perform multiple subject tracking, and then an image of a specific subject is extracted from among these multiple subjects.
[0011] Figure 1 is a schematic diagram showing an example of the overall configuration of the imaging device 100 according to this embodiment, an external device 120 connected to the imaging device 100, and a plurality of people who are the subjects to be imaged by the imaging device 100. In the example of Figure 1, the imaging device 100 images people on a stage as subjects 111a to 111c, frame by frame in chronological order. In this embodiment, each of the subjects 111a to 111c holds (wears, etc.) a wireless slave unit. In the example of Figure 1, subject 111a holds wireless slave unit 110a, subject 111b holds wireless slave unit 110b, and subject 111c holds wireless slave unit 110c. In the following description, when subjects 111a to 111c are not specified, they will be referred to as "subject 111," and similarly, when wireless slave units 110a to 110c are not specified, they will be referred to as "wireless slave unit 110."
[0012] The configuration shown in Figure 1 is merely an example and is not limited to it. In Figure 1, the three wireless slave units 110a to 113c are each held by one of the three subjects, but the number of wireless slave units is not limited to three, and one subject may hold two or more wireless slave units. In other words, the number of subjects and the number of wireless slave units do not have to match. Also, in this embodiment, the external device 120 is assumed to be a personal computer with a monitor, but it is not limited to this, and may be a tablet terminal or a server, for example. Furthermore, the number of external devices 120 connected to the imaging device 100 is not limited to one but may be multiple, and the external device 120 is not essential and may be omitted, and the imaging device 100 may have the functions of the external device 120.
[0013] The imaging device 100 includes an imaging unit that can change the imaging direction and field of view, i.e., the imaging target area (imaging target range), by pan-tilt-zoom (PTZ) operation. The imaging device 100 also communicates wirelessly with the wireless slave units 110 and calculates the direction and distance to each wireless slave unit 110. Wireless communication technologies such as Bluetooth® or ultra-wideband (UWB) can be used for wireless communication between the imaging device 100 and the wireless slave units 110.
[0014] Figure 2 is a diagram showing a configuration example of the imaging apparatus 100. The imaging apparatus 100 includes an imaging unit 201, an image processing unit 202, a driving unit 203, a control unit 204, a storage unit 205, a communication unit 206, an operation information acquisition unit 207, and the like. The imaging apparatus 100 receives operation information input by a user to the external apparatus 120, and performs PTZ control based on the operation information to change the imaging direction and the angle of view. Further, the imaging apparatus 100 may have, for example, a preset function that sets the imaging direction and the angle of view to pre-registered imaging direction and angle of view. In addition, the imaging apparatus 100 transmits captured images and various types of information to the external apparatus 120 in response to a request from the external apparatus 120.
[0015] The imaging unit 201 includes an imaging optical system including a focus lens, a zoom lens, and the like, an imaging element, a mechanical drive system for driving these components, a circuit, and the like. Further, in the present embodiment, the imaging unit 201 also includes a pan-tilt head function (for example, a motorized pan-tilt head) for changing the imaging direction of the imaging apparatus 100 by pan and tilt driving. The imaging element is configured to include an image sensor such as a CCD or CMOS and its peripheral circuits, and outputs an imaging signal obtained by photoelectrically converting light formed into an image on the light receiving surface by the imaging optical system to the image processing unit 202.
[0016] The image processing unit 202 performs image processing such as noise removal and gamma correction on the imaging signal input from the imaging unit 201 to generate an image signal (hereinafter referred to as a captured image, or simply an image). Further, in the case of the present embodiment, the image processing unit 202 also has a digital PTZ function for cutting out a partial sub-image of the captured image. Then, the image processing unit 202 sends the image cut out by the digital PTZ function to the control unit 204. Hereinafter, in the present embodiment, the partial image cut out by the digital PTZ function is referred to as a cut-out image.
[0017] Furthermore, the image processing unit 202 can also perform processing based on control commands received from the control unit 204. For example, when the image processing unit 202 receives a control command instructing image quality adjustment from the control unit 204, it changes the image processing parameters to perform image quality adjustment. In addition, the image processing unit 202 outputs setting contents such as exposure settings and time information to the control unit 204.
[0018] The drive unit 203 performs PTZ operations that change the pan / tilt direction and zoom magnification of the imaging unit 201 of the imaging apparatus 100. That is, the drive unit 203 includes a mechanical drive system that drives the pan / tilt direction of the pan / tilt head included in the imaging unit 201, a motor as a drive source, as well as a mechanical drive system for changing the zoom magnification (that is, focal length) of the zoom lens of the imaging unit 201, a motor as a drive source, and the like. The drive unit 203 operates the drive system that moves the imaging direction of the imaging unit 201 of the imaging apparatus 100 in the pan / tilt direction, based on the pan / tilt (PT) drive control command sent from the control unit 204. In addition, the drive unit 203 operates the drive system that moves the zoom lens of the imaging unit 201, based on the zoom drive control command sent from the control unit 204.
[0019] Furthermore, the drive unit 203 also has a sensor (encoder) for specifying the posture of the imaging apparatus 100, that is, the PTZ position of the imaging unit 201. Note that in the present embodiment, the PTZ position refers to the pan / tilt position (hereinafter referred to as PT position) corresponding to the imaging direction of the imaging unit 201, and the position of the zoom lens indicating the focal length of the zoom lens (hereinafter referred to as Z position). In the present embodiment, the sensor (encoder) for specifying the posture of the imaging apparatus 100 (the PTZ position of the imaging unit 201) is assumed to be, for example, a photo interrupter (PI). Note that although the photo interrupter (PI) is taken as an example in the present embodiment, the present invention is not limited to this, and other sensors may be used as long as the posture of the imaging apparatus 100 can be specified.
[0020] The drive unit 203 then calculates the PT position based on the PT drive amount and the detected PI value, and identifies the Z position from the zoom lens drive amount, thereby obtaining the PTZ position (PT position and Z position) of the imaging unit 201. The drive unit 203 sends the obtained PTZ position value to the control unit 204, and the control unit 204 outputs the PTZ position value to the storage unit 205 for storage. In this embodiment, the drive unit 203 is shown as acquiring the PTZ position value, but it may also acquire any converted value that can identify the PTZ position.
[0021] Furthermore, although this embodiment shows an example in which the imaging unit 201 of the imaging device 100 has a PT function, the imaging device 100 may also be configured in which the PT drive of the imaging device 100 is performed by an external pan / tilt head (not shown) separate from the imaging device 100, that is, a configuration in which the imaging function and the PT drive function are separated. In that case, the drive unit 203 outputs a PT drive control command from the control unit 204 to the external pan / tilt head (not shown), and the control unit 204 stores the PT drive control command output by the drive unit 203 to the external pan / tilt head in the storage unit 205.
[0022] The control unit 204 functions as a control device that controls the entire imaging device. The control unit 204 is a central processing unit and can be realized, for example, by a CPU executing a control program. The control unit 204 may also be configured in combination with an application-specific integrated circuit (ASIC). The control unit 204 also performs analysis processing on various control commands acquired via the communication unit 206, wireless signals from the wireless slave unit 110, and images captured by the imaging unit 201. Based on these analysis results, the control unit 204 controls the imaging device 100. In addition to the functions of the control unit 204, the control device according to this embodiment also includes the image extraction function by the image processing unit 202 described above.
[0023] For example, if the control unit 204 receives a control command related to PTZ drive control transmitted from the external device 120 via the communication unit 206, it outputs a PTZ drive control command to the drive unit 203 based on that control command. The control unit 204 also stores information related to the PTZ position obtained from the drive unit 203, various acquired data, information about the lens of the imaging unit 201, and preset registration contents by the user (described later) in the storage unit 205. Furthermore, as will be described in detail later, the control unit 204 identifies the position for extracting a partial image from the image captured by the imaging unit 201 and controls the image processing unit 202 to generate a partial image of that position as an extracted image from the captured image.
[0024] The memory unit 205 has two functions: it serves as a temporary storage area for the work area and data of the control unit 204, and it stores various programs, various setting information, and images captured by the imaging unit 201. For example, the function as a temporary storage area for the work area and data is implemented by random access memory (RAM), while the function of storing various programs, various setting information, and images captured by the imaging unit 201 is implemented by flash memory storage or the like. The memory unit 205 stores, for example, PTZ position information from the drive unit 203, control commands and preset registration contents from the control unit 204, and data of captured images and cropped images after image processing by the image processing unit 202. This information and data are stored in the memory unit 205 together with the system information within the imaging device.
[0025] The communication unit 206 receives various control commands from the external device 120 via the network 140, appropriately discriminates these received control commands, and transmits them to the control unit 204 or the operation information acquisition unit 207. The communication unit 206 also acquires PTZ position information, preset information, captured images and their thumbnail images, cropped images, and imaging device settings from the storage unit 205 and transmits them to the external device 120. Furthermore, the communication unit 206 communicates wirelessly with the wireless slave unit 110 and sends the information acquired from the wireless slave unit 110 to the control unit 204 and the storage unit 205.
[0026] In this embodiment, the control unit 204 is assumed to acquire various control commands from the external device 120 via the network 140, but it may acquire control commands from other external devices. For example, control commands related to image quality and white balance may be acquired via the network 140 from an external device different from the external device 120. Furthermore, the control unit 204 is not required to receive various control commands.
[0027] The operation information acquisition unit 207 receives control commands related to the operation of the imaging device 100 from the communication unit 206, among the various control commands received by the communication unit 206. In this embodiment, control commands related to operation include, for example, operation commands related to PTZ drive, and operation commands related to presets, such as preset registration and updating, which will be described later. These control commands are generated in the external device 120 by input via a GUI (Graphical User Interface), which will be described later. The operation information acquisition unit 207 also acquires control commands generated by, for example, an infrared remote control, a wireless remote control, wireless input from an external terminal, or hardware switch operation of the imaging device 100, as control commands related to operation. The control commands related to operation described above are just examples and are not limited to these.
[0028] Figure 3 is a flowchart showing the flow of control processing by the control unit 204 and image extraction processing by the image processing unit 202 according to this embodiment. The control processing shown in the flowchart of Figure 3 is assumed to start, for example, when the control unit 204 recognizes that the communication unit 206 has received a wireless signal from the wireless slave unit 110.
[0029] Figures 4 and 5 show an example of operation of the imaging device 100 according to the first embodiment. In the example in Figures 4 and 5, as also shown in Figure 1, it is assumed that three people, subjects 111a to 111c, who move around on the stage, each have wireless sub-units 110a to 110c attached to their chests. For the sake of simplicity, it is assumed that the imaging device 100 and each wireless sub-unit 110 attached to each subject 111 are at the same height from the ground (i.e., the tilt direction is fixed at "0 degrees" for all). Therefore, only the control of the pan direction will be explained below. It is also assumed that the imaging device 100 is capturing images of each subject 111 on the stage from near the front of the stage, frame by frame in chronological order. The control process of the flowchart shown in Figure 3 will be explained below with reference to Figures 4 and 5.
[0030] First, in step S301, the control unit 204 derives position information, which is represented by the direction and distance of each wireless slave unit 110 relative to the imaging device 100, based on the wireless signals exchanged between the communication unit 206 and the wireless slave units 110. That is, the control unit 204 acquires the position information of the wireless slave units 110 held by each of the multiple subjects (people). For example, the control unit 204 sets the optical axis direction of the imaging optical system of the imaging device 100 as the reference angle (0 degrees), and acquires the relative angle value of each wireless slave unit 110 with respect to that reference angle as the direction of each wireless slave unit 110 relative to the imaging device 100. The control unit 204 also acquires the distance from the imaging device 100 to each wireless slave unit 110 based on the time difference and phase difference of the wireless signals transmitted and received between the communication unit 206 and the wireless slave units 110. In step S301, the control unit 204 also acquires PTZ position information from the drive unit 203. In this embodiment, the control unit 204 performs the process related to step S301 repeatedly and asynchronously, for example, at predetermined time intervals. The process related to step S301 may also be performed in synchronization with the imaging frame of the imaging device 100.
[0031] In the example shown in Figure 4, the position information of wireless slave unit 110a is assumed to indicate that the angle in the pan direction relative to the reference angle of the imaging device 100 is 0 degrees and the distance is 10 m. Similarly, the position information of wireless slave unit 110b is assumed to indicate that the angle in the pan direction is 28 degrees (-28°) to the left of the reference angle and the distance is 14 m. Furthermore, the position information of wireless slave unit 110c is assumed to indicate that the angle in the pan direction is 30 degrees (+30°) to the right of the reference angle and the distance is 14 m.
[0032] Next, in step S302, the control unit 204 associates the position information of each wireless slave unit 110 with each person in the subject 111 that is within the imaging field of view of the imaging device 100. To perform this association, the control unit 204 first detects each person in the subject 111 that is within the imaging field of view of the imaging device 100 by performing person detection processing on the captured images for each frame in the time series. The person detection processing is a known process, so its explanation will be omitted. Furthermore, for each person detected by the person detection processing, the control unit 204 obtains a position represented by the direction and distance from the imaging device 100.
[0033] In this embodiment, the control unit 204 obtains the direction and distance of each person by checking how far the position of each person in the subject within the imaging field of view of the imaging device 100 is shifted from the center of the captured image. For example, if the panning angle in the imaging field of view of the imaging device 100 is 60 degrees and the subject 111 is located near the edge of the captured image, the control unit 204 determines that the direction of the person in the subject relative to the imaging device 100 is approximately 30 degrees shifted from the center of the image. The control unit 204 also obtains the distance from the imaging device 100 to the person in the subject based on the depth of field corresponding to the focal length of the imaging optical system of the imaging device 100 and the focus position relative to the subject.
[0034] Then, the control unit 204 compares the position represented by the direction and distance obtained for the subject 111 with the position represented by the direction and distance obtained for each wireless slave unit 110 in step S301, and derives the most likely combination of subject 111 and wireless slave unit 110.
[0035] Figure 5 shows an example of an image 500 acquired by the imaging device 100. In Figure 5, image 510 shows an example of an image within the field of view corresponding to, for example, the position information of the wireless slave unit 110a and the person of subject 111a detected by person detection on the image 500. The control unit 204 compares the position information of each wireless slave unit 110 with the person detection results of each subject 111 and determines which subject 111 in the image 500 corresponds to which wireless slave unit 110, thereby deriving the most likely combination of subject and wireless slave unit.
[0036] Next, in step S303, the control unit 204 determines whether all wireless slave units 110 are within the imaging field of view of the imaging device 100. That is, the control unit 204 checks whether it has been able to derive combinations of all subjects and wireless slave units within the imaging field of view of the imaging device 100. If the control unit 204 determines that all wireless slave units 110 for which combinations have been derived are within the imaging field of view, it proceeds to step S305. On the other hand, if it determines that there are wireless slave units 110 that are not within the imaging field of view, it proceeds to step S304.
[0037] If the process proceeds to step S304, the control unit 204 controls the drive unit 203 using PTZ drive control to adjust either the imaging direction and / or field of view of the imaging device 100 so that all wireless slave units 110 for which combinations have been derived are within the imaging field of view of the imaging device 100. After step S304, the control unit 204 returns to step S302. In this way, imaging is performed with an imaging field of view adjusted to include the subject 111a to 111c holding all wireless slave units 110a to 110c, resulting in the acquisition image 500 shown in Figure 5.
[0038] Furthermore, if the process proceeds to step S305, the control unit 204 identifies the location of a subject that is the target of partial image extraction by the image processing unit 202. For example, based on the captured image sent from the image processing unit 202, the location information of each wireless slave unit 110, and the result of the combination derived in step S302, the control unit 204 identifies the location of each subject that may be subject to extraction within the captured image. Then, among the subjects that may be subject to extraction, the control unit 204 identifies one or more subjects, for example, that the user has instructed via the external device 120, as the location of the subject that will be subject to partial image extraction. In addition, the control unit 204 may identify, for example, a subject holding a specific wireless slave unit as a subject to be extracted. Alternatively, the control unit 204 may identify a subject to be extracted based on the facial information of a person obtained by the image processing unit 202 from image analysis processing. Furthermore, the control unit 204 may identify a subject that is presumed to have made a specific action or sound as a subject to be extracted. Note that the methods for identifying the subject to be cropped are not limited to these examples.
[0039] Next, in step S306, the control unit 204 issues a cropping command to the image processing unit 202, instructing it to crop a partial image at the cropping position of the subject identified in step S305, for example, within a range corresponding to the size of the subject. For example, if multiple subjects are selected as cropping targets, the control unit 204 may set a single cropping position and cropping range that includes all of those subjects. Alternatively, the control unit 204 may set a cropping position and cropping range for each of the multiple subjects.
[0040] Upon receiving a cropping command from the control unit 204, the image processing unit 202 generates a cropped image by cropping a portion of the captured image from the captured image using digital PTZ processing according to the cropping position and range specified by the cropping command. For example, if the position of the subject 111a holding the wireless slave unit 110a is specified as the cropping position, then image 510, as shown in Figure 5, is generated as the cropped image. The control unit 204 then transmits the cropped image generated by the image processing unit 202 to the external device 120 via the communication unit 206. In addition to being transmitted to the external device 120, the cropped image may also be sent to, for example, the storage unit 205 and stored within the imaging device 100.
[0041] Next, in step S307, the control unit 204 sends the cropped image from the image processing unit 202 to the communication unit 206, and the communication unit 206 transmits the cropped image to the external device 120 via the network 104. As a result, the cropped image containing a specific subject from among multiple subjects is transmitted to the external device 120. The control unit 204 may also send the cropped image to, for example, the storage unit 205 for internal storage. Alternatively, the control unit 204 may divide the cropped image into multiple streams and transmit the data divided into these multiple streams to the external device 120 via the communication unit 206.
[0042] Next, in step S308, the control unit 204 checks whether the termination flag is set. For example, if the termination flag is set, the control unit 204 terminates the process shown in the flowchart of Figure 3, while if the termination flag is not set, it returns to step S301. In this embodiment, the termination flag is set, for example, when the system receives an imaging termination instruction from the user at any given time.
[0043] The imaging device 100 can perform the image extraction and transmission described above, for example, in a sequential order for each frame, to transmit extracted images that track a specific moving subject to an external device 120 or the like. Here, for example, suppose that in response to instructions from the user via the external device 120, the subject to be extracted is switched from subject 111a holding the wireless slave unit 110a to another subject 111c holding the wireless slave unit 110c. In other words, suppose the subject (person) being tracked is switched from subject 111a to subject 111c. In this case, the imaging device 100 performs the same extraction process as described above, with subject 111c holding the wireless slave unit 110c as the new subject to be extracted, thereby generating an extracted image with subject 111c as the tracked subject, as shown in image 520 of Figure 5. The imaging device 100 then switches the extracted image to be transmitted to the external device 120 from image 510 to image 520. Since images 510 and 520 are extracted from the captured image 500, the image transmitted to the external device 120 is an image in which the subject being tracked is instantly switched from subject 111a to subject 111c. In other words, the imaging device 100 of this embodiment can output tracking images without image distortion or missed shots, even when the subject being tracked is switched from subject 111a to subject 111c.
[0044] As described above, the imaging device 100 according to the first embodiment acquires positional information representing the position and distance of subjects, and performs multiple subject tracking with the optimal imaging direction and field of view that captures all subjects based on that positional information. The imaging device 100 then identifies the position of a specific subject from among the multiple subjects, and extracts the image portion of that specific subject from the captured image, thereby transmitting the extracted image containing the specific subject to an external device 120 or the like. Furthermore, according to this embodiment, even when switching to another subject, it is possible to switch to an image containing the other subject quickly and seamlessly without interruption or distortion of the image.
[0045] In the first embodiment, positional information representing the location and distance of the subject was obtained using wireless communication with a wireless slave unit, but this is not the only method. For example, the control unit 204 of the imaging device 100 may create a 3D map based on images captured by another imaging device that captures an overview of multiple subjects, and derive the coordinates of each subject based on that 3D map.
[0046] Alternatively, for example, infrared sensors and pressure sensors may be comprehensively arranged on the stage, and the control unit 204 of the imaging device 100 may derive positional information for each subject based on the sensor information detected by these infrared sensors and pressure sensors.
[0047] Furthermore, while the first embodiment describes an example where the control unit 204 within the imaging device 100 determines the position of the subject to be cropped, etc., it is not limited to this. For example, an external device 120 may acquire the captured image and the position information of the wireless slave unit from the imaging device 100, derive information indicating the cropping position and cropping range in the same manner as described above, and transmit it to the imaging device 100.
[0048] Furthermore, while this embodiment does not impose any restrictions on the tracking process of multiple subjects, it is not limited to this. For example, the imaging device 100 may have a function that allows the user to pre-set the imaging control range by adjusting the imaging direction and field of view. If a subject exceeds that imaging control range, that subject may be excluded from the multiple subject tracking.
[0049] <Second Embodiment> In the second embodiment, an example will be described in which the imaging device 100 has a function to correct how the subject appears in the image, that is, the composition of the subject. In the first embodiment, as an example, when image 520 in Figure 5 is used as a cropped image, depending on the position of the subject within the imaging angle of view, the position of the subject in the cropped image may be off-center. That is, if the user intended to obtain a cropped image in which the subject is located in the center of the image, image 520 in Figure 5 will be an image in which the subject appears (composition) in a way that the user did not intend. In addition, in some cases, part of the subject may be outside the image in the cropped image (the subject may be cut off from the cropped image).
[0050] Therefore, in the second embodiment, in order to prevent the generation of cropped images with compositions unintended by the user, an example is described in which cropped images can be generated while maintaining the composition of a pre-specified subject. Note that the overall system configuration and the configuration of the imaging device 100 in the second embodiment are the same as those in Figures 1 and 2 described above, so their illustration and explanation are omitted.
[0051] Figure 6 is a flowchart showing the flow of control processing by the control unit 204 and image extraction processing by the image processing unit 202 of the imaging device 100 according to the second embodiment. In the flowchart of Figure 6, steps S301 to S308 are the same as in the flowchart of Figure 3, so their explanation is omitted. The flowchart of Figure 6 according to the second embodiment differs from the flowchart of Figure 3 in that it includes processing steps S601 to S606 according to the composition setting of the subject.
[0052] First, in step S601, the control unit 204 of the imaging device 100 determines whether a composition of the subject has been set. In this embodiment, it is explained that there are two types of composition settings: a default composition and a composition arbitrarily set by the user via an external device 120, but this is not limited to these. If the control unit 204 determines that no composition has been set by the user, it proceeds to step S602; on the other hand, if it determines that a composition has been set by the user, it proceeds to step S301.
[0053] If the process proceeds to step S602, the control unit 204 recalls the default composition settings. After step S602, the control unit 204 proceeds to step S301. In the flowchart of Figure 6, after steps S301 and the following step S302, as described above, the control unit 204 proceeds to step S603. The following explanation will use the case where it is determined that the composition has been set by the user in step S601 as an example. In the following explanation, the user's composition setting is assumed to be a composition in which the main subject is cropped and captured in the center of the frame, but it is not limited to this.
[0054] When the process proceeds to step S603, the control unit 204 determines whether all subjects are within the imaging field of view of the imaging device 100 and whether the composition of the subjects when a cropped image is generated satisfies the composition set by the user.
[0055] For example, in the case of the captured image 500 shown in Figure 5, the cropped image of subject 111c will not capture subject 111c in the center of the cropped image, as shown in image 520. Therefore, if the control unit 204 finds that even one subject among all the subjects within the imaging field of view will not satisfy the user's composition settings in the cropped image, it proceeds to step S604. On the other hand, if the control unit 204 determines that the user's composition settings can be satisfied regardless of which subject's image is cropped from among all the subjects, it proceeds to step S305 as described above.
[0056] When the process proceeds to step S604, the control unit 204 checks whether the composition setting has been updated from the composition setting determined in step S601. Note that this composition setting update check process is not limited to this; for example, if an update occurs, the process may be performed as an interrupt process midway through this flowchart. If the control unit 204 determines that the composition setting has been updated, it returns to step S601; on the other hand, if it determines that it has not been updated, it proceeds to step S605.
[0057] When the process proceeds to step S605, the control unit 204 controls the drive unit 203 using PTZ control so that the imaging direction and field of view are such that cropped images satisfying the user's composition settings can be generated for all subjects. After step S605, the control unit 204 returns the process to step S302.
[0058] In the flowchart of Figure 6, after step S305, the control unit 204 proceeds to step S606. Upon reaching step S606, the control unit 204 sends a cropping command to the image processing unit 202 to generate a cropped image of the subject that matches the composition setting set by the user. As a result, the image processing unit 202 can generate a cropped image of the subject according to that composition setting. After step S606, the control unit 204 proceeds to step S307. Furthermore, in the flowchart of Figure 6, if the control unit 204 determines in step S308 that the termination flag is not set, it returns the process to step S601.
[0059] Figure 7 shows an example of a GUI (Graphical User Interface) screen 700 that can be used when the user sets the composition of the subject in the cropped image, as in the second embodiment. In this embodiment, the GUI screen 700 is shown on a monitor attached to the external device 120, but it is not limited to this, and for example, if the imaging device 100 has a built-in monitor, it may be shown on that built-in monitor.
[0060] In the GUI screen 700, the cropped image frame 701 represents the overall size of the cropped image, and the display position frame 702 represents the display position of the subject. The user can, for example, place the display position frame 702 at any position within the cropped image frame 701 by drag-and-drop operation using a mouse attached to an external device 120, and can also change the size of the frame.
[0061] Furthermore, by pressing buttons 704a to 704d displayed on the GUI screen 700, the user can easily select how the subject 703 is displayed in the cropped image, that is, how the subject 703 appears in the cropped image. In other words, buttons 704a to 704d are virtual buttons for selecting how the subject appears in the cropped image. For example, button 704a is a button for selecting a setting in the imaging device 100 to crop an image showing a close-up of the subject's face. Similarly, button 704b is a button for selecting a setting in the imaging device 100 to crop an image showing the upper body of the subject. Likewise, button 704c is a button for selecting a setting to crop an image showing the entire subject, and button 704d is a button for selecting a setting to crop an image showing the entire subject in a wide shot.
[0062] In this embodiment, buttons 704a to 704d are provided for selecting how the subject appears in the cropped image, but the system is not limited to these. The cropped image according to the composition setting may not only be one in which the subject is generally positioned in the center of the image, but may also be one in which the subject is positioned at any arbitrary position within the cropped image. Alternatively, for example, the cropped image may be one in which the size of the subject is arbitrary.
[0063] Figure 8 is a diagram used to explain the field of view and the process of generating cropped images in the second embodiment. In the example in Figure 8, as with the example in Figure 5 described above, we assume a situation where three people, each subject 111, have a wireless sub-unit 110 attached to their chest. Also, in the second embodiment, for the sake of simplicity, as with the previous description, we assume that the height from the ground of the imaging device 100 and the wireless sub-units 110 attached to the subjects 111 are all the same (the tilt direction is fixed at "0 degrees" for all), and that their positions are also the same. Furthermore, in this embodiment, we will explain assuming that the user has selected a composition setting in which the entire subject is largely visible in the center of the image, as illustrated in the GUI screen 700 of Figure 7. Also, in the second embodiment, as with the example in the first embodiment, we will give an example of switching the cropping target from a state in which subject 111a holding a wireless sub-unit 110a is selected as the cropping target to subject 111c holding a wireless sub-unit 110c.
[0064] In this embodiment, the control unit 204 of the imaging device 100 adjusts the PTZ value so that, regardless of which subject's image is being extracted from among all subjects within the imaging field of view, an extracted image with the subject in the center of the image is obtained. That is, even when, for example, an image of a subject located near the edge of the imaging field of view is being extracted, the control unit 204 adjusts the PTZ value to an imaging field of view that allows for the acquisition of an image in which an extracted image with the subject in the center of the image can be obtained.
[0065] The imaging field of view 800 in Figure 8 represents the imaging field of view corresponding to the adjusted PTZ value, and is an adjusted field of view that is wider than the imaging field of view corresponding to the image 500 in Figure 5, which is shown by the dotted line in Figure 5.
[0066] In Figure 8, image 810 represents a cropped image of subject 111a holding the wireless slave unit 110a. Image 820 represents a cropped image of subject 111c when the subject from which the cropped image is generated is switched from subject 111a holding the wireless slave unit 110c to subject 111c holding the wireless slave unit 110c. In other words, the cropped image 820, which is cropped from the image captured with the imaging angle 800 shown in Figure 8, is a cropped image in which subject 111c holding the wireless slave unit 110c is captured in the center of the image.
[0067] As described above, in the second embodiment, the imaging angle of view is adjusted according to the composition set by the user. According to this embodiment, no matter which subject's image is extracted from all the subjects, an extracted image can be obtained in which the subject is positioned according to the user's composition setting. Furthermore, according to this embodiment, even when switching the subject from which the image is extracted, the extracted image of the subject after the switch will be an extracted image in which the subject is positioned according to the user's composition setting, enabling a seamless transition without image interruption or distortion.
[0068] In this embodiment, the composition setting was explained using the example of a case where there is only one subject in the cropped image, but it is not limited to this. For example, it is also possible to set the composition to combine multiple subjects into a single cropped image. Also, for example, if there is another subject other than the subject to be cropped at the edge of the cropped image, the cropped image may be made to include that other subject.
[0069] Furthermore, while the second embodiment provides an example of adjusting the imaging angle of view to enable the acquisition of a cropped image that satisfies the composition setting, it is not limited to this. For example, if the range that can be photographed is predetermined, and it is difficult to obtain a cropped image that matches the composition setting within that range, the image cropping to match the composition setting may be omitted.
[0070] <Third Embodiment> In the third embodiment, an example of an imaging device 100 having a function to prevent image quality degradation will be described. In the first and second embodiments described above, for example, when the distance between subjects is large, the image will be zoomed out (the imaging angle of view will be widened) so that all of the multiple subjects are included in the image. However, the image extracted from the image captured with a wide angle of view will be grainy, and the extracted image may become a low-quality image.
[0071] Therefore, in the third embodiment, if a decrease in image quality of the cropped image is expected, at least one subject from among the multiple subjects that are the target of multiple subject tracking is excluded from the subjects to be included in the shooting direction and field of view for multiple subject tracking. In other words, in this embodiment, the imaging device 100 excludes subjects that are outside the specified shooting range, as described later, from the targets of multiple subject tracking. Note that the system configuration and the configuration of the imaging device 100 in the third embodiment are the same as those in Figures 1 and 2 described above, so their illustration and description are omitted.
[0072] Figure 9 is a flowchart showing the flow of control processing by the control unit 204 and image extraction processing by the image processing unit 202 according to the third embodiment. In the flowchart of Figure 9, steps S301 to S308 and steps S601 to S606 are the same as in the flowcharts of Figures 3 and 6, so their explanations are omitted. The flowchart of Figure 9 according to the third embodiment differs from the flowcharts described above in that it includes steps S901 and S902 as processing steps related to the process of excluding subjects outside the defined shooting range from the target of multiple subject tracking.
[0073] Figure 10 is a diagram used to explain an example of operation of the imaging device 100 according to the third embodiment. In the example in Figure 10, it is assumed that multiple subjects 111a to 111d (four people in the example in Figure 10) are on a stage, similar to the embodiment described above, and each has a wireless sub-unit 110a to 110d attached to their chest. In this embodiment as well, it is assumed that the imaging device 100 and each wireless sub-unit 110a to 110d are at the same height from the ground (all with a fixed tilt direction of "0 degrees"), and that the imaging device 100 is capturing images from near the front of the stage.
[0074] In the third embodiment as well, in step S301, the control unit 204 communicates with each wireless slave unit 110a to 110d via the communication unit 206 and acquires position information, which is expressed as direction and distance to each wireless slave unit 110, as described above.
[0075] In the example shown in Figure 10, the position information of wireless slave unit 110a is assumed to indicate that the angle in the pan direction relative to the reference angle of the imaging device 100 is 0 degrees and the distance is 10 m. The position information of wireless slave unit 110b is assumed to indicate that the angle in the pan direction is 28 degrees (-28°) to the left of the reference angle and the distance is 21 m. The position information of wireless slave unit 110c is assumed to indicate that the angle in the pan direction is 30 degrees (+30°) to the right of the reference angle and the distance is 14 m. Furthermore, the position information of wireless slave unit 110d is assumed to indicate that the angle in the pan direction is 45 degrees (-45°) to the left of the reference angle and the distance is 14 m.
[0076] In the third embodiment, after step S301, the control unit 204 proceeds to step S901. When the process proceeds to step S901, the control unit 204 determines, based on the position information of each wireless slave unit 110 acquired in step S301, whether they are within the specified shooting range according to this embodiment. If the control unit 204 determines that each wireless slave unit 110 is within the specified shooting range, the process proceeds to step S302. On the other hand, if it determines that they are outside the shooting range (not within the shooting range), the process proceeds to step S902.
[0077] When the process proceeds to step S902, the control unit 204 excludes the person holding the wireless sub-unit 110 that is not within the specified shooting range from the multiple subject tracking. After step S902, the control unit 204 proceeds to step S302.
[0078] In this embodiment, the defined shooting range is defined as a range where the distance between the imaging device 100 and the wireless sub-unit 110 is a certain distance (e.g., 15m) or less, or where the distance between subjects as seen from the imaging device 100, that is, the angle between each wireless sub-unit 110, is a certain angle (60°) or less. Note that these are just examples, and the certain distance is not limited to 15m, nor is the certain angle limited to 60 degrees.
[0079] In the example shown in Figure 10, the wireless slave unit 110b is 21m away from the imaging device 100, which exceeds the specified shooting range of 15m. Therefore, the control unit 204 determines that the subject 111b holding the wireless slave unit 110b is outside the specified shooting range and excludes the subject 111b from the multiple subject tracking.
[0080] In the example shown in Figure 10, the panning angle of the wireless sub-unit 110d is -45 degrees relative to the reference angle of the imaging device 100, and the panning angle of the wireless sub-unit 110c is +30 degrees. In other words, from the perspective of the imaging device 100, the wireless sub-units 110d and 110c are located 75 degrees apart, exceeding the specified shooting range of 60 degrees. In this case, the control unit 204 excludes either the subject 111d holding the wireless sub-units 110d or the subject 111c holding the wireless sub-units 110c from the multiple subject tracking. For example, the control unit 204 excludes the subject holding the wireless sub-unit with the lower imaging priority from the multiple subject tracking. The imaging priority of each wireless sub-unit 110 may be set by the user via an external device 120, or it may be predetermined. For example, if the imaging priority of wireless slave unit 110d is set lower than that of wireless slave unit 110c, the control unit 204 excludes the subject 111d holding wireless slave unit 110d from the multiple subject tracking target.
[0081] In this embodiment, the control unit 204 checks whether each of the wireless sub-units 110a to 110d is within a specified shooting range based on the position information of the wireless sub-units 110a to 110d. The control unit 204 then excludes subjects holding wireless sub-units 110 that are outside the specified shooting range from the multiple subject tracking. In other words, the control unit 204 excludes subjects holding wireless sub-units 110 that are outside a certain distance from the imaging device 100, or where the distance between each wireless sub-unit as viewed from the imaging device 100 is greater than a certain angle, from the multiple subject tracking.
[0082] Furthermore, if there are three or more wireless sub-units whose shooting range exceeds the specified 60 degrees, the control unit 204 will exclude them from tracking multiple subjects in order of their lowest imaging priority. In addition, the imaging priority of each wireless sub-unit may be changed according to its distance from the imaging device 100, or the imaging priority of wireless sub-units held by individuals with specific characteristics may be increased.
[0083] Furthermore, in this embodiment, an example was given in which, if the distance (angle) between wireless sub-units 110 as seen from the imaging device 100 is outside the specified imaging range, subjects holding wireless sub-units with a lower imaging priority are excluded from the multiple-subject tracking target. However, this is not the only example. For instance, even if the distance (angle) between wireless sub-units exceeds a certain angle, subjects holding wireless sub-units within a certain distance from the imaging device 100 may be included in the multiple-subject tracking target.
[0084] In the third embodiment, from step S302 onward, the control unit 204 performs PTZ control of the drive unit 203 based on the position information of the remaining wireless slave units 110 that have not been excluded from tracking multiple subjects, so that the subjects of those wireless slave units 110 come within the imaging field of view. The subsequent processing is the same as described above.
[0085] As explained above, in the third embodiment, subjects that may lead to a decrease in the quality of the cropped image are excluded from the target of multi-subject tracking in advance, and the imaging field of view is determined for the remaining subjects to be tracked. As a result, according to the third embodiment, it is possible to suppress the decrease in the quality of the cropped image caused by, for example, the imaging field of view becoming too wide.
[0086] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions. The embodiments described above are merely examples of how the Disclosure may be implemented, and the technical scope of the Disclosure should not be limited by them. In other words, the Disclosure can be implemented in various ways without departing from its technical concept or its main features.
[0087] This embodiment includes the following configurations, methods, and programs. (Composition 1) A control device for controlling an imaging device, A means for acquiring location information for each of multiple subjects, Based on the positional information of the plurality of subjects, an adjustment means adjusts either the imaging direction and / or field of view of the imaging device so that the plurality of subjects fall within the field of view of the imaging device, A cropping means for extracting a partial image corresponding to at least one of the multiple subjects from an image captured by the imaging device, which is adjusted so that the multiple subjects are within the field of view, A control device characterized by having the following features. (Configuration 2) The control device according to configuration 1, characterized in that the cropping means changes the subject from which the partial image is to be cropped from the captured image in accordance with instructions from the user. (Composition 3) The control device according to configuration 1 or 2, characterized in that the adjustment means excludes subjects that exceed the imaging control range of the imaging device from the plurality of subjects. (Composition 4) The control device according to configuration 3, characterized in that the adjustment means changes the imaging control range according to the user's instructions. (Composition 5) The control device according to any one of configurations 1 to 4, characterized in that the cropping means crops the partial image which includes two or more subjects. (Composition 6) The control device according to any one of configurations 1 to 5, characterized in that, if the position of the subject in the partial image cut out corresponding to any of the multiple subjects deviates from a preset position, the adjustment means adjusts either or both of the imaging direction and field of view of the imaging device so that a partial image is obtained in which the position of the subject is at the preset position. (Composition 7) The control device according to any one of configurations 1 to 6, characterized in that the adjustment means excludes at least one subject from the plurality of subjects when a decrease in the image quality of the partial image is expected. (Composition 8) The control device according to configuration 7, characterized in that the adjustment means excludes subjects whose position information is outside a specified shooting range from the plurality of subjects. (Composition 9) The control device according to configuration 8, characterized in that the shooting range is defined by the distance between the imaging device and the subject and the angle representing the distance between subjects as seen from the imaging device. (Composition 10) The control device according to configuration 9, characterized in that the adjustment means changes the distance between the imaging device and the subject and the angle representing the distance between subjects as seen from the imaging device, which define the shooting range, in accordance with the user's instructions. (Composition 11) The control device according to any one of configurations 8 to 10, characterized in that, when there are multiple subjects outside the defined shooting range, the adjustment means determines which subjects to exclude based on the priority set for the subjects outside the shooting range. (Composition 12) The control device according to any one of configurations 1 to 11, characterized in that the acquisition means acquires location information of the subject based on a wireless signal obtained by wireless communication between the subject and a wireless slave unit held by the subject. (Composition 13) The acquisition means acquires the position of the wireless slave unit based on the wireless signal, and acquires a corresponding combination of the subject and the wireless slave unit based on the position of the subject detected from the captured image and the position of the wireless slave unit. The control device according to configuration 12, characterized in that the adjustment means adjusts either or both the imaging direction and the field of view of the imaging device so that all of the wireless slave units in the combination corresponding to the subject are included. (Method 1) A control method executed by a control device that controls an imaging device, The acquisition process involves obtaining location information for each of multiple subjects, An adjustment step of adjusting either the imaging direction and / or field of view of the imaging device so that the multiple subjects are within the field of view of the imaging device, based on the positional information of the multiple subjects; A cropping step of extracting a partial image corresponding to at least one of the multiple subjects from an image captured by the imaging device, which is adjusted so that the multiple subjects are within the field of view, A control method characterized by having the following features. (Program 1) A program that causes a computer to function as a control device described in any one of configurations 1 to 13. [Explanation of symbols]
[0088] 100: Imaging device, 110: Wireless slave unit, 120: External device, 140: Network, 201: Imaging unit, 202: Image processing unit, 203: Drive unit, 204: Control unit, 205: Storage unit, 206: Communication unit, 207: Operation information acquisition unit
Claims
1. A control device for controlling an imaging device, A means for acquiring location information for each of multiple subjects, Based on the positional information of the plurality of subjects, an adjustment means adjusts either the imaging direction and / or field of view of the imaging device so that the plurality of subjects fall within the field of view of the imaging device, A cropping means for extracting a partial image corresponding to at least one of the multiple subjects from an image captured by the imaging device, which is adjusted so that the multiple subjects are within the field of view, A control device characterized by having the following features.
2. The control device according to claim 1, characterized in that the cropping means changes the subject from which the partial image is to be cropped from the captured image in accordance with instructions from the user.
3. The control device according to claim 1, characterized in that the adjustment means excludes subjects that exceed the imaging control range of the imaging device from the plurality of subjects.
4. The control device according to claim 3, characterized in that the adjustment means changes the imaging control range according to the user's instructions.
5. The control device according to claim 1, characterized in that the cropping means crops the partial image which includes two or more subjects.
6. The control device according to claim 1, characterized in that the adjustment means adjusts either or both the imaging direction and the field of view of the imaging device so that, if the position of the subject in the partial image cut out corresponding to any of the multiple subjects deviates from a preset position, a partial image is obtained in which the position of the subject becomes the preset position.
7. The control device according to claim 1, characterized in that the adjustment means excludes at least one subject from the plurality of subjects when a decrease in the image quality of the partial image is expected.
8. The control device according to claim 7, characterized in that the adjustment means excludes subjects whose position information is outside the specified shooting range from the plurality of subjects.
9. The control device according to claim 8, characterized in that the shooting range is a range defined by the distance between the imaging device and the subject and the angle representing the distance between subjects as seen from the imaging device.
10. The control device according to claim 9, characterized in that the adjustment means changes the distance between the imaging device and the subject and the angle representing the distance between subjects as seen from the imaging device, which define the shooting range, in accordance with the user's instructions.
11. The control device according to claim 8, characterized in that, when there are multiple subjects outside the defined shooting range, the adjustment means determines the subject to be excluded based on the priority set for the subjects outside the shooting range.
12. The control device according to claim 1, wherein the acquisition means acquires location information of the subject based on a wireless signal obtained by wireless communication between the subject and a wireless slave unit held by the subject.
13. The acquisition means acquires the position of the wireless slave unit based on the wireless signal, and acquires a corresponding combination of the subject and the wireless slave unit based on the position of the subject detected from the captured image and the position of the wireless slave unit. The control device according to claim 12, characterized in that the adjustment means adjusts either or both the imaging direction and the field of view of the imaging device so that all of the wireless slave units in the combination corresponding to the subject are included.
14. A control method executed by a control device that controls an imaging device, The acquisition process involves obtaining location information for each of multiple subjects, An adjustment step of adjusting either the imaging direction and / or field of view of the imaging device so that the multiple subjects are within the field of view of the imaging device, based on the positional information of the multiple subjects; A cropping step of extracting a partial image corresponding to at least one of the multiple subjects from an image captured by the imaging device, which is adjusted so that the multiple subjects are within the field of view, A control method characterized by having the following features.
15. Computers, A control device for controlling an imaging device, A means for acquiring location information for each of multiple subjects, Based on the positional information of the plurality of subjects, an adjustment means adjusts either the imaging direction and / or field of view of the imaging device so that the plurality of subjects fall within the field of view of the imaging device, A cropping means for extracting a partial image corresponding to at least one of the multiple subjects from an image captured by the imaging device, which is adjusted so that the multiple subjects are within the field of view, A program that causes a control device to function as such.
Citation Information
Patent Citations
Imaging device, control method of imaging device, and program
JP2021087059A
