Imaging system, imaging device, control device, control method, and control program

Abstract

Provided are an imaging system, an imaging device, a control device, a control method, and a control program capable of improving performance in detecting and tracking a subject.　The present invention comprises: a wide-angle camera (110) having a first detection unit (51) that detects a specific subject; a telephoto zoom camera (120) having a second detection unit (52) that detects a specific subject, the telephoto zoom camera (120) being capable of performing imaging at a focal length greater than that of the wide-angle camera (110); and a personal computer (130) having a control unit (60). The control unit (60) controls the imaging range of the telephoto zoom camera (120) on the basis of the detection result obtained by the second detection unit (52), and when the detection result obtained by the second detection unit (52) corresponds to a first condition, controls the imaging range of the telephoto zoom camera (120) on the basis of the specific-subject detection result obtained by the first detection unit (51).

Description

Imaging system, imaging device, control device, control method, and control program

[0001] The present invention relates to an imaging system, an imaging device, a control device, a control method, and a control program.

[0002] Patent Document 1 describes an imaging device that includes a multi-lens wide-angle camera that captures wide-angle images and a monocular telephoto camera that captures detailed images, and by updating the wide-angle images more frequently than the detailed images, when a moving object to be tracked is lost by the monocular telephoto camera, it prevents a delay in the timing at which the multi-lens wide-angle camera re-detects the object to be tracked, which would otherwise result in a decrease in tracking accuracy.

[0003] Patent document 2 describes a surveillance camera that is equipped with a wide-angle lens and a zoom lens that can pan and tilt, in which light incident on the wide-angle lens is constantly converted into a wide-angle video signal by a first image sensor, and when the "position" and "size" of a face are detected by a face detection unit, the pan angle, tilt angle, and zoom amount of the zoom lens are determined by a coordinate conversion unit, the zoom lens is driven based on these determined values, and face recognition processing is performed based on the zoom video signal output from a second image sensor.

[0004] Patent Document 3 describes a surveillance system that is equipped with multiple AI (Artificial Intelligence) cameras, which not only take pictures of the surveillance area but also have the function of analyzing the images based on an AI model, and the AI ​​models include a general-purpose AI model that can be used by multiple AI cameras and a dedicated AI model specialized for each AI camera, with one of the AI ​​models being set in each AI camera.

[0005] Patent Document 4 describes an information processing device that performs image processing using an imaging device (AI camera) that processes captured images using an artificial intelligence model, thereby distributing processing and reducing the amount of communication data.

[0006] Japanese Patent Publication No. 2019-080226 International Publication No. WO2018 / 109869 International Publication No. WO2022 / 059122 International Publication No. WO2023 / 090037

[0007] One embodiment of the technique of the present disclosure provides an imaging system, an imaging device, a control device, a control method, and a control program that can improve the performance of detecting and tracking a subject.

[0008] (1) An imaging system comprising: a first imaging device having a first detection unit that detects a specific subject; a second imaging device that is capable of imaging at a longer focal length than the first imaging device and has a second detection unit that detects the specific subject; and a control device having a processor, wherein the processor controls the imaging range of the second imaging device based on the detection result by the second detection unit, and when the detection result by the second detection unit satisfies a first condition, controls the imaging range based on the detection result of the specific subject by the first detection unit.

[0009] (2) The imaging system according to (1), wherein the first condition is a condition relating to whether or not the specific subject is detected.

[0010] (3) The imaging system according to (2), wherein the first condition is a condition that is satisfied when the specific subject is not detected.

[0011] (4) The imaging system according to any one of (1) to (3), wherein the second detection unit detects the specific subject in parallel with the detection of the specific subject by the first detection unit.

[0012] (5) The imaging system according to any one of (1) to (4), wherein the detection of the specific subject by the first detection unit and the second detection unit is detection by a machine learning model.

[0013] (6) The imaging system according to (5), wherein the machine learning model differs between the first detection unit and the second detection unit.

[0014] (7) The imaging system according to (6), wherein the second machine learning model of the second detection unit is a machine learning model capable of detecting a subject in more detail than the first machine learning model of the first detection unit.

[0015] (8) The imaging system according to any one of (1) to (7), further comprising a rotation device that rotates the second imaging device, and the control of the imaging range includes control of the rotation device.

[0016] (9) The imaging system according to any one of (1) to (8), wherein the second imaging device has a variable focal length for imaging, and the control of the imaging range includes control of the focal length.

[0017] (10) An imaging system according to any one of (1) to (9), wherein the processor, when the specific subject is not detected by the second detection unit after controlling the imaging range based on a first result of the detection of the specific subject by the first detection unit, controls the imaging range based on the first result and a detection result of the specific subject by the first detection unit subsequent to the first result.

[0018] (11) An imaging system according to (10), wherein, if the specific subject is not detected by the second detection unit after controlling the imaging range based on the first result, the processor controls the imaging range based on a detection result of the specific subject by the first detection unit after the first result, excluding the specific subject included in the first result.

[0019] (12) The imaging system according to any one of (1) to (11), in which the control device is provided in the second imaging device.

[0020] (13) A control device having a processor included in an imaging system including a first imaging device having a first detection unit that detects a specific subject, and a second imaging device that is capable of imaging at a longer focal length than the first imaging device and has a second detection unit that detects the specific subject in parallel with the detection of the specific subject by the first detection unit, wherein the processor controls the imaging range of the second imaging device based on the detection result by the second detection unit, and when the detection result by the second detection unit satisfies a first condition, controls the imaging range based on the detection result of the specific subject by the first detection unit.

[0021] (14) A control method by a control device having a processor included in an imaging system including a first imaging device having a first detection unit that detects a specific subject, and a second imaging device that is capable of imaging at a longer focal length than the first imaging device and has a second detection unit that detects the specific subject in parallel with the detection of the specific subject by the first detection unit, wherein the processor controls the imaging range of the second imaging device based on the detection result by the second detection unit, and when the detection result by the second detection unit satisfies a first condition, controls the imaging range based on the detection result of the specific subject by the first detection unit.

[0022] (15) A control program for a control device having a processor, which is included in an imaging system including a first imaging device having a first detection unit that detects a specific subject, and a second imaging device that is capable of imaging at a longer focal length than the first imaging device and has a second detection unit that detects the specific subject in parallel with the detection of the specific subject by the first detection unit, wherein the processor controls the imaging range of the second imaging device based on the detection result by the second detection unit, and when the detection result by the second detection unit satisfies a first condition, controls the imaging range based on the detection result of the specific subject by the first detection unit.

[0023] According to the present invention, it is possible to provide an imaging system, an imaging device, a control device, a control method, and a control program that are capable of improving the performance of detecting and tracking a subject.

[0024] 10 is a diagram showing an example of the configuration of an imaging system 100 according to an embodiment. FIG. 11 is a diagram showing an example of the external appearance of the imaging system 100. FIG. 12 is a diagram showing an example of the flow of information in the imaging system 100. FIG. 13 is a block diagram showing an example of the configuration of the electrical system of a swivel base 140 and a personal computer 130. FIG. 14 is a block diagram showing an example of the configuration of the optical system and electrical system of a telephoto zoom camera 120. FIG. 15 is a diagram showing an example of images captured by a wide-angle camera 110 and a telephoto zoom camera 120. FIG. 16 is a flowchart showing an example of processing by a control unit 60 of a personal computer 130. FIG. 17 is a diagram showing an example of a subject that can be determined by AI detection. FIG. 18 is a flowchart showing a modified example of processing by the control unit 60 of a personal computer 130. FIG. 19 is a diagram showing a modified example of a control device in the imaging system 100. FIG. 19 is a diagram showing an example of the flow of information in the imaging system 100 of the modified example shown in FIG. 10. FIG. 19 is a diagram showing a modified example of communication between a personal computer 130 and a swivel base 140. FIG. 20 is a diagram showing a modified example of how a wide-angle camera 110 is installed. FIG. 21 is a diagram showing an example of how a control program for imaging control is installed in a control unit 60 of a personal computer 130 from a storage medium on which the control program is stored.

[0025] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

[0026] <Imaging System of the Embodiment> FIG. 1 is a diagram showing an example of the configuration of an imaging system 100 of the embodiment. As shown in FIG. 1, the imaging system 100 includes a wide-angle camera 110, which is an example of the "first imaging device" of the present invention, a telephoto zoom camera 120, which is an example of the "second imaging device" of the present invention, and a personal computer (PC) 130, which is an example of the "control device" of the present invention. The imaging system 100 also includes a swivel 140 capable of rotating the telephoto zoom camera 120. The swivel 140 is an example of the "swivel device" of the present invention. The wide-angle camera 110, the telephoto zoom camera 120, and the swivel 140 are connected to the personal computer 130 via a switching hub 150, for example, via a LAN (Local Area Network). The second imaging device may be, for example, a telephoto prime camera.

[0027] FIG. 2 is a diagram showing an example of the appearance of imaging system 100 shown in FIG. 2. In imaging system 100 shown in FIG. 2, wide-angle camera 110 is installed, for example, on an indoor or outdoor pillar, a wall, or part of a building (e.g., a rooftop), and captures an image of a specific subject. The specific subject is, for example, an object to be tracked. Wide-angle camera 110 is a camera with a shorter focal length (wider angle of view) than telephoto zoom camera 120. Wide-angle camera 110 transmits detection information related to the specific subject detected by the image capture to personal computer 130 via communication line 12.

[0028] The telephoto zoom camera 120, together with the wide-angle camera 110, is installed indoors or outdoors on a pillar, a wall, or part of a building (e.g., a rooftop), and captures images of a specific subject. As with the wide-angle camera 110, the specific subject is a subject to be tracked. The telephoto zoom camera 120 is installed via a swivel base 140. The telephoto zoom camera 120 is a camera capable of capturing images at a longer focal length than the wide-angle camera 110. In this example, of the wide-angle camera 110 and the telephoto zoom camera 120, only the telephoto zoom camera 120 is attached to the swivel base 140, and the rotation of the swivel base 140 enables, for example, panning and tilting. The wide-angle camera 110 is installed in a fixed position below the telephoto zoom camera 120, as shown in FIG. 2 . The telephoto zoom camera 120 transmits detection information regarding the specific subject detected by the image capture to the personal computer 130 via the communication line 12.

[0029] The personal computer 130 includes a display 131, a keyboard 132, a mouse 133, and a secondary storage device 134. Examples of the display 131 include a liquid crystal display, a plasma display, and an organic EL (Electro-Luminescence) display.

[0030] An example of the secondary storage device 134 is a hard disk drive (HDD). The secondary storage device 134 is not limited to an HDD, and may be a non-volatile memory such as a flash memory, a solid state drive (SSD), or an electrically erasable and programmable read-only memory (EEPROM).

[0031] The personal computer 130 receives detection information relating to a specific subject transmitted from the wide-angle camera 110 and the telephoto zoom camera 120, and displays the received detection information on the display 131 or stores it in the secondary storage device 134.

[0032] The personal computer 130 performs imaging control to control imaging by the wide-angle camera 110 and the telephoto zoom camera 120. The personal computer 130 also performs rotation control to control the rotation of the telephoto zoom camera 120 by the swivel base 140.

[0033] For example, personal computer 130 performs imaging control by communicating with wide-angle camera 110 and telephoto zoom camera 120 via communication line 12. Imaging control involves setting imaging parameters for wide-angle camera 110 and telephoto zoom camera 120 to capture images, and causing wide-angle camera 110 and telephoto zoom camera 120 to capture images. The imaging parameters include parameters related to the width of the imaging range (focal length) of telephoto zoom camera 120, parameters related to the exposure of wide-angle camera 110 and telephoto zoom camera 120, and the like.

[0034] The personal computer 130 also performs rotation control by communicating with the swivel base 140 via the communication line 12. The rotation control is a control in which rotation parameters for the swivel base 140 to rotate the telephoto zoom camera 120 are set in the swivel base 140, and the swivel base 140 is caused to rotate the telephoto zoom camera 120. The rotation parameters include parameters relating to the direction (pan and tilt) of the imaging range of the telephoto zoom camera 120, and the like.

[0035] The personal computer 130 can also set the rotation direction, rotation amount, rotation speed, etc. of the telephoto zoom camera 120 in response to, for example, operation of the keyboard 132 or mouse 133 or touch operation on the screen of the display 131 .

[0036] 3 is a diagram showing an example of the flow of information in the imaging system 100. As shown in FIG. 3, the wide-angle camera 110, the telephoto zoom camera 120, and the swivel base 140 are each connected to a personal computer 130 so as to be able to communicate with each other.

[0037] Wide-angle camera 110 has a first detection unit 51 that detects a specific subject. Wide-angle camera 110 detects the specific subject using a machine learning model with first detection unit 51. Wide-angle camera 110 transmits the detection result of the specific subject by first detection unit 51 to personal computer 130.

[0038] The telephoto zoom camera 120 has a second detection unit 52 that detects a specific subject and a zoom control unit 53 that controls the zoom state of the telephoto zoom camera 120. The telephoto zoom camera 120 detects the specific subject using a machine learning model with the second detection unit 52. The telephoto zoom camera 120 transmits the detection result of the specific subject by the second detection unit 52 to the personal computer 130. The telephoto zoom camera 120 controls the zoom state of the telephoto zoom camera 120 with the zoom control unit 53 based on instructions from the personal computer 130. The telephoto zoom camera 120 transmits the zoom control state to the personal computer 130.

[0039] The swivel base 140 has a pan / tilt control unit 54 that controls the rotation state of the swivel base 140. Based on instructions from the personal computer 130, the swivel base 140 controls the rotation state of the swivel base 140 using the pan / tilt control unit 54. The swivel base 140 transmits the results of the rotation control to the personal computer 130.

[0040] The personal computer 130 has a control unit 60 that controls the operation of the entire imaging system 100. The personal computer 130 transmits a request signal (Watch) requesting the detection results of a specific subject to the wide-angle camera 110 and the telephoto zoom camera 120. The personal computer 130 transmits an instruction to the telephoto zoom camera 120 to adjust the width of the imaging range (focal length) based on the detection results of the specific subject transmitted from the wide-angle camera 110 and the telephoto zoom camera 120. The personal computer 130 also transmits an instruction to the swivel base 140 to adjust the rotation based on the detection results of the specific subject transmitted from the wide-angle camera 110 and the telephoto zoom camera 120.

[0041] If the telephoto zoom camera 120 can detect a specific subject using the second detection unit 52, the personal computer 130 controls the imaging range of the telephoto zoom camera 120 based on the detection result of the second detection unit 52. If the telephoto zoom camera 120 cannot detect a specific subject using the second detection unit 52, the personal computer 130 controls the imaging range of the telephoto zoom camera 120 based on the detection result of the specific subject using the first detection unit 51 of the wide-angle camera 110. Controlling the imaging range includes at least one of controlling the width of the imaging range (focal length) and controlling the direction of the imaging range (pan and tilt). The personal computer 130 normally processes the imaging range of the telephoto zoom camera 120 based on the detection result of the second detection unit 52 of the telephoto zoom camera 120, but if the target subject is not detected by the detection of the telephoto zoom camera 120, it processes the imaging range based on the detection result of the first detection unit 51 of the wide-angle camera 110.

[0042] The second detection unit 52 of the telephoto zoom camera 120 detects a specific subject in parallel with the detection of a specific subject by the first detection unit 51 of the wide-angle camera 110. "Parallel" does not necessarily mean that the detections are performed strictly at the same time. For example, when the imaging range is controlled based on the detection result of the specific subject by the first detection unit 51, the second detection unit 52 may detect the specific subject even during the period when the imaging range is being controlled by the first detection unit 51. Furthermore, when the imaging range is controlled based on the detection result of the specific subject by the second detection unit 52, the first detection unit 51 may detect the specific subject even during the period when the imaging range is being controlled by the second detection unit 52.

[0043] <Configuration of Electrical Systems of Swivel Base 140 and Personal Computer 130> Figure 4 is a block diagram showing an example of the configuration of the electrical systems of the swivel base 140 and personal computer 130. As shown in Figure 4, the swivel base 140 includes a yaw axis rotation mechanism 71, a pitch axis rotation mechanism 72, motors 73 and 74, drivers 75 and 76, and a communication I / F 78.

[0044] The yaw axis rotation mechanism 71 rotates the swivel base 140, to which the telephoto zoom camera 120 is attached, in the yaw direction. The motor 73 generates power by being driven under the control of a driver 75. The yaw axis rotation mechanism 71 rotates the swivel base 140, to which the telephoto zoom camera 120 is attached, in the yaw direction by receiving the power generated by the motor 73. The pitch axis rotation mechanism 72 rotates the swivel base 140, to which the telephoto zoom camera 120 is attached, in the pitch direction. The motor 74 generates power by being driven under the control of a driver 76. The pitch axis rotation mechanism 72 rotates the swivel base 140, to which the telephoto zoom camera 120 is attached, in the pitch direction by receiving the power generated by the motor 74.

[0045] The communication I / F 78 is, for example, a network interface. The communication I / F 78 controls the transmission of various information between the swivel base 140 and the personal computer 130 via the switching hub 150. This network is, for example, a LAN or a WAN. The communication I / F 78 performs communication between the swivel base 140 and the personal computer 130.

[0046] The personal computer 130 includes a display 131, a secondary storage device 134, a control unit 60, a receiving device 62, and communication I / Fs 66, 67, and 68. The control unit 60 includes a CPU 60A, a storage 60B, and a memory 60C. The CPU 60A is an example of the "processor" of the present invention.

[0047] The accepting device 62, display 131, secondary storage device 134, CPU 60A, storage 60B, memory 60C, and communication I / Fs 66 to 68 are each connected to a bus 70. Note that, for convenience of illustration, one bus is shown as the bus 70 in the example shown in Fig. 4, but multiple buses may be used. The bus 70 may be a serial bus or a parallel bus including a data bus, an address bus, a control bus, etc.

[0048] The memory 60C temporarily stores various types of information and is used as a work memory. An example of the memory 60C is a RAM, but it is not limited to this and other types of storage devices may be used. The storage 60B stores various programs for the personal computer 130 (hereinafter simply referred to as "personal computer programs").

[0049] The CPU 60A reads the personal computer program from the storage 60B and executes the read personal computer program on the memory 60C, thereby controlling the entire personal computer 11. The personal computer program includes the "control program" of the present invention.

[0050] The communication I / Fs 66, 67 are, for example, network interfaces. The communication I / F 66 is communicatively connected to a communication I / F (not shown) of the wide-angle camera 110 via the switching hub 150, and controls the transmission of various information between the wide-angle camera 110 and the wide-angle camera 110. The communication I / F 67 is communicatively connected to a communication I / F 34 of the telephoto zoom camera 120 via the switching hub 150, and controls the transmission of various information between the telephoto zoom camera 120 and the wide-angle camera 110. The communication I / F 68 is, for example, a network interface. The communication I / F 68 is communicatively connected to a communication I / F 78 of the swivel base 140 via the switching hub 150, and controls the transmission of various information between the yaw axis rotation mechanism 71 and the pitch axis rotation mechanism 72.

[0051] CPU 60A receives detection information and the like relating to a specific subject from wide-angle camera 110 via communication I / F 66. CPU 60A receives detection information and the like relating to a specific subject from telephoto zoom camera 120 via communication I / F 67 and communication I / F 34. CPU 60A also controls the rotation operation of yaw axis rotation mechanism 71 by controlling driver 75 and motor 73 of swivel base 140 via communication I / F 68 and communication I / F 78, and controls the rotation operation of pitch axis rotation mechanism 72 by controlling driver 76 and motor 74 of swivel base 140.

[0052] Reception device 62 is, for example, keyboard 132, mouse 133, or the touch panel of display 131, and receives various instructions from the user. CPU 60A acquires the various instructions received by reception device 62 and operates in accordance with the acquired instructions. For example, when reception device 62 receives processing content for wide-angle camera 110, telephoto zoom camera 120, and swivel head 140, CPU 60A operates wide-angle camera 110, telephoto zoom camera 120, and swivel head 140 in accordance with the instruction content received by reception device 62.

[0053] The display 131 displays various information under the control of the CPU 60A. Examples of the various information displayed on the display 131 include the contents of various instructions accepted by the accepting device 62 and detection information related to a specific subject received by the communication I / Fs 66 and 67. The CPU 60A causes the display 131 to display the contents of various instructions accepted by the accepting device 62 and the detection information related to a specific subject received by the communication I / Fs 66 and 67.

[0054] The secondary storage device 134 is, for example, a non-volatile memory, and stores various types of information under the control of the CPU 60 A. Examples of the various types of information stored in the secondary storage device 134 include detection information about a specific subject received by the communication I / Fs 66 and 67. The CPU 60 A stores the detection information about the specific subject received by the communication I / Fs 66 and 67 in the secondary storage device 134.

[0055] <Configuration of Optical System and Electrical System of Telephoto Zoom Camera 120> FIG. 5 is a block diagram showing an example of the configuration of the optical system and electrical system of the telephoto zoom camera 120. As shown in FIG. 5, the telephoto zoom camera 120 includes an optical system 15 and an image sensor 25. The image sensor 25 is located after the optical system 15. The optical system 15 includes an objective lens 15A and a lens group 15B. The objective lens 15A and the lens group 15B are arranged in this order along the optical axis OA of the optical system 15 from the target subject side (object side) to the light receiving surface 25A side (image side) of the image sensor 25. The lens group 15B includes an anti-vibration lens 15B1, a focus lens (not shown), a zoom lens 15B2, and the like. The zoom lens 15B2 is supported by a lens actuator 21 (described later) so as to be movable along the optical axis OA. The vibration-proof lens 15B1 is supported by a lens actuator 17 (to be described later) so as to be movable in a direction perpendicular to the optical axis OA.

[0056] By increasing the focal length with zoom lens 15B2, telephoto zoom camera 120 is set to the telephoto side, resulting in a smaller angle of view (narrower imaging range). By decreasing the focal length with zoom lens 15B2, telephoto zoom camera 120 is set to the wide-angle side, resulting in a larger angle of view (wider imaging range).

[0057] The optical system 15 may include various lenses (not shown) in addition to the objective lens 15A and the lens group 15B. The optical system 15 may also include an aperture. The positions of the lenses, lens group, and aperture included in the optical system 15 are not limited, and the technology of the present disclosure is valid even if they are located at positions different from those shown in FIG. 5 .

[0058] The vibration-proof lens 15B1 is movable in a direction perpendicular to the optical axis OA, and the zoom lens 15B2 is movable along the optical axis OA.

[0059] The optical system 15 includes lens actuators 17 and 21. The lens actuator 17 applies a force to the vibration-proof lens 15B1 that fluctuates in a direction perpendicular to the optical axis of the vibration-proof lens 15B1. The lens actuator 17 is controlled by an OIS (Optical Image Stabilizer) driver 23. When the lens actuator 17 is driven under the control of the OIS driver 23, the position of the vibration-proof lens 15B1 fluctuates in a direction perpendicular to the optical axis OA.

[0060] The lens actuator 21 applies a force to the zoom lens 15B2 to move it along the optical axis OA of the optical system 15. The lens actuator 21 is controlled by a lens driver 28. When the lens actuator 21 is driven under the control of the lens driver 28, the position of the zoom lens 15B2 moves along the optical axis OA. When the position of the zoom lens 15B2 moves along the optical axis OA, the focal length of the telephoto zoom camera 120 changes.

[0061] In addition, if the outline of the captured image is, for example, a rectangle with a short side in the pitch axis PA direction (see Figure 2) and a long side in the yaw axis YA direction (see Figure 2), the angle of view in the pitch axis PA direction is narrower than the angle of view in the yaw axis YA direction and is narrower than the angle of view of the diagonal.

[0062] By the optical system 15 configured in this manner, light representing the imaging area is focused on the light receiving surface 25A of the imaging element 25, and the imaging area is imaged by the imaging element 25.

[0063] Incidentally, vibrations imparted to the telephoto zoom camera 120 include, if outdoors, vibrations caused by passing cars, vibrations caused by wind, vibrations caused by road construction, etc., and if indoors, vibrations caused by the operation of an air conditioner, vibrations caused by people entering and exiting a building, etc. Therefore, the telephoto zoom camera 120 shakes due to vibrations imparted to the telephoto zoom camera 120 (hereinafter simply referred to as "vibrations").

[0064] In this embodiment, "shake" refers to a phenomenon in which the image of a target subject on the light-receiving surface 25A of the image sensor 25 in the telephoto zoom camera 120 fluctuates due to a change in the positional relationship between the optical axis OA and the light-receiving surface 25A. In other words, "shake" can also be said to be a phenomenon in which the optical image formed on the light-receiving surface 25A fluctuates due to the tilt of the optical axis OA caused by vibrations applied to the telephoto zoom camera 120. The fluctuation of the optical axis OA means, for example, that the optical axis OA is tilted with respect to a reference axis (e.g., the optical axis OA before the shake occurs). Hereinafter, shake caused by vibrations will also be simply referred to as "shake."

[0065] Shake is included in the captured image as a noise component and affects the image quality of the captured image. Therefore, in order to remove the noise component included in the captured image due to shake, the telephoto zoom camera 120 is equipped with a lens-side shake correction mechanism 29, an image sensor-side shake correction mechanism 45, and an electronic shake correction unit 33, which are used to correct shake.

[0066] The lens-side shake correction mechanism 29 and the image sensor-side shake correction mechanism 45 are mechanical shake correction mechanisms that apply power generated by a drive source such as a motor (for example, a voice coil motor) to a shake correction element (for example, the vibration-proof lens 15B1 and / or the image sensor 25) to move the shake correction element in a direction perpendicular to the optical axis of the imaging optical system, thereby correcting shake.

[0067] Specifically, the lens-side image stabilization mechanism 29 applies power generated by a drive source such as a motor (e.g., a voice coil motor) to the image stabilization lens 15B1, thereby moving the image stabilization lens 15B1 in a direction perpendicular to the optical axis of the imaging optical system, thereby correcting shake. The image sensor-side image stabilization mechanism 45 applies power generated by a drive source such as a motor (e.g., a voice coil motor) to the image sensor 25, thereby moving the image sensor 25 in a direction perpendicular to the optical axis of the imaging optical system, thereby correcting shake. The electronic image stabilization unit 33 corrects shake by performing image processing on the captured image based on the amount of shake. In other words, the image stabilization unit (image stabilization component) performs shake correction mechanically or electronically using a hardware and / or software configuration. Here, mechanical shake correction refers to shake correction achieved by mechanically moving shake correction elements such as the anti-vibration lens 15B1 and / or the image sensor 25 using power generated by a driving source such as a motor (e.g., a voice coil motor), and electronic shake correction refers to shake correction achieved, for example, by image processing performed by a processor.

[0068] As an example, as shown in FIG. 5, the lens-side shake correction mechanism 29 includes an anti-vibration lens 15B1, a lens actuator 17, an OIS driver 23, and a position sensor 39.

[0069] Various well-known methods can be used to correct shake using the lens-side shake correction mechanism 29. In this embodiment, the method used is to correct shake by moving the vibration-proof lens 15B1 based on the amount of shake detected by a shake amount detection sensor 40 (described later). Specifically, shake correction is performed by moving the vibration-proof lens 15B1 in a direction that cancels out the shake and by an amount that cancels out the shake.

[0070] A lens actuator 17 is attached to the vibration-proof lens 15B1. The lens actuator 17 is a shift mechanism equipped with a voice coil motor, and by driving the voice coil motor, the vibration-proof lens 15B1 is moved in a direction perpendicular to the optical axis of the vibration-proof lens 15B1. Note that, although a shift mechanism equipped with a voice coil motor is used as the lens actuator 17 here, the technology of the present disclosure is not limited to this, and other power sources such as a stepping motor or a piezoelectric element may be used instead of the voice coil motor.

[0071] The lens actuator 17 is controlled by an OIS driver 23. When the lens actuator 17 is driven under the control of the OIS driver 23, the position of the vibration-proof lens 15B1 is mechanically changed within a two-dimensional plane perpendicular to the optical axis OA.

[0072] The position sensor 39 detects the current position of the vibration-proof lens 15B1 and outputs a position signal indicating the detected current position. Here, a device including a Hall element is used as an example of the position sensor 39. Here, the current position of the vibration-proof lens 15B1 refers to the current position within a two-dimensional plane of the vibration-proof lens. The vibration-proof lens two-dimensional plane refers to a two-dimensional plane perpendicular to the optical axis of the vibration-proof lens 15B1. Note that in this embodiment, a device including a Hall element is used as an example of the position sensor 39, but the technology of the present disclosure is not limited to this, and a magnetic sensor, a photosensor, or the like may be used instead of the Hall element.

[0073] The lens-side image stabilization mechanism 29 corrects shake by moving the vibration-proof lens 15B1 along at least one of the pitch axis PA direction and the yaw axis YA direction within the range that is actually captured. In other words, the lens-side image stabilization mechanism 29 corrects shake by moving the vibration-proof lens 15B1 within the two-dimensional vibration-proof lens plane by an amount corresponding to the amount of shake.

[0074] The image sensor side shake correction mechanism 45 includes the image sensor 25 , a BIS (Body Image Stabilizer) driver 22 , an image sensor actuator 27 , and a position sensor 47 .

[0075] As with the shake correction method used by the lens-side shake correction mechanism 29, various well-known methods can be used to correct shake using the image sensor-side shake correction mechanism 45. In this embodiment, the shake correction method used is to correct shake by moving the image sensor 25 based on the amount of shake detected by the shake amount detection sensor 40. Specifically, shake correction is performed by moving the image sensor 25 in a direction that cancels out the shake and by an amount that cancels out the shake.

[0076] An imaging element actuator 27 is attached to the imaging element 25. The imaging element actuator 27 is a shift mechanism equipped with a voice coil motor, and by driving the voice coil motor, the imaging element 25 is moved in a direction perpendicular to the optical axis of the vibration-proof lens 15B1. Note that, although a shift mechanism equipped with a voice coil motor is used as the imaging element actuator 27 here, the technology of the present disclosure is not limited to this, and other power sources such as a stepping motor or a piezoelectric element may be applied instead of the voice coil motor.

[0077] The imaging element actuator 27 is controlled by the BIS driver 22. When the imaging element actuator 27 is driven under the control of the BIS driver 22, the position of the imaging element 25 is mechanically moved in a direction perpendicular to the optical axis OA.

[0078] The position sensor 47 detects the current position of the image sensor 25 and outputs a position signal indicating the detected current position. Here, a device including a Hall element is used as an example of the position sensor 47. Here, the current position of the image sensor 25 refers to the current position within a two-dimensional plane of the image sensor. The two-dimensional plane of the image sensor refers to a two-dimensional plane perpendicular to the optical axis of the vibration-proof lens 15B1. Note that in this embodiment, a device including a Hall element is used as an example of the position sensor 47, but the technology of the present disclosure is not limited to this, and a magnetic sensor, a photosensor, or the like may be used instead of the Hall element.

[0079] The telephoto zoom camera 120 includes a computer 19, a DSP (Digital Signal Processor) 31, an image memory 32, an electronic shake correction unit 33, a communication I / F 34, a shake amount detection sensor 40, and a UI (User Interface) device 43. The computer 19 includes a memory 35, a storage 36, and a CPU (Central Processing Unit) 37. The telephoto zoom camera 120 detects a specific subject using a machine learning model with a processor within the computer 19. The processor may be, for example, the CPU 37 or a separate processor.

[0080] The image sensor 25, DSP 31, image memory 32, electronic shake correction unit 33, communication I / F 34, memory 35, storage 36, CPU 37, shake amount detection sensor 40, and UI device 43 are connected to a bus 38. The OIS driver 23 is also connected to the bus 38. Note that, although one bus is shown as the bus 38 in the example shown in Fig. 5 for convenience of illustration, multiple buses may be used. The bus 38 may be a serial bus or a parallel bus such as a data bus, an address bus, and a control bus.

[0081] The memory 35 temporarily stores various types of information and is used as a work memory. An example of the memory 35 is a random access memory (RAM), but other types of storage devices may also be used. The storage 36 stores various programs for the telephoto zoom camera 120. The CPU 37 reads the various programs from the storage 36 and executes them on the memory 35 to control the entire telephoto zoom camera 120. Examples of the storage 36 include flash memory, SSD, EEPROM, and HDD. Furthermore, various types of non-volatile memory, such as magnetoresistive memory and ferroelectric memory, may be used instead of or in combination with flash memory.

[0082] The imaging element 25 is a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging element 25 captures an image of a target subject at a predetermined frame rate under the direction of the CPU 37. The "predetermined frame rate" here refers to, for example, several tens to several hundreds of frames per second. The imaging element 25 itself may have a built-in control device (image sensor control device), in which case the image sensor control device performs detailed control of the imaging element 25 in accordance with imaging instructions output by the CPU 37. The imaging element 25 may also capture an image of a target subject at a predetermined frame rate under the direction of the DSP 31, in which case the image sensor control device performs detailed control of the imaging element 25 in accordance with imaging instructions output by the DSP 31. The DSP 31 is sometimes referred to as an ISP (Image Signal Processor).

[0083] The light receiving surface 25A of the image sensor 25 is formed by a plurality of photosensitive pixels (not shown) arranged in a matrix. In the image sensor 25, each photosensitive pixel is exposed to light, and photoelectric conversion is performed for each photosensitive pixel. The electric charge obtained by photoelectric conversion for each photosensitive pixel is an analog image signal representing the target subject. Here, a plurality of photoelectric conversion elements (e.g., photoelectric conversion elements with color filters) sensitive to visible light are used as the photosensitive pixels. In the image sensor 25, a plurality of photoelectric conversion elements are used, including a photoelectric conversion element sensitive to R (red) light (e.g., a photoelectric conversion element with an R filter corresponding to R), a photoelectric conversion element sensitive to G (green) light (e.g., a photoelectric conversion element with a G filter corresponding to G), and a photoelectric conversion element sensitive to B (blue) light (e.g., a photoelectric conversion element with a B filter corresponding to B). In the telephoto zoom camera 120, these photosensitive pixels are used to capture images based on visible light (e.g., light on the short wavelength side of approximately 700 nanometers or less). However, this embodiment is not limited to this, and images may also be captured based on infrared light (e.g., light on the long wavelength side of approximately 700 nanometers). In this case, a plurality of photoelectric conversion elements sensitive to infrared light may be used as the photosensitive pixels. In particular, for capturing images of short-wavelength infrared (SWIR), an InGaAs sensor and / or a Type-II Quantum Well (T2SL: Simulation of Type-II Quantum Well) sensor may be used, for example.

[0084] The image sensor 25 performs signal processing such as A / D (Analog / Digital) conversion on the analog image signal to generate a digital image, which is a digital image signal. The image sensor 25 is connected to the DSP 31 via a bus 38, and outputs the generated digital image to the DSP 31 via the bus 38 in frame units.

[0085] Note that, although a CMOS image sensor is described here as an example of the image sensor 25, the technology of the present disclosure is not limited thereto, and a CCD (Charge Coupled Device) image sensor may also be applied as the image sensor 25. In this case, the image sensor 25 is connected to the bus 38 via an AFE (Analog Front End) (not shown) that has a built-in CCD driver. The AFE generates a digital image by performing signal processing such as A / D conversion on the analog image signal obtained by the image sensor 25, and outputs the generated digital image to the DSP 31. The CCD image sensor is driven by a CCD driver built into the AFE. Of course, the CCD driver may also be provided separately.

[0086] The DSP 31 performs various types of digital signal processing on the digital image. The various types of digital signal processing refer to, for example, demosaic processing, noise removal processing, gradation correction processing, color correction processing, etc. The DSP 31 outputs the digital image after digital signal processing to the image memory 32 for each frame. The image memory 32 stores the digital image from the DSP 31.

[0087] The shake amount detection sensor 40 is a device that includes, for example, a gyro sensor, and detects the amount of shake of the telephoto zoom camera 120. In other words, the shake amount detection sensor 40 detects the amount of shake in each of a pair of axial directions. The gyro sensor detects the amount of rotational shake around each of the pitch axis PA, the yaw axis YA, and the roll axis RA (an axis parallel to the optical axis OA) (see FIG. 2 ). The shake amount detection sensor 40 detects the amount of shake of the telephoto zoom camera 120 by converting the amount of rotational shake around the pitch axis PA and the amount of rotational shake around the yaw axis YA detected by the gyro sensor into the amount of shake in a two-dimensional plane parallel to the pitch axis PA and the yaw axis YA.

[0088] Here, a gyro sensor is used as an example of the shake amount detection sensor 40, but this is merely an example, and the shake amount detection sensor 40 may also be an acceleration sensor. The acceleration sensor detects the amount of shake in a two-dimensional plane parallel to the pitch axis PA and the yaw axis YA. The shake amount detection sensor 40 outputs the detected amount of shake to the CPU 37.

[0089] Furthermore, although an example in which the amount of shake is detected by a physical sensor, namely, the shake amount detection sensor 40, is given here, the technology of the present disclosure is not limited to this. For example, a motion vector obtained by comparing successive captured images stored in the image memory 32 in chronological order may be used as the amount of shake. Furthermore, the amount of shake that is ultimately used may be derived based on the amount of shake detected by the physical sensor and the motion vector obtained by image processing.

[0090] The CPU 37 acquires the amount of shake detected by the shake amount detection sensor 40, and controls the lens-side shake compensation mechanism 29, the image sensor-side shake compensation mechanism 45, and the electronic shake compensation unit 33 based on the acquired amount of shake. The amount of shake detected by the shake amount detection sensor 40 is used for shake compensation by each of the lens-side shake compensation mechanism 29 and the electronic shake compensation unit 33.

[0091] The electronic shake correction unit 33 is a device including an ASIC (Application Specific Integrated Circuit). The electronic shake correction unit 33 corrects shake by performing image processing on the captured image in the image memory 32 based on the amount of shake detected by the shake amount detection sensor 40.

[0092] Note that, although a device including an ASIC is exemplified as the electronic shake correction unit 33 here, the technology of the present disclosure is not limited to this. For example, the electronic shake correction unit 33 may be a device including a field programmable gate array (FPGA) or a programmable logic device (PLD). Furthermore, for example, the electronic shake correction unit 33 may be a device including multiple ASICs, FPGAs, and PLDs. Furthermore, a computer including a CPU, storage, and memory may be employed as the electronic shake correction unit 33. There may be a single CPU or multiple CPUs. Furthermore, the electronic shake correction unit 33 may be realized by a combination of hardware and software configurations.

[0093] The communication I / F 34 is, for example, a network interface, and controls the transmission of various information between the telephoto zoom camera 120 and the personal computer 130 via a network. This network may be, for example, a wide area network (WAN) such as the Internet, or a local area network (LAN). The communication I / F 34 performs communication between the telephoto zoom camera 120 and the personal computer 130.

[0094] The UI device 43 includes a reception device 43A and a display 43B. The reception device 43A is, for example, a hard key or a touch panel, and receives various instructions from the user. The CPU 37 acquires the various instructions received by the reception device 43A and operates in accordance with the acquired instructions.

[0095] The display 43B displays various information under the control of the CPU 37. Examples of the various information displayed on the display 43B include the contents of various instructions accepted by the accepting device 43A, captured images, and the like.

[0096] Although not shown, the configuration of wide-angle camera 110 is similar to that of telephoto zoom camera 120 shown in Fig. 5, except that optical system 15 has wider-angle optical characteristics. Also, wide-angle camera 110 may omit the zoom function and image stabilization function.

[0097] <Captured Image of Subject> Figure 6 is a diagram showing an example of an image captured by the wide-angle camera 110 and the telephoto zoom camera 120. As shown in Figure 6, the wide-angle camera 110 and the telephoto zoom camera 120 capture images in parallel to detect a specific drone 10 that is a target subject.

[0098] As shown in the images captured by wide-angle camera 110, drone 10 flies while moving its position irregularly. For this reason, drone 10 is detected (tracked) at times t=t0, t1, t2, and t3 in the images captured by wide-angle camera 110, but is not tracked at times t=t0 and t2 among times t=t0, t1, t2, and t3 in the images captured by telephoto zoom camera 120 (it is outside the imaging angle of view).

[0099] Even while the drone 10 is being imaged by the telephoto zoom camera 120, the drone 10 continues to be detected / tracked in the image captured by the wide-angle camera 110, making it possible to confirm the movement (position, speed, direction of movement, etc.) of the drone 10. This makes it possible to use the detection result of the drone 10 by the wide-angle camera 110 to enable the telephoto zoom camera 120 to detect the drone 10 again (for example, t = t1, t3) even if it becomes impossible to track the drone 10 in the image captured by the telephoto zoom camera 120 (for example, t = t0, t2).

[0100] <Processing by the control unit 60 of the personal computer 130> Figure 7 is a flowchart showing an example of processing by the control unit 60 of the personal computer 130. For example, the processing for detecting a specific drone invading a predetermined area as a target subject will be described below. The specific drone is assumed to be, for example, a drone manufactured by a specific manufacturer. The control unit 60 of the personal computer 130 starts the processing shown in Figure 7 when, for example, a detection start button is pressed to start subject detection.

[0101] First, the control unit 60 acquires the AI ​​detection results of the telephoto zoom camera 120 from the telephoto zoom camera 120 (step S11). The AI ​​detection results are detection results obtained using a machine learning model. The control unit 60 acquires the results of the AI ​​detection process performed by the telephoto zoom camera 120 from the telephoto zoom camera 120. The telephoto zoom camera 120 repeatedly performs AI detection using the machine learning model.

[0102] Next, the control unit 60 determines whether a specific drone, which is the target subject, has been detected in the AI ​​detection result of the telephoto zoom camera 120 acquired in step S11 (step S12). The condition of whether a specific drone (subject) has been detected is an example of the "first condition" of the present invention.

[0103] In step S12, if a specific drone that is the target subject is detected (step S12: Yes), the control unit 60 determines the zoom position (focal length) of the telephoto zoom camera 120 and the movement amount (pan angle and tilt angle) of the swivel base 140 based on the AI ​​detection result of the telephoto zoom camera 120 (step S13). Note that the AI ​​detection by the telephoto zoom camera 120 can even determine that the subject is a drone made by a specific manufacturer, for example.

[0104] Next, the control unit 60 controls the zoom position of the telephoto zoom camera 120 based on the determined zoom position (step S14). The control unit 60 controls the width of the imaging range (focal length) of the telephoto zoom camera 120, for example.

[0105] Next, the control unit 60 controls the rotation position of the swivel base 140 to which the telephoto zoom camera 120 is attached based on the determined amount of movement of the swivel base 140 (step S15). The control unit 60 controls, for example, the direction (pan direction and tilt direction) of the imaging range of the telephoto zoom camera 120.

[0106] Next, the control unit 60 determines whether or not a termination condition for terminating subject detection has been met (step S16). If the termination condition has not been met in step S16, the control unit 60 returns to step S11 and executes each process. If the termination condition has been met in step S16, the control unit 60 terminates the subject detection process. The termination condition is, for example, a condition in which a detection stop button for terminating subject detection has been pressed.

[0107] On the other hand, if a specific drone that is the target subject is not detected in step S12 (step S12: No), the control unit 60 acquires the AI ​​detection result of the wide-angle camera 110 from the wide-angle camera 110 (step S17). The AI ​​detection result is a detection result using a machine learning model by the first detection unit 51 of the wide-angle camera 110. The control unit 60 acquires the result of the AI ​​detection process performed by the wide-angle camera 110 from the wide-angle camera 110. The control unit 60 normally processes the imaging range of the telephoto zoom camera 120 using the AI ​​detection result by the second detection unit 52 of the telephoto zoom camera 120. However, if the target subject is not detected by the AI ​​detection of the telephoto zoom camera 120, the control unit 60 processes the imaging range of the telephoto zoom camera 120 using the AI ​​detection result of the first detection unit 51 of the wide-angle camera 110. The wide-angle camera 110 repeatedly performs AI detection using a machine learning model. The condition that a specific drone (subject) is not detected is an example of the "first condition" of the present invention.

[0108] Next, the control unit 60 determines whether or not a drone, which is the target subject, has been detected in the AI ​​detection results of the wide-angle camera 110 obtained in step S17 (step S18).

[0109] In step S18, if a drone is not detected (step S18: No), the control unit 60 returns to step S11 and executes each process.

[0110] If a drone is detected in step S18 (step S18: Yes), the control unit 60 determines the zoom position (focal length) of the telephoto zoom camera 120 and the movement amount (pan angle and tilt angle) of the swivel base 140 based on the AI ​​detection result of the wide-angle camera 110 (step S19). However, the target subject that can be detected from the AI ​​detection result of the wide-angle camera 110 is, for example, a determination of whether the subject is a drone or not, and cannot determine whether the subject is a drone manufactured by a specific manufacturer or not. The machine learning models of the first detection unit 51 and the second detection unit 52 are machine learning models with different content. The machine learning model of the second detection unit 52 is a machine learning model that is capable of detecting subjects in more detail than the machine learning model of the first detection unit 51. Note that, in determining the imaging range of the telephoto zoom camera 120, the control unit 60 is assumed to be able to recognize the correspondence between the coordinates of the imaging area of ​​the wide-angle camera 110 in the imaging space and the pan and tilt states of the movement amount of the swivel base 140.

[0111] Next, the control unit 60 proceeds to step S14 and executes the subsequent processes.

[0112] In the above example, the target subject is a single individual (e.g., a specific drone), but is not limited to this. For example, the target subject may be one or more individual individuals (e.g., any type of drone).

[0113] <Subject Determination by AI Detection> FIG. 8 is a diagram showing an example of a subject that can be determined by AI detection. When a subject is detected by AI detection using the first detection unit 51 in the wide-angle camera 110, it is possible to determine whether the detected subject is a drone, a person, a vehicle, or the like, as shown in FIG. 8 . In contrast, when a subject is detected by AI detection using the second detection unit 52 in the telephoto zoom camera 120, it is possible to make a detailed determination, such as whether the drone detected as the subject is a drone manufactured by company A, a drone manufactured by company B, or a drone manufactured by company C. Furthermore, when the subject is a person, it is possible to determine whether the person is male or female, or whether the person is a child or an adult. Furthermore, when the subject is a car, it is possible to determine whether the car is manufactured by company X, company Y, or company Z, and even whether the car is a passenger car or a truck.

[0114] As described above, imaging system 100 of this embodiment normally controls the imaging range of telephoto zoom camera 120 based on the detection result of a specific subject by second detection unit 52 of telephoto zoom camera 120, and if second detection unit 52 is unable to detect a specific subject, controls the imaging range of telephoto zoom camera 120 based on the detection result of a specific subject by first detection unit 51 of wide-angle camera 110. That is, in imaging system 100, subject detection processing based on images captured by telephoto zoom camera 120 and subject detection processing based on images captured by wide-angle camera 110 are independently performed by the detection unit of telephoto zoom camera 120 and the detection unit of wide-angle camera 110, and each detection result is transmitted to personal computer 130, which controls the imaging range of telephoto zoom camera 120 based on the detection result. With this configuration, even if the performance of control unit 60 in personal computer 130 is not particularly high, it is possible for each detection unit to perform subject detection based on the image captured by telephoto zoom camera 120 and subject detection based on the image captured by wide-angle camera 110 in parallel. This allows control unit 60 of personal computer 130 to shorten the time required to switch between control based on the subject detection result based on the image captured by telephoto zoom camera 120 and control based on the subject detection result based on the image captured by wide-angle camera 110, thereby improving the subject detection and tracking performance.

[0115] 9 is a flowchart showing a modified example of the processing by the control unit 60 of the personal computer 130. The processing by the control unit 60 according to this modified example differs from the processing by the control unit 60 according to the embodiment described in FIG.

[0116] As shown in Fig. 9, in this modified example, the processes from step S11 to step S16 are the same as the processes from step S11 to step S16 in the embodiment described in Fig. 7. Therefore, the description of steps S11 to S16 will be omitted.

[0117] In step S12, if a specific drone that is the target subject is not detected (step S12: No), in this modification, control unit 60 acquires the AI ​​detection results of wide-angle camera 110 from wide-angle camera 110, and if the detected subjects in the acquired AI detection results include a most recently stored subject, excludes that subject from the detected subjects (step S17). The most recently stored subject is a subject that was detected as a target subject in the AI ​​detection results acquired in the previous processing of step S17 and stored in step S19A described below.

[0118] Next, the control unit 60 determines whether or not a drone, which is the target subject, has been detected in the AI ​​detection result of the wide-angle camera 110 in step S17 (step S18). If a drone has not been detected in step S18 (step S18: No), the control unit 60 returns to step S11 and executes each process.

[0119] If a drone is detected in step S18 (step S18: Yes), the control unit 60 stores drone information about the detected subject in the secondary storage device 134 (step S19A). For example, assume that three drones (drones A, B, and C) are detected in the AI ​​detection results of the wide-angle camera 110, and the likelihood that these drones are a specific drone is drone A: 70%, drone B: 50%, and drone C: 30%. In this case, the control unit 60 first stores information about drone A, which has the highest likelihood, in the secondary storage device 134. Therefore, in the next processing of step S17, drone A is excluded from the detected subjects.

[0120] Next, the control unit 60 determines the zoom position (focal length) of the telephoto zoom camera 120 and the movement amount (pan angle and tilt angle) of the swivel base 140 based on the AI ​​detection results of the wide-angle camera 110 (step S19B), and proceeds to step S14 to execute each subsequent process.

[0121] In this modified example, if the control unit 60 controls the imaging range of the telephoto zoom camera 120 based on an AI detection result in which a target subject is detected by the first detection unit 51 of the wide-angle camera 110, and then the telephoto zoom camera 120 fails to detect the target subject by the second detection unit 52, the control unit 60 controls the imaging range of the telephoto zoom camera 120 based on the AI ​​detection result of the target subject by the first detection unit 51 of the wide-angle camera 110 subsequent to the AI ​​detection result of the wide-angle camera 110 and the AI ​​detection result of the wide-angle camera 110. Specifically, the control unit 60 controls the imaging range of the telephoto zoom camera 120 based on the AI ​​detection result of the target subject by the first detection unit 51 of the wide-angle camera 110 subsequent to the AI ​​detection result of the wide-angle camera 110, excluding the target subject included in the AI ​​detection result of the wide-angle camera 110.

[0122] "After the AI ​​detection result of wide-angle camera 110" means immediately after AI detection is performed. "After controlling the imaging range of telephoto zoom camera 120 based on the AI ​​detection result in which first detection unit 51 of wide-angle camera 110 detected a target subject, the telephoto zoom camera 120 is unable to detect the target subject using second detection unit 52" means a case in which a specific subject is not detected in detection based on a telephoto image immediately after controlling the imaging range based on the detection result in the wide-angle image (a case in which a specific subject is erroneously detected in detection based on the wide-angle image).

[0123] That is, because the AI ​​detection by the first detection unit 51 of the wide-angle camera 110 is highly abstract (has low detection accuracy), it may erroneously detect a non-target subject as a target subject. In such a case, even if the imaging range (focal length, pan angle, and tilt angle) of the telephoto zoom camera 120 is determined based on the erroneous AI detection result by the first detection unit 51 of the wide-angle camera 110 and AI detection is performed by the second detection unit 52 of the telephoto zoom camera 120, the target subject may not be found. In this case, the imaging range (focal length, pan angle, and tilt angle) of the telephoto zoom camera 120 is determined again based on the AI ​​detection result by the first detection unit 51 of the wide-angle camera 110, which would result in the same subject being erroneously detected again. Therefore, to avoid this, when determining the imaging range of the telephoto zoom camera 120 next time, the target subject that was included in the most recent AI detection result of the wide-angle camera 110 is excluded from the subjects included in the next AI detection result of the wide-angle camera 110.

[0124] As described above, according to the modified example of processing by control unit 60, if a subject detected by AI detection of wide-angle camera 110 is a non-target subject, the imaging range of telephoto zoom camera 120 can be determined based on the next detection result in which the non-target subject is excluded from the subjects detected by AI detection of wide-angle camera 110. Therefore, control unit 60 can accurately detect a specific subject based on the detection result of wide-angle camera 110, and improve its performance in tracking it.

[0125] <Modification of the Control Device in the Imaging System 100> Fig. 10 is a diagram showing a modification of the control device in the imaging system 100. In the imaging system 100 of the above embodiment, a personal computer 130 is used as the control device, but this is not limiting. For example, as shown in Fig. 10, a telephoto zoom camera 120 may be used as the control device. In this system configuration, the personal computer 130 does not need to be provided.

[0126] Fig. 11 is a diagram showing an example of the flow of information in the modified imaging system 100 shown in Fig. 10. As shown in Fig. 11, the wide-angle camera 110 and the swivel base 140 are each communicably connected to the telephoto zoom camera 120, which functions as a control device for the imaging system 100, via a switching hub 150.

[0127] 3 in that the wide-angle camera 110 has a first detection unit 51 that performs AI detection of the subject. The wide-angle camera 110 in this modification transmits the AI ​​detection result of the subject by the first detection unit 51 to the telephoto zoom camera 120.

[0128] The swivel base 140 has a pan / tilt control unit 54 that controls the rotation state, similar to the swivel base 140 of the embodiment shown in Fig. 3. The swivel base 140 in this modification controls the rotation state of the swivel base 140 in response to instructions from the telephoto zoom camera 120, and transmits the results of the rotation control to the telephoto zoom camera 120.

[0129] The telephoto zoom camera 120 is similar to the telephoto zoom camera 120 of the embodiment shown in FIG. 3 in that it includes a second detection unit 52 that performs AI detection of the subject and a zoom control unit 53 that controls the zoom state of the telephoto zoom camera 120. The telephoto zoom camera 120 of this modification includes a control unit 60 that controls the operation of the entire imaging system 100. In addition to the processing of the telephoto zoom camera 120 of the embodiment shown in FIG. 3, the telephoto zoom camera 120 also performs processing of the personal computer 130 shown in FIG. 3. For example, the telephoto zoom camera 120 performs AI detection processing of the subject by itself and determines the imaging range (width and direction) of the telephoto zoom camera 120 based on the AI ​​detection results. Furthermore, if the telephoto zoom camera 120 cannot detect the subject through its own AI detection processing, it determines the imaging range (width and direction) of the telephoto zoom camera 120 based on the AI ​​detection results of the wide-angle camera 110 received from the wide-angle camera 110.

[0130] As described above, according to the modified example in which the telephoto zoom camera 120 is used as the control device, when the swivel base 140 connected to the telephoto zoom camera 120 is caused to perform a swivel operation, the telephoto zoom camera 120 itself can transmit a swivel instruction signal to the swivel base 140, thereby improving the subject tracking performance compared to transmitting a swivel instruction signal to the swivel base 140 via a network from, for example, a remotely installed personal computer 130. Note that, although the telephoto zoom camera 120 is used as the control device in this modified example, the control device is not limited to this, and the wide-angle camera 110, for example, may also be used as the control device.

[0131] <Modification of Communication with Swivel Base 140> Figure 12 is a diagram showing a modification of communication between the personal computer 130 and the swivel base 140. In the imaging system 100 shown in Figure 1, an example has been described in which the personal computer 130 is connected to the swivel base 140 via a LAN via the switching hub 150, but this is not limiting. For example, as shown in Figure 12, the personal computer 130 may be communicably connected to the swivel base 140 via the switching hub 150 and the telephoto zoom camera 120. Note that the configuration in which the personal computer 130 is connected to the wide-angle camera 110 and the telephoto zoom camera 120 via a LAN via the switching hub 150 is the same as the configuration of the embodiment shown in Figure 1.

[0132] The swivel head 140 receives rotation instructions from the personal computer 130 via the telephoto zoom camera 120. The swivel head 140 is connected to the telephoto zoom camera 120, for example, serially, so that they can communicate with each other. The swivel head 140 rotates the telephoto zoom camera 120 based on instructions from the personal computer 130 received from the telephoto zoom camera 120. This allows the telephoto zoom camera 120 to transmit a rotation instruction signal to the serially connected swivel head 140 without delay, thereby improving the ability to track a subject.

[0133] In the configuration shown in FIG. 12, the personal computer 130 may not be provided, and the telephoto zoom camera 120 may be used as the control device, as described with reference to FIG.

[0134] <Modification of Installation of Wide-Angle Camera 110> Figure 13 is a diagram showing a modification of the installation method of wide-angle camera 110. In the imaging system 100 shown in Figure 1, of the wide-angle camera 110 and the telephoto zoom camera 120, only the telephoto zoom camera 120 is installed on the swivel base 140 and is rotatable, but this is not limiting. For example, as shown in Figure 13, the wide-angle camera 110 may also be installed on the swivel base and be rotatable.

[0135] For example, telephoto zoom camera 120 is mounted on swivel base 140A, and wide-angle camera 110 is mounted on swivel base 140B, which is separate from swivel base 140A. Swivel base 140A is connected to personal computer 130 via a LAN via switching hub 150, and rotates telephoto zoom camera 120 based on instructions from personal computer 130. Swivel base 140B is connected to personal computer 130 via a LAN via switching hub 150, and rotates wide-angle camera 110 based on instructions from personal computer 130. This allows wide-angle camera 110 to be rotated based on the AI ​​detection results, thereby enabling more accurate detection of specific subjects and improving tracking performance.

[0136] In the configuration shown in Fig. 13, the telephoto zoom camera 120 may be used as a control device without providing the personal computer 130, as described in Fig. 10. Furthermore, the telephoto zoom camera 120 and the swivel base 140A, and the wide-angle camera 110 and the swivel base 140B may be serially connected, as described in Fig. 12.

[0137] <Storage medium for control program> In each of the above imaging controls, an example has been given in which the control program of each embodiment is stored in the storage 60B of the control unit 60, and the CPU 60A of the control unit 60 executes the control program in the memory 60C, but the technology of the present disclosure is not limited to this.

[0138] 14 is a diagram showing an example of how a control program for image capture control is installed from a storage medium storing the control program into the control unit 60 of the personal computer 130. As an example, as shown in FIG. 14, a control program 221 may be stored in a storage medium 220, which is a non-transitory storage medium. In the example shown in FIG. 14, the control program 221 stored in the storage medium 220 is installed into the control unit 60, and the CPU 60A executes the above-described processes in accordance with the control program 221.

[0139] Although various embodiments have been described above, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention. Furthermore, the components of the above embodiments may be combined in any manner as long as they do not deviate from the spirit of the invention.

[0140] This application is based on a Japanese patent application (Patent Application No. 2023-140992) filed on August 31, 2023, the contents of which are incorporated herein by reference.

[0141] 10 Drone 11, 130 Personal computer 12 Communication line 15 Optical system 15A Objective lens 15B Lens group 15B1 Anti-vibration lens 15B2 Zoom lens 17, 21 Lens actuator 19 Computer 22 BIS driver 23 OIS driver 25 Image sensor 25A Light receiving surface 27 Image sensor actuator 28 Lens driver 29, 45 Correction mechanism 31 DSP 32 Image memory 33 Electronic shake correction unit 34, 66 to 68, 78 Communication I / F 35, 60C Memory 36, 60B Storage 37, 60A CPU 38, 70 Bus 39, 47 Position sensor 40 Shake amount detection sensor 43 UI device 43A, 62 Reception device 43B, 131 Display 51 First detection unit 52 Second detection unit 53 Zoom control unit 54 Pan / tilt control unit 60 Control unit 71 Yaw axis rotation mechanism 72 Pitch axis rotation mechanism 73, 74 Motors 75, 76 Drivers 100 Imaging system 110 Wide-angle camera 120 Telephoto zoom camera 132 Keyboard 133 Mouse 134 Secondary storage device 140, 140A, 140B Swivel base 150 Switching hub 220 Storage medium 221 Control program

Claims

1. An imaging system comprising: a first imaging device having a first detection unit that detects a specific subject; a second imaging device capable of imaging at a longer focal length than the first imaging device and having a second detection unit that detects the specific subject; and a control device having a processor, wherein the processor controls the imaging range of the second imaging device based on the detection result by the second detection unit, and when the detection result by the second detection unit satisfies a first condition, controls the imaging range based on the detection result of the specific subject by the first detection unit.

2. An imaging system according to claim 1, wherein the first condition is a condition relating to whether or not the specific subject is detected.

3. An imaging system according to claim 2, wherein the first condition is a condition when the specific subject is not detected.

4. An imaging system according to claim 1, wherein the second detection section detects the specific subject in parallel with detection of the specific subject by the first detection section.

5. An imaging system according to claim 1, wherein the detection of the specific subject by the first detection unit and the second detection unit is detection by a machine learning model.

6. An imaging system according to claim 5, wherein the machine learning model is different between the first detection unit and the second detection unit.

7. An imaging system according to claim 6, wherein the second machine learning model of the second detection unit is a machine learning model capable of subject detection in greater detail than the first machine learning model of the first detection unit.

8. An imaging system according to claim 1, further comprising a rotation device for rotating said second imaging device, and said control of said imaging range includes control of said rotation device.

9. An imaging system according to claim 1, wherein the second imaging device has a variable focal length for imaging, and the control of the imaging range includes control of the focal length.

10. An imaging system as described in claim 1, wherein the processor, when the specific subject is not detected by the second detection unit after controlling the imaging range based on a first result in which the specific subject is detected by the first detection unit, controls the imaging range based on the detection result of the specific subject by the first detection unit after the first result and the first result.

11. An imaging system as described in claim 10, wherein, if the specific subject is not detected by the second detection unit after controlling the imaging range based on the first result, the processor controls the imaging range based on a detection result of the specific subject by the first detection unit after the first result, excluding the specific subject included in the first result.

12. An imaging system according to any one of claims 1 to 11, wherein the control device is provided in the second imaging device.

13. A control device having a processor included in an imaging system including a first imaging device having a first detection unit that detects a specific subject, and a second imaging device capable of imaging at a longer focal length than the first imaging device and having a second detection unit that detects the specific subject in parallel with the detection of the specific subject by the first detection unit, wherein the processor controls the imaging range of the second imaging device based on the detection result by the second detection unit, and when the detection result by the second detection unit satisfies a first condition, controls the imaging range based on the detection result of the specific subject by the first detection unit.

14. A control method by a control device having a processor included in an imaging system including a first imaging device having a first detection unit that detects a specific subject, and a second imaging device capable of imaging at a longer focal length than the first imaging device and having a second detection unit that detects the specific subject in parallel with the detection of the specific subject by the first detection unit, wherein the processor controls the imaging range of the second imaging device based on the detection result by the second detection unit, and when the detection result by the second detection unit satisfies a first condition, controls the imaging range based on the detection result of the specific subject by the first detection unit.

15. A control program for a control device having a processor, which is included in an imaging system including a first imaging device having a first detection unit that detects a specific subject, and a second imaging device that is capable of imaging at a longer focal length than the first imaging device and has a second detection unit that detects the specific subject in parallel with the detection of the specific subject by the first detection unit, wherein the processor controls the imaging range of the second imaging device based on the detection result by the second detection unit, and when the detection result by the second detection unit satisfies a first condition, controls the imaging range based on the detection result of the specific subject by the first detection unit.

Images