Information processing device, control device, information processing method, and information processing program

Abstract

Provided is an information processing device, a control device, an information processing method, and an information processing program capable of supporting work on a subject.　The present invention comprises a CPU6 (0A) capable of carrying out communication with an imaging device (10) and a turning device (16) that turns the imaging device (10). The CPU6 (0A): acquires first position information on the basis of a turning condition of the turning device (16) corresponding to a defect site (92) in a power transmission line (91); acquires second position information on the basis of a turning condition of the turning device (16) corresponding to an end part A of the power transmission line (91) connected to a tower (81); acquires third position information on the basis of a turning condition of the turning device (16) corresponding to an end part B of the power transmission line (91) connected to a tower (82); and, on the basis of the first position information, the second position information, and the third position information, outputs distance information for the distance between the defect site (92) and one of the end part A of the power transmission line (91) and the end part B of the power transmission line (91).

Description

Information processing device, control device, information processing method, and information processing program

[0001] The present invention relates to an information processing device, a control device, an information processing method, and an information processing program.

[0002] Patent Document 1 describes an overhead line photography device and an imaging device that includes a telescope for telephotographing overhead lines, a video camera device for capturing images of the overhead lines, a pan head mechanism that can rotate in the yaw and tilt directions, a yaw direction control unit, a tilt direction control unit, and a slide control unit for focusing, and that takes telephoto photographs of overhead lines from the ground over a predetermined length in order to determine the damage status of the overhead lines.

[0003] Patent Document 2 describes a master camera that has a distance measurement function using autofocus, is capable of measuring the distance to a target object, and is capable of measuring the position of the target object in three-dimensional world coordinates from the line of sight (pan angle / tilt angle) and the measured distance.

[0004] Patent document 3 describes an overhead wire sag monitoring system that has a camera, a known data capture means, a coordinate conversion unit, an operation control unit, a screen display unit, and a monitoring / calculation unit, and that performs a calibration process (image correction process) on a still image of the steel tower to be measured taken by the camera so that it matches the structural drawing of the steel tower captured by the known data capture means, and then calculates the coordinates of the bottom of the overhead wire sag from the calibrated still image.

[0005] Japanese Utility Model Registration No. 3220181 Japanese Patent Application Laid-Open No. 2000-083246 Japanese Patent Application Laid-Open No. 2002-112421

[0006] One embodiment of the technique of the present disclosure provides an information processing device, a control device, an information processing method, and an information processing program that are capable of assisting a task on a subject.

[0007] (1) An information processing device including a processor capable of communicating with an imaging device and a rotation device that rotates the imaging device, wherein the processor: acquires first position information based on a rotation condition of the rotation device corresponding to a first subject; acquires second position information based on a rotation condition of the rotation device corresponding to a second subject; acquires third position information based on a rotation condition of the rotation device corresponding to a third subject; and outputs distance information between the first subject and either the second subject or the third subject based on the first position information, the second position information, and the third position information.

[0008] (2) The information processing device according to (1), wherein the distance information is distance information between the first subject and one of the second subject and the third subject that is closer to the imaging device.

[0009] (3) The information processing device according to (1) or (2), in which the first subject, the second subject, and the third subject are present on a first surface.

[0010] (4) An information processing device according to any one of (1) to (3), wherein the processor acquires the first position information based on the rotation conditions of the rotation device and the imaging conditions of the imaging device corresponding to the first subject, acquires the second position information based on the rotation conditions of the rotation device and the imaging conditions of the imaging device corresponding to the second subject, and acquires the third position information based on the rotation conditions of the rotation device and the imaging conditions of the imaging device corresponding to the third subject.

[0011] (5) An information processing device according to any one of (1) to (3), wherein the second subject is a subject closer to the imaging device than the third subject, and the processor acquires the second position information based on the rotation conditions of the rotation device corresponding to the second subject and the imaging conditions of the imaging device, and acquires the third position information based on the rotation conditions of the rotation device corresponding to the second subject and the imaging conditions of the imaging device, and the rotation conditions of the rotation device corresponding to the third subject.

[0012] (6) An information processing device according to any one of (1) to (3), wherein the second subject is a subject closer to the imaging device than the third subject, and the processor acquires the second position information based on the rotation conditions of the rotation device corresponding to the second subject and the imaging conditions of the imaging device, and acquires the third position information based on the second position information, the distance between the second subject and the third subject, and the rotation conditions of the rotation device corresponding to the third subject.

[0013] (7) An information processing device according to (5) or (6), wherein the first subject, the second subject, and the third subject are present on a first surface, and the processor acquires the first position information based on the second position information, the third position information, and a rotation condition of the rotation device corresponding to the first subject.

[0014] (8) An information processing device according to any one of (5) to (7), wherein the processor determines that the second subject is closer to the imaging device than the third subject based on an imaging condition of the imaging device corresponding to the second subject and an imaging condition of the imaging device corresponding to the third subject.

[0015] (9) The information processing device according to any one of (5) to (8), wherein the processor acquires the third position information based on information indicating a relative height relationship between the second subject and the third subject.

[0016] (10) The information processing device according to any one of (1) to (9), wherein the rotation condition of the rotation device includes at least one of a pan state and a tilt state of the rotation device.

[0017] (11) The information processing device according to any one of (1) to (10), in which the imaging conditions of the imaging device include focus information of the imaging device.

[0018] (12) A control device that controls the imaging device and / or the turning device based on information generated by the information processing device according to any one of (1) to (11).

[0019] (13) An information processing method by an information processing device having a processor capable of communicating with an imaging device and a turning device that turns the imaging device, wherein the processor: acquires first position information based on turning conditions of the turning device corresponding to a first subject; acquires second position information based on turning conditions of the turning device corresponding to a second subject; acquires third position information based on turning conditions of the turning device corresponding to a third subject; and outputs distance information between either the second subject or the third subject and the first subject based on the first position information, the second position information, and the third position information.

[0020] (14) An information processing program for an information processing device having a processor capable of communicating with an imaging device and a turning device that turns the imaging device, the information processing program causing the processor to execute the following processes: acquire first position information based on turning conditions of the turning device corresponding to a first subject; acquire second position information based on turning conditions of the turning device corresponding to a second subject; acquire third position information based on turning conditions of the turning device corresponding to a third subject; and output distance information between either the second subject or the third subject and the first subject based on the first position information, the second position information, and the third position information.

[0021] According to the present invention, it is possible to provide an information processing device, a control device, an information processing method, and an information processing program that are capable of assisting in work on a subject.

[0022] 11 is a diagram showing an example of an imaging system 1 equipped with a control device of the present embodiment. FIG. 12 is a block diagram showing an example of the configuration of an electrical system of a swivel device 16 and a personal computer 11. FIG. 13 is a block diagram showing an example of the configuration of an optical system and an electrical system of an imaging device 10. FIG. 14 is a schematic diagram showing an example of maintenance work on a power transmission line. FIG. 15 is a flowchart showing an example of processing by a CPU 60A of an information processing device 60. FIG. 16 is a diagram showing an example of distance information displayed on a display 13a of a personal computer 11. FIG. 17 is a diagram showing a modified example of distance information displayed on a display 13a of a personal computer 11. FIG. 18 is a diagram explaining an error that occurs when the distance from the imaging device to a steel tower is calculated based on focus information. FIG. 19 is a flowchart showing a first modified example of processing by a CPU 60A of an information processing device 60. FIG. 19 is a diagram explaining calculation of spatial coordinates of a long-distance reference position in the first modified example of FIG. 9. FIG. 20 is a flowchart showing a second modified example of processing by a CPU 60A of an information processing device 60. FIG. 21 is a diagram explaining calculation of spatial coordinates of a long-distance reference position in the second modified example of FIG. FIG. 22 is a diagram showing an example of how an information processing program for imaging control is installed in an information processing device 60 of a personal computer 11 from a storage medium on which the information processing program is stored.

[0023] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

[0024] <Imaging System of the Embodiment> Fig. 1 is a diagram showing an example of an imaging system 1 equipped with a control device of the present embodiment. As shown in Fig. 1, the imaging system 1 includes an imaging device 10, a personal computer (PC) 11, and a turning device 16. The personal computer (PC) 11 is an example of the "control device" in the present invention.

[0025] The imaging device 10 is installed on a pillar, a wall, or a part of a building (e.g., a rooftop) indoors or outdoors via a swivel device 16, and captures an image of a subject. The imaging device 10 is, for example, a camera capable of capturing long-distance images. The imaging device 10 transmits the captured image and information related to the image to a personal computer 11 via a communication line 12.

[0026] The personal computer 11 includes a display 13a, a keyboard 13b, a mouse 13c, and a secondary storage device 14. Examples of the display 13a include a liquid crystal display, a plasma display, and an organic EL (Electro-Luminescence) display.

[0027] An example of the secondary storage device 14 is a hard disk drive (HDD). The secondary storage device 14 is not limited to an HDD, and may be a non-volatile memory such as a flash memory, a solid state drive (SSD), or an electrically erasable and programmable read-only memory (EEPROM).

[0028] The personal computer 11 receives the captured images and information relating to imaging transmitted from the imaging device 10 , and displays the received captured images and information relating to imaging on the display 13 a and stores them in the secondary storage device 14 .

[0029] For example, the personal computer 11 performs imaging control by communicating with the imaging device 10 via the communication line 12. The imaging control is a control for setting imaging parameters for the imaging device 10 to capture an image, and causing the imaging device 10 to capture an image. The imaging parameters include parameters related to exposure, parameters related to the width of the imaging range (focal length), and the like.

[0030] The personal computer 11 also performs rotation control by communicating with the rotation device 16 via the communication line 12. The rotation control is a control for setting rotation parameters in the rotation device 16 for the rotation device 16 to rotate the image capture device 10, and causing the rotation device 16 to rotate the image capture device 10. The rotation parameters include parameters related to the direction (pan / tilt) of the imaging range of the image capture device 10, etc.

[0031] The personal computer 11 sets the rotation direction, rotation amount, rotation speed, etc. of the imaging device 10 in response to, for example, operation of the keyboard 13b or mouse 13c, or touch operation on the screen of the display 13a.

[0032] <Configuration of Electrical Systems of Swing Device 16 and Personal Computer 11> Figure 2 is a block diagram showing an example of the configuration of the electrical systems of the swing device 16 and personal computer 11. As shown in Figure 2, the swing device 16 includes a yaw axis swing mechanism 71, a pitch axis swing mechanism 72, motors 73 and 74, drivers 75 and 76, and a communication I / F 78.

[0033] The yaw axis rotation mechanism 71 rotates the rotation device 16 to which the imaging device 10 is attached in the yaw direction. The motor 73 generates power by being driven under the control of a driver 75. The yaw axis rotation mechanism 71 rotates the rotation device 16 to which the imaging device 10 is attached in the yaw direction by receiving the power generated by the motor 73. The pitch axis rotation mechanism 72 rotates the rotation device 16 to which the imaging device 10 is attached in the pitch direction. The motor 74 generates power by being driven under the control of a driver 76. The pitch axis rotation mechanism 72 rotates the rotation device 16 to which the imaging device 10 is attached in the pitch direction by receiving the power generated by the motor 74.

[0034] The communication I / F 78 is, for example, a network interface. The communication I / F 78 controls the transmission of various information between the personal computer 11 and the swivel device 16. This network may be, for example, a wide area network (WAN) such as the Internet, or a local area network (LAN). The communication I / F 78 performs communication between the swivel device 16 and the personal computer 11.

[0035] The personal computer 11 includes a display 13a, a secondary storage device 14, an information processing device 60, a receiving device 62, and communication I / Fs 66 and 68. The information processing device 60 includes a CPU 60A, a storage 60B, and a memory 60C.

[0036] The accepting device 62, the display 13a, the secondary storage device 14, the CPU 60A, the storage 60B, the memory 60C, and the communication I / Fs 66 and 68 are each connected to a bus 69. Note that, for convenience of illustration, one bus is shown as the bus 69 in the example shown in Fig. 2, but multiple buses may be used. The bus 69 may be a serial bus or a parallel bus including a data bus, an address bus, a control bus, etc.

[0037] The memory 60C temporarily stores various types of information and is used as a work memory. An example of the memory 60C is a RAM, but other types of storage devices may be used. The storage 60B stores various programs for the personal computer 11 (hereinafter simply referred to as "personal computer programs").

[0038] The CPU 60A reads the personal computer program from the storage 60B and executes the read personal computer program on the memory 60C to control the entire personal computer 11. The personal computer program includes the "information processing program" of the present invention. The CPU 60A is an example of the "processor" of the present invention.

[0039] The communication I / F 66 is, for example, a network interface. The communication I / F 66 is communicatively connected to a communication I / F (not shown) of the imaging device 10, and controls the transmission of various information between the imaging device 10. The communication I / F 68 is, for example, a network interface. The communication I / F 68 is communicatively connected to a communication I / F 78 of the turning device 16, and controls the transmission of various information between the yaw axis turning mechanism 71 and the pitch axis turning mechanism 72.

[0040] The CPU 60A receives captured images, information related to imaging, and the like from the imaging device 10 via the communication I / F 66. Furthermore, the CPU 60A controls the driver 75 and motor 73 of the turning device 16 via the communication I / F 68 and the communication I / F 78 to control the turning operation of the yaw axis turning mechanism 71, and controls the driver 76 and motor 74 of the turning device 16 to control the turning operation of the pitch axis turning mechanism 72.

[0041] The reception device 62 is, for example, the keyboard 13b, the mouse 13c, or the touch panel of the display 13a, and receives various instructions from the user. The CPU 60A acquires the various instructions received by the reception device 62 and operates in accordance with the acquired instructions. For example, when the reception device 62 receives processing content for the imaging device 10 or the turning device 16, the CPU 60A operates the imaging device 10 or the turning device 16 in accordance with the instruction content received by the reception device 62.

[0042] The display 13a displays various types of information under the control of the CPU 60A. Examples of the various types of information displayed on the display 13a include the contents of various instructions accepted by the acceptance device 62, and captured images of subjects and information related to imaging received by the communication I / F 66. The CPU 60A causes the display 13a to display the contents of various instructions accepted by the acceptance device 62, and captured images of subjects and information related to imaging received by the communication I / F 66.

[0043] The secondary storage device 14 is, for example, a non-volatile memory, and stores various types of information under the control of the CPU 60 A. Examples of the various types of information stored in the secondary storage device 14 include captured images of a subject and information related to the imaging received by the communication I / F 66. The CPU 60 A stores the captured images of a subject and information related to the imaging received by the communication I / F 66 in the secondary storage device 14.

[0044] <Configuration of Optical System and Electrical System of Imaging Device 10> FIG. 3 is a block diagram showing an example of the configuration of the optical system and electrical system of the imaging device 10. As shown in FIG. 3, the imaging device 10 includes an optical system 15 and an image sensor 25. The image sensor 25 is located downstream of the optical system 15. The optical system 15 includes an objective lens 15A and a lens group 15B. The objective lens 15A and the lens group 15B are arranged in this order along the optical axis OA of the optical system 15 from the target subject side (object side) to the light receiving surface 25A side (image side) of the image sensor 25. The lens group 15B includes an anti-vibration lens 15B1, a focus lens (not shown), a zoom lens 15B2, and the like. The zoom lens 15B2 is supported by a lens actuator 21 (described later) so as to be movable along the optical axis OA. The anti-vibration lens 15B1 is supported by a lens actuator 17 (described later) so as to be movable in a direction perpendicular to the optical axis OA.

[0045] By increasing the focal length with the zoom lens 15B2, the image capture device 10 becomes telephoto side, so the angle of view becomes smaller (the image capture range becomes narrower). By decreasing the focal length with the zoom lens 15B2, the image capture device 10 becomes wide-angle side, so the angle of view becomes larger (the image capture range becomes wider).

[0046] The optical system 15 may include various lenses (not shown) in addition to the objective lens 15A and the lens group 15B. The optical system 15 may also include an aperture. The positions of the lenses, lens group, and aperture included in the optical system 15 are not limited, and the technology of the present disclosure is valid even if they are located in positions different from those shown in FIG. 3 .

[0047] The vibration-proof lens 15B1 is movable in a direction perpendicular to the optical axis OA, and the zoom lens 15B2 is movable along the optical axis OA.

[0048] The optical system 15 includes lens actuators 17 and 21. The lens actuator 17 applies a force to the vibration-proof lens 15B1 that fluctuates in a direction perpendicular to the optical axis of the vibration-proof lens 15B1. The lens actuator 17 is controlled by an OIS (Optical Image Stabilizer) driver 23. When the lens actuator 17 is driven under the control of the OIS driver 23, the position of the vibration-proof lens 15B1 fluctuates in a direction perpendicular to the optical axis OA.

[0049] The lens actuator 21 applies a force to the zoom lens 15B2 to move it along the optical axis OA of the optical system 15. The lens actuator 21 is controlled by a lens driver 28. When the lens actuator 21 is driven under the control of the lens driver 28, the position of the zoom lens 15B2 moves along the optical axis OA. When the position of the zoom lens 15B2 moves along the optical axis OA, the focal length of the imaging device 10 changes.

[0050] In addition, if the outline of the captured image is, for example, a rectangle with a short side in the pitch axis PA direction (see Figure 1) and a long side in the yaw axis YA direction (see Figure 1), the angle of view in the pitch axis PA direction is narrower than the angle of view in the yaw axis YA direction and is narrower than the angle of view of the diagonal.

[0051] By the optical system 15 configured in this manner, light representing the imaging area is focused on the light receiving surface 25A of the imaging element 25, and the imaging area is imaged by the imaging element 25.

[0052] Incidentally, vibrations that are applied to the imaging device 10 include, if outdoors, vibrations caused by passing cars, vibrations caused by wind, vibrations caused by road construction, etc., and if indoors, vibrations caused by the operation of an air conditioner, vibrations caused by people entering and exiting, etc. Therefore, in the imaging device 10, shaking occurs due to vibrations applied to the imaging device 10 (hereinafter also simply referred to as "vibrations").

[0053] In this embodiment, "shake" refers to a phenomenon in which the image of a target subject on the light-receiving surface 25A of the image sensor 25 in the imaging device 10 fluctuates due to a change in the positional relationship between the optical axis OA and the light-receiving surface 25A. In other words, "shake" can also be said to be a phenomenon in which the optical image formed on the light-receiving surface 25A fluctuates due to the tilt of the optical axis OA caused by vibrations applied to the imaging device 10. The fluctuation of the optical axis OA means, for example, that the optical axis OA is tilted with respect to a reference axis (e.g., the optical axis OA before the shake occurs). Hereinafter, shake caused by vibrations will also be simply referred to as "shake."

[0054] Shake is included in the captured image as a noise component and affects the image quality of the captured image. Therefore, in order to remove the noise component included in the captured image due to shake, the imaging device 10 is equipped with a lens-side shake correction mechanism 29, an image sensor-side shake correction mechanism 45, and an electronic shake correction unit 33, which are used to correct shake.

[0055] The lens-side shake correction mechanism 29 and the image sensor-side shake correction mechanism 45 are mechanical shake correction mechanisms that apply power generated by a drive source such as a motor (for example, a voice coil motor) to a shake correction element (for example, the vibration-proof lens 15B1 and / or the image sensor 25) to move the shake correction element in a direction perpendicular to the optical axis of the imaging optical system, thereby correcting shake.

[0056] Specifically, the lens-side image stabilization mechanism 29 applies power generated by a drive source such as a motor (e.g., a voice coil motor) to the image stabilization lens 15B1, thereby moving the image stabilization lens 15B1 in a direction perpendicular to the optical axis of the imaging optical system, thereby correcting shake. The image sensor-side image stabilization mechanism 45 applies power generated by a drive source such as a motor (e.g., a voice coil motor) to the image sensor 25, thereby moving the image sensor 25 in a direction perpendicular to the optical axis of the imaging optical system, thereby correcting shake. The electronic image stabilization unit 33 corrects shake by performing image processing on the captured image based on the amount of shake. In other words, the image stabilization unit (image stabilization component) performs shake correction mechanically or electronically using a hardware and / or software configuration. Here, mechanical shake correction refers to shake correction achieved by mechanically moving shake correction elements such as the anti-vibration lens 15B1 and / or the image sensor 25 using power generated by a driving source such as a motor (e.g., a voice coil motor), and electronic shake correction refers to shake correction achieved by, for example, image processing performed by a processor.

[0057] As an example, as shown in FIG. 3, the lens-side shake correction mechanism 29 includes an anti-vibration lens 15B1, a lens actuator 17, an OIS driver 23, and a position sensor 39.

[0058] Various well-known methods can be used to correct shake using the lens-side shake correction mechanism 29. In this embodiment, the method used is to correct shake by moving the vibration-proof lens 15B1 based on the amount of shake detected by a shake amount detection sensor 40 (described later). Specifically, shake correction is performed by moving the vibration-proof lens 15B1 in a direction that cancels out the shake and by an amount that cancels out the shake.

[0059] A lens actuator 17 is attached to the vibration-proof lens 15B1. The lens actuator 17 is a shift mechanism equipped with a voice coil motor, and by driving the voice coil motor, the vibration-proof lens 15B1 is moved in a direction perpendicular to the optical axis of the vibration-proof lens 15B1. Note that, although a shift mechanism equipped with a voice coil motor is used as the lens actuator 17 here, the technology of the present disclosure is not limited to this, and other power sources such as a stepping motor or a piezoelectric element may be used instead of the voice coil motor.

[0060] The lens actuator 17 is controlled by an OIS driver 23. When the lens actuator 17 is driven under the control of the OIS driver 23, the position of the vibration-proof lens 15B1 is mechanically changed within a two-dimensional plane perpendicular to the optical axis OA.

[0061] The position sensor 39 detects the current position of the vibration-proof lens 15B1 and outputs a position signal indicating the detected current position. Here, a device including a Hall element is used as an example of the position sensor 39. Here, the current position of the vibration-proof lens 15B1 refers to the current position within a two-dimensional plane of the vibration-proof lens. The vibration-proof lens two-dimensional plane refers to a two-dimensional plane perpendicular to the optical axis of the vibration-proof lens 15B1. Note that in this embodiment, a device including a Hall element is used as an example of the position sensor 39, but the technology of the present disclosure is not limited to this, and a magnetic sensor, a photosensor, or the like may be used instead of the Hall element.

[0062] The lens-side image stabilization mechanism 29 corrects shake by moving the vibration-proof lens 15B1 along at least one of the pitch axis PA direction and the yaw axis YA direction within the range that is actually captured. In other words, the lens-side image stabilization mechanism 29 corrects shake by moving the vibration-proof lens 15B1 within the two-dimensional vibration-proof lens plane by an amount corresponding to the amount of shake.

[0063] The image sensor side shake correction mechanism 45 includes the image sensor 25 , a BIS (Body Image Stabilizer) driver 22 , an image sensor actuator 27 , and a position sensor 47 .

[0064] As with the shake correction method used by the lens-side shake correction mechanism 29, various well-known methods can be used to correct shake using the image sensor-side shake correction mechanism 45. In this embodiment, the shake correction method used is to correct shake by moving the image sensor 25 based on the amount of shake detected by the shake amount detection sensor 40. Specifically, shake correction is performed by moving the image sensor 25 in a direction that cancels out the shake and by an amount that cancels out the shake.

[0065] An imaging element actuator 27 is attached to the imaging element 25. The imaging element actuator 27 is a shift mechanism equipped with a voice coil motor, and by driving the voice coil motor, the imaging element 25 is moved in a direction perpendicular to the optical axis of the vibration-proof lens 15B1. Note that, although a shift mechanism equipped with a voice coil motor is used as the imaging element actuator 27 here, the technology of the present disclosure is not limited to this, and other power sources such as a stepping motor or a piezoelectric element may be applied instead of the voice coil motor.

[0066] The imaging element actuator 27 is controlled by the BIS driver 22. When the imaging element actuator 27 is driven under the control of the BIS driver 22, the position of the imaging element 25 is mechanically moved in a direction perpendicular to the optical axis OA.

[0067] The position sensor 47 detects the current position of the image sensor 25 and outputs a position signal indicating the detected current position. Here, a device including a Hall element is used as an example of the position sensor 47. Here, the current position of the image sensor 25 refers to the current position within a two-dimensional plane of the image sensor. The two-dimensional plane of the image sensor refers to a two-dimensional plane perpendicular to the optical axis of the vibration-proof lens 15B1. Note that in this embodiment, a device including a Hall element is used as an example of the position sensor 47, but the technology of the present disclosure is not limited to this, and a magnetic sensor, a photosensor, or the like may be used instead of the Hall element.

[0068] The imaging device 10 includes a computer 19, a DSP (Digital Signal Processor) 31, an image memory 32, an electronic shake correction unit 33, a communication I / F 34, a shake amount detection sensor 40, and a UI (User Interface) device 43. The computer 19 includes a memory 35, a storage 36, and a CPU (Central Processing Unit) 37. The imaging device 10 detects a specific subject using a machine learning model with a processor in the computer 19. The processor may be, for example, the CPU 37 or another processor.

[0069] The image sensor 25, DSP 31, image memory 32, electronic shake correction unit 33, communication I / F 34, memory 35, storage 36, CPU 37, shake amount detection sensor 40, and UI device 43 are connected to a bus 38. The OIS driver 23 is also connected to the bus 38. Note that, although one bus is shown as the bus 38 in the example shown in Fig. 3 for convenience of illustration, multiple buses may be used. The bus 38 may be a serial bus or a parallel bus such as a data bus, an address bus, and a control bus.

[0070] The memory 35 temporarily stores various types of information and is used as a work memory. An example of the memory 35 is a random access memory (RAM), but other types of storage devices may be used. The storage 36 stores various programs for the imaging device 10. The CPU 37 reads the various programs from the storage 36 and executes them on the memory 35 to control the entire imaging device 10. Examples of the storage 36 include a flash memory, an SSD, an EEPROM, and an HDD. Furthermore, various types of non-volatile memory, such as a magnetoresistive memory or a ferroelectric memory, may be used instead of or in combination with the flash memory.

[0071] The imaging element 25 is a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging element 25 captures an image of a target subject at a predetermined frame rate under the direction of the CPU 37. The "predetermined frame rate" here refers to, for example, several tens to several hundreds of frames per second. The imaging element 25 itself may have a built-in control device (image sensor control device), in which case the image sensor control device performs detailed control of the imaging element 25 in accordance with imaging instructions output by the CPU 37. The imaging element 25 may also capture an image of a target subject at a predetermined frame rate under the direction of the DSP 31, in which case the image sensor control device performs detailed control of the imaging element 25 in accordance with imaging instructions output by the DSP 31. The DSP 31 is sometimes referred to as an ISP (Image Signal Processor).

[0072] The light receiving surface 25A of the image sensor 25 is formed by a plurality of photosensitive pixels (not shown) arranged in a matrix. In the image sensor 25, each photosensitive pixel is exposed to light, and photoelectric conversion is performed for each photosensitive pixel. The electric charge obtained by photoelectric conversion for each photosensitive pixel is an analog image signal representing the target subject. Here, a plurality of photoelectric conversion elements (e.g., photoelectric conversion elements with color filters) sensitive to visible light are used as the photosensitive pixels. In the image sensor 25, a plurality of photoelectric conversion elements are used, including a photoelectric conversion element sensitive to R (red) light (e.g., a photoelectric conversion element with an R filter corresponding to R), a photoelectric conversion element sensitive to G (green) light (e.g., a photoelectric conversion element with a G filter corresponding to G), and a photoelectric conversion element sensitive to B (blue) light (e.g., a photoelectric conversion element with a B filter corresponding to B). In the imaging device 10, these photosensitive pixels are used to capture images based on visible light (e.g., light on the short wavelength side of approximately 700 nanometers or less). However, this embodiment is not limited to this, and imaging based on infrared light (e.g., light on the long wavelength side of approximately 700 nanometers) may also be performed. In this case, a plurality of photoelectric conversion elements sensitive to infrared light may be used as the photosensitive pixels. In particular, for imaging SWIR (Short-wavelength infrared), an InGaAs sensor and / or a Type-II Quantum Well (T2SL: Simulation of Type-II Quantum Well) sensor may be used, for example.

[0073] The image sensor 25 performs signal processing such as A / D (Analog / Digital) conversion on the analog image signal to generate a digital image, which is a digital image signal. The image sensor 25 is connected to the DSP 31 via a bus 38, and outputs the generated digital image to the DSP 31 via the bus 38 in frame units.

[0074] Note that, although a CMOS image sensor is described here as an example of the image sensor 25, the technology of the present disclosure is not limited thereto, and a CCD (Charge Coupled Device) image sensor may also be applied as the image sensor 25. In this case, the image sensor 25 is connected to the bus 38 via an AFE (Analog Front End) (not shown) that has a built-in CCD driver. The AFE generates a digital image by performing signal processing such as A / D conversion on the analog image signal obtained by the image sensor 25, and outputs the generated digital image to the DSP 31. The CCD image sensor is driven by a CCD driver built into the AFE. Of course, the CCD driver may also be provided separately.

[0075] The DSP 31 performs various types of digital signal processing on the digital image. The various types of digital signal processing refer to, for example, demosaic processing, noise removal processing, gradation correction processing, color correction processing, etc. The DSP 31 outputs the digital image after digital signal processing to the image memory 32 for each frame. The image memory 32 stores the digital image from the DSP 31.

[0076] The shake amount detection sensor 40 is a device that includes, for example, a gyro sensor, and detects the amount of shake of the imaging device 10. In other words, the shake amount detection sensor 40 detects the amount of shake in each of a pair of axial directions. The gyro sensor detects the amount of rotational shake around each of the pitch axis PA, the yaw axis YA, and the roll axis RA (an axis parallel to the optical axis OA) (see FIG. 1 ). The shake amount detection sensor 40 detects the amount of shake of the imaging device 10 by converting the amount of rotational shake around the pitch axis PA and the amount of rotational shake around the yaw axis YA detected by the gyro sensor into the amount of shake in a two-dimensional plane parallel to the pitch axis PA and the yaw axis YA.

[0077] Here, a gyro sensor is used as an example of the shake amount detection sensor 40, but this is merely an example, and the shake amount detection sensor 40 may also be an acceleration sensor. The acceleration sensor detects the amount of shake in a two-dimensional plane parallel to the pitch axis PA and the yaw axis YA. The shake amount detection sensor 40 outputs the detected amount of shake to the CPU 37.

[0078] Furthermore, although an example in which the amount of shake is detected by a physical sensor, namely, the shake amount detection sensor 40, is given here, the technology of the present disclosure is not limited to this. For example, a motion vector obtained by comparing successive captured images stored in the image memory 32 in chronological order may be used as the amount of shake. Furthermore, the amount of shake that is ultimately used may be derived based on the amount of shake detected by the physical sensor and the motion vector obtained by image processing.

[0079] The CPU 37 acquires the amount of shake detected by the shake amount detection sensor 40, and controls the lens-side shake compensation mechanism 29, the image sensor-side shake compensation mechanism 45, and the electronic shake compensation unit 33 based on the acquired amount of shake. The amount of shake detected by the shake amount detection sensor 40 is used for shake compensation by each of the lens-side shake compensation mechanism 29 and the electronic shake compensation unit 33.

[0080] The electronic shake correction unit 33 is a device including an ASIC (Application Specific Integrated Circuit). The electronic shake correction unit 33 corrects shake by performing image processing on the captured image in the image memory 32 based on the amount of shake detected by the shake amount detection sensor 40.

[0081] Note that, although a device including an ASIC is exemplified as the electronic shake correction unit 33 here, the technology of the present disclosure is not limited to this and may be, for example, a device including an FPGA (Field Programmable Gate Array) or a PLD (Programmable Logic Device). Furthermore, for example, the electronic shake correction unit 33 may be a device including multiple ASICs, FPGAs, and PLDs. Furthermore, a computer including a CPU, storage, and memory may be employed as the electronic shake correction unit 33. There may be a single CPU or multiple CPUs. Furthermore, the electronic shake correction unit 33 may be realized by a combination of hardware and software configurations.

[0082] The communication I / F 34 is, for example, a network interface, and controls the transmission of various information between the imaging device 10 and the personal computer 11 via a network. This network is, for example, a WAN or a LAN. The communication I / F 34 performs communication between the imaging device 10 and the personal computer 11.

[0083] The UI device 43 includes a reception device 43A and a display 43B. The reception device 43A is, for example, a hard key or a touch panel, and receives various instructions from the user. The CPU 37 acquires the various instructions received by the reception device 43A and operates in accordance with the acquired instructions.

[0084] The display 43B displays various information under the control of the CPU 37. Examples of the various information displayed on the display 43B include the contents of various instructions accepted by the accepting device 43A, captured images, and the like.

[0085] <Example of Maintenance Work for Defective Location> FIG. 4 is a schematic diagram showing an example of maintenance work for a power transmission line. As shown in FIG. 4 , for example, a power transmission line 91 connected between a steel tower 81 and a steel tower 82 may be inspected using a camera (imaging device 10). In this case, for example, a telephoto camera may be used to capture an image of the power transmission line 91 and confirm whether or not there is a defect. Furthermore, when performing maintenance on a defective location identified during inspection at a later date, for example, a worker may go to the site, capture an image of the power transmission line 91 using a camera in the same way as during inspection, and perform maintenance work on the defective location 92. In this case, it is desirable to be able to quickly and accurately determine the location of the defective location on the power transmission line 91. The present invention makes it possible to appropriately provide a user with information regarding the detected location of a defective location detected in a subject such as the power transmission line 91.

[0086] Specifically, during inspection of the power transmission line 91, an image of the power transmission line 91 is captured by a camera installed at the imaging device position 10a in Fig. 4. The captured image of the power transmission line 91 is configured as an image obtained by continuously joining together a plurality of divided captured images that are sequentially captured by dividing the power transmission line 91 from end A of the power transmission line 91 connected to the steel tower 81 to end B of the power transmission line 91 connected to the steel tower 82 according to the width of the imaging range (focal length) of the camera.

[0087] An inspection of the divided captured images is performed while capturing an image of the power line 91, and if a defect 92 is detected in the power line 91, rotation information and imaging information related to the defect 92 are stored. The rotation information is the pan value and tilt value of the camera associated with the divided captured image including the detected defect 92. The rotation information is the rotation information of the camera at the time when the defect 92 was captured. The rotation information is an example of a "rotation condition" in the present invention. The pan value and tilt value of the camera are the same as the pan value and tilt value of, for example, the rotation device 16 to which the camera is attached. The imaging information is focus information of the camera associated with the divided captured image including the detected defect 92. The focus information is information that makes it possible to calculate the distance from the camera to the divided captured image. The imaging information is the imaging information of the camera at the time when the defect 92 was captured. The imaging information is an example of an "imaging condition" in the present invention.

[0088] Similarly, rotation information and imaging information for end A of power line 91 connected to steel tower 81, and rotation information and imaging information for end B of power line 91 connected to steel tower 82 are stored. The rotation information for end A is the pan value and tilt value of the camera associated with the divided captured image including end A of power line 91. The imaging information for end A is the focus information of the camera associated with the divided captured image including end A. Furthermore, the rotation information for end B is the pan value and tilt value of the camera associated with the divided captured image including end B of power line 91. The imaging information for end B is the focus information of the camera associated with the divided captured image including end B. The rotation information for defective location 92 and the rotation information for end A and end B of power line 91 can be measured based on the rotation information when the camera rotation amount is 0 (zero).

[0089] In the present invention, based on this rotation information and imaging information, it is possible to provide the user with location information of the defective area 92 as the distance from the tower to the defective area 92, for example, the distance D (Figure 4) from the end B of the power transmission line 91 on the tower 82 to the defective area 92.

[0090] <Processing by CPU 60A of Information Processing Device 60> Figure 5 is a flowchart showing an example of processing by CPU 60A of information processing device 60. To start this processing, a user displays, for example, a captured image of a power line on which maintenance work is to be performed on the display 13a of the personal computer 11. As described with reference to Figure 4, the captured image of the power line is an image composed of a plurality of divided captured images obtained by dividing, for example, a power line 91 connected between steel towers 81 and 82 according to the width of the camera's imaging range (focal length). In the captured image of the power line displayed on the display 13a, the user determines a defective portion of the power line and a first and second candidate reference positions that serve as reference points for the power line. The user can determine the defective portion, the first and second candidate reference positions by selecting divided captured images that include the defective portion, the first and second candidate reference positions from the captured image of the power line. The selection of the divided captured images may be performed, for example, by a user touching the defect location, the first reference candidate position, and the second reference candidate position on the display 13a, or may be performed by recognizing the divided captured images and detecting them using AI (Artificial Intelligence). The first and second reference candidate positions may be selected from the first and last divided captured images among the plurality of divided captured images. When the defect location, the first reference candidate position, and the second reference candidate position in the power transmission line are selected, the CPU 60A of the information processing device 60 starts the process shown in FIG. 5 .

[0091] The defective location is an example of a "first subject" in the present invention, such as defective location 92 shown in Fig. 4. The first reference candidate position is an example of a "second subject" in the present invention, such as the power line holding portion of pylon 82 to which end B of power line 91 shown in Fig. 4 is connected. The second reference candidate position is an example of a "third subject" in the present invention, such as the power line holding portion of pylon 81 to which end A of power line 91 shown in Fig. 4 is connected.

[0092] First, the CPU 60A acquires the pan value, tilt value, and focus information when the defective portion is captured from the secondary storage device 14 (step S11). The pan value, tilt value, and focus information are the pan value, tilt value, and focus information of the camera associated with the divided captured image including the defective portion.

[0093] Next, the CPU 60A calculates the spatial coordinates of the defect location based on the pan value, tilt value, and focus information acquired in step S11 (step S12). The spatial coordinates are an example of "position information" in the present invention. The spatial coordinates are coordinates indicating a position in real space, and are, for example, geographic coordinates consisting of latitude, longitude, and altitude. The spatial coordinates of the defect location can be calculated as coordinates obtained by moving the spatial coordinates indicating the installation position of the camera by a focal length determined based on the focus information of the camera in an imaging direction specified by the pan value and tilt value of the camera associated with the divided captured image including the defect location, using the pan value and tilt value when the camera's rotation amount is 0 (zero) as the reference imaging direction.

[0094] Next, the CPU 60A acquires the pan value, tilt value, and focus information when the first reference candidate position was imaged from the secondary storage device 14 (step S13). The pan value, tilt value, and focus information are the pan value, tilt value, and focus information of the camera associated with the divided captured image that includes the first reference candidate position.

[0095] Next, the CPU 60A calculates the spatial coordinates of the first reference candidate position based on the pan value, tilt value, and focus information acquired in step S13 (step S14). The spatial coordinates of the first reference candidate position can be calculated in the same way as the spatial coordinates of the defect location.

[0096] Next, the CPU 60A acquires the pan value, tilt value, and focus information when the second reference candidate position is captured from the secondary storage device 14 (step S15). The pan value, tilt value, and focus information are the pan value, tilt value, and focus information of the camera associated with the divided captured image that includes the second reference candidate position.

[0097] Next, the CPU 60A calculates the spatial coordinates of the second reference candidate position based on the pan value, tilt value, and focus information acquired in step S15 (step S16). The spatial coordinates of the second reference candidate position can be calculated in the same way as the spatial coordinates of the defect location.

[0098] Next, CPU 60A compares the distance from the camera (image capture device) to the first reference candidate position with the distance from the camera to the second reference candidate position to determine whether the first reference candidate position is closer to the camera than the second reference candidate position (step S17). The determination of the distance from the camera to the first reference candidate position and the distance to the second reference candidate position may be made based on the focus information of the first reference candidate position acquired in step S13 and the focus information of the second reference candidate position acquired in step S15, or may be made based on the spatial coordinates of the first reference candidate position calculated in step S14 and the spatial coordinates of the second reference candidate position calculated in step S16.

[0099] If the first reference candidate position is closer to the camera (step S17: Yes), the CPU 60A calculates the distance between the first reference candidate position and the defective portion (step S18) from the spatial coordinates of the first reference candidate position calculated in step S14 and the spatial coordinates of the defective portion calculated in step S12.

[0100] If the second reference candidate position is closer to the camera (step S17: No), the CPU 60A calculates the distance between the second reference candidate position and the defective portion (step S19) from the spatial coordinates of the second reference candidate position calculated in step S16 and the spatial coordinates of the defective portion calculated in step S12.

[0101] Next, the CPU 60A outputs the distance between the reference candidate position and the defective portion calculated in step S18 or step S19 to, for example, the display 13a (step S20). The output on the display 13a will be described later with reference to FIGS.

[0102] In the above example, the power line is divided into sections and pan, tilt, and focus information is stored in association with each divided captured image, but the present invention is not limited to this. For example, a user may control the imaging device 10 via the personal computer 11 at a maintenance work site for the power line 91, capture images of three locations, namely, the defective section, the first reference candidate position, and the second reference candidate position, and store the pan, tilt, and focus information at that time.

[0103] 6 is a diagram showing an example of distance information displayed on the display 13a of the personal computer 11. The personal computer 11 displays, on the display 13a, distance information calculated by the CPU 60A of the information processing device 60, for example, indicating the distance between the defective portion and a reference candidate position close to the defective portion.

[0104] For example, as shown in FIG. 6 , the personal computer 11 displays, on the display 13a, distance information 100 indicating the distance between a power transmission line holding portion 82a of a steel tower 82 to which end portion B of the power transmission line 91 is connected and a defective portion 92 of the power transmission line 91. In this example, the power transmission line holding portion 82a of the steel tower 82 to which end portion B of the power transmission line 91 is connected is the reference candidate position closest to the defective portion 92. The distance information may be the straight-line distance between the two, or may be the distance between the two resolved in the horizontal and vertical directions. The personal computer 11 also displays, on the display 13a, a divided captured image 101a including the defective portion 92 of the power transmission line 91 and a divided captured image 101e including the power transmission line holding portion 82a of the steel tower 82 to which end portion B of the power transmission line 91 is connected. In this example, the distance information 100 indicates that the straight-line distance between the power line holding portion 82a of the steel tower 82 to which the end B of the power line 91 is connected and the defective portion 92 of the power line 91 is 15 m.

[0105] Fig. 7 is a diagram showing a modified example of distance information displayed on the display 13a of the personal computer 11. In the example shown in Fig. 6, only the divided captured image 101a including the defective portion 92 of the power line 91 and the divided captured image 101e including the power line holding portion 82a of the pylon 82 to which the end B of the power line 91 is connected are displayed on the display 13a, but in this modified example, an image of the power line 91 existing between the power line 91 in the divided captured image 101a and the power line 91 in the divided captured image 101e is also displayed on the display 13a. Specifically, as shown in Fig. 7, divided captured images 101b, 101c, and 101d of the power line 91 existing between the divided captured image 101a including the defective portion 92 and the divided captured image 101e including the power line holding portion 82a of the pylon 82 to which the end B of the power line 91 is connected are displayed on the display 13a.

[0106] As described above, the information processing device 60 of this embodiment can output distance information between the reference candidate position and the defective portion 92 based on position information acquired based on the rotation conditions (pan-tilt values) of the rotation device 16 with respect to the defective portion 92 and the imaging conditions (focus information) of the imaging device 10, and on position information acquired based on the rotation conditions of the rotation device 16 with respect to the reference candidate position (the power line holding portion of the steel towers 81, 82) and the imaging conditions of the imaging device 10. Therefore, it is possible to easily identify the position in real space of the defective portion 92 of the power transmission line 91, and maintenance work on the defective portion 92 of the power transmission line 91 can be performed smoothly.

[0107] <Errors when calculating distance from focus information> Fig. 8 is a diagram illustrating errors that occur when the distance from the imaging device to the steel tower is calculated based on focus information. When the distance from the imaging device to the subject is calculated based on focus information, the accuracy of the calculated distance generally decreases as the subject becomes farther away.

[0108] 8 , when the distance from the imaging device 10 (camera) to the steel tower 81 is greater than the distance from the imaging device 10 to the steel tower 82, the distance between the imaging device 10 and the steel tower calculated based on the focus information is less accurately calculated for the steel tower 81, which is farther away, than for the steel tower 82, which is closer. Specifically, while the correct distance from the imaging device 10 to the end A of the power transmission line 91 connected to the steel tower 81 is distance D1, when the distance to the end A of the power transmission line 91 is calculated based on the focus information acquired by the imaging device 10, the calculated distance may be distance D2, which is longer than distance D1. In this case, the position of the end of the power transmission line 91 calculated based on distance D2 calculated from the focus information is set to the position of end A2, which is farther away than end A1.

[0109] 9 is a flowchart showing a first modified example of processing by the CPU 60A of the information processing device 60. In the first modified example, taking into consideration the low accuracy of the distance calculated based on the focus information of a subject far from the imaging device 10 (camera), the focus information is not used to calculate the position information of the subject far from the imaging device 10, and distance information from the defective part of the power transmission line 91 to the reference position is output. Note that the operation performed by the user to start this modified processing is the same as the operation described in FIG. 5.

[0110] First, the CPU 60A sets the reference candidate position, either the first or second reference candidate position, which is closer to the camera (image capture device 10) as the close-distance reference position, and sets the farther reference candidate position as the long-distance reference position (step S31).Which of the first or second reference candidate position is closer to the camera may be determined by the CPU 60A from the focus information of the camera when the first reference candidate position was imaged and the focus information of the camera when the second reference candidate position was imaged, or may be specified by the user.

[0111] Next, the CPU 60A acquires the pan value, tilt value, and focus information from the secondary storage device 14 when the short-distance reference position is captured (step S32). The pan value, tilt value, and focus information are the pan value, tilt value, and focus information of the camera associated with the divided captured image that includes the short-distance reference position. For example, the short-distance reference position is the position of the power line holding unit that holds the end B of the power line 91 on the steel tower 82 shown in Figure 8.

[0112] Next, the CPU 60A calculates the spatial coordinates of the close-distance reference position based on the pan value, tilt value, and focus information acquired in step S32 (step S33). The spatial coordinates of the close-distance reference position can be calculated in the same way as the spatial coordinates of the defect location described in step S12 of FIG. 5 above.

[0113] Next, the CPU 60A acquires the pan and tilt values ​​used when the long-distance reference position was captured from the secondary storage device 14 (step S34). The pan and tilt values ​​are the pan and tilt values ​​of the camera associated with the divided captured image that includes the long-distance reference position. For example, the long-distance reference position is the position of the power line holding part that holds the end A of the power line 91 on the steel tower 81 shown in FIG. 8.

[0114] Next, the CPU 60A calculates the spatial coordinates of the long-distance reference position based on the pan and tilt values ​​for the long-distance reference position acquired in step S34 and the pan, tilt, and focus information for the short-distance reference position acquired in step S32 (step S35). The calculation of the spatial coordinates of the long-distance reference position will be described later with reference to FIG. 10.

[0115] Next, the CPU 60A calculates a vertical plane including the short-distance reference position calculated in step S33 and the long-distance reference position calculated in step S35 (step S36). The defective portion, which is a location on the power transmission line 91, exists on the vertical plane including the short-distance reference position and the long-distance reference position. The vertical plane is a common plane on which the short-distance reference position, the long-distance reference position, and the defective portion exist. The vertical plane is a common plane in which an error is allowed.

[0116] Next, the CPU 60A acquires the pan and tilt values ​​used when the defective area was captured from the secondary storage device 14 (step S37). The pan and tilt values ​​are the pan and tilt values ​​of the camera associated with the divided captured image containing the defective area. For example, the defective area is defective area 92 shown in FIG. 8.

[0117] Next, the CPU 60A calculates the spatial coordinates of the defect location based on the pan and tilt values ​​acquired in step S37 and the vertical plane calculated in step S36 (step S38). The spatial coordinates of the defect location can be calculated by determining the intersection of a straight line determined based on the pan and tilt values ​​acquired in step S37 and the vertical plane calculated in step S36.

[0118] Next, the CPU 60A calculates the distance between the short-distance reference position and the defective portion (step S39) from the spatial coordinates of the short-distance reference position calculated in step S33 and the spatial coordinates of the defective portion calculated in step S38.

[0119] Next, the CPU 60A outputs the distance between the short-distance reference position and the defective location calculated in step S39 to, for example, the display 13a (step S40).

[0120] Figure 10 is a diagram illustrating calculation of spatial coordinates of the long-distance reference position in the first modified example of Figure 9. As shown in Figure 10, a power transmission line 91 is connected between a steel tower 81 and a steel tower 82. End A of the power transmission line 91 is held by a power transmission line holding part of the steel tower 81, and end B of the power transmission line 91 is held by a power transmission line holding part of the steel tower 82. The power transmission line holding part of the steel tower 81 to which end A of the power transmission line 91 is connected is the long-distance reference position, and the power transmission line holding part of the steel tower 82 to which end B of the power transmission line 91 is connected is the short-distance reference position.

[0121] For example, if the height hA of the long-distance reference position and the height hB of the short-distance reference position are the same (hA = hB = h), the distance (distA) from the image capture device 10 to the long-distance reference position can be calculated using the following formula:

[0122] hA=hB h=distB×sin(tiltB) distA=h / sin(tiltA) =distB×sin(tiltB) / sin(tiltA)

[0123] Furthermore, for example, if the height hA of the long-distance reference position and the height hB of the short-distance reference position have a predetermined height ratio (hA = hB × r), the distance (distA) from the image capture device 10 to the long-distance reference position can be calculated using the following formula:

[0124] hA=hB×r hB=distB×sin(tiltB) distA=hA / sin(tiltA) =distB×sin(tiltB)×r / sin(tiltA)

[0125] In this way, the distance (distA) from the image capture device 10 to the long-distance reference position can be calculated without using focus information of the long-distance reference position. As a result, the spatial coordinates of the long-distance reference position can be calculated as coordinates moved from the spatial coordinates of the image capture device 10 by the distance (distA) to the long-distance reference position in an imaging direction specified by the pan value and tilt value associated with the divided captured image that includes the long-distance reference position. Note that information indicating the height relationship between the long-distance reference position and the short-distance reference position (same height, height ratio) can be specified, for example, by an input operation by the user.

[0126] 10 , a vertical plane 110 indicated by diagonal lines and including end A and end B of the power transmission line 91 is the vertical plane calculated in step S36 in FIG. A short-distance reference position, a long-distance reference position, and a defective portion 92 of the power transmission line 91 exist on the vertical plane 110. Therefore, from the spatial coordinates of the imaging device 10, the intersection of a straight line extended in an imaging direction specified by the pan value and tilt value associated with the divided captured image including the defective portion 92 and the vertical plane 110 can be calculated as the spatial coordinate of the defective portion 92. Then, based on the calculated spatial coordinates, the distance from the short-distance reference position to the defective portion 92 of the power transmission line 91 can be calculated.

[0127] As described above, according to the first modified example of processing by the CPU 60A, position information of the long-distance reference position can be acquired based on the rotation conditions (pan / tilt values) of the rotation device 16 and the imaging conditions (focus information) of the imaging device 10 relative to the short-distance reference position, and the rotation conditions of the rotation device 16 relative to the long-distance reference position. Therefore, when the distance from the imaging device 10 to the long-distance reference position is long and the accuracy of the distance information based on the imaging conditions of the long-distance reference position is low, the distance information of the long-distance reference position can be acquired with high accuracy without using the imaging conditions of the long-distance reference position. Furthermore, position information of the defect point 92 can be acquired based on the position information of the short-distance reference position, the position information of the long-distance reference position, and the rotation conditions of the rotation device 16 relative to the defect point 92. Therefore, when the distance from the imaging device 10 to the defect point 92 is long and the accuracy of the distance information based on the imaging conditions of the defect point 92 is low, the distance information of the defect point 92 can be acquired with high accuracy without using the imaging conditions of the defect point 92. This makes it possible to calculate the distance from the defective part 92 to the short-distance reference position, and to easily identify the position in real space of the defective part 92 of the power transmission line 91. Therefore, maintenance work on the defective part 92 can be carried out smoothly.

[0128] 11 is a flowchart showing a second modified example of processing by the CPU 60A of the information processing device 60. Like the first modified example, the second modified example calculates position information of a subject that is far from the imaging device 10 (camera) without using focus information of the far subject. However, the second modified example differs from the first modified example in that the distance between a close subject and a far subject is used to calculate the position information of the far subject.

[0129] As shown in FIG. 11, in the second modified example, the processes from step S31 to step S34 are the same as the processes from step S31 to step S34 in the first modified example described with reference to FIG.

[0130] Next, the CPU 60A acquires the distance L between the pylon 81 to which the end A of the power transmission line 91 is connected and the pylon 82 to which the end B of the power transmission line 91 is connected (step S35A). The distance between the pylon 81 and the pylon 82 is a distance that is input and specified by the user by operating the personal computer 11, for example.

[0131] Next, CPU 60A calculates the spatial coordinates of the long-distance reference position based on the distance L between pylons 81 and 82 acquired in step S35A and the pan value and tilt value for the long-distance reference position acquired in step S34 (step S35B). The calculation of the spatial coordinates of the long-distance reference position will be described later with reference to FIG. 12.

[0132] The processes from step S36 to step S40 are the same as the processes from step S36 to step S40 in the first modified example described with reference to FIG.

[0133] FIG. 12 is a diagram illustrating the calculation of spatial coordinates of the long-distance reference position in the second modified example of FIG. 11 . In FIG. 12 , end A of power line 91 held by a power line holding portion of pylon 81, end B of power line 91 held by a power line holding portion of pylon 82, and imaging device position 10a where imaging device 10 (camera) is installed are viewed from above. In a horizontal plane represented by the x-axis and y-axis, the intersection of the x-axis and y-axis is defined as imaging device position 10a, and the y-axis direction is defined as the reference imaging direction specified by the pan value when the camera rotation amount is 0 (zero). Furthermore, the distance between end A of power line 91 connected to pylon 81 and end B of power line 91 connected to pylon 82 is defined as the distance L between the pylons. End B of power line 91 is the short-distance reference position in FIG. 11 , and end A of power line 91 is the long-distance reference position in FIG. 11 .

[0134] The distance on the xy horizontal plane from the imaging device position 10a to the end B of the power transmission line 91 can be expressed as distB × cos(tiltB) (see FIG. 10). Therefore, the position of the end B of the power transmission line 91 is expressed as (distB × cos(tiltB) × sin(PanB), distB × cos(tiltB) × cos(PanB)).

[0135] Therefore, the position of end A of power transmission line 91 on the xy horizontal plane can be found by drawing a circle 120 with a radius equal to distance L and centered at end B of power transmission line 91, and calculating the intersection of circle 120 and the imaging direction of end A of power transmission line 91 specified by Pan A. This makes it possible to calculate the distance from imaging device position 10a to end A of power transmission line 91.

[0136] In this way, the distance from the imaging device 10 to the long-distance reference position can be calculated without using focus information for the long-distance reference position by using the distance L between the steel towers. As a result, the spatial coordinates of the long-distance reference position can be calculated as coordinates moved vertically from the position of end A of the power transmission line 91 on the xy horizontal plane by the height hA of the long-distance reference position.

[0137] As described above, according to the second modified example of processing by the CPU 60A, position information of the long-distance reference position can be acquired based on position information acquired based on the rotation conditions (pan / tilt values) of the rotation device 16 relative to the short-distance reference position and the imaging conditions (focus information) of the imaging device 10, the distance L between the short-distance reference position and the long-distance reference position, and the rotation conditions of the rotation device 16 relative to the long-distance reference position. Therefore, as with the first modified example, even if the distance from the imaging device 10 to the long-distance reference position is long, distance information of the long-distance reference position can be acquired with high accuracy. Furthermore, even if the distance from the imaging device 10 to the defective portion 92 is long, distance information of the defective portion 92 can be acquired with high accuracy. This makes it possible to easily identify the location of the defective portion 92 of the power transmission line 91 in real space, thereby enabling smooth maintenance work on the defective portion 92.

[0138] In the above embodiment, the case where the defective part is located in the power transmission line has been described, but the present invention is not limited to this. The defective part may be located on the wall surface of a building, for example. In other words, the present invention can also be applied to cases such as inspection and maintenance of the wall surface of a building.

[0139] <Storage medium for information processing program> In each of the above imaging controls, an example has been described in which the information processing program of each embodiment is stored in the storage 60B of the information processing device 60, and the CPU 60A of the information processing device 60 executes the information processing program in the memory 60C, but the technology of the present disclosure is not limited to this.

[0140] 13 is a diagram showing an example of how an information processing program for imaging control is installed from a storage medium storing the information processing program into the information processing device 60 of the personal computer 11. As an example, as shown in Fig. 13, an information processing program 221 may be stored in a storage medium 220, which is a non-transitory storage medium. In the example shown in Fig. 13, the information processing program 221 stored in the storage medium 220 is installed into the information processing device 60, and the CPU 60A executes the above-described processes in accordance with the information processing program 221.

[0141] Although various embodiments have been described above, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention. Furthermore, the components of the above embodiments may be combined in any manner as long as they do not deviate from the spirit of the invention.

[0142] This application is based on a Japanese patent application (Patent Application No. 2023-185446) filed on October 30, 2023, the contents of which are incorporated herein by reference.

[0143] 1 Imaging system 10 Imaging device 10a Imaging device position 11 Personal computer 12 Communication line 13a, 43B Display 13b Keyboard 13c Mouse 14 Secondary storage device 15 Optical system 15B Lens group 15B1 Anti-vibration lens 15B2 Zoom lens 16 Swivel device 17, 21 Lens actuator 19 Computer 22 BIS driver 23 OIS driver 25 Imaging element 25A Light receiving surface 27 Imaging element actuator 28 Lens driver 29, 45 Correction mechanism 31 DSP 32 Image memory 33 Correction unit 34, 66, 68, 78 Communication I / F 35, 60C Memory 36, 60B Storage 37, 60A CPU 38, 69 Bus 39, 47 Position sensor 40 Quantity detection sensor 43 UI device 43A, 62 Reception device 60 Information processing device 71 Yaw axis rotation mechanism 72 Pitch axis rotation mechanism 73, 74 Motor 75, 76 Driver 81, 82 Steel tower 82a Power transmission line holding unit 91 Power transmission line 92 Defective part 100 Distance information 101a, 101b, 101c, 101d, 101e Divided captured image 110 Vertical plane 120 Circle 220 Storage medium 221 Information processing program D1, D2 Distance A1, A2 End

Claims

1. An information processing device having a processor capable of communicating between an imaging device and a rotation device that rotates the imaging device, wherein the processor acquires first position information based on a rotation condition of the rotation device corresponding to a first subject, acquires second position information based on a rotation condition of the rotation device corresponding to a second subject, acquires third position information based on a rotation condition of the rotation device corresponding to a third subject, and outputs distance information between the first subject and either the second subject or the third subject based on the first position information, the second position information, and the third position information.

2. An information processing device according to claim 1, wherein the distance information is distance information between the first subject and one of the second subject and the third subject that is closer to the imaging device.

3. An information processing device according to claim 1, wherein the first subject, the second subject, and the third subject are present on a first surface.

4. An information processing device as described in claim 1, wherein the processor acquires the first position information based on the rotation conditions of the rotation device and the imaging conditions of the imaging device corresponding to the first subject, acquires the second position information based on the rotation conditions of the rotation device and the imaging conditions of the imaging device corresponding to the second subject, and acquires the third position information based on the rotation conditions of the rotation device and the imaging conditions of the imaging device corresponding to the third subject.

5. An information processing device as described in claim 1, wherein the second subject is a subject closer to the imaging device than the third subject, and the processor acquires the second position information based on the rotation conditions of the rotation device corresponding to the second subject and the imaging conditions of the imaging device, and acquires the third position information based on the rotation conditions of the rotation device corresponding to the second subject and the imaging conditions of the imaging device, and the rotation conditions of the rotation device corresponding to the third subject.

6. An information processing device as described in claim 1, wherein the second subject is a subject closer to the imaging device than the third subject, and the processor acquires the second position information based on the rotation conditions of the rotation device corresponding to the second subject and the imaging conditions of the imaging device, and acquires the third position information based on the second position information, the distance between the second subject and the third subject, and the rotation conditions of the rotation device corresponding to the third subject.

7. An information processing device according to claim 5, wherein the first subject, the second subject, and the third subject are present on a first surface, and the processor acquires the first position information based on the second position information, the third position information, and a rotation condition of the rotation device corresponding to the first subject.

8. An information processing device according to claim 5, wherein the processor determines that the second subject is closer to the imaging device than the third subject based on imaging conditions of the imaging device corresponding to the second subject and imaging conditions of the imaging device corresponding to the third subject.

9. An information processing device according to claim 5, wherein the processor acquires the third position information based on information indicating a relative height relationship between the second subject and the third subject.

10. An information processing device according to claim 1, wherein the rotation conditions of the rotation device include at least one of a pan state and a tilt state of the rotation device.

11. An information processing device according to claim 1, wherein the imaging conditions of the imaging device include focus information of the imaging device.

12. A control device that controls the imaging device and / or the rotation device based on information generated by an information processing device according to any one of claims 1 to 11.

13. An information processing method by an information processing device having a processor capable of communicating between an imaging device and a rotation device that rotates the imaging device, wherein the processor acquires first position information based on a rotation condition of the rotation device corresponding to a first subject, acquires second position information based on a rotation condition of the rotation device corresponding to a second subject, acquires third position information based on a rotation condition of the rotation device corresponding to a third subject, and outputs distance information between the first subject and either the second subject or the third subject based on the first position information, the second position information, and the third position information.

14. An information processing program by an information processing device having a processor capable of communicating between an imaging device and a rotation device that rotates the imaging device, the information processing program causing the processor to execute the following processes: acquire first position information based on the rotation conditions of the rotation device corresponding to a first subject; acquire second position information based on the rotation conditions of the rotation device corresponding to a second subject; acquire third position information based on the rotation conditions of the rotation device corresponding to a third subject; and output distance information between either the second subject or the third subject and the first subject based on the first position information, the second position information, and the third position information.

Images