Image processing device and image processing method

Abstract

The image processing device comprises: a receiving unit that receives position information of a photographic object and a photographic image captured by at least one camera and showing the photographic object; a prediction unit that predicts the position of the photographic object within a photographic range of the camera based on the position information of the photographic object; a detection unit that detects the photographic object by reading a photographic image of a limited range that is a part of the photographic range from the photographic image of the photographic range based on the predicted position of the photographic object; a measuring unit that measures the position of the detected photographic object; and an output unit that outputs the difference between the measured position of the photographic object and the predicted position.

Description

Technical Field

[0001] The present disclosure relates to an image processing device and an image processing method. Background Art

[0002] Patent document 1 discloses a component mounting coordinate correction method, in which when mounting an electronic component on a printed circuit board, an operator measures the coordinates of a mark used as a reference for positioning, and inputs the coordinates into the printed circuit board. In this method, the coordinates of two points of an electronic component mounting position pattern close to the marked pattern position are obtained, and the true mark position is determined with the aid of a camera unit based on the deviation between the true coordinate position of the mounting position pattern and the coordinate position including an error based on the marked coordinates, and the component mounting coordinates are corrected based on the true mark position.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2001-284899 Summary of the invention

[0006] Problem that the invention aims to solve

[0007] However, in the structure of patent document 1, it is impossible to correct the error caused by external reasons such as the movement error when moving from the marking position to the component mounting coordinate after the coordinate is corrected, so the accuracy of the correction of the position information is limited. In addition, in the structure of patent document 1, for example, image processing of the camera image captured by the camera is performed to calculate the deviation between the designed coordinates and the actual coordinates to correct the coordinate error. However, in the method of correcting the coordinate error using the camera image, due to the camera speed, the reading of the camera image and the processing of the read image, etc., it takes a specified time before the coordinate error is calculated, which may become a reason for limiting the increase in the operation speed of other devices (such as the installation speed of electronic components).

[0008] The present disclosure has been made in view of the above-mentioned conventional situation, and an object thereof is to provide an image processing device and an image processing method that perform efficient image processing on an image of an object captured by a camera and calculate a position error of the object with higher accuracy.

[0009] Solutions for solving problems

[0010] The present disclosure provides an image processing device, comprising: a receiving unit, which receives position information of a camera object and a camera image captured by at least one camera and showing the camera object; a prediction unit, which predicts the position of the camera object within the camera range of the camera based on the position information of the camera object; a detection unit, which detects the camera object by reading a camera image of a limited range that is a part of the camera range from the camera image of the camera range based on the predicted position of the camera object; a measuring unit, which measures the detected position of the camera object; and an output unit, which outputs the difference between the measured position of the camera object and the predicted position.

[0011] In addition, the present disclosure provides an image processing device, comprising: a receiving unit, which receives position information of a camera and a camera image captured by at least one of the cameras; a detecting unit, which reads out a camera image of a limited range that is a part of the camera range of the camera from at least one of the camera images to detect a camera object that serves as a reference for the position of the camera; a measuring unit, which measures the position of the detected camera object; a predicting unit, which predicts the position of the camera object that appears in a camera image captured after the camera image used to detect the camera object based on the measured position of the camera object; and an output unit, which outputs the difference between the predicted position of the camera object and the measured position of the camera object.

[0012] In addition, the present disclosure provides an image processing method performed by an image processing device connected to at least one camera, in which position information of a camera object and a camera image captured by the camera and containing the camera object are received, the position of the camera object within the camera range of the camera is predicted based on the position information of the camera object, and based on the predicted position of the camera object, a specified limited range within the camera range of the camera containing the predicted position is read out to detect the camera object, the detected position of the camera object is measured, and the difference between the measured position of the camera object and the predicted position is output.

[0013] In addition, the present disclosure provides an image processing method performed by an image processing device connected to at least one camera, in which a camera image captured by the camera and containing the camera object is received, at least one camera image of a limited range that is part of the camera range of the camera is read out to detect the camera object serving as a reference for the position of the camera, the position of the detected camera object is measured, and based on the measured position of the camera object, the position of the camera object reflected in a camera image captured after the camera image used to detect the camera object is predicted, and the difference between the predicted position of the camera object and the measured position of the camera object is output.

[0014] Effects of the Invention

[0015] According to the present disclosure, efficient image processing can be performed on an image of an object captured by a camera, and a position error of the object can be calculated with higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is an explanatory diagram of an example of usage of the image processing system according to the first embodiment.

[0017] Figure 2 FIG. 1 is a timing chart showing an example of image reading and image processing according to a comparative example.

[0018] Figure 3 FIG. 1 is a timing chart showing an example of image reading and image processing in the image processing device according to the first embodiment.

[0019] Figure 4 This is a diagram showing an example of each range of the imaging range and the limited range.

[0020] Figure 5 The diagram shows an example of how the imaging subject shown in each of a plurality of limited ranges changes over time.

[0021] Figure 6 This is a sequence diagram for explaining an example of the operation procedure of the image processing system according to the first embodiment.

[0022] Figure 7 This is a flowchart for explaining an example of a basic operation procedure of the image processing device according to the first embodiment.

[0023] Figure 8 This is an explanatory diagram of an example of usage of an image processing system including a plurality of cameras according to the second embodiment.

[0024] Fig. 9This is a flowchart for explaining an example of the operation procedure of an image processing device including a plurality of cameras according to the second embodiment.

[0025] Fig.10 2 is a diagram showing an example of feature point detection.

[0026] Fig.11 This is a flowchart for explaining an example of the operation procedure of the image processing device according to the second embodiment for detecting feature points.

[0027] Fig.12 This is an explanatory diagram of an example of usage of the image processing system including an unmanned aircraft according to the second embodiment.

[0028] Fig.13 This is a flowchart illustrating an example of the tracking and detection operation process of the image processing device involved in embodiment 2.

[0029] Fig.14 This is a diagram for explaining an example of switching between the tracking limited range and the detection limited range.

[0030] Fig.15 This is a diagram for explaining an example of tracking and detection of an imaging object. DETAILED DESCRIPTION

[0031] (Process of realizing the contents of Implementation Mode 1)

[0032] For example, there is a component mounting coordinate correction method for correcting component mounting coordinates when electronic components are mounted on a printed circuit board. In such a component mounting coordinate correction method, the coordinates of a mark used as a reference for positioning are measured by an operator and input into a printed circuit board, and the true mark position is determined by using a camera unit based on the deviation amount between the mark position and the coordinate position containing an error, and the component mounting coordinates are corrected based on the true mark position. However, errors caused by external factors such as movement errors when moving from the mark position to the component mounting coordinates after the coordinates are corrected cannot be corrected, so the accuracy of the correction of the position information is limited. In addition, the component mounting coordinate correction performed by the camera unit requires a predetermined time before the coordinate error is calculated due to the camera speed, the reading of the camera image, and the processing of the read image, so the improvement of the movement speed of other devices, such as the installation speed of electronic components, is limited. That is, in such a component mounting coordinate correction method using a camera image, when considering the influence on the movement speed of other devices, the number of camera images to be processed is limited, and it is difficult to increase the number of samples for achieving higher accuracy error correction. In the above-mentioned Patent Document 1, the method of coordinate correction using the imaging unit does not take into consideration the reduction of the time required for image processing.

[0033] As described above, an example of an image processing device and an image processing method in which an image processing device performs efficient image processing on an image of an object captured by a camera and calculates a position error of the object with higher accuracy will be described.

[0034] Below, with appropriate reference to the drawings, embodiment 1 that specifically discloses the structure and function of the image processing device and image processing method involved in the present disclosure is described in detail. However, unnecessary detailed descriptions are sometimes omitted. For example, detailed descriptions of well-known matters and repeated descriptions of substantially the same structures are sometimes omitted. This is to avoid the following description from becoming too lengthy and to make it easy for those skilled in the art to understand. In addition, the drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and it is not intended to limit the subject matter described in the claims through these drawings and descriptions.

[0035] (Implementation Method 1)

[0036] Figure 1 1 is an explanatory diagram of an example of use of the image processing system according to Embodiment 1. The image processing system includes a control device 1 , an actuator 2 , a camera 3 , and an image processing device 4 . The control device 1 is a device for controlling the actuator 2 , the camera 3 , and the image processing device 4 .

[0037] First, the control device 1 will be described. The control device 1 includes a control unit 10, a memory 11, and a region data 12. The control device 1 and the actuator 2 are connected to each other in a communicable manner.

[0038] The control unit 10 is configured using, for example, a CPU (Central Processing Unit) or an FPGA (Field Programmable Gate Array), and performs various processes and controls in cooperation with the memory 11. Specifically, the control unit 10 refers to the program and data stored in the memory 11, and implements the function of the area data 12 described later by executing the program. The control unit 10 is connected to the control unit 20 of the actuator 2 in a communicable manner. The control unit 10 controls the actuator 2 based on the area data 12 input by the user operation.

[0039] The memory 11 includes, for example, a RAM (Random Access Memory) as a working memory used when executing each process of the control unit 10, and a ROM (Read Only Memory) storing programs and data that define the operation of the control unit 10. Data or information generated or acquired by the control unit 10 is temporarily stored in the RAM. Programs for defining the operation of the control unit 10 (for example, a method of reading data and programs written in the area data 12 and controlling the actuator 2 based on these data and programs, etc.) are written in the ROM.

[0040] The area data 12 is, for example, data created using a design support tool such as CAD (Computer Aided Design). The area data 12 is data having design information or position information (e.g., position information about the imaging object Tg1 captured by the camera 3 and stored in the area data 12, position information for the work unit 5 to place, weld, or fuse components, etc.), and a program for operating a drive device such as the actuator 2 is written therein.

[0041] Next, the actuator 2 is described. The actuator 2 is, for example, a drive device capable of electric control or flight control. The actuator 2 is connected to the control device 1 and the image processing device 4 in a communicable manner. The actuator 2 is configured to include a control unit 20, a memory 21, a drive unit 22, and an arm 24. In addition, the working unit 5 is not a necessary structure and can be omitted.

[0042] The control unit 20 is configured using, for example, a CPU or an FPGA, and performs various processing and control in cooperation with the memory 21. Specifically, the control unit 20 refers to the program and data stored in the memory 21, and implements the function of the error correction unit 23 by executing the program. The control unit 20 is connected to the control unit 10, the control unit 40, and the receiving unit 42 in a communicable manner. The control unit 20 drives the driving unit 22 based on the control signal received from the control device 1, so that the working unit 5 performs a predetermined control.

[0043] When the actuator 2 is activated, the control unit 20 performs initial position alignment based on the reference mark Pt0 between the camera 3 driven by the driving unit 22 and the working unit 5. In addition, the initial position alignment can be performed according to any timing specified by the user, such as a change in the imaging object, the completion of the work performed by the working unit 5, etc.

[0044] The control unit 20 sends various information such as the position information of the imaging object Tg1 and the position information of the camera 3 included in the area data 12 received from the control device 1 to the image processing device 4. The various information includes information such as the frame rate, imaging range IA1, and zoom factor of the camera 3. In addition, when the imaging position of the camera 3 is moved based on the program written in the area data 12, the control unit 20 sends information that can estimate the position of the camera 3 (for example, the position information of the camera 3 or the moving speed information of the camera 3) to the image processing device 4. In addition, for example, when the camera 3 is fixed or when the imaging range IA1 of the camera 3 includes all possible positions where the imaging object Tg1 may be located, the information that can estimate the position of the camera 3 may be omitted.

[0045] The control unit 20 receives difference information (in other words, position error information) about the position of the imaging object Tg1 based on the imaging image captured by the camera 3 from the image processing device 4. The control unit 20 causes the error correction unit 23 to perform error correction based on the received difference information.

[0046] The memory 21 includes, for example, a RAM as a working memory used when executing each process of the control unit 20, and a ROM storing programs and data that define the operation of the control unit 20. Data or information generated or acquired by the control unit 20 is temporarily stored in the RAM. Programs for defining the operation of the control unit 20 (for example, a method for moving the camera 3 and the working unit 5 to a predetermined position based on a control signal from the control device 1) are written in the ROM.

[0047] The driving unit 22 moves the camera 3 and the working unit 5 based on the position information of the imaging object Tg1 with the reference mark Pt0 as the base point. The driving unit 22 transmits the moving speeds of the camera 3 and the working unit 5 to the image processing device 4 via the control unit 20 .

[0048] The error correction unit 23 corrects the positions of the camera 3 and the working unit 5 moved by the driving unit 22 based on the difference information received from the image processing device 4. In addition, when the camera 3 and the working unit 5 are fixedly installed, the error correction unit 23 corrects the position information of the imaging object Tg1 stored in the area data 12 (that is, CAD data, etc.) based on the received difference information.

[0049] The arm 24 is connected to a support stand 26 that integrally supports the camera 3 and the working unit 5. The arm 24 is driven by the driving unit 22, and moves the camera 3 and the working unit 5 integrally via the support stand 26.

[0050] The camera 3 has a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) as an imaging element. The camera 3 has a focus lens (not shown) capable of adjusting the focal length, a zoom lens (not shown) capable of changing the zoom ratio, and a gain adjustment unit (not shown) capable of adjusting the sensitivity of the imaging element.

[0051] The camera 3 is configured using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field Programmable Gate Array). The camera 3 performs predetermined signal processing using the electrical signal of the camera image, thereby generating data (frames) of the camera image specified by RGB (Red Green Blue) or YUV (brightness / color difference) that can be recognized by humans. The camera 3 sends the data of the camera image (hereinafter referred to as the camera image) to the image processing device 4. In addition, the camera image captured by the camera 3 is stored in the memory 41.

[0052] In addition, the camera 3 has an imaging range IA1. The camera 3 is a high-speed camera that generates data (frames) of an imaging object Tg1 at a predetermined frame rate (e.g., 120fps (frame per second)). In addition, the predetermined frame rate can be arbitrarily set by the user according to the imaging range IA1 and the size of the limited range described later. Specifically, the predetermined frame rate can also be, for example, 60fps, 240fps, etc.

[0053] also, Figure 1 The camera 3 shown is configured to be able to change the camera position through the arm 24, and can be fixedly installed on the bottom or side of the actuator 2 according to the purpose, or can be fixedly installed on other support platforms (not shown) that can capture the camera object Tg1. Figure 1 The imaging range IA1 of the camera 3 shown in the figure shows a range including the reference mark Pt0 and the imaging object Tg1, but when the imaging position of the camera 3 is set to be variable, the reference mark Pt0 and the imaging object Tg1 may be captured at different predetermined imaging positions. That is, the camera 3 involved in the first embodiment may be set to be able to capture the reference mark Pt0 and the imaging object Tg1 or have an imaging range IA1 that can capture.

[0054] Furthermore, when the imaging position of the camera 3 is fixed and the imaging range IA1 of the camera 3 can capture all positions where the imaging object Tg1 can be disposed, the reference mark Pt0 may be omitted. That is, in such a case, the camera 3 according to the first embodiment only needs to capture the imaging object Tg1.

[0055] Next, the image processing device 4 will be described. The image processing device 4 is communicably connected to the actuator 2 and the camera 3. The image processing device 4 includes a control unit 40, a memory 41, and a receiving unit 42.

[0056] The control unit 40 is configured using, for example, a CPU or an FPGA, and performs various processes and controls in cooperation with the memory 41. Specifically, the control unit 40 refers to the program and data stored in the memory 41, and implements the functions of each unit by executing the program. Each unit includes a prediction unit 43, a detection unit 44, a measurement unit 45, and an output unit 46.

[0057] The memory 41 includes, for example, a RAM as a working memory used when executing each process of the control unit 40, and a ROM for storing programs and data that define the operation of the control unit 40. Data or information generated or acquired by the control unit 40 is temporarily stored in the RAM. Programs for defining the operation of the control unit 40 (for example, a method for predicting the position of the received imaging object Tg1, a method for detecting the imaging object Tg1 from the read limited range, and a method for measuring the position of the detected imaging object Tg1) are written in the ROM. In addition, the memory 41 stores the received imaging image, the position information of the imaging object Tg1, and the limited range described later.

[0058] The receiving unit 42 is connected to the control unit 20 of the actuator 2 and the camera 3 in a communicable manner. The receiving unit 42 receives the position information of the imaging object Tg1 and the information that can estimate the position of the camera 3 (for example, the position information of the camera 3 or the moving speed information of the camera 3) from the control unit 20, outputs the received position information of the imaging object Tg1 and the information that can estimate the position of the camera 3 to the prediction unit 43, and outputs the received position information of the imaging object Tg1 to the output unit 46. The receiving unit 42 receives the data of the camera image captured by the camera 3, and outputs the received camera image data to the detection unit 44.

[0059] In addition, the receiving unit 42 outputs the received various information of the camera 3 to the control unit 40. The various information outputted from the receiving unit 42 is further outputted to each unit through the control unit 40.

[0060] The prediction unit 43 predicts the position of the object Tg1 shown in the received camera image based on the position information of the object Tg1 stored in the area data 12 output from the receiving unit 42 and the information that can estimate the position of the camera 3 after being moved by the actuator 2. Specifically, the prediction unit 43 predicts the position of the object Tg1 in the image sensor of the camera 3. The prediction unit 43 outputs the predicted position of the object Tg1 to the detection unit 44 and the output unit 46. In addition, the position of the object Tg1 predicted by the prediction unit 43 is not only the position of the object Tg1 captured in the next frame (specifically, the image captured after the image used when detecting the object), but may also be the position of the object Tg1 captured several frames later.

[0061] The detection unit 44 reads out the limited range of the image sensor as a part of the imaging range IA1 including the predicted position predicted by the prediction unit 43 (that is, the predicted position of the imaging object Tg1 in the image sensor) in the imaging image captured and received by the camera 3, and detects the imaging object Tg1 reflected in the limited range of the imaging image. The detection unit 44 outputs the detection result to the measurement unit 45. In addition, the limited range may be a predetermined range pre-set in the memory 41, or a predetermined range centered on the predicted position. The limited range will be described later.

[0062] As described above, the detection unit 44 reads out the limited range in the imaging range IA1 in a limited manner, thereby shortening the time required for the readout process compared to the readout process of the entire area of ​​the imaging image in the comparative example. In addition, the detection unit 44 can reduce the load required for the readout process by reducing the readout range. Therefore, the image processing device 4 according to the first embodiment can perform efficient image processing on the image of the imaging object Tg1 captured by the camera 3, and can calculate the position error of the imaging object Tg1 with higher accuracy.

[0063] Furthermore, in the coordinate error correction method using the camera image of the comparative example, the readout process takes time, and therefore the number of camera images to be processed is limited in consideration of the influence on the operation speed of other devices. However, the image processing device 4 according to the first embodiment can shorten the readout time by restrictively reading the limited range in the camera range IA1, and thus can suppress the influence on the operation speed of other devices. In addition, the image processing device 4 according to the first embodiment can increase the number of samples by shortening the readout time, and thus can realize the position error correction with higher accuracy.

[0064] The measuring unit 45 measures the position of the imaging target Tg1 that appears in the limited range on the imaging image detected by the detecting unit 44. The measuring unit 45 outputs the measured position of the imaging target Tg1 to the output unit 46.

[0065] The output unit 46 outputs the difference between the predicted position of the imaging object Tg1 in the image sensor and the measured position of the imaging object Tg1 in the actually captured image. Thus, the output unit 46 can output the error between the position of the imaging object Tg1 received from the actuator 2 and the actually detected position.

[0066] The output unit 46 sends the calculated difference information (in other words, error information) to the error correction unit 23 of the actuator 2. The error correction unit 23 corrects the error related to the position of the arm 24 driven by the drive unit 22 (in other words, the imaging position of the camera 3 and the working position of the working unit 5) based on the received difference information.

[0067] The working unit 5 is, for example, a component mounting head capable of mounting electronic components, a soldering iron capable of performing soldering, or a welding rod capable of performing welding, etc. The working unit 5 is driven to change its position by the driving unit 22. In addition, the working unit 5 can be provided as a working unit capable of replacing the above-mentioned user-required operation.

[0068] The imaging target Tg1 is set based on the area data 12. Figure 1 In the description of FIG. 3 , the imaging object Tg1 is assumed to stay at a predetermined position, but the present invention is not limited thereto. The imaging object Tg1 is, for example, a component, and its position may change according to a fixed speed of a conveying track or the like. In this case, the image processing device 4 receives the moving speed information of the camera 3 and the moving speed information of the imaging object Tg1, and performs image processing in consideration of the relative speed.

[0069] Next, refer to Figure 2 and Figure 3 The time required for image processing by the camera of the comparative example is compared with the time required for image processing by the camera 3 according to the first embodiment. Figure 2 FIG. 1 is a timing chart showing an example of image reading and image processing according to a comparative example. Figure 3 FIG. 1 is a timing chart showing an example of image reading and image processing in the image processing device according to the first embodiment. Figure 2 and Figure 3 The transmission in the processing performed by the image processing device 4 shown in the figure indicates the processing of reading the captured image. The calculation indicates the processing until the captured object Tg1 is detected from the captured image read out and the position of the detected captured object Tg1 is measured to calculate and output the difference from the designed position of the captured object Tg1. Figure 2The camera of the comparative example shown and Figure 3 The imaging range of the camera 3 according to the first embodiment shown is assumed to be an imaging range IA1 .

[0070] Figure 2 The camera of the comparative example shown is in a non-exposure state from time 0 (zero) to time s2, and becomes exposed from time s2 to time s3. When the exposure state of the camera of the comparative example ends, the image processing device of the comparative example reads out the entire area of ​​the imaging range IA1 from time s3 to time s6, and performs image processing from time s6 to time s7. That is, it takes time s7 for the image processing system using the camera and the image processing device of the comparative example to output a primary error.

[0071] on the other hand, Figure 3 The camera 3 involved in the embodiment 1 shown ends the exposure state during the period from time 0 (zero) to time s1. The image processing device 4 starts the readout process from time s1 when the camera 3 ends the exposure state. The image processing device 4 ends the readout process during the period from time s1 to time s2 by limiting the reading of only the limited area in the captured imaging range IA1, and completes the image processing during the period from time s2 to time s3. That is, it takes time s3 for the image processing system involved in the embodiment 1 to output an error. Therefore, the image processing system involved in the embodiment 1 can shorten the time required for the readout transmission, so as Figure 3 As shown, the camera 3 can quickly repeat the exposure state, so that more errors can be output quickly.

[0072] As described above, the image processing system according to the first embodiment can shorten the time required for the readout process by limiting the readout of the image in the image processing device 4 to a limited range, thereby being able to set the frame rate of the camera 3 to be faster. In addition, the image processing system according to the first embodiment can thereby obtain a greater number of samples (in other words, the number of error information outputted) in the same time, and thus can make the accuracy of the position error correction higher.

[0073] In addition, the camera 3 may not be Figure 3 The exposure state is continuously repeated as shown, and there may be a time when it is a non-exposure state.

[0074] Figure 441. FIG. 4 is a diagram showing an example of the imaging range IA1 and the limited ranges Ar1, Ar2, ..., Ar(n-2), Ar(n-1), and Arn. Each of the plurality of limited ranges Ar1, ..., and Arn is a part of the imaging range IA1. Each of the plurality of limited ranges Ar1, ..., and Arn can be set and stored in advance in the memory 41. Figure 4 Each of the plurality of limited ranges Ar1, . . . , Arn shown is an example in which the imaging range IA1 is divided into rectangular shapes, but may be, for example, in a square shape.

[0075] Furthermore, the limit range may not be Figure 4 The predetermined range shown may also be a specified range centered on the predicted position. The limited range may be, for example, a circular shape having a specified radius centered on the predicted position of the imaging object Tg1 predicted by the prediction unit 43, or a quadrilateral shape having the predicted position of the imaging object Tg1 as the intersection of two diagonal lines.

[0076] Figure 5 1 and 2 are diagrams showing an example of a change over time of the imaging object Tg1 reflected in each of the plurality of limited ranges Ar1, . . . , Arn. Figure 5 The horizontal axis shown represents time T. In addition, Figure 5 The imaging object Tg1 in the imaging range IA1 does not move relative to the predetermined position. The vector RT0 indicates the position of the imaging object Tg1 in the next frame.

[0077] The camera 3 captures the image of the object Tg1 while moving at a predetermined speed in the direction opposite to the vector RT0 by the driving unit 22. The image of the object Tg1 at time t1 is located in the limited range Ar1. The image of the object Tg1 at time t2 is located in the limited range Ar2. The image of the object Tg1 at time t(n-2) is located in the limited range Ar(n-2). The image of the object Tg1 at time t(n-1) is located in the limited range Ar(n-1). The image of the object Tg1 at time tn is located in the limited range Arn.

[0078] As described above, the prediction unit 43 in the image processing device 4 can predict the position of the imaging object Tg1 in the imaging range IA1 based on the information that can estimate the position of the camera 3 and the position information of the imaging object Tg1 received from the actuator 2. In addition, the detection unit 44 restrictively reads the restricted range containing the predicted position of the imaging object Tg1 in each of the plurality of restricted ranges Ar1, ..., Arn described above based on the predicted position. As a result, the image processing device 4 can restrictively and efficiently perform image processing on the restricted range of the imaging range IA1, thereby reducing the time and load required for image processing.

[0079] Figure 6 This is a sequence diagram for explaining an example of the operation procedure of the image processing system according to the first embodiment.

[0080] The control device 1 generates a control signal based on the area data 12 input by the user, and transmits it to the actuator 2. Specifically, the control device 1 transmits the position information of the imaging object Tg1 to the actuator 2 based on the area data 12 (T1).

[0081] The control device 1 generates a control signal for controlling the driving of the camera 3 and a control signal for instructing movement based on the position information of the imaging object Tg1, and transmits the control signals to the actuator 2 (T2).

[0082] The actuator 2 performs initial position alignment based on the reference mark Pt0 (T3). Specifically, the actuator 2 moves the camera 3 to the shooting position of the reference mark Pt0. The actuator 2 causes the moved camera 3 to shoot the reference mark Pt0, and sends the position information of the reference mark Pt0 to the image processing device 4. The camera 3 sends the captured camera image of the reference mark Pt0 to the image processing device 4. The image processing device 4 detects the reference mark Pt0 based on the received camera image, and measures the position of the reference mark Pt0. The image processing device 4 calculates the difference between the measured measurement position and the position of the reference mark Pt0 received from the actuator 2, and sends it to the actuator 2. The actuator 2 corrects the position of the camera 3 based on the received difference.

[0083] The actuator 2 transmits the position information of the imaging object Tg1 received from the control device 1 to the image processing device 4 (T4).

[0084] The actuator 2 moves the camera 3 to a position where the imaging target Tg1 can be imaged based on the position information of the imaging target Tg1 ( T5 ).

[0085] The image processing device 4 predicts the position (T6) of the camera object Tg1 reflected in the camera image with the camera range IA1 based on the received position information of the camera object Tg1 and information that can estimate the position of the camera 3 (for example, the position information of the camera 3, the moving speed information of the camera 3, etc.).

[0086] The camera 3 transmits a captured image having an imaging range IA1 in which the imaging object Tg1 is captured to the image processing device 4 ( T7 ).

[0087] Based on the predicted position of the imaging object Tg1, the image processing device 4 restrictively reads out a limited range including the predicted position from among the plurality of limited ranges Ar1, ..., Arn as part of the imaging range IA1 (T8).

[0088] The image processing device 4 detects the imaging object Tg1 from the read limited range, and measures the position of the detected imaging object Tg1 ( T9 ).

[0089] The image processing device 4 outputs the difference between the measured position and the predicted position of the imaging object Tg1 ( T10 ).

[0090] The image processing device 4 transmits the output result (difference information) to the actuator 2 (T11).

[0091] The actuator 2 corrects the current position of the camera 3 based on the output result (difference information) (T12).

[0092] The actuator 2 moves the camera 3 to the next position based on the corrected position information of the camera 3 and the position information of the imaging object Tg1 ( T13 ).

[0093] After executing the operation process of step T13, the actuator 2 returns to the operation process of step T5 and repeats the operation process of the repetition process TRp of steps T5 to T13 until the imaging object Tg1 is changed. Figure 4 In the operation procedure shown, when the imaging target Tg1 is changed to another imaging target, the processing of step T3 can be omitted.

[0094] In addition, the process of the steps shown in the sequence diagram is not limited to the above order. For example, the action processes performed in step T6 and step T7 can be performed in the opposite order.

[0095] As described above, the image processing system according to the first embodiment can shorten the time required for the readout process by limiting the readout of the image in the image processing device 4 to a limited range, thereby being able to set the frame rate of the camera 3 to be faster. In addition, the image processing system according to the first embodiment can thereby obtain a greater number of samples (in other words, the number of error information outputted) in the same time, and thus can make the accuracy of the position error correction higher.

[0096] Figure 7 This is a flowchart for explaining an example of a basic operation procedure of the image processing device 4 according to the first embodiment.

[0097] The receiving unit 42 receives the position information of the imaging object Tg1 and information capable of estimating the position of the camera 3 (for example, the position information of the camera 3 , the moving speed information of the camera 3 , etc.) from the actuator 2 ( St11 ).

[0098] The prediction unit 43 predicts the position of the imaging target Tg1 shown in the captured image of the camera 3 having the imaging range IA1 based on the received position information of the imaging target Tg1 and the information capable of estimating the position of the camera 3 ( St12 ).

[0099] Based on the predicted position of the imaging target Tg1 , the detection unit 44 quickly reads out the limited range including the predicted position among the plurality of limited ranges Ar1 , . . . , Arn as part of the imaging range IA1 ( St13 ).

[0100] The detection unit 44 detects the imaging target Tg1 from the read limited range, and measures the position of the detected imaging target Tg1. The detection unit 44 outputs the difference between the measured position and the predicted position of the measured imaging target Tg1 (St14).

[0101] After executing the process of step St14, the image processing device 4 returns to the process of step St12. Figure 7 The image processing device 4 shown in the figure operates until there is a user instruction (for example, until the imaging target Tg1 is changed to another imaging target, until a predetermined number of differences are output) or the operation of the program stored in the area data 12 is terminated.

[0102] As described above, the image processing device 4 according to the first embodiment can shorten the time required for the readout process by limiting the readout of the image to a limited range, thereby being able to set a higher frame rate for the camera 3. In addition, the image processing device 4 according to the first embodiment can thereby obtain a greater number of samples (in other words, the number of error information outputted) in the same amount of time, thereby being able to achieve a higher accuracy in position error correction.

[0103] (Implementation Method 2)

[0104] In the second embodiment, an image processing system for each camera is described, which is provided with a plurality of cameras having different imaging ranges in addition to the first embodiment. The image processing device 4 according to the second embodiment can output an error in the moving speed of the camera or an error in the moving position of the camera based on feature points extracted from a predetermined limited range in the imaging range. In addition, the structure of the image processing system according to the second embodiment is substantially the same as that of the image processing system according to the first embodiment, so the same symbols are attached to the same structures and the description is simplified or omitted, and the different contents are described.

[0105] Figure 8 This is an explanatory diagram of an example of usage of an image processing system including a plurality of cameras 3a, 3b, and 3c according to the second embodiment. Figure 8The internal structure of the control device 1 according to the second embodiment shown in FIG. Figure 1 The structures shown are the same, so simplified diagrams are shown. In addition, regarding the actuator 2 and the image processing device 4 according to the second embodiment, the same contents as those described in the first embodiment are simplified or omitted, and the different contents are described.

[0106] The control unit 20 outputs a control signal to each of the plurality of cameras 3a, 3b, 3c based on the data and program stored in the area data 12. In addition, the control unit 20 outputs a control signal for moving each of the plurality of cameras 3a, 3b, 3c based on the data and program stored in the area data 12 to the drive unit 22. Figure 8 Although three cameras are shown, it goes without saying that the number of cameras is not limited to three.

[0107] Furthermore, the control unit 20 transmits information of the camera performing the imaging and information that can estimate the position of the camera (eg, camera position information, camera moving speed information, etc.) to the receiving unit 42 of the image processing device 4 .

[0108] The memory 21 stores the arrangement of each of the plurality of cameras 3a, 3b, and 3c and each of the imaging ranges IB1, IB2, and IB3.

[0109] Each of the plurality of arms 24a, 24b, 24c includes a plurality of cameras 3a, 3b, 3c and is controlled by the drive unit 22. Alternatively, for example, each of the plurality of cameras 3a, 3b, 3c may be provided on one arm 24a.

[0110] Each of the plurality of cameras 3a, 3b, and 3c moves in conjunction with the driving of each of the plurality of arms 24a, 24b, and 24c under the control of the driving unit 22. Each of the plurality of cameras 3a, 3b, and 3c is configured to be able to capture a different imaging range. The camera 3a has an imaging range IB1. The camera 3b has an imaging range IB2. The camera 3c has an imaging range IB3.

[0111] In addition, the operation of each of the plurality of cameras 3a, 3b, and 3c is the same as the operation of the camera 3 according to the first embodiment, and thus the description thereof is omitted.

[0112] Each of the plurality of imaging ranges IB1, IB2, and IB3 is a different imaging range. Figure 8 Although each of the plurality of imaging ranges IB1, IB2, and IB3 shown is shown as an adjacent imaging range, each imaging range moves according to the position of each of the plurality of cameras 3a, 3b, and 3c.

[0113] The image processing device 4 further includes a camera switching unit 47 in comparison with the image processing device 4 according to the first embodiment.

[0114] The receiving unit 42 outputs various information of the camera received from the actuator 2 to the predicting unit 43, the detecting unit 44, the output unit 46, and the camera switching unit 47. The various information includes the frame rate of each of the plurality of cameras 3a, 3b, and 3c, information on each of the plurality of imaging ranges IB1, IB2, and IB3, and zoom factor information of each of the plurality of cameras 3a, 3b, and 3c.

[0115] The detection unit 44 according to the second embodiment extracts feature points described below when an imaging target is not set in an initial state.

[0116] The detection unit 44 reads out a specified limited range set in the first frame of at least two frames captured continuously to extract each feature point of a plurality of feature points having a specified feature amount. The detection unit 44 extracts the imaging object Tg2 as a feature point with a large feature amount from each feature point of the extracted plurality of feature points. In addition, when the detection unit 44 cannot extract feature points in the first frame, it extracts feature points (imaging objects) in other limited ranges, or corrects the limited range and reads it out again to extract feature points (imaging objects). The correction of the limited range is performed by the detection unit 44 based on the distribution of each feature point of the extracted plurality of feature points. The correction of the limited range is performed, for example, by expanding or shifting the limited range in a direction where the density (density) of feature points in the distribution of each feature point of the plurality of feature points within the limited range is high.

[0117] After extracting the imaging object Tg2, the detection unit 44 reads the same limited range in the second frame to detect the imaging object Tg2. In addition, if the detection unit 44 fails to detect the imaging object Tg2 in the second frame, it performs reading in another limited range, or performs reading again after correcting the limited range. In addition, the detection unit 44 may also set the imaging object Tg2 as the imaging object.

[0118] The above-mentioned predetermined feature quantity is set in advance by the user and is stored in the memory 11 of the control device 1. The image processing device 4 receives information on the predetermined feature quantity from the control device 1 via the actuator 2.

[0119] The measuring unit 45 measures the position Pt1 of the imaging subject Tg2 shown in the first frame (ie, the first captured image) and the position Pt1 of the imaging subject Tg2 shown in the second frame (ie, the second captured image).

[0120] The output unit 46 calculates the motion speed of the imaging object Tg2 based on the motion amount of the imaging object Tg2 measured based on each of the two frames and the frame rates of the plurality of cameras 3a, 3b, 3c received by the receiving unit 42. The output unit 46 outputs the speed difference between the calculated motion speed of the imaging object Tg2 and the moving speed of the camera or the actuator 2 that captured the imaging object Tg2. The output unit 46 sends the output result to the error correction unit 23 of the actuator 2.

[0121] The error correction unit 23 outputs a control signal for correcting the speed error of the camera capturing the imaging object Tg2 to the driving unit 22 based on the received speed difference.

[0122] The camera switching unit 47 includes a switch of each of the plurality of switches SW1, SW2, SW3 connected to each of the plurality of cameras 3a, 3b, 3c and a switch SW for outputting a captured image to the receiving unit 42. The camera switching unit 47 switches between each of the plurality of switches SW1, SW2, SW3 connected to the switch SW (that is, each of the plurality of cameras 3a, 3b, 3c) based on the predicted position of the imaging object Tg2 predicted by the prediction unit 43 or the control signal input from the control unit 20.

[0123] Fig. 9 1 is a flowchart for explaining an example of the operation process of the image processing device 4 including the plurality of cameras 3a, 3b, and 3c according to the second embodiment. Fig. 9 In the flowchart shown, the image processing device 4 is set with an imaging object.

[0124] The receiving unit 42 receives from the actuator 2 the position information of the imaging object (not shown), the information of a certain camera among the multiple cameras 3a, 3b, 3c that photograph the imaging object, and the information capable of estimating the respective positions of the multiple cameras 3a, 3b, 3c (for example, the respective position information of the multiple cameras 3a, 3b, 3c, the respective moving speed information of the multiple cameras 3a, 3b, 3c, etc.) (St21).

[0125] The prediction unit 43 predicts the position where the object appears on the image sensor of the camera photographing the object based on the received position information of the object, information of the camera photographing the object, and information that can estimate the position of the camera ( St22 ).

[0126] The camera switching unit 47 switches the switch connected to the switch SW based on the received information of the camera capturing the imaging object ( St23 ).

[0127] Based on the predicted position of the imaging subject on the image sensor, the detection unit 44 reads out at high speed a limited range including the predicted position from among a predetermined limited range that is a part of the imaging range ( St24 ).

[0128] The detection unit 44 detects an imaging target having a predetermined feature value from the captured image of the read limited range. The measurement unit 45 measures the position of the detected imaging target (St25).

[0129] The output unit 46 outputs the difference between the measured position of the imaging object on the captured image and the predicted position of the imaging object on the image sensor ( St26 ).

[0130] After executing the process of step St26, the image processing device 4 returns to the process of step St22. Figure 7 The operation of the image processing device 4 shown in the figure continues until the imaging target is changed to another imaging target or until the operation of the program stored in the area data 12 is terminated.

[0131] In the following description, refer to Fig.10 and Fig.11 An image processing system for extracting feature points by image processing when an imaging object is not set in advance will be described.

[0132] As described above, the image processing device 4 according to the second embodiment can shorten the time required for the readout process by limiting the readout of the image to a limited range, thereby being able to set the frame rate of the camera to be faster. In addition, the image processing device 4 according to the second embodiment can thereby obtain a greater number of samples (in other words, the number of error information outputted) in the same time, thereby being able to make the accuracy of the position error correction higher.

[0133] Fig.10 It is a diagram showing an example of detection of a feature point (imaging target Tg2). Fig.11 This is a flowchart for explaining an example of the operation procedure of the image processing device 4 according to the second embodiment for detecting a feature point (imaging target Tg2).

[0134] Fig.10 The image shown is an image in which the motion of each feature point of a plurality of feature points appearing in a video image between two frames read out continuously within the same limited range Ar is extracted, and shows a situation in which the video object Tg2 is extracted as a feature point from each feature point of the plurality of feature points. Fig.10 The images shown are obtained by Fig.11 Generated by the processing performed in step St34.

[0135] The imaging object Tg2 is located at a position Pt1 represented by coordinates (X1, Y1) in the imaging image in the first frame by each of the plurality of cameras 3a, 3b, and 3c, which are high-speed cameras, and is located at a position Pt2 represented by coordinates (X2, Y2) in the imaging image in the second frame. The amount of motion Δα of the imaging object Tg2 is represented by the change in coordinates between the position Pt1 and the position Pt2 or the magnitude of a vector from the position Pt1 to the position Pt2.

[0136] Next, use Fig.11 The flowchart shown is used to illustrate the detection process of the imaging object Tg2.

[0137] The receiving unit 42 receives information related to the camera, such as the camera range, moving speed, frame rate, and zoom factor, from the actuator 2, and outputs the information to the detecting unit 44, the measuring unit 45, and the output unit 46. The detecting unit 44 sets the camera range based on the input camera information (St31).

[0138] The detection unit 44 reads out a predetermined limited range in the imaging range captured by the first frame of the two most recently continuously captured frames at high speed ( St32 ).

[0139] The detection unit 44 reads out a predetermined limited range in the imaging range captured by the second frame of the two most recently captured consecutive frames at high speed ( St33 ).

[0140] In addition, the readout can be performed within a limited range as described by Figure 4 As described above, the limit range is one of the plurality of limit ranges Ar1, . . . , Arn preset by the actuator 2, but the limit range may be a limit range set by the user.

[0141] The detection unit 44 detects each of a plurality of feature points that appear in the read captured image of the limited range based on the read results in each of the two frames that were captured continuously most recently ( St34 ).

[0142] The detection unit 44 performs weighting (extraction of feature amounts) on each of the plurality of feature points detected in step St34 to extract a predetermined imaging object Tg2 having a predetermined feature amount from each of the plurality of feature points. The measurement unit 45 measures the motion amount Δα (for example, based on the motion amount Δα) of the extracted predetermined imaging object Tg2. Fig.10 The output unit 46 calculates the movement speed of the imaging object Tg2 based on the frame rate of the camera received from the actuator 2 and the measured movement amount Δα (St35).

[0143] The output unit 46 outputs the calculated difference between the movement speed of the predetermined imaging object Tg2 and the moving speed of the camera, and transmits the output speed difference to the actuator 2 ( St36 ).

[0144] After executing the process in step St36 , the image processing device 4 returns to the process in step St32 , and extracts each of a plurality of feature points having a predetermined feature amount from the same limited range.

[0145] In addition, when the result of executing the processing in step St35 is that no feature point with the specified feature value is obtained from the limited range, the limited range for reading can be changed to another limited range to execute the processing shown after step St32 again.

[0146] According to the above, the image processing device 4 according to the second embodiment can shorten the time required for the readout process by limiting the readout of the image to a limited range, thereby being able to set the frame rate of the camera to be faster. In addition, the image processing device 4 according to the second embodiment can obtain more sampling numbers (in other words, the number of error information outputted) in the same time, thereby being able to make the accuracy of the speed error correction higher.

[0147] (Other Modifications)

[0148] In other modified examples, as an application example of the second embodiment, an image processing system is shown in which the actuator is set as an unmanned aircraft capable of flight control. In addition, the image processing system in other modified examples tracks the feature points detected from a predetermined limited range while detecting other feature points in other limited ranges. In addition, the structure of the image processing system involved in other modified examples is substantially the same as that of the image processing system involved in the second embodiment, so the same symbols are marked on the same structure and the description is simplified or omitted, and the different contents are described.

[0149] Fig.12 It is an explanatory diagram of an example of usage of the image processing system provided with the unmanned aircraft 2A. Fig.12 The internal structure of the control device 1 in the other modified example shown is different from Figure 1 The structures shown are the same, so a simplified diagram is shown. In the control device 1 of other modified examples, the same contents as those described in the first embodiment are simplified or omitted, and the different contents are described.

[0150] The control device 1 in another modified example is, for example, a proportional controller (Japanese: プロポ) (so-called remote controller) used by an operator (user) of the unmanned aircraft 2A, and remotely controls the flight of the unmanned aircraft 2A based on the area data 12. The control device 1 is connected to the unmanned aircraft 2A via a wireless N / W, generates a control signal for controlling the flight of the unmanned aircraft 2A based on the area data 12, and transmits the control signal.

[0151] The area data 12 in other modified examples is configured to include, for example, information on the flight path of the unmanned aircraft 2A.

[0152] In addition, the control device 1 may be operated by a user. In this case, the control device 1 remotely controls the flight of the unmanned aircraft 2A based on the user's operation. The control device 1 is connected to the unmanned aircraft 2A via a wireless N / W, and generates and transmits a control signal related to the flight control of the unmanned aircraft 2A.

[0153] The unmanned aircraft 2A is, for example, an unmanned aerial vehicle, and flies based on a control signal transmitted from the control device 1 according to an input operation of a user. The unmanned aircraft 2A includes a plurality of cameras 3a and 3b. The unmanned aircraft 2A includes a control unit 20, a memory 21, a drive unit 22, an error correction unit 23, and a communication unit 25.

[0154] The communication unit 25 includes an antenna Ant1 and is connected to the control device 1 and the image processing device 4 via a wireless N / W (for example, a wireless communication network using Wifi (registered trademark)) to transmit and receive information and data.

[0155] The communication unit 25 receives control-related signals such as the moving direction and flight altitude of the unmanned aircraft 2A through communication with the control device 1. The communication unit 25 transmits a satellite positioning signal indicating the position information of the unmanned aircraft 2A received through the antenna Ant1 to the control device 1. The antenna Ant1 will be described later.

[0156] In addition, the communication unit 25 transmits, for example, setting information related to feature quantities required for feature point extraction, setting information of each of the plurality of cameras 3a and 3b (for example, information related to the imaging range, frame rate, zoom factor, and limited range), speed information of the unmanned aircraft 2A, etc., through communication with the image processing device 4. The communication unit 25 receives, through communication with the image processing device 4, speed difference (error) information related to the speed between the speed information of the unmanned aircraft 2A and the moving speed of the imaging object Tg2 shown in the imaging image captured by each of the plurality of cameras 3a and 3b. The communication unit 25 outputs the received difference (error) information to the error correction unit 23.

[0157] Antenna Ant1 is, for example, an antenna capable of receiving satellite positioning signals transmitted from artificial satellites (not shown). Signals that antenna Ant1 can receive are not limited to signals from the United States' GPS (Global Positioning System), but can also be signals transmitted from artificial satellites that can provide satellite positioning services, such as Russia's GLONASS (Global Navigation Satellite System) or Europe's Galileo. In addition, antenna Ant1 can also receive signals from quasi-zenith satellites, which are used to transmit satellite positioning signals that can enhance or correct satellite positioning signals transmitted by the artificial satellites that provide satellite positioning services.

[0158] The driving unit 22 drives the unmanned aircraft 2A to fly based on the control signal received from the control device 1 via the communication unit 25. The driving unit 22 is at least one rotor, and controls the lift generated by the rotation to fly. Fig.12 In the figure, the driving unit 22 is shown on the top surface of the unmanned aircraft 2A, but the installation location is not limited to the top surface, as long as it is a place where the flight of the unmanned aircraft 2A can be controlled, for example, it is installed at the bottom or side of the unmanned aircraft 2A.

[0159] The error correction section 23 corrects the flight speed of the driving section 22 based on the speed difference (error) information between the flight speed of the unmanned aircraft 2A and the moving speed of the imaging object Tg3 received from the output section 46 in the image processing device 4 .

[0160] Each of the plurality of cameras 3a and 3b is a camera for capturing different camera ranges IB1 and IB2. Each of the plurality of cameras 3a and 3b may be fixedly mounted on the unmanned aircraft 2A, or may be configured to capture images at various angles. In addition, each of the plurality of cameras 3a and 3b may be mounted on any of the side, bottom, and top surfaces of the unmanned aircraft 2A. For example, each of the plurality of cameras 3a and 3b may be mounted on different surfaces such as the top and bottom surfaces of the unmanned aircraft 2A, or different side surfaces.

[0161] also, Fig.12 The imaging ranges IB1 and IB2 shown are continuous imaging ranges, but each imaging range may be changed based on the installation location of each of the plurality of cameras 3a and 3b, and the imaging range may not be continuous.

[0162] Each of the plurality of cameras 3 a and 3 b transmits a captured image to the camera switching unit 47 in the image processing device 4 via the communication unit 25 .

[0163] The receiving unit 42 receives setting information related to each camera of the multiple cameras 3a, 3b, such as the frame rate of each camera 3a, 3b, the imaging range, and each limited range set on the image sensor, the camera images captured by each camera of the multiple cameras 3a, 3b, and setting information related to feature points (for example, the feature quantity required to detect feature points in the limited range read out in the camera image, etc.) through communication with the unmanned aircraft 2A.

[0164] The detection unit 44 sets the tracking limited range for tracking the imaging target Tg3 and the detection limited range for detecting other imaging targets in the image sensor based on the setting information of each of the plurality of cameras 3a, 3b received by the receiving unit 42. Fig.13 In addition, the detection unit 44 sets a tracking camera for tracking the imaging target Tg3 and a detection camera for detecting another imaging target Tg4, and sets a tracking limited range for tracking the imaging target Tg3 for the tracking camera (in Fig.13 A detection limited range is set for the detection camera to detect other imaging objects Tg4.

[0165] The detection unit 44 in another modified example does not set the imaging target Tg3 in the initial state. Therefore, the setting of the imaging target Tg3 will be described below.

[0166] The detection unit 44 reads the captured image of the limited tracking range set on the image sensor, and extracts each of the plurality of feature points having a predetermined feature value. The detection unit 44 sets one feature point having a large feature value among the extracted plurality of feature points as the image capture target Tg3.

[0167] The detection unit 44 reads out the camera image of the detection limited range set on the image sensor, and extracts each feature point of a plurality of feature points having a specified feature amount. The detection unit 44 determines whether each feature point of the plurality of feature points included in the detection limited range is more than each feature point of the plurality of feature points included in the tracking limited range. In addition, the detection unit 44 may also make a determination based on the feature point with the largest feature amount among the plurality of feature points included in the detection limited range and the feature amount of the imaging object Tg3. The detection unit 44 sets the limited range where the result of the determination is that the number of feature points of the plurality of feature points is larger or the limited range containing feature points with a larger feature amount as the tracking limited range. Furthermore, the detection unit 44 sets the other limited range as the detection limited range. In addition, the image processing device 4 also performs the same processing when the tracking camera and the detection camera are set by the detection unit 44.

[0168] In addition, the detection unit 44 may also correct the tracking limited range based on the distribution of each of the plurality of feature points included in the tracking limited range. Thus, when the imaging target Tg3 has feature points with a larger feature value near the boundary of the tracking imaging range, the detection unit 44 may set it as another imaging target Tg4.

[0169] The prediction unit 43 predicts the position of the imaging object Tg3 captured in the next two frames on the image sensor based on the detected movement amount of the imaging object Tg3 and the flight direction of the unmanned aircraft 2A. The prediction unit 43 outputs the predicted position of the imaging object Tg3 to the detection unit 44.

[0170] Furthermore, when the predicted position is transferred to the imaging range of another camera or the limited range of another camera, the prediction unit 43 may output information on the camera serving as the transfer target or the limited range set on the image sensor of the camera serving as the transfer target to the detection unit 44 and the camera switching unit 47. Furthermore, when the predicted position of the imaging object Tg3 is outside the imaging range, the prediction unit 43 may output to the detection unit 44 and the camera switching unit 47 that the predicted position of the imaging object Tg3 has moved outside the imaging range.

[0171] The output unit 46 calculates the moving speed of the imaging object Tg3 based on the position of the imaging object Tg3 in the imaging image measured by the measuring unit 45. Fig.13 The output unit 46 transmits the speed difference between the flight speed of the unmanned aircraft 2A received by the receiving unit 42 and the speed of the moving object Tg3 to the error correction unit 23 via the communication unit 25 .

[0172] The camera switching unit 47 switches the camera that captures the set tracking limited range and detection limited range frame by frame, and does not switch the camera when the set tracking limited range and detection limited range are within the imaging range of the same camera. In addition, when each of the plurality of cameras 3a and 3b is set as a tracking camera and a detection camera, the camera switching unit 47 also switches the camera frame by frame.

[0173] Fig.13 4 is a flowchart illustrating an example of the tracking and detection operation process of the image processing device 4 according to the second embodiment. Fig.13 In the description of the flowchart shown in the figure, Fig.12 The example of the operation process of the image processing device 4 when each of the multiple cameras 3a and 3b of the unmanned aircraft 2A shown in the figure receives the image is explained, but the number of cameras is not limited to two, it can also be three or more, and it can also be one when the viewing angle of the camera is not fixed.

[0174] The receiving unit 42 receives the setting information of each of the plurality of cameras 3a, 3b, such as the frame rate, the imaging range, and the limited range, and the setting information related to the feature point (such as the feature amount required to detect the feature point) through wireless communication with the unmanned aircraft 2A. The camera switching unit 47 sets the limited range for tracking based on the setting information of each of the plurality of cameras 3a, 3b received by the receiving unit 42 (St41). In addition, when one of the plurality of cameras 3a, 3b is set as a tracking camera, the limited range in the imaging range of the tracking camera is set as the tracking limited range.

[0175] The camera switching unit 47 sets the detection limited range based on the setting information of each of the plurality of cameras 3a and 3b received by the receiving unit 42 (St42). In addition, when the user sets one of the plurality of cameras 3a and 3b as a detection camera, the limited range in the imaging range of the detection camera is set as the detection limited range. In addition, the detection limited range and the detection camera may be not one but multiple.

[0176] The camera switching unit 47 switches the connection of the switch SW to the set tracking limited range (in other words, the camera including the tracking detection range in the imaging range). The receiving unit 42 receives the captured image from the camera connected by the switching of the camera switching unit 47, and outputs it to the detecting unit 44. The detecting unit 44 reads the set tracking limited range in the input imaging range in a limited manner at high speed (St43).

[0177] The camera switching unit 47 switches the connection of the switch SW to the set detection limited range (in other words, the camera including the detection range in the imaging range). The receiving unit 42 receives the captured image from the camera connected by the switching of the camera switching unit 47, and outputs it to the detecting unit 44. The detecting unit 44 reads the set detection limited range in the input imaging range in a limited manner at high speed (St44).

[0178] The detection unit 44 extracts each of a plurality of feature points (image pickup target) having a predetermined feature amount from the captured image of the read detection limited range ( St45 ).

[0179] The detection unit 44 compares each of the plurality of feature points of the tracking limited range extracted by the processing in step St44 with each of the plurality of feature points of the detection limited range extracted by the processing in step St45 to determine whether each of the plurality of feature points included in the detection limited range is greater than each of the plurality of feature points included in the tracking limited range (St46). In addition, the determination method may be the number of feature points or the size of the maximum feature amount of the feature points in each limited range.

[0180] When the result of determination in step St46 is that the number of feature points included in the detection limited range is greater than the number of feature points included in the tracking limited range (St46, "Yes"), the detection unit 44 causes the camera switching unit 47 to change the current tracking limited range to the detection limited range, and to change the current detection limited range to the tracking limited range (St47).

[0181] When the result of determination in step St46 is that the number of feature points included in the detection limited range is less than the number of feature points included in the tracking limited range (St46, "No"), or after executing the processing in step St47, the camera switching unit 47 changes the current detection limited range to another limited range (specifically, a limited range other than the limited range that is not set as the tracking limited range and includes the predicted position of the camera object) (St48).

[0182] The camera switching unit 47 switches the connection of the switch SW to the set tracking limited range. The receiving unit 42 outputs the frame of the camera switched by the camera switching unit 47 to the detection unit 44. The detection unit 44 reads the set tracking limited range in the input imaging range at a limited speed (St49).

[0183] The detection unit 44 extracts each of the plurality of feature points from the camera image of the limited tracking range read out by executing the process in step St43. The detection unit 44 sets one of the extracted plurality of feature points as the camera object Tg3, and detects the camera object Tg3 from the camera image of the limited tracking range read out by executing the process in step St49. The measurement unit 45 measures the position of the camera object Tg3 detected in step St43 and the position of the camera object Tg3 detected in step St49 based on the setting information of each of the plurality of cameras 3a and 3b received by the receiving unit 42. The output unit 46 calculates the movement speed of the camera object Tg3 based on the difference between the measured position of the camera object Tg3 detected in step St43 and the position of the camera object Tg2 detected in step St49 (St50).

[0184] Here, the moving speed of the imaging subject calculated in step St50 will be described.

[0185] In step St46, when the number of feature points included in the detection limited range is greater than the number of feature points included in the tracking limited range (St46, "Yes"), the detection unit 44 changes the current detection limited range to the tracking limited range through the processing in step St47, and reads out the same limited range as in step St44 through the processing in step St49. Therefore, the output unit 46 continuously reads out the same limited range, and thus calculates the motion speed of the imaging subject based on the position of the imaging subject that changes between two frames.

[0186] On the other hand, when the number of feature points included in the detection limited range in step St46 is not more than the number of feature points included in the tracking limited range (St46, "No"), the detection unit 44 reads out the same tracking limited range as in step St43 through the processing in step St49. In such a case, the detection unit 44 reads out another limited range once through step St44. Therefore, the position of the imaging object (feature point) detected by step St49 is the position of the imaging object two frames later from the imaging object detected by step St44. Therefore, the output unit 46 reads out another limited range once, thereby calculating the motion speed of the imaging object based on the position of the imaging object that changes between three frames.

[0187] Furthermore, the output unit 46 outputs the speed difference between the speed information of the unmanned aircraft 2A input from the receiving unit 42 and the moving speed of the imaging object Tg3, and transmits it to the unmanned aircraft 2A (St51).

[0188] After executing the process in step St51, the image processing device 4 returns to the process in step St44. In the process in step St46 after the second loop, the detection unit 44 detects another imaging object Tg4 containing more feature amounts than the current imaging object Tg3. In addition, when the imaging object Tg3 is outside the imaging range of each of the plurality of cameras 3a and 3b, the detection unit 44 may return to the process in step St41.

[0189] In addition, the detection unit 44 may also correct the tracking limited range based on the distribution of each of the plurality of feature points detected in the tracking limited range after executing the processing in step St51 (St52). In addition, in such a case, the image processing device 4 also returns to the processing of step St44 after executing the processing in step St52.

[0190] According to the above, the image processing device 4 involved in other modified examples can simultaneously perform tracking of the imaging object Tg3 and detection of other imaging objects. As a result, the unmanned aircraft 2A can obtain the imaging object Tg3 (target) in the imaging range when performing the posture control of the unmanned aircraft 2A. In addition, when using the above-mentioned image processing device 4, the unmanned aircraft 2A can obtain information related to the posture of the unmanned aircraft 2A by comparing information such as the moving speed or moving direction of the unmanned aircraft 2A with information such as the moving speed or moving direction (vector) of the imaging object Tg3 (target).

[0191] Reference Fig.14 and Fig.15 The tracking and detection of the imaging object in other modified examples will be described. Fig.14 This is a diagram for explaining an example of switching between the tracking limited range and the detection limited range. Fig.14 The horizontal axis shown represents frames. Fig.15 is a diagram illustrating an example of tracking and detection of an imaged object. Fig.14 In the frame F1 shown, the image processing device 4 executes the process of step St44 after executing the process up to step St51 or step St52.

[0192] Fig.14 The camera switching unit 47 switches the tracking limited range and the set detection limited range frame by frame based on the predicted position of the imaging target Tg3 obtained by the prediction unit 43 .

[0193] Fig.15Each of the multiple camera objects Tg3 and Tg4 shown is a feature point having a specified feature quantity extracted by the detection unit 44. The camera object Tg3 is a feature point that has been extracted by the detection unit 44 and is set as the camera object at the time point of the frame F1. The camera object Tg3 changes its position in a manner that moves on the track RT1 frame by frame due to the flight (movement) of the unmanned aircraft 2A. The camera object Tg4 is a feature point that is not detected by the detection unit 44 in the initial state and has a specified feature quantity. The camera object Tg4 is located outside the camera range of each of the multiple cameras 3a and 3b in the frame F1. The camera object Tg4 changes its position in a manner that moves on the track RT2 frame by frame due to the flight (movement) of the unmanned aircraft 2A.

[0194] In the frame F1, the camera switching unit 47 switches the connection destination of the switch SW to the camera 3a including the detection limited range Ar11 in the imaging range. The detection unit 44 reads the detection limited range Ar11 at high speed and extracts feature points having a predetermined feature amount. The detection unit 44 determines based on the extraction result that no feature points exceeding the feature amount of the imaging object Tg3 in the previous tracking limited range (not shown) have been extracted, and changes the detection limited range Ar11 to the adjacent detection limited range Ar12.

[0195] In the frame F2, the prediction unit 43 predicts the predicted position of the imaging object Tg3 as the position Ps31 (tracking limited range Ar13), and outputs the prediction result to the camera switching unit 47 and the detection unit 44. The camera switching unit 47 keeps the connection destination of the switch SW at the camera 3a including the tracking limited range Ar13 in the imaging range. The detection unit 44 reads the tracking limited range Ar13 at high speed and detects the imaging object Tg3. Based on the detection result, the measurement unit 45 measures the movement amount of the imaging object Tg3 based on the position of the imaging object Tg3 photographed in the previous tracking limited range (not shown) and the position of the imaging object Tg3 photographed in the tracking limited range Ar13. The output unit 46 calculates the movement speed of the imaging object Tg3 based on the measured movement amount of the imaging object Tg3, outputs the speed difference between the movement speed of the imaging object Tg3 and the flight speed of the unmanned aircraft 2A, and transmits it to the error correction unit 23 via the communication unit 25.

[0196] In the frame F3, the camera switching unit 47 keeps the connection destination of the switch SW at the camera 3a including the detection limited range Ar12 in the imaging range. The detection unit 44 reads the detection limited range Ar12 at high speed and extracts feature points having a predetermined feature amount. Based on the extraction result, the detection unit 44 determines that no feature points exceeding the feature amount of the imaging object Tg3 in the previous tracking limited range Ar13 have been extracted, and changes the detection limited range Ar12 to the adjacent detection limited range Ar13.

[0197] In the frame F4, the prediction unit 43 predicts the predicted position of the imaging object Tg3 as the position Ps32 (tracking limited range Ar21), and outputs the prediction result to the camera switching unit 47 and the detection unit 44. The camera switching unit 47 switches the connection destination of the switch SW to the camera 3b including the tracking limited range Ar21 in the imaging range. The detection unit 44 reads the tracking limited range Ar21 at high speed and detects the imaging object Tg3. Based on the detection result, the measurement unit 45 measures the movement amount of the imaging object Tg3 based on the position of the imaging object Tg3 photographed in the previous tracking limited range Ar13 and the position of the imaging object Tg3 photographed in the tracking limited range Ar21. The output unit 46 calculates the movement speed of the imaging object Tg3 based on the measured movement amount of the imaging object Tg3, outputs the speed difference between the movement speed of the imaging object Tg3 and the flight speed of the unmanned aircraft 2A, and transmits it to the error correction unit 23 via the communication unit 25.

[0198] In frame F5, the camera switching unit 47 switches the connection destination of the switch SW to the camera 3a including the detection limited range Ar13 in the imaging range. The detection unit 44 reads the detection limited range Ar13 at high speed and extracts feature points having a predetermined feature amount. Based on the extraction result, the detection unit 44 determines that no feature points exceeding the feature amount of the imaging object Tg3 in the previous tracking limited range Ar21 have been extracted, and changes the detection limited range Ar12 to the adjacent detection limited range Ar13.

[0199] In frame F6, the prediction unit 43 predicts the predicted position of the imaging object Tg3 as the position Ps33 (tracking limited range Ar22), and outputs the prediction result to the camera switching unit 47 and the detection unit 44. The camera switching unit 47 switches the connection destination of the switch SW to the camera 3b including the tracking limited range Ar22 in the imaging range. The detection unit 44 reads the tracking limited range Ar22 at high speed and detects the imaging object Tg3. Based on the detection result, the measurement unit 45 measures the movement amount of the imaging object Tg3 based on the position of the imaging object Tg3 photographed in the previous tracking limited range Ar21 and the position of the imaging object Tg3 photographed in the tracking limited range Ar22. The output unit 46 calculates the movement speed of the imaging object Tg3 based on the measured movement amount of the imaging object Tg3, outputs the speed difference between the movement speed of the imaging object Tg3 and the flight speed of the unmanned aircraft 2A, and transmits it to the error correction unit 23 via the communication unit 25.

[0200] In frame F7, the camera switching unit 47 keeps the connection destination of the switch SW at the camera 3b including the detection limited range Ar21 in the imaging range. The detection unit 44 reads the detection limited range Ar21 at high speed. The detection unit 44 extracts a feature point located at the position Ps42 and having a predetermined feature amount as the imaging object Tg4. Based on the extraction result, the detection unit 44 compares the imaging object Tg4 in the detection limited range Ar21 with the imaging object Tg3 in the previous tracking limited range Ar22. As a result of the comparison, the detection unit 44 determines that no feature point exceeding the feature amount of the imaging object Tg3 in the previous tracking limited range Ar22 has been extracted, and changes the detection limited range Ar12 to the adjacent detection limited range Ar13.

[0201] In frame F8, the prediction unit 43 predicts the predicted position of the imaging object Tg3 as the position Ps34 (tracking limited range Ar23), and outputs the prediction result to the camera switching unit 47 and the detection unit 44. The camera switching unit 47 keeps the connection destination of the switch SW at the camera 3b including the tracking limited range Ar23 in the imaging range. The detection unit 44 reads the tracking limited range Ar23 at high speed and detects the imaging object Tg3. Based on the detection result, the measurement unit 45 measures the movement amount of the imaging object Tg3 based on the position of the imaging object Tg3 photographed in the previous tracking limited range Ar22 and the position of the imaging object Tg3 photographed in the tracking limited range Ar23. The output unit 46 calculates the movement speed of the imaging object Tg3 based on the measured movement amount of the imaging object Tg3, outputs the speed difference between the movement speed of the imaging object Tg3 and the flight speed of the unmanned aircraft 2A, and transmits it to the error correction unit 23 via the communication unit 25.

[0202] In frame F9, the camera switching unit 47 keeps the connection destination of the switch SW at the camera 3b including the detection limitation range Ar22 in the imaging range. The detection unit 44 reads the detection limitation range Ar22 at high speed. The detection unit 44 extracts the imaging object Tg4 located at the position Ps43 and having a predetermined feature amount. Based on the extraction result, the detection unit 44 compares the imaging object Tg4 in the detection limitation range Ar22 with the imaging object Tg3 in the previous tracking limitation range Ar23. As a result of the comparison, the detection unit 44 determines that a feature point exceeding the feature amount of the imaging object Tg3 in the previous tracking limitation range Ar23 has been extracted, and changes the imaging object from the current imaging object Tg3 to the next imaging object Tg4. In addition, the detection unit 44 changes the detection limitation range Ar22 to the tracking limitation range Ar22, and changes the next detection limitation range to another adjacent detection limitation range Ar23.

[0203] In addition, the image processing device 4 at the time of the frame F10 may predict the position of the imaging object Tg4 in the frame F11 by the prediction unit 43, and set the limited range Ar23 including the position Ps45 as the predicted position as the tracking limited range Ar23. In this case, the detection limited range Ar23 changed in the frame F10 may be further changed to another detection limited range Ar11.

[0204] In frame F10, the camera switching unit 47 keeps the connection destination of the switch SW at the camera 3b including the same tracking limited range Ar22 as in frame F9 in the imaging range. The detection unit 44 reads the tracking limited range Ar22 at high speed. The detection unit 44 detects the imaging object Tg4 located at the position Ps44. The measurement unit 45 measures the amount of motion of the imaging object Tg4 based on the position Ps43 of the imaging object Tg4 in frame F9 and the position Ps44 of the imaging object Tg4 in frame F10 based on the detection result. The output unit 46 calculates the motion speed of the imaging object Tg4 based on the measured motion speed of the imaging object Tg4, outputs the speed difference between the motion speed of the imaging object Tg4 and the flight speed of the unmanned aircraft 2A, and transmits it to the error correction unit 23 via the communication unit 25.

[0205] Furthermore, since the imaging object Tg4 is located at the boundary line of the limited range Ar22 in the frame F10, the image processing device 4 may also perform Fig.13 The processing of step St52 in the flowchart shown is used to correct the range of the limited range Ar22 or the limited range Ar23.

[0206] In frame F11, the camera switching unit 47 keeps the connection destination of the switch SW at the camera 3b including the detection limitation range Ar23 in the imaging range. The detection unit 44 reads the detection limitation range Ar23 at high speed. The detection unit 44 extracts the feature point located at the position Ps45 and having a predetermined feature amount as the imaging object Tg4. The detection unit 44 determines that the imaging object Tg4 is the imaging object Tg4 based on the extraction result, and determines that the feature point is not extracted from the detection limitation range Ar23, and recursively changes the detection limitation range Ar23 to the detection limitation range Ar11.

[0207] In addition, the image processing device 4 in frame F11 can also determine the extracted camera object Tg4 as the camera object Tg4, and calculate the movement amount and movement speed of the camera object Tg4 based on the position Ps44 of the camera object Tg4 in frame F10 and the position Ps45 of the camera object Tg4 in frame F11.

[0208] In the above Fig.14 and Fig.15 In the description of , an example is shown in which the detection limited range is sequentially changed from the limited range Ar11 to the limited range Ar23, but the detection limited range can also be randomly changed (set). Fig.14 and Fig.15 The description of FIG. 4 shows an example in which the prediction unit 43 predicts the position of the object to be photographed at the timing of switching each of the plurality of cameras 3a and 3b, but the timing of the prediction is not limited thereto. The prediction unit 43 may also predict the position of the object to be photographed before changing the tracking detection range and the detection limited range in the next frame, for example.

[0209] According to the above, the image processing device 4 involved in other modified examples can be changed to reflect the tracking limited range and the detection limited range of the predicted position, so that the imaging object can be tracked more efficiently and other imaging objects can be detected. In addition, the image processing device 4 involved in other modified examples can obtain more sampling numbers (in other words, the number of error information output) in the same time, so the accuracy of position error correction can be made higher.

[0210] According to the above, the image processing device 4 involved in embodiment 1 includes: a receiving unit 42, which receives the position information of the imaging object Tg1 and the camera image captured by at least one camera 3 and showing the imaging object Tg1; a prediction unit 43, which predicts the position of the imaging object Tg1 within the imaging range IA1 of the camera 3 based on the position information of the imaging object Tg1; a detection unit 44, which detects the imaging object Tg1 by reading the camera image of the limited range Ar1 which is a part of the imaging range IA1 from the camera image of the imaging range IA1 based on the predicted position of the predicted imaging object Tg1; a measuring unit 45, which measures the position of the detected imaging object Tg1; and an output unit 46, which outputs the difference between the measured position and the predicted position of the imaging object Tg1.

[0211] Thus, the image processing device 4 can perform efficient image processing on the image of the imaging object Tg1 captured by the camera 3, and can calculate the position error of the imaging object Tg1 with higher accuracy. In addition, the image processing device 4 according to the first embodiment can shorten the reading time by restrictively reading the limited range in the imaging range IA1, thereby suppressing the influence on the operation speed of other devices. Thus, the image processing device 4 according to the first embodiment can increase the number of samples by shortening the reading time, thereby realizing the position error correction with higher accuracy.

[0212] In addition, the image processing device 4 involved in embodiment 2 and other variants includes: a receiving unit 42, which receives the position information of each of the multiple cameras 3a, 3b and the camera images captured by at least one camera; a detection unit 4, which reads out a camera image of a limited range that is a part of the camera range of the camera in at least one camera image to detect a feature point (camera object Tg3) that serves as a reference for the position of the camera; a measuring unit 45, which measures the position of the detected camera object; a prediction unit 43, which predicts the position of the camera object reflected in a camera image captured after the camera image used when detecting the camera object based on the measured position of the camera object; and an output unit 46, which outputs the difference between the predicted position of the camera object and the measured position of the camera object.

[0213] Thus, the image processing device 4 according to the second embodiment and other modified examples performs efficient image processing on the image of the imaging object Tg3 captured by the camera, and can calculate the position error of the imaging object with higher accuracy. Furthermore, the image processing device 4 according to the second embodiment and other modified examples restrictively reads out the limited range in the imaging range of the camera, thereby shortening the reading time, thereby suppressing the influence on the operation speed of other devices. Thus, the image processing device 4 according to the second embodiment and other modified examples can increase the number of samples by shortening the reading time, thereby achieving higher accuracy position error correction. Therefore, when the image processing device 4 according to the second embodiment and other modified examples is used, the unmanned aircraft 2A can perform posture control in flight based on the output position difference.

[0214] In addition, the image processing device 4 according to the second embodiment and other modified examples further includes a camera switching unit 47 that switches the connection between the image processing device 4 and each of the plurality of cameras having different imaging ranges, and the camera switching unit 47 switches to a camera that can capture the predicted position among the plurality of cameras according to the predicted position. Thus, the image processing device 4 according to the first embodiment, the second embodiment and other modified examples can switch each of the plurality of cameras 3a and 3b according to the position of the imaging object Tg3 predicted by the prediction unit 43, thereby shortening the time taken by the movement of each of the plurality of cameras 3a and 3b, and performing efficient image processing on the image of the imaging object Tg3 captured. Therefore, when the image processing device 4 according to the second embodiment and other modified examples is used, the unmanned aircraft 2A can receive more position differences in a certain period of time, and can perform more accurate posture control based on each of these position differences.

[0215] In addition, the camera switching unit 47 in the image processing device 4 according to the second embodiment and other modified examples sets the camera that reads the limited range including the predicted position of the imaging target Tg3 to track the imaging target Tg3 as the tracking camera, and sets the other camera that reads the other limited range outside the imaging range of the tracking camera to detect the other imaging target Tg4 as the detection camera, thereby switching the tracking camera and the detection camera. As a result, the image processing device 4 according to the second embodiment and other modified examples can efficiently perform the tracking of the imaging target Tg3 and the detection of the other imaging target Tg4 by switching the camera by the camera switching unit 47, and can perform efficient image processing. In addition, the image processing device 4 can suppress the reduction of the number of samples of the imaging target Tg3 by performing the tracking of the imaging target Tg3 and the detection of the other imaging target Tg4 at the same time, and can correct the position error with maintaining accuracy. Therefore, when the image processing device 4 according to the second embodiment and other modified examples is used, the unmanned aircraft 2A can always receive the position difference, so that the posture control can be performed more stably.

[0216] In addition, the camera switching unit 47 in the image processing device 4 according to the second embodiment and other modified examples sets the limited range including the predicted position of the imaging object Tg3 among the multiple limited ranges of each of the multiple cameras as the tracking limited range, and sets at least one limited range among the other limited ranges other than the tracking limited range as the detection limited range for detecting the other imaging object Tg4, thereby switching the tracking limited range and the detection limited range. Thus, the image processing device 4 according to the second embodiment and other modified examples sets the tracking limited range for tracking the imaging object Tg3 and the detection limited range for detecting the other imaging object, thereby enabling the camera switching unit 47 to perform the switching of the cameras more efficiently. Therefore, the image processing device 4 can efficiently perform the reading process of the captured imaging image. In addition, the image processing device 4 simultaneously performs the tracking of the imaging object Tg3 and the detection of the other imaging object Tg4, thereby enabling the reduction of the number of samples of the imaging object Tg3 to be suppressed, and the position error can be corrected with the accuracy maintained. Therefore, when the image processing device 4 according to the second embodiment and other modified examples is used, the unmanned aircraft 2A can always receive the position difference, and can perform posture control more stably.

[0217] In addition, the detection unit 44 in the image processing device 4 involved in the second embodiment and other modified examples detects at least one feature point with a specified feature quantity contained in each of the limited ranges of at least two camera images. Thus, the image processing device 4 involved in the second embodiment and other modified examples can detect at least one feature point with a specified feature quantity from the camera image, so that a target with high reliability can be set even when there is no camera object. Therefore, the image processing device 4 can perform efficient image processing on the image of the camera object captured by the camera, and can calculate the position error of the camera object with higher accuracy. Therefore, when using the image processing device 4 involved in the second embodiment and other modified examples, the unmanned aircraft 2A can receive a more reliable position difference, so that it can perform posture control based on the difference.

[0218] In addition, the detection unit 44 in the image processing device 4 involved in the second embodiment and other modified examples corrects the limited range based on the distribution of each of the plurality of feature points detected. Thus, when the set limited range is inappropriate (for example, the feature point with more feature quantity is not located in the center of the limited range but is located at the end), the image processing device 4 involved in the second embodiment and other modified examples can correct the limited range based on the distribution of each of the plurality of feature points detected from the read camera image. Therefore, the image processing device 4 can perform the correction of the read range, thereby detecting more reliable feature points. Therefore, when using the image processing device 4 involved in the second embodiment and other modified examples, the unmanned aircraft 2A can receive a more reliable position difference, thereby performing a posture control based on the difference.

[0219] In addition, the detection unit 44 in the image processing device 4 according to the second embodiment and other modified examples sets the detected feature point as another imaging object. Thus, the image processing device 4 according to the second embodiment and other modified examples can set a more reliable feature point as the imaging object. Therefore, the image processing device 4 can calculate the position error of the imaging object with higher accuracy. Therefore, when the image processing device 4 according to the second embodiment and other modified examples is used, the unmanned aircraft 2A can receive a more reliable position difference, and can perform posture control based on the difference.

[0220] In addition, the measuring unit 45 in the image processing device 4 according to the second embodiment and other modified examples measures the amount of motion of the imaged object Tg2 based on the detected positions of the imaged object Tg2, and the output unit 46 calculates the motion speed of the imaged object Tg2 based on the measured amount of motion of the imaged object Tg2, and outputs the motion speed of the imaged object Tg2. Thus, the image processing device 4 according to the second embodiment and other modified examples can calculate the motion speed of the imaged object Tg3. Therefore, the image processing device 4 can predict the position of the imaged object Tg3 with higher accuracy. In addition, the image processing device 4 can more efficiently control the operation of the camera switching unit 47 based on the predicted position, so that the next imaged object can be set efficiently before the imaged object is lost. Therefore, when the image processing device 4 according to the second embodiment and other modified examples is used, the unmanned aircraft 2A can always receive the position difference, so that the posture control can be performed more stably.

[0221] In addition, the receiving unit 42 in the image processing device 4 according to the second embodiment and other modified examples also receives the moving speed information of the camera, and the output unit 46 calculates and outputs the difference between the calculated moving speed of the object being photographed and the moving speed information of the camera. Thus, the image processing device 4 according to the second embodiment and other modified examples can correct not only the error of the position of the object being photographed, but also the control error of the actuator 2 that moves the camera. In addition, the actuator 2 can correct the position error of the camera based on the output speed difference. Therefore, the image processing device 4 can calculate the position error of the object being photographed with higher accuracy, and can calculate the control error of other devices (such as the actuator 2, etc.). Therefore, when the image processing device 4 according to the second embodiment and other modified examples is used, the unmanned aircraft 2A can always receive the position difference and the speed difference, so that the posture control can be performed more stably, and the flight control error of the unmanned aircraft 2A can be corrected.

[0222] Various embodiments have been described above with reference to the accompanying drawings, but the present disclosure is not limited to the above examples. If it is a person skilled in the art, it should be clear that various changes, corrections, substitutions, additions, deletions, and equivalents can be thought of within the scope described in the claims, and it can be understood that these also belong to the technical scope of the present disclosure. In addition, the various constituent elements in the above various embodiments can also be arbitrarily combined within the scope of the main purpose of the invention.

[0223] In addition, the present application is an application based on the Japanese patent application (Japanese Patent Application No. 2019-127912) filed on July 9, 2019, and the contents thereof are incorporated herein by reference.

[0224] Industrial Applicability

[0225] The present disclosure is useful in proposing an image processing device and an image processing method, as an image processing device and an image processing method that perform efficient image processing on an image of an object captured by a camera and calculate a position error of the object with higher accuracy.

[0226] Description of Reference Numerals

[0227] 1: control device; 10, 20, 40: control unit; 11, 21, 41: memory; 12: area data; 2: actuator; 22: driving unit; 23: error correction unit; 24: arm; 3: camera; 4: image processing device; 42: receiving unit; 43: prediction unit; 44: detection unit; 45: measurement unit; 46: output unit; 5: operation unit; IA1: imaging range; Pt0: reference mark; Tg1: imaging object.

Claims

1. An image processing device comprising: a prediction unit that acquires position information of an image capturing object from pre-input area data and predicts a position of the image capturing object within an image capturing range of a camera that moves in a predetermined direction based on the position information of the image capturing object; an image receiving unit that receives a camera image showing the imaging object, which is captured by the camera that moves based on the position information of the imaging object; a detection unit that detects the imaging subject by reading an imaging image of a limited range that is a part of the imaging range from the imaging image of the imaging range based on the predicted predicted position of the imaging subject; a measuring unit that measures the detected position of the imaging object; and an output unit that outputs a difference between the measured position of the imaging object and the predicted position, in, The prediction unit further predicts the position of the imaging object captured after the imaging image of the measurement position of the imaging object is measured, and outputs the predicted position to the output unit. The predetermined direction is a direction opposite to a vector that starts at the measurement position of the imaging target and ends at the position of the imaging target.

2. The image processing device according to claim 1, characterized in that The image processing device further includes a camera switching unit that switches the connection between the image processing device and each of the plurality of cameras having different imaging ranges. The camera switching unit switches, based on the predicted position, to a camera that can capture an image of the predicted position among the plurality of cameras.

3. The image processing device according to claim 2, characterized in that: The camera switching unit sets the camera that reads out the limited range including the predicted position of the camera object to track the camera object as a tracking camera, and sets other cameras that read out other limited ranges outside the camera range of the tracking camera to detect other camera objects as detection cameras, to switch the tracking camera and the detection camera.

4. The image processing device according to claim 2, characterized in that: The camera switching unit sets the limited range including the predicted position of the camera object among each of the multiple limited ranges of each of the multiple cameras as a limited range for tracking, and sets at least one of the other limited ranges other than the limited range for tracking as a limited range for detection for detecting other camera objects, so as to switch the limited range for tracking and the limited range for detection.

5. The image processing device according to claim 1, characterized in that: The detection unit detects at least one feature point having a predetermined feature value included in each of the limited ranges of at least two captured images.

6. The image processing device according to claim 5, characterized in that: The detection unit corrects the limited range based on the distribution of each of the plurality of detected feature points.

7. The image processing device according to claim 6, characterized in that: The detection unit sets the detected feature points as other imaging targets.

8. The image processing device according to claim 7, characterized in that: The measuring unit measures the amount of movement of the imaging subject based on the detected positions of the imaging subject. The output unit calculates a motion speed of the imaging subject based on the measured motion amount of the imaging subject, and outputs the motion speed of the imaging subject.

9. The image processing device according to claim 8, characterized in that: The receiving unit also receives the moving speed information of the camera. The output unit calculates and outputs a difference between the calculated moving speed of the imaging object and the moving speed information of the camera.

10. An image processing method, which is performed by an image processing device connected to at least one camera, in which: Acquire the location information of the camera object from the pre-entered area data, predicting the position of the imaging object within the imaging range of the camera moving in a predetermined direction based on the position information of the imaging object, receiving a camera image showing the imaging object captured by the camera that moves based on the position information of the imaging object, Based on the predicted position of the imaging object, a predetermined limited range including the predicted position is read out in the imaging range of the camera to detect the imaging object, measuring the detected position of the imaging object, outputting a difference between the measured position of the imaged object and the predicted position, in, further predicting and outputting the position of the imaging object photographed after the imaging image of the measurement position of the imaging object is measured, The predetermined direction is a direction opposite to a vector that starts at the measurement position of the imaging target and ends at the position of the imaging target.

Images