Image analysis apparatus, image analysis method, and program

Abstract

To provide an image analysis apparatus, an image analysis method, and a program which can discriminate a kind of a flying body flying in a wide range with satisfactory accuracy.SOLUTION: An image analysis apparatus 1 according to the present invention includes a detection unit 312 for detecting a flying body from time-serial images to generate a first displacement signal indicating time change of displacement of the flying body, and a discrimination unit 313 for discriminating a kind of the flying body based on the first displacement signal. The detection unit 312 generates the first displacement signal with using displacement of a geometric center of the flying body as displacement of the flying body. The discrimination unit 313 decomposes the first displacement signal with empirical mode decomposition and then integrates unique mode function of figures less than a predetermined threshold value, thereby generating a second displacement signal representing displacement relating to fluttering of the flying body. Thus, the discrimination unit analyzes a frequency of the second displacement signal to discriminate the kind of the flying body.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image analysis device, an image analysis method, and a program, and more particularly to an image analysis device, an image analysis method, and a program for discriminating the type of flying object such as a bird or an insect. [Background technology]

[0002] Bird and animal damage leads to a lot of economic losses, and is therefore considered an important issue for agricultural workers. For example, vegetable and fruit farms suffer losses due to reduced yields caused by bird damage. When dealing with such bird and animal damage, effective responses differ depending on the number and type of birds, so it is important to know the number of birds flying in and to distinguish their types. There is also a need to know the status of insect arrivals for pest control, and the status of honeybees in beekeeping.

[0003] For example, in Patent Document 1, the trajectory of a bird's wings is obtained from successive captured images, and based on the trajectory, the depth, pitch, number of flappings per unit time, etc., of the flapping, it is determined whether the photographed bird is a large bird of prey or not.

[0004] In addition, Patent Document 2 proposes to estimate the birds and animals based on their size, body shape, color, calls, movements, etc. obtained from image and sound data collected using a drone. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2021-60694 A [Patent Document 2] JP 2021-40519 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, when identifying flying birds based on captured images as in Patent Document 1, it is difficult to obtain high-resolution images that constantly capture the movements of flying birds. Therefore, it is difficult not only to determine a specific type of flying object, such as whether it is a large bird of prey, but also to accurately identify the type of flying object.

[0007] In addition, when collecting image and sound data using a drone as in Patent Document 2, it is difficult to collect data continuously for a long period of time. In addition, since the spatial directivity of the microphone is low, it is difficult to identify the exact location of birds and animals over a wide area from a remote location. Therefore, when there is a lot of audio noise or when there are multiple birds and animals in the target area, it is difficult to estimate the birds and animals.

[0008] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide an image analysis device, an image analysis method, and a program that can accurately distinguish the type of flying object flying over a wide area. [Means for solving the problem]

[0009] In order to achieve the above object, an image analyzing apparatus according to a first aspect of the present invention comprises: A detection unit that detects a flying object from a time-series image and generates a first displacement signal that is a time change in displacement of the flying object; a discrimination unit that discriminates the type of the flying object based on the first displacement signal, The detection unit is generating the first displacement signal by taking the displacement of the geometric center of the flying object as the displacement of the flying object; The discrimination unit is Decomposing the first displacement signal by empirical mode decomposition and integrating a number of intrinsic mode functions equal to or less than a predetermined threshold value to generate a second displacement signal representing a displacement related to flapping of the flying object; The type of the flying object is identified by performing a frequency analysis on the second displacement signal.

[0010] In addition, the detection unit calculating a first foreground image and a second foreground image represented by the following formula using a first frame image and a second frame image at a predetermined time interval and an intermediate frame image between the first frame image and the second frame image in the time series images;

number

[0011] In addition, the detection unit a dilation operation for the region representing the flying object, and then a contraction operation using a kernel smaller than that of the dilation operation to detect the flying object; This may also be the case.

[0012] In addition, the discrimination unit performing a Fourier transform on the second displacement signal to calculate a frequency spectrum, and identifying the type of the flying object by pattern matching based on the calculated frequency spectrum; This may also be the case.

[0013] In addition, the discrimination unit performing a continuous wavelet transform on the second displacement signal to calculate a time change in power at a peak frequency; and calculating a duty ratio of a binary signal generated by binarizing the signal representing the time change in power at the peak frequency using a predetermined threshold value; Identifying the type of the flying object by classification based on the calculated duty ratio and the peak frequency. This may also be the case.

[0014] In addition, in an image analysis method according to a second aspect of the present invention, Detecting a flying object from the time series images and generating a first displacement signal which is a time change in displacement of a geometric center of the flying object; Decomposing the first displacement signal by empirical mode decomposition and integrating a number of intrinsic mode functions equal to or less than a predetermined threshold value to generate a second displacement signal representing a displacement related to flapping of the flying object; The type of the flying object is identified by performing a frequency analysis on the second displacement signal.

[0015] Moreover, the program according to the third aspect of the present invention comprises: Computer, a detection unit that detects a flying object from a time-series image and generates a first displacement signal that is a time change in displacement of a geometric center of the flying object; Decomposing the first displacement signal by empirical mode decomposition and integrating a number of intrinsic mode functions equal to or less than a predetermined threshold value to generate a second displacement signal representing a displacement related to flapping of the flying object; a discrimination unit that discriminates the type of the flying object by performing frequency analysis on the second displacement signal; Function as. Effect of the Invention

[0016] According to the image analysis device, image analysis method, and program of the present invention, the change in displacement of the flying object over time is decomposed using empirical mode decomposition, and a displacement signal related to the flapping component is generated by integrating intrinsic mode functions up to a predetermined threshold number, and the type of flying object is identified by frequency analysis, thereby enabling type identification to be performed with high accuracy. [Brief description of the drawings]

[0017] [Figure 1] 1 is a functional block diagram of an image analysis device according to an embodiment of the present invention. [Diagram 2] 4 is a flowchart showing a flow of image analysis according to the embodiment. [Diagram 3] 4A and 4B are diagrams showing examples of imaging target areas according to the embodiment; [Figure 4] 11A and 11B are diagrams showing examples of a hollow phenomenon in a discrimination target region. [Diagram 5] 4 is a diagram showing a processing flow of an inter-frame difference method according to the embodiment; [Figure 6] A conceptual diagram showing the COB association process. [Figure 7] 11A and 11B are diagrams showing examples of tracking results when a plurality of discrimination targets exist. [Figure 8] FIG. 13 is a diagram showing an example of COB association. [Figure 9] 4A and 4B are diagrams showing examples of displacement signals in the x and y directions of a COB. [Figure 10] 10 is a diagram showing an example of a signal obtained by integrating the x-direction displacement signal and the y-direction displacement signal shown in FIG. 9 . [Figure 11] FIG. 13 is a diagram showing an example of an input signal for empirical mode decomposition. [Figure 12] FIG. 12 is a diagram showing an example of a result of processing the input signal of FIG. 11 by empirical mode decomposition. [Figure 13] 6A to 6C are diagrams showing examples of second displacement signals for three types of flying objects. [Figure 14] 14 is a diagram showing an example of a frequency spectrum calculated by subjecting the second displacement signal of FIG. 13 to a fast Fourier transform; [Figure 15] (A) is an example of the second displacement signal, and (B) is a diagram showing the spectrum of the second displacement signal in (A) obtained by continuous wavelet transform. [Figure 16] (A) is a graph showing the time variation of the power at the peak frequency in Figure 15(B), and (B) is a graph showing the binarized time variation of the power in (A). [Figure 17] FIG. 13 is a diagram showing an example in which the relationship between the peak frequency and the duty ratio is plotted for each discrimination target. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an image analyzing device 1 according to an embodiment of the present invention will be described with reference to the drawings. The image analyzing device 1 is a device that distinguishes the type of flying object, such as a bird or insect, flying in a predetermined shooting target area.

[0019] As shown in the functional block diagram of FIG. 1, the image analyzing device 1 includes a camera 10 and a control unit 30.

[0020] The camera 10 is a camera that captures an image of a target area for identifying flying objects. The size of the target area is not particularly limited, and may be set within a range in which the captured flying object can be detected as a target for identification based on the resolution, frame rate, etc. of the camera 10. The camera 10 according to the present embodiment is a DFK38UX255 manufactured by Imaging Source, and captures images at a frame rate of 100 fps and a resolution of 4096×904 pixels, for example. This allows accurate identification of the type of target for identification by vibration analysis using the image I for flying objects such as birds with a flapping frequency of about 2 to 40 Hz. The camera 10 also transmits data of the captured image I to the control unit 30.

[0021] As shown in FIG. 1, the control unit 30 includes a control unit 31, a storage unit 32, a display unit 33, and an input unit .

[0022] The control unit 31 is composed of a central processing unit (CPU), a graphics processing unit (GPU), a read only memory (ROM), a random access memory (RAM), etc., and controls the operation of the camera 10. The control unit 31 also detects a flying object to be discriminated based on an image I captured by the camera 10, and discriminates the type of the detected object to be discriminated. The control unit 31 loads various operation programs and data stored in the ROM, storage unit 32, etc. of the control unit 31 into the RAM and operates the CPU, thereby realizing each function of the control unit 31 shown in FIG. 1. In this way, the control unit 31 operates as a camera control unit 311, a detection unit 312, and a discrimination unit 313.

[0023] The camera control unit 311 controls the camera 10 to capture the target area and obtain the image I. The size of the target area is not particularly limited, and may be set according to the size and flight speed of the object to be identified, the range to be observed, and the like. When detecting flying objects such as birds and insects, the flying objects may move quickly. In addition, birds, insects, and the like have a higher degree of freedom in the direction of movement compared to animals that move on the ground, so that the direction of movement is changed frequently and it is difficult to predict the direction of movement. Therefore, in identifying flying objects, it is required to capture a relatively wide range. For example, when identifying birds flying into a field, the width can be set to about 100 m and the height to about 20 m, and when identifying bees around a hive, the width can be set to about 5 m and the height to about 3 m. Typically, the target area is set to a width of about 1 to 100 m.

[0024] The detection unit 312 detects a discrimination target for discriminating the type based on the acquired image I. When detecting a specific target from a time series of images, a process using an inter-frame difference method is generally performed in which the difference between different frames is calculated and the target foreground is separated from the rest of the background.

[0025] In order to capture a wide range as the target area, it is often necessary to relatively reduce the resolution due to restrictions imposed by the processing power of the camera. When the resolution is reduced, the effect of noise increases, making it difficult to detect the target within the target area from a single still image. In addition, when the frame rate is high, there is a risk that the target overlaps in the difference between different frames and cannot be detected. Therefore, in this embodiment, the frame difference method is improved to increase the detection accuracy of the target. Details of the image processing for detection will be described later.

[0026] The detection unit 312 generates a first displacement signal that indicates a change in the displacement of the detected discrimination target over time. The position of the discrimination target is expressed as coordinates of an image I that is set in advance in correspondence with the subject area captured by the camera 10.

[0027] The discrimination unit 313 discriminates the type of the discrimination target based on the image I. More specifically, the discrimination unit 313 performs vibration analysis using the first displacement signal generated by the detection unit 312 as vibration information based on the time-series images I of each discrimination target. The first displacement signal based on the displacement of the discrimination target contains an overall movement component, a fluctuation component due to flapping, high-frequency noise, and the like. The discrimination unit 313 according to this embodiment performs processing to extract components that represent characteristics of vibration due to flapping from the first displacement signal. Then, features that serve as discrimination indexes are extracted from a second displacement signal consisting of the extracted components to discriminate the type of the discrimination target.

[0028] Furthermore, the control unit 30 communicates with the camera 10 via a communication interface (communication I / F). Specifically, the camera control unit 311 transmits control information to the camera 10 and acquires image data of the captured image I.

[0029] The memory unit 32 is a non-volatile memory such as a hard disk or flash memory, and stores information such as data of the image I captured by the camera 10, a detection program for detecting the object to be discriminated from the image I, and a discrimination program for discriminating the type of the object to be discriminated.

[0030] The display unit 33 is a display device, such as a liquid crystal display, provided in the control unit 30. The display unit 33 displays the image I captured by the camera 10, information about the discrimination target detected by the detection unit 312, and the like.

[0031] The input unit 34 is an input device for inputting parameters related to shooting such as the frame rate of the camera 10 and the shooting time, etc. The input unit 34 is a keyboard, a touch panel, a mouse, etc. provided in the control unit 30.

[0032] Hereinafter, the process of discriminating a flying object using the image analysis method according to the present embodiment will be specifically described with reference to the flow chart of FIG.

[0033] In the discrimination process according to this embodiment, as shown in Fig. 3, a range of about 100 m wide, more specifically, a range with an observation viewing angle of about 60° and a maximum effective measurement distance of about 150 m is set as the imaging target area, and discrimination of birds flying in this imaging target area is described as an example. In this embodiment, discrimination of birds is described as an example, but by selecting the imaging range, resolution, etc. based on the size, flight range, flight speed, etc. of the discrimination target, it is possible to discriminate various flying objects such as insects that fly by flapping their wings, drones with rotors, etc. using the vibration analysis according to this embodiment.

[0034] (Image acquisition step) In the image analyzing device 1 according to the present embodiment, the camera control section 311 of the control unit 30 controls the camera 10 to capture an image I of the target area (step S11). In the present embodiment, the image I of the target area captured by the camera 10 has a frame rate of 100 fps and a resolution of 4096×904 pixels.

[0035] (First displacement signal generating step) The control unit 30 receives data of the captured image I from the camera 10, and detects, as a discrimination target, a flying object flying within the capture target area.

[0036] In this embodiment, in order to detect the discrimination target, a region representing the discrimination target is calculated using an inter-frame difference method (step S12). The inter-frame difference method usually calculates the pixel difference between two consecutive frames, more specifically, the difference in luminance of the pixel at each coordinate, and extracts the foreground by binarizing it based on a preset threshold value.

[0037] In this embodiment, a vibration analysis is performed using the time change of the displacement related to the movement of the discrimination target as vibration information to detect the type of the discrimination target. Therefore, in order to improve the accuracy of discrimination of the discrimination target, it is preferable to capture the image I at a high frame rate. However, when the frame difference method is applied to the image I captured at a high frame rate, the discrimination target may overlap between two frames. When the frame difference method is executed in such a situation, if the difference in the luminance value within the object of the discrimination target is smaller than a preset threshold, a part of the foreground representing the discrimination target is missing, causing a hollow phenomenon as shown in FIG. 4.

[0038] Therefore, in this embodiment, as shown in Fig. 5, a span is enlarged from two consecutive frames (Fig. 5(A) and (C)), and the inter-frame difference method is applied. In addition, an intermediate frame image (Fig. 5(B)) is selected from between the enlarged frames, and the shadow is removed.

[0039] The first foreground image, which is the difference between the first and second frame images with an expanded span (FIG. 5(D)), is calculated using the following formula.

number

[0040] Then, to remove the shadow, we select the intermediate frame and use it as the second foreground image I d2 Here, FIG. 5(E) shows the mask image I t0 I d1 ,Figure 5(F) shows the mask image I (t0-1 / 2·T) I d1 ,Figure 5(G) shows the second foreground image I, which is the difference mask image. d2 Represents.

number

[0041] Second foreground image Id2 After obtaining, two foreground images I d1 ,I d2 By calculating the overlap of the two, the binary mask image I target can be obtained.

number

[0042] As described above, in this embodiment, by expanding the frame interval of the frame difference method, it is possible to suppress the occurrence of the hollow phenomenon caused by the overlap of the discrimination object between frames. In addition, since the discrimination object is detected by calculating the area representing the discrimination object by the frame difference method using the intermediate frame, it is possible to obtain a binary image without the residual image of the discrimination object that occurs in the general frame difference method.

[0043] In some cases, such as when the background contains complex shadows, the difference in luminance between the foreground region representing the discrimination target calculated by the frame difference method and the background may be relatively small. In this case, missing parts may appear in the discrimination target region. In extreme cases, these missing parts may result in the discrimination target region being divided into several parts. If the discrimination target region is divided, each divided region may be recognized as a separate discrimination target in subsequent processing, which may significantly reduce the recognition accuracy.

[0044] In order to avoid such problems caused by region division, the detection unit 312 according to the present embodiment detects the binary image I target (Step S13). Specifically, the detection unit 312 performs a dilation operation as shown in the following formula to fill in missing parts in the region to be discriminated and prevent the region from being divided. The size of the kernel to be applied is not particularly limited, and may be determined in consideration of the size of missing parts expected from the frame rate, the span of the interframe difference method, etc.

number

[0045] In order to remove the redundant dilated contour generated by the dilation calculation of formula (4), the detection unit 312 performs an erosion calculation using a kernel smaller than the kernel used in the dilation calculation. If the discrimination target region is enlarged by the above dilation calculation, there is a risk that the calculation accuracy of the center displacement described below will decrease. By performing the following erosion calculation, it is possible to fill in missing parts without excessively enlarging the discrimination target region.

number

[0046] As described above, the detection unit 312 detects the binary image I target By performing expansion and contraction calculations on the target object, the missing parts of the foreground are filled in. This makes it possible to maintain the size of the area representing the target object while suppressing defects such as division due to missing areas.

[0047] Next, the displacement of the discrimination target is calculated from the detected image of the discrimination target. When tracking a flying object such as a bird or insect, the geometric center of body (COB: Center Of Body) of the image area of ​​the entire body including wings changes with flapping of wings. In other words, flapping characteristics that differ depending on the type of flying object are reflected in the displacement of the COB. Therefore, in this embodiment, the COB of the flying object is calculated as the displacement (step S14).

[0048] COB is calculated based on the image moments of the binary flying object regions that have been morphologically calculated. First, the original moments of the region to be discriminated are calculated.

number

[0049] The coordinates of the center of gravity representing the COB are calculated as follows:

number

[0050] The above procedure is repeated for each frame, thereby obtaining the coordinates of all COBs in the regions to be discriminated for each frame.

[0051] Since the COBs calculated in step S14 are all discrete, they must be associated with each discrimination target. In this embodiment, discrimination targets are associated and labeled based on Euclidean distance.

[0052] As shown in Fig. 6, first, at time t n-1 Select one of the COBs obtained in . The selected COB and the time t n The Euclidean distance between each COB and all other COBs obtained in is calculated and they are sorted from smallest to largest.

[0053] The new COB with the shortest Euclidean distance is associated with the old COB if the distance is within a preset search range threshold, and the associated chain of COBs belongs to each discrimination target. The above procedure is repeated until all COBs are associated. Any COB that is not in any search range is assigned as a new discrimination target.

[0054] Figure 7 shows an example of a tracking result when there are multiple discrimination targets. Figure 8 shows an example of COB association. The trajectory of the associated COB can be expressed as follows:

number

[0055] Each point P of the COB recorded in time series n (x tn ,y tn ), the corresponding vector can be expressed as:

number

[0056] From time t=t0 to t=t n The displacement of the COB up to can be expressed individually as follows:

number

[0057] In this case, Δx and Δy can be regarded as separate discrete signals (FIG. 9). In reality, the flapping motion of a bird is not necessarily orthogonal to the coordinate axes. Therefore, if only one signal is used, some of the kinematic information will be lost. In this embodiment, the two displacement signals are integrated as follows to reduce the dimension of the calculation, and a first displacement signal SigΔ representing the displacement of the object to be discriminated is generated (step S15).

number

[0058] (Second displacement signal generating step) Next, the discrimination unit 313 performs signal processing for discriminating the first displacement signal SigΔ. The first displacement signal SigΔ obtained in step S15 contains not only the displacement due to the flapping motion but also a drift component due to the overall movement of the bird. In this embodiment, signal processing is performed to remove redundant frequency components before analyzing the displacement signal.

[0059] In general, the Fourier transform is applied to decompose and analyze regular infinite periodic signals, and is not suitable for irregular non-stationary signals. Usually, a bird's flight period consists of several phases, and the wings flap periodically in the flapping phase. That is, the recorded bird's displacement signal is considered to be a non-stationary non-periodic signal having partially periodic characteristics. Therefore, the discrimination unit 313 according to this embodiment decomposes the first displacement signal SigΔ, which is a non-stationary non-periodic signal, into intrinsic mode functions (IMFs) representing several main signal components using empirical mode decomposition (EMD).

[0060] Using empirical mode decomposition, a function can be decomposed (filtered) as follows: In the filtering process, to obtain the upper envelope of the function, all the local maxima are identified and connected with a cubic spline line. The lower envelope is obtained by identifying all the local minima and connecting them with a cubic spline line. Let the average of the upper and lower envelopes be m1 and the remaining function be h1.

number

[0061] If we consider h1 as a new function, we can decompose it as follows:

number

[0062] The above filtering allows to eliminate surfed waves and make the wave profile more symmetrical. This filtering process is repeated until all local maxima are on the zero line.

number

[0063] The first IMF component c1 is specified as:

number

[0064] The first component of the function, c1, contains the information about the shortest period with high frequency. The rest of the original function can be found as follows:

number

[0065] The residual r1 contains the information of the longer period components. The residual r1 is treated as a new function, and the above filtering process is repeated multiple times as follows:

number

[0066] residual r n When becomes small enough, the decomposition is terminated. Adding all the above equations together, we get

number

[0067] In this way, by using empirical mode decomposition, a non-periodic non-stationary signal can be decomposed into multiple components with various time scales. Figure 11 shows an example of an input signal, and Figure 12 shows its IMF components.

[0068] In the case of a signal representing a bird's flight, essential information about the motion is generally contained in the first few IMFs, and the residual components contain the bird's global motion. When applying empirical mode decomposition, signal components of similar scales may appear in different IMFs, or one IMF may contain signal components of various scales. Therefore, even if the actual flapping frequency of the bird is known, it is not clear which IMF constitutes a suitable flapping signal. Therefore, in this embodiment, a second displacement signal is generated that contains multiple IMF components and represents the time change of displacement so as not to damage the flapping signal. The second displacement signal Sig EMDcan be calculated as follows (step S16):

[0069]

number

number

[0070] Coefficient k i is a weighting coefficient that can be adjusted according to the type of bird. For example, the average flapping frequency of a crow is about 3.8Hz, and in this case, most of the features of the flapping signal fall within IMF3. In contrast, the average flapping frequency of a swallow is 13.2Hz, and most of the features of the flapping signal fall within IMF2. Therefore, the coefficient k i By adjusting the size of the pixel, the type of the object to be discriminated can be discriminated with higher accuracy.

[0071] Furthermore, the variable m in equation (19) represents the number of IMFs to be integrated, and is set to be equal to or less than a predetermined threshold. The threshold and variable m are set, for example, to about 4 (IMF1 to IMF4), and can be set according to the type of object to be discriminated. When the object to be discriminated is clearer, the variable m is set to a number less than the threshold (for example, 3), and fewer IMFs are integrated to obtain the second displacement signal Sig EMD It is also possible to generate the following.

[0072] The first displacement signal SigΔ and the second displacement signal Sig EMD There is a large amount of inherent noise caused by the brightness gain during recording, as well as high frequency noise due to partially present background disturbances in the second displacement signal Sig EMD The second displacement signal Sig is filtered (step S17). EMD Since is a discrete signal, a discrete wavelet transform (DWT) is applied as a low-pass filter.

[0073] First, the second displacement signal Sig EMD is passed through a low-pass filter with impulse response g and convolution is performed.

number

[0074] After half the frequencies are removed from the signal according to the sampling theorem, the filter output is subsampled and further processed by passing it through a new low-pass filter g and a high-pass filter h as follows:

number

[0075] Then, the high frequency components of the second displacement signal are limited by a coefficient K (0≦K≦1) as shown in the following equation.

number

[0076] (Discrimination step) Next, the discrimination unit 313 outputs the filtered second displacement signal Sig filtered In this embodiment, the discrimination unit 313 performs discrimination processing based on the second displacement signal Sig filtered is Fourier transformed to obtain a frequency spectrum (step S18). The obtained frequency spectrum is subjected to pattern matching to determine the type of the object to be determined (step S19).

[0077] As described above, in the image analysis device and image analysis method according to the present embodiment, the first displacement signal that indicates the time change of the displacement of the flying object is decomposed by empirical mode decomposition, and the second displacement signal related to the flapping component is generated by integrating the intrinsic mode functions (IMFs) whose number is equal to or less than a preset threshold value. This makes it possible to extract the flapping component related to the displacement of the flying object with high accuracy.

[0078] In the image analysis device and image analysis method according to the present embodiment, the type of flying object is identified by frequency analysis of the generated second displacement signal related to the flapping component. This allows the type to be identified based on the second displacement signal related to the flapping component that represents the characteristics of the type of flying object, thereby improving the accuracy of the identification.

[0079] In addition, in this embodiment, the frame difference method is improved to detect the discrimination target by the frame difference method using three frames, as shown in Fig. 5. This makes it possible to eliminate overlapping of discrimination targets between frames from which differences are taken, thereby improving the detection accuracy of the discrimination target.

[0080] In this embodiment, the region representing the discrimination target obtained by the frame difference method is expanded by a morphological operation, and then contracted by a smaller kernel than that of the expansion operation to extract the discrimination target. This makes it possible to fill in the gaps without excessively expanding the region representing the discrimination target, thereby suppressing problems caused by region division.

[0081] In this embodiment, the second displacement signal is Fourier transformed and the type of flying object is identified by pattern matching based on the frequency characteristics. This makes it possible to easily and accurately identify the type of flying object based on the second displacement signal that accurately represents the frequency characteristics of the fluctuation caused by the flapping of the flying object.

[0082] (Analysis example) This section describes a field experiment in which flying birds were photographed and their species were identified using image analysis under the following environmental settings. The camera used in the experiment was a DFK 38UX255 from The Imaging Source. The resolution was 4096 x 904 pixels, and the frame rate was 100 fps. The lens used was a Ricoh FL-BC1218A-VG, with a focal length of 12.5 mm and a viewing angle of 55.4° x 42.6°.

[0083] Fig. 13 is an example of a graph of the second displacement signal for three types of flying objects, and Fig. 14 is an example of a frequency spectrum calculated by performing a fast Fourier transform on the second displacement signal of Fig. 13.

[0084] As shown in Fig. 14, the frequency of flapping of birds is related to the mass ratio between the wing and the body, and it is considered that the frequency characteristics differ depending on the bird type. In this embodiment, as described above, since the vibration analysis is performed based on the second displacement signal related to the flapping component, the frequency characteristics of each type are well reflected in the frequency spectrum, and it is found that the type of flying object can be accurately identified.

[0085] (Modification) In the above embodiment, the second displacement signal is Fourier transformed to calculate a frequency spectrum, and the type of the flying object is identified by pattern matching, but the method of identifying the type of the flying object is not limited to this. For example, the type of the flying object may be identified by classification based on the frequency representing the flapping of wings and the time interval of the flapping of wings. This will be specifically described below.

[0086] The discrimination unit 313 performs a continuous wavelet transform (CWT) on the second displacement signal. Fig. 15(A) shows an example of the second displacement signal, and Fig. 15(B) shows a spectrum obtained by the CWT of the second displacement signal of Fig. 15(A). As shown in Fig. 15(B), by performing the CWT process on the second displacement signal, it is possible to select a peak frequency.

[0087] Next, the power fluctuation over time at the peak frequency is extracted (Fig. 16(A)). The extracted power fluctuation is then binarized using a preset threshold to generate a binary signal, and the duty ratio of the binary signal is calculated (Fig. 16(B)).

[0088] FIG. 17 is a diagram in which the relationship between the peak frequency and the duty ratio is plotted for each discrimination target. As shown in FIG. 17, the plot area of ​​the relationship between the peak frequency and the duty ratio is divided depending on the type of discrimination target. In this modification, the relationship between the peak frequency, the duty ratio, and the type of discrimination target is stored in advance in the database of the storage unit 12. Then, the discrimination unit 313 calculates the peak frequency and the duty ratio for the detected second displacement signal of the discrimination target, and discriminates the type of the discrimination target based on the data in the database. The classification method for discriminating the type is not particularly limited, and known methods such as k-means, KNN, and DBSCAN can be used.

[0089] As described above, in this modification, the second displacement signal is subjected to a continuous wavelet transform to calculate the peak frequency and the duty ratio of the power at the peak frequency. Then, the type of the object to be discriminated is discriminated based on the calculated peak frequency and duty ratio. This allows for more accurate discrimination that takes into account the proportion of time involved in flapping the flying object during flight.

[0090] Furthermore, the image analyzing device 1 according to the above embodiment includes the camera 10, and discriminates the type of flying object in the photographed area photographed by the camera 10, but is not limited to this. For example, the type of flying object may be discriminated by performing image analysis on time-series images photographed in advance and stored in the storage unit 32, or time-series images acquired via a network. This makes it possible to appropriately perform image analysis on images photographed in the past, images photographed in a remote location, etc.

[0091] The discrimination of flying objects by image analysis according to the above embodiment can be realized by using a normal computer system. For example, a computer program for executing the image analysis according to the above embodiment can be distributed via a network, and the computer program can be installed in a computer to cause the computer to function as the image analysis device 1 that executes the above image analysis. [Industrial Applicability]

[0092] The present invention is suitable for an image analysis device that distinguishes flying objects over a wide range, and is particularly suitable for an image analysis device that distinguishes flying objects when a plurality of objects to be distinguished exist in an image capture area. [Explanation of symbols]

[0093] 1 image analysis device, 10 camera, 30 control unit, 31 control section, 311 camera control section, 312 detection section, 313 discrimination section, 32 storage section, 33 display section, 34 input section

Claims

1. A detection unit that detects a flying object from a time series of images and generates a first displacement signal that is a time change in displacement of the flying object; a discrimination unit that discriminates the type of the flying object based on the first displacement signal, The detection unit is generating the first displacement signal by taking the displacement of the geometric center of the flying object as the displacement of the flying object; The discrimination unit is Decomposing the first displacement signal by empirical mode decomposition and integrating a number of intrinsic mode functions equal to or less than a predetermined threshold value to generate a second displacement signal representing a displacement related to flapping of the air vehicle; The type of the flying object is identified by subjecting the second displacement signal to frequency analysis.

1. An image analysis device comprising:

2. The detection unit is calculating a first foreground image and a second foreground image represented by the following formula using a first frame image and a second frame image at a predetermined time interval and an intermediate frame image between the first frame image and the second frame image in the time series images; [0010] Here, t ０ is the current time, T is the adjustable delay time, I ｔ０ is the first frame image, I （ｔ０－Ｔ） is the second frame image, I （ｔ０－１／２・Ｔ） is the intermediate frame image, I ｄ１ is the first foreground image, I ｄ２ represents the second foreground image. extracting an area representing the flying object by multiplying the first foreground image and the second foreground image, thereby detecting the flying object; 2. The image analysis device according to claim 1 .

3. The detection unit is a dilation operation for the region representing the flying object, and then a contraction operation using a kernel smaller than that of the dilation operation to detect the flying object; 3. The image analysis device according to claim 2.

4. The discrimination unit is performing a Fourier transform on the second displacement signal to calculate a frequency spectrum, and identifying the type of the flying object by pattern matching based on the calculated frequency spectrum; 4. The image analysis device according to claim 1, wherein the image analysis device is a digital camera.

5. The discrimination unit is performing a continuous wavelet transform on the second displacement signal to calculate a time change in power at a peak frequency; and calculating a duty ratio of a binary signal generated by binarizing the signal representing the time change in power at the peak frequency using a predetermined threshold value; Identifying the type of the flying object by classification based on the calculated duty ratio and the peak frequency.

4. The image analysis device according to claim 1, wherein the image analysis device is a computer.

6. Detecting a flying object from the time-series images and generating a first displacement signal that is a time change in displacement of a geometric center of the flying object; Decomposing the first displacement signal by empirical mode decomposition and integrating a number of intrinsic mode functions equal to or less than a predetermined threshold value to generate a second displacement signal representing a displacement related to flapping of the air vehicle; The type of the flying object is identified by subjecting the second displacement signal to frequency analysis.

1. An image analysis method comprising:

7. Computer, a detection unit that detects a flying object from a time-series image and generates a first displacement signal that is a time change in displacement of a geometric center of the flying object; Decomposing the first displacement signal by empirical mode decomposition and integrating a number of intrinsic mode functions equal to or less than a predetermined threshold value to generate a second displacement signal representing a displacement related to flapping of the air vehicle; a discrimination unit that discriminates the type of the flying object by performing frequency analysis on the second displacement signal; A program that functions as a

Images