Image analysis apparatus, control method of image analysis apparatus, and program

Abstract

To associate a plurality of sections constituting a target object in an image, in a more appropriate manner.SOLUTION: A feature extraction unit 320 extracts features of a target object from an image obtained by imaging the target object. A detection region estimation unit 330 detects a plurality of sections of the target object from the features extracted by the feature extraction unit 320. An angle estimation unit 340 estimates angles formed by semi-straight lines extending in directions in which the target object is extended based on target sections, the target sections being the detected sections, and a straight line extending in a reference direction in the image. An association determination unit 370 associates two sections having associations, out of the plurality of sections of the target object, on the basis of detection results of the plurality of sections by the detection region estimation unit 330 and estimation results of the angles by the angle estimation unit 340.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an image analyzing device, a control method for an image analyzing device, and a program. [Background technology]

[0002] In recent years, a technology has been proposed that extracts various information by performing recognition processing on an image of a target object. In particular, a technology that uses a multi-layered neural network, also called a deep net (e.g., a deep neural network constructed based on deep learning), has attracted attention and is being actively researched.

[0003] When detecting an arbitrary object in an image, in addition to a method of detecting the entire object as one object, there is a method of detecting multiple partial regions of the object and inferring the correspondence between the detected partial regions. In this disclosure, the process of inferring the correspondence between multiple partial regions of the object is also referred to as a matching process for convenience. The matching process makes it possible to detect the object with high accuracy, and by using information on the correspondence between the multiple partial regions, it becomes possible to realize various functions. Patent Document 1 discloses a technology for detecting multiple parts of a target person and using the detection results of the multiple parts to control the autofocus (AF) function of an imaging device. Patent Document 2 discloses a technology for matching joint points for multiple people. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-48077 [Patent Document 2] JP 2020-194294 A [Non-patent literature]

[0005] [Non-Patent Document 1] Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields, Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh(2016) [Non-Patent Document 2] Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation., Ross Girshick et al.,2014 IEEE Conference on Computer Vision and Pattern Recognition [Non-Patent Document 3] SSD: Single Shot MultiBox Detector, Wei Liu et al., Computer Vision - ECCV 2016 [Non-Patent Document 4] You Only Look Once: Unified, Real-Time Object Detection, Joseph Redmon et al., 2016 IEEE Conference on Computer Vision and Pattern Recognition Summary of the Invention [Problem to be solved by the invention]

[0006] In order to detect the parts of a detection target using deep learning and perform matching processing of each part of the same target, ground truth information (GT) such as the position and size of each part is used in addition to the learning image of the detection target. In such a case, when training a neural network, in addition to the training image and the ground truth image, GT information that indicates the correspondence between parts is prepared and training is performed. On the other hand, in learning a neural network, in general matching methods as typified by Patent Document 2 and Non-Patent Document 1, matching is determined based on matching information of a straight line region between two points of the object to be determined. Therefore, in a situation where the object has a special shape different from the general shape of the object, such as a situation where a part of the object is curved (for example, a person's upper body is curved), the accuracy of the matching process of each part constituting the object may decrease.

[0007] In view of the above problems, an object of the present invention is to realize correspondence between multiple parts constituting an object in an image in a more suitable manner. [Means for solving the problem]

[0008] The image analysis device of the present invention is characterized by having an extraction means for extracting features of an object from an image in which the object is captured; a detection means for detecting multiple parts of the object from the features extracted by the extraction means; an angle estimation means for estimating, with each of the multiple parts detected by the detection means as a target part, an angle formed between a half line extending from the target part as a base point in the direction in which the object extends and a line extending in a reference direction in the image; and a correspondence means for matching two of the multiple parts of the object that are linked to each other based on the detection results of the multiple parts by the detection means and the estimation results of the angle by the angle estimation means. Effect of the Invention

[0009] According to the present invention, it is possible to realize, in a more preferable manner, correspondence between a plurality of parts constituting an object in an image. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of an imaging device. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an imaging device and an image analyzing device. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of an imaging device and an image analyzing device. [Figure 4] 10 is a flowchart showing an example of processing performed by the imaging device. [Diagram 5] FIG. 1 is a diagram showing an example of a network structure of a neural network. [Figure 6] FIG. 2 is a diagram illustrating an example of a functional configuration of a learning device. [Figure 7] 11 is a flowchart showing an example of a process of a learning device. [Figure 8] FIG. 2 is a diagram illustrating an example of a functional configuration of an imaging device and an image analyzing device. [Figure 9] FIG. 2 is a diagram illustrating an example of a functional configuration of a learning device. [Figure 10] FIG. 11 is a diagram showing an example of a method for calculating an association score. [Figure 11] FIG. 11 is a diagram for explaining an example of association processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0012] <Matching multiple parts that make up an object> First, in order to more clearly explain the characteristics of the technology according to an embodiment of the present disclosure, an overview of a process for detecting a plurality of parts constituting an object and associating the plurality of parts will be described. As described above, an example of a method for detecting an arbitrary object in an image is a method for detecting a plurality of partial regions of the object and inferring the correspondence between the detected partial regions. The association process makes it possible to detect the object with high accuracy, and also makes it possible to realize various functions by using information on the association of the plurality of partial regions.

[0013] For example, Patent Document 1 discloses a technology in which, when selecting an object to focus on using the AF function, multiple parts of the object are detected, and even if one of the multiple parts is lost, AF is continued by using information on the other parts. Furthermore, Patent Document 2 discloses an example of a technique for classifying the body parts of a plurality of people based on the distance when associating the joint points of the people.

[0014] Here, an example of a function that is realized by using the result of the association process will be described below. As a specific example, when a person's tracking process is performed, the head and torso of the person are associated with each other. Under such a premise, when the person's head is being tracked as a target, the head is lost due to obstruction by an obstacle, the person's unusual posture, etc. Even under such a situation, it is possible to continue tracking the target person by, for example, retargeting the torso that was associated with the head that was the target of the tracking process as the tracking target. Another example is a technology that detects body parts of a person as partial regions and estimates the posture of the person by using the detection results of the body parts. As a specific example, partial regions corresponding to body parts such as the head, neck, shoulders, elbows, waist, and knees are detected, and the partial regions corresponding to these body parts of the same person are associated with each other, so that the posture of the person can be estimated based on the positional relationship between the body parts.

[0015] As described above, in order to detect the parts of a detection target using deep learning and perform matching processing of each part of the same target, ground truth information (GT) such as the position and size of each part is used in addition to the learning image of the detection target. In such a case, when learning a neural network, in addition to the learning image and the ground truth image, learning is performed by preparing GT information that indicates the matching between parts. On the other hand, in learning a neural network, in general matching methods as typified by Patent Document 2 and Non-Patent Document 1, matching is determined based on matching information of a straight line region between two points of the object to be determined. Therefore, in a situation where the object has a special shape different from the general shape of the object, such as a situation where a part of the object is curved (for example, a person's upper body is curved), the accuracy of the matching process of each part constituting the object may decrease.

[0016] For example, Fig. 11 is a diagram for explaining an example of the matching process, and shows a schematic diagram of a situation in which the upper body part is curved due to a person taking a back flip posture. In addition, in the example shown in Fig. 11(a), since the upper body part of the person is curved, if a straight line is drawn between the centers of the head and the torso, the area corresponding to the straight line will be outside the area of ​​the human body to be detected, and the score related to the matching between the head and the torso will be lowered. Therefore, in the example shown in Fig. 11(a), it may be difficult to correctly match the head and torso of the target person.

[0017] As an example of a method for solving the above problem, there is a method of increasing the number of parts to be detected. By increasing the number of parts to be detected, it becomes possible to express parts that have been approximated by a straight line with multiple straight lines. As a result, even if the target object has a special shape, the position of the area used for determining the correspondence is less likely to deviate significantly from the object. For example, in the example shown in FIG. 11(b), a part to be detected is added between the center of the head and the torso, and even if the upper body part is curved, it becomes possible to perform correspondence between parts more accurately. On the other hand, as shown in FIG. 11(b), by increasing the parts to be detected, the amount of calculation for the correspondence process increases. Therefore, for example, in a situation where the process is performed by a device (e.g., an imaging device, etc.) that has restrictions on calculation power and calculation time, it may be difficult to actually realize the correspondence process as shown in FIG. 11(b).

[0018] In view of the above circumstances, this disclosure proposes an example of a technology for suppressing increases in costs related to calculations and time when training a neural network or performing inference using the neural network, and for realizing matching processing that is robust to special shapes.

[0019] <First embodiment> The image analysis device according to the first embodiment of the present disclosure will be described with a focus on an example in which the image analysis device is realized as an imaging device such as a so-called digital camera. In the following, objects with the same reference numerals in the drawings may operate in the same manner, and may not be described again. In addition, the configuration and processing described in this embodiment are merely examples, and do not limit the configuration and processing of the image analysis device according to this embodiment. In this embodiment, the area of ​​a plurality of parts of a person is detected as a frame from a moving image or a still image frame captured continuously, and the correspondence between each detected part is inferred, and the person is tracked using the detection frame and inference information. Note that, in this embodiment, the case where a person is the detection target is described, but the application target of the technology according to this embodiment is not limited. As a specific example, the technology according to this embodiment can be applied even when various objects other than a person, such as animals, birds, fish, and insects, are the detection target.

[0020] (Hardware configuration) An example of a schematic configuration for implementing the functions and processing of the imaging device according to this embodiment will be described with reference to Fig. 1. The imaging device 100 according to this embodiment has an image analysis device 110 that performs various image processing on a captured image to detect a target object (e.g., a person) from the image. The image analysis device 110 is implemented by a so-called neural network, and learns based on machine learning. The learning device 150 is a schematic device that performs learning of the image analysis device 110 (neural network learning), and executes, for example, various processes related to learning parameters for the image analysis device 110 to detect a target object from an image.

[0021] An example of the hardware configuration of the imaging device 100 and the image analyzing device 110 according to this embodiment will be described with reference to FIG. The arithmetic processing device 101 controls the operations of the imaging device 100 and the image analyzing device 110, and realizes various functions by executing programs stored in the storage device 102. The arithmetic processing device 101 may also control the operation of the learning device 150. The arithmetic processing device 101 may be realized by various processors such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The storage device 102 is a storage area for storing programs loaded based on the operation of the arithmetic processing device 101 and various data. The storage device 102 can be realized by a storage device such as a magnetic storage device or a semiconductor memory. In this embodiment, the arithmetic processing device 101 deploys and executes the programs stored in the storage device 102, thereby realizing various functions of the imaging device 100 and processing of the imaging device 100, which will be described later. The storage device 102 can also be used as a storage area for storing images to be processed by the imaging device 100 and various processing results. The input device 103 is an input interface that accepts various instructions from a user, and can be realized by, for example, a touch panel device, buttons, and the like. The output device 104 is an output interface that outputs various information to present the information to a user, and can be realized by, for example, a liquid crystal panel, an external monitor, or the like. 2 is merely an example and does not limit the hardware configuration of the imaging device 100 according to this embodiment. As a specific example, the imaging device 100 may include an I / O device for communicating between various devices. Examples of the I / O device include an input / output unit to which a memory card, a USB cable, or the like is connected, and a transmitting / receiving unit for transmitting and receiving various data via wired or wireless communication.

[0022] (Functional configuration) An example of the functional configuration of the imaging device 100 and image analysis device 110 according to this embodiment will be described with reference to Fig. 3. The imaging device 100 includes the image analysis device 110 and a tracking processing unit 380. The image analysis device 110 includes an image acquisition unit 310, a feature extraction unit 320, a detection region estimation unit 330, an angle estimation unit 340, an association likelihood estimation unit 350, an association determination region determination unit 360, and an association determination unit 370.

[0023] The image acquisition unit 310 acquires an image to be processed. For example, the image acquisition unit 310 may acquire a moving image captured in time series by an imaging unit (not shown) (in other words, a series of still images captured sequentially in time series) as the image to be processed. In this embodiment, for convenience, it is assumed that the image acquisition unit 310 acquires image data of a moving image in full HD (1920×1280 pixels) in real time (60 frames per second).

[0024] The feature extraction unit 320 performs various analysis processes on the image acquired by the image acquisition unit 310 to acquire a feature amount map in which feature amounts indicating features of objects in the image extracted from the image are mapped to positions in the image. In this embodiment, the feature extraction unit 320 acquires a center map, a size map, a correspondence likelihood map, and an angle map as the feature amount maps. The center map is data in which the position of the center of each of a series of objects (for example, detection targets or parts of the detection targets) detected in an image is mapped to a position in the image. A size map is data in which information indicating the size of each of the objects detected in an image is linked to the position of the object in the image (e.g., the position of the center of the object) and mapped (e.g., data linked to information indicating the size as pixel values). The angle map and the association likelihood map will be described in detail later. The center map and the size map are output in the number of objects to be detected. In addition, when the object to be detected is divided into a plurality of parts, the center map and the size map may be output for each part. In addition, the number of angle maps corresponds to the number of pairs of parts having a connection relationship among the series of parts of the detected object. The number of angle maps will be described in detail later together with the details of the angle maps. In addition, at least one association likelihood map is output.

[0025] The detection area estimation unit 330 estimates an area (hereinafter also referred to as a detection area) corresponding to the detection target in the target image based on the feature amount extracted by the feature extraction unit 320. In this embodiment, the detection area indicates the area of ​​the parts constituting the object to be detected. As a specific example, when the object to be detected is a human, the head, torso, hands, feet, etc. correspond to the parts of the object. The detection area estimation unit 330 estimates, as information related to the detection area, the image coordinate value of the center of a frame (e.g., a bounding box) that includes the target part in the image, the width of the frame, and the height of the frame. More specifically, the detection area estimation unit 330 calculates the peak position of the feature amount on the center map and estimates this position as the center position of the detection target. The detection area estimation unit 330 also obtains the value of the size map at the same position as the center position of the detection target and obtains this as the size of the detection target. Through the above processing, the detection area estimation unit 330 outputs information indicating the position and size of the detection target as information related to the detection area.

[0026] The angle estimation unit 340 estimates (infers) an angle indicating the direction in which a target object (part) that is linked to another object (part) exists (in other words, the direction in which the target object extends) at each coordinate in the image based on the features extracted by the feature extraction unit 320. Here, with reference to Fig. 11(c), the angle estimated by the angle estimation unit 340 will be described in more detail. Note that in the example shown in Fig. 11(c), the horizontal direction of the drawing corresponds to the bottom side of the target image, and the angle is estimated with the bottom side as the reference direction in the image. Specifically, the angle estimation unit 340 calculates an angle θ between a line parallel to the bottom side of the image and a half line 1103 extending from the head as a base point toward the torso connected to the head in the direction in which the person's body extends. 1101 The angle estimation unit 340 estimates an angle θ between a line parallel to the bottom side of the image and a half line 1104 that extends from the torso as a base point toward the head connected to the torso in the direction in which the person's body extends. 1102 In this way, the angle estimation unit 340 estimates an angle between a half line extending in a direction in which the body of the person to be detected extends, starting from the target part, and a line extending in a reference direction in the image, with each part of the body being a target part as a base point, and the half line extending in a reference direction in the image. Note that the angle estimation unit 340 does not necessarily need to hold information on the half lines such as the half lines 1103 and 1104 when inferring the angle, and can indirectly estimate information on the half lines by estimating the above angle. This process is performed by the association determination region determination unit 360. The above-mentioned angle map corresponds to data in which information indicating the above-mentioned angles estimated by the angle estimation unit 340 for each of a series of detection targets (e.g., body parts of a person) in an image is mapped to the positions (e.g., center positions) of the detection targets in the image. Due to such characteristics, angle maps may be generated as many times as there are pairs of objects that are linked to each other (e.g., pairs of parts that have a connection relationship with each other). As another example, an angle map may be generated separately for each pair of objects that are linked to each other. In this case, the number of angle maps is twice the number of pairs of objects that are linked to each other.

[0027] Based on the features extracted by the feature extraction unit 320, the correspondence likelihood estimation unit 350 calculates the likelihood (hereinafter also referred to as correspondence likelihood) that each pixel in the image is located in an area connecting two objects that are linked to each other (for example, two parts that are connected to each other). The aforementioned correspondence likelihood map corresponds to data in which the correspondence likelihood calculated for each pixel in an image is mapped to the position of the pixel in the image (for example, linked as a pixel value). For example, in the example shown in FIG. 11(c), the upper half of the person's body is in a bent state. In such a case, for example, the correspondence likelihood between the region corresponding to the head and torso, which are in a connected relationship with each other, and the part where the head and the torso are connected becomes higher. Therefore, in the correspondence likelihood map in the example shown in FIG. 11(c), the pixels in the region in the image corresponding to the bent upper half of the body will show a higher correspondence likelihood value than the other pixels. The correspondence determination area determination unit 360 determines the correspondence map area to be referred to in the detection area correspondence determination described later, based on the detection area information estimated by the detection area estimation unit 330 and the angle map estimated by the angle estimation unit 340. At this time, the correspondence determination area determination unit 360 determines the correspondence map area to be referred to for each combination of two objects (e.g., two parts) that are the targets of the detection area correspondence determination. Specifically, the correspondence map area is determined as an area on a line segment that extends a half line from the center coordinates of each detection area in the direction of the angle map value at the coordinates, for each of the two detection areas, and connects the intersections where the half lines extended from the center coordinates of the two points intersect with each center coordinate. Note that a situation in which the two half lines extended from the center coordinates of the two detection areas do not intersect may also be assumed. In this case, the extension is terminated when one of the extended ray segments passes the x or y coordinate of the center coordinate that is the starting point of the other ray segment, and the areas on the two lines formed as a result are determined as the correspondence area map.

[0028] Association determination section 370 performs association determination as to whether or not two target objects (e.g., two target body parts) are associated, based on the association likelihood of pixels in the association determination area determined by association likelihood estimation section 350 and association determination area determination section 360. In this way, association determination section 370 performs a determination as to whether or not the two objects are associated, for each combination of two objects (e.g., two target body parts) corresponding to the association determination area. Specifically, the association determination unit 370 first calculates an association score for the association determination area to be targeted. Here, an example of a method for calculating the association score will be described with reference to FIG. 10. The association determination unit 370 obtains the association likelihood of a point on the association determination area that is closest to each of a series of points obtained by equally dividing a series of points obtained by connecting two half-lines extended from the central coordinates of each of the two detection areas related to the definition of the association determination area into N (N is an integer) parts. For example, in the example shown in FIG. 10, the association likelihood of a point on the association determination area that is closest to each of the five points (division points) obtained by equally dividing a series of points obtained by connecting two half-lines into six parts is obtained. Then, the association determination unit 370 determines the average value of the association likelihood obtained for each of the series of points (for example, the five points illustrated in FIG. 10) as the association score of the association determination area to be targeted.

[0029] The tracking processing unit 380 executes a tracking process to track a target object (subject) in an image based on the detection area information estimated by the detection area estimation unit 330 and the correspondence relationship between the detection areas determined by the correspondence determination unit 370.

[0030] (process) With reference to FIG. 4, an example of a tracking process for detecting a region of a noteworthy subject from a captured moving image and tracking the subject will be described as an example of the process of the imaging device according to this embodiment.

[0031] In S401, the image acquisition unit 310 sequentially acquires a series of images captured in time series (e.g., still images corresponding to each frame of a moving image). The images acquired in S401 may be acquired as bitmap data expressed in RGB 8 bits, for example. In S402, the feature extraction unit 320 performs a predetermined analysis process on the image acquired in S401 to extract feature amounts indicating features of an object (detection target or a part of the detection target) in the image from the image. In this embodiment, the feature extraction unit 320 extracts the feature amounts from the target image by using a neural network that is constructed in advance based on machine learning and that receives an image as input and outputs feature amounts indicating features of an object in the image.

[0032] Here, referring to FIG. 5, an example of the network structure of the neural network used by the feature extraction unit 320 to extract features from an image will be described. The neural network illustrated in FIG. 5, which the feature extraction unit 320 uses to extract features from an image, is a neural network that performs a recognition task by repeating convolution processing and pooling processing. As such a neural network, for example, the neural networks disclosed in Non-Patent Document 2, Non-Patent Document 3, and Non-Patent Document 4 can be used. In this embodiment, an example in which features are extracted from an image using a neural network as illustrated in FIG. 5 will be described, but as long as the target features can be extracted from the image, the method is not particularly limited. As a specific example, the feature may be extracted from the target image by using a known feature extraction method such as SIFT (Scale-Invariant Feature Transform) or HOG (Histgrams of Oriented Gradients). 5 is composed of multiple convolution processes (511, 513, 515) and multiple pooling processes (512, 514), and extracts features from an input image 530 based on parameters representing weighting coefficients in the multiple convolution processes. The parameters representing weighting coefficients in the multiple convolution processes are set by learning, the details of which will be described later.

[0033] In the convolution layer, for example, a filter of size 3×3 is set for multiple channels for an input image or feature map, a convolution operation is performed centering on a pixel of interest, and multiple feature maps corresponding to the multiple channels are output. In the example shown in FIG. 5, for convolution process 1 (511), the filter size is 3×3 and there are 4 channels. For convolution process 2 (513), the filter size is 3×3 and there are 12 channels. For convolution process 3 (515), the filter size is 3×3 and there are 24 channels. In the pooling process, the feature map output from the convolution process is reduced. For example, when pooling is performed in a 2×2 range, the feature map is reduced to 1 / 4 times. As a pooling process method, for example, maximum value pooling, average value pooling, etc. can be used. In the example shown in FIG. 5, pooling is performed in a 2×2 range in both pooling process 1 (512) and pooling process 2 (514). Thus, in the example shown in FIG. 5, since there are two pooling processes in a 2×2 range, the resolution of the input image 530 is reduced to 1 / 16 times at the output layer stage. It should be noted that the network structure shown in Fig. 5 is merely an example, and does not limit the network structure of the neural network applied to the feature extraction unit 320. As a specific example, a multi-layer network structure may be applied, or the number of channels may be changed, rather than the example shown in Fig. 5. It should be noted that in this embodiment, it is assumed that the output layer 520 outputs one or more center maps, one or more size maps, and one or more angle maps, and one or more association likelihood maps.

[0034] 4 again. In S403, the detection area estimation unit 330 estimates the position, width, and height of the detection target or a frame (for example, a bounding box) that includes the detection target or a part of the detection target, based on the feature amount extracted by the feature extraction unit 320. In S404, the angle estimation unit 340 estimates an angle indicating the direction in which a target object (part) associated with another object (part) exists at each coordinate in the image based on the feature amount extracted by the feature extraction unit 320. Note that the angle here indicates the angle formed by a straight line parallel to the bottom of the image with the reference pixel as the base point and a half line extending in the direction in which the object associated with the pixel exists (in other words, the direction in which the target object extends). In S405, the association likelihood estimation unit 350 calculates the association likelihood for each pixel in the target image based on the feature extracted by the feature extraction unit 320. As described above, the association likelihood indicates the likelihood that the referenced pixel is present in an area connecting two objects that are linked to each other (for example, two parts that are connected to each other). The learning method of the feature extraction unit 320 (specifically, the learning method of the neural network used by the feature extraction unit 320 to extract feature amounts from an image) will be described in detail later. In S406, the correspondence determination area determination unit 360 determines a correspondence map area to be referenced for each combination of objects (parts) for which detection area correspondence determination is performed, based on the detection area estimated in S403 and the angle map estimated in S404. In S407, matching determination unit 370 performs matching determination as to whether or not the two target objects (parts) are matched, based on the matching likelihood of pixels within the matching determination area determined by matching likelihood estimation unit 350 and matching determination area determination unit 360.

[0035] In S408, a control unit (not shown) of the imaging device determines whether or not to execute tracking processing related to tracking of a subject in an image, based on a control signal according to a previous setting or an instruction from a user. When the control unit of the imaging device determines in S408 that the tracking process is to be executed, the process proceeds to S409. In S409, the tracking processing unit 380 starts the tracking process for tracking the subject in the image based on the information of the area detected by the detection area estimation unit 330 and the information according to the determination result of the correspondence between the areas by the correspondence determination unit 370. On the other hand, if the control unit of the imaging device determines in S408 that the tracking process is not to be executed, the process proceeds to S410. In this case, the process of S409 is skipped.

[0036] In S410, a control unit (not shown) of the imaging device determines whether or not to continue the operation of the imaging device. As a specific example, the control unit of the imaging device may determine not to continue the operation of the imaging device when an instruction related to the presentation of capturing an image (e.g., a moving image) is received from a user. If the control unit of the imaging device determines in S410 that the operation of the imaging device is to be continued, the process proceeds to S401. In this case, the processes from S401 onwards are executed again. Then, if the control unit of the imaging device determines in S410 not to continue the operation of the imaging device, the control unit ends the series of processes shown in FIG.

[0037] (How to learn) A learning method of the neural network applied to the image analysis device according to the present embodiment will be described below. First, with reference to FIG. 6, an example of the functional configuration of a learning device that performs learning of the neural network applied to the image analysis device according to the present embodiment will be described. The learning device 150 includes a learning data storage unit 601, a learning data acquisition unit 602, an image acquisition unit 603, a feature extraction unit 604, a detection area estimation unit 605, an angle estimation unit 606, and a matching likelihood estimation unit 607. The learning device 150 also includes an area error calculation unit 608, an angle error calculation unit 609, a matching likelihood error calculation unit 610, and a parameter learning unit 611. Note that the feature extraction unit 604, the detection area estimation unit 605, the angle estimation unit 606, and the matching likelihood estimation unit 607 are substantially similar to the feature extraction unit 320, the detection area estimation unit 330, the angle estimation unit 340, and the matching likelihood estimation unit 350 shown in FIG. 3, and therefore detailed description thereof will be omitted.

[0038] The learning data storage unit 601 stores and holds learning data used by the learning device 150 to learn the neural network. The learning data consists of a set of a learning image, area information of a detection target or a part of a detection target in the learning image to be analyzed, the angle information described above, and the association likelihood information described above. The learning data storage unit 601 stores a sufficient amount of learning data for the learning device 150 to learn the image analyzing device 110 (in other words, learn the neural network). The learning data acquisition unit 602 acquires the learning data stored in the learning data storage unit 601 . The image acquiring unit 603 acquires, from the learning data acquiring unit 602 , the learning images included in the learning data acquired by the learning data acquiring unit 602 .

[0039] The area error calculation unit 608 calculates the error (hereinafter also referred to as area error) between the estimated value of the detection area acquired by the detection area estimation unit 605 and the area information included in the learning data. Specifically, the area information is information on a frame (e.g., a bounding box) that represents the area of ​​the detection target in the image, and is composed of center coordinates and size (e.g., width and height). The angle error calculation unit 609 calculates the error (hereinafter also referred to as angle error) between the estimated value of the angle acquired by the angle estimation unit 606 and the angle information included in the learning data. In addition, the area subject to the calculation of the angle error at this time is limited to the area of ​​the entire image that is defined as the area where the detection target exists by the area information included in the learning data. The matching likelihood error calculation unit 610 calculates the error (hereinafter also referred to as the matching likelihood error) between the estimated result of the likelihood map obtained by the matching likelihood estimation unit 607 and the matching likelihood map calculated from the matching likelihood information contained in the learning data. The parameter learning unit 611 updates the parameters of the feature extraction unit 604, the detection area estimation unit 605, the angle estimation unit 606, and the association likelihood estimation unit 607 so as to reduce the area error, the angle error, and the association likelihood error.

[0040] Next, an example of processing by learning device 150 will be described with reference to FIG. 7, focusing on processing related to learning of a neural network applied to image analyzing device 110.

[0041] In S701, the learning data acquisition unit 602 acquires learning data stored in the learning data storage unit 601. Furthermore, the image acquisition unit 603 acquires, from the learning data acquisition unit 602, learning images included in the learning data acquired by the learning data acquisition unit 602. As described above, the learning data includes learning images and correct answer information for each learning image. The correct answer information includes area information indicating the area of ​​the detection target or part of the detection target captured in the learning image, information on the angle calculated for the detection target or part of the detection target, and information on the likelihood of association corresponding to pixels in the area (for example, an association likelihood map). It is assumed that the learning data has been prepared in advance.

[0042] In S702, the feature extraction unit 604 performs a predetermined analysis process on the training image acquired by the image acquisition unit 603, thereby extracting features indicating the characteristics of an object (detection target or a part of the detection target) in the training image from the training image. In S703, the detection area estimation unit 605 estimates a detection area candidate of the detection target or a part of the detection target based on the feature amount extracted by the feature extraction unit 604. In S704, the angle estimation unit 606 estimates an angle indicating the direction in which a target object (part) associated with another object (part) exists at each coordinate in the learning image based on the feature amount extracted by the feature extraction unit 604. As described above, the angle indicates the angle formed by a straight line parallel to the bottom of the image with the reference pixel as the base point and a half line extending in the direction in which an object associated with the pixel exists (in other words, the direction in which the target object extends). In S705, the association likelihood estimation unit 607 calculates the association likelihood for each pixel in the training image based on the feature amount extracted by the feature extraction unit 604. As described above, the association likelihood indicates the likelihood that the referenced pixel exists in an area connecting two objects that are linked to each other (for example, two parts that are connected to each other).

[0043] In S706, the area error calculation unit 608 calculates the error (area error) between the estimated value of the detection area acquired by the detection area estimation unit 605 and the area information included in the learning data. Note that, as a method for calculating the area error, for example, the method disclosed in Non-Patent Document 2 can be applied. As a specific example, the area error may be found by adding up the deviation amounts with respect to the center coordinates and sizes of the detection frame estimated by the detection area estimation unit 605 and the correct answer frame including the center coordinates closest to the detection frame. In S707, the angle error calculation unit 609 calculates the error (angle error) between the estimated value of the angle acquired by the angle estimation unit 606 and the angle information included in the learning data. In addition, the area subject to the calculation of the angle error at this time is limited to the area of ​​the entire image that is defined as the area where the detection target exists by the area information included in the learning data. In S708, the matching likelihood error calculation unit 610 calculates the error (likelihood error) between the estimation result of the likelihood map obtained by the matching likelihood estimation unit 607 and the matching likelihood map calculated from the matching likelihood information included in the learning data. In S709, the parameter learning unit 611 updates the parameters so that the area error calculated in S706, the angle error calculated in S707, and the likelihood error calculated in S708 become smaller. The parameters to be updated may be, for example, weight coefficients of the convolution layer of the neural network. The parameter update is performed, for example, by calculating a loss value by taking the sum of the area error calculated in S706, the angle error calculated in S707, and the likelihood error calculated in S708, and based on the error backpropagation method to which the loss value is applied. Through the series of processes from S701 to S709 described above, learning of the neural network applied to the image analyzing device 110 using the learning data acquired in S701 as the target is completed.

[0044] In S710, the learning device 150 determines whether or not to end the learning of the neural network applied to the image analyzing device 110. If the learning device 150 determines in S710 not to end the learning of the neural network, the process proceeds to S701. In this case, the processes from S701 onwards (for example, a series of processes from S701 to S709 for other learning data) are executed again. Then, when learning device 150 determines in S710 to end the learning of the neural network, it ends the series of processes shown in FIG.

[0045] As described above, in the method according to the present embodiment, unlike the conventional method, the association likelihood information of the straight line region connecting the two regions of interest is not used, but the above-mentioned angle information is added as a new value, and the association determination region is calculated taking the angle into consideration. In addition, in the method according to the present embodiment, by using the association likelihood in the association determination region, it is possible to suppress an increase in the cost of learning the neural network, and to determine between regions that are linked to each other even in a situation where the target object has a special shape.

[0046] (Modification 1 of the first embodiment) In the first embodiment described above, as described with reference to Fig. 10, the association score for the target association determination region is calculated based on the association likelihood of each of five points obtained by dividing a series of lines connecting two half-lines into six equal parts. Meanwhile, the number of points from which the association likelihood is obtained when calculating the association score is not necessarily limited to five points as in the above example, and may be changed to, for example, four points or six points. Furthermore, the method of determining these points is not limited to the method of equally dividing a series of lines connecting two half-lines, and, for example, each point may be determined so that the lengths of the line segments divided by each point are biased.

[0047] (Modification 2 of the first embodiment) In the first embodiment described above, an example in which the center map, size map, matching likelihood map, and angle map are used in the matching process has been described, but it is also possible to realize a substantially similar matching process without using the size map. In this case, the detection target output by the detection region estimation unit 330 can be treated as information on the coordinates of the center, rather than as information on a frame (bounding box).

[0048] <Second embodiment> A second embodiment of the present disclosure will be described below. In the above-mentioned first embodiment, an example of a case where a correspondence score between two objects of interest (e.g., a detection target or a part of the detection target) is calculated by learning and inferring a correspondence likelihood map in addition to the angle of each detected part is described. In contrast, in this embodiment, an example of a case where a correspondence score between two detected targets is calculated by using the angle information without using a correspondence likelihood map is described. Note that, for convenience, in this embodiment, the configuration and processing will be described assuming that correspondence is performed between two parts of a detection target. In this embodiment, the correspondence score of the two parts is calculated based on the distance between the intersection of a half line extending from a reference point (e.g., a center position) in the area corresponding to each of the two parts in the direction of the angle estimated for each object, and the midpoint of the line connecting the reference points. The correspondence score is calculated so that the closer the distance is, the higher the likelihood is (i.e., the more it indicates that the two target parts are linked to each other), and the farther the distance is, the lower the likelihood is, and the lower the score is. Note that the hardware configurations of the imaging device 100 and image analysis device 110 according to this embodiment are substantially similar to those of the imaging device 100 and image analysis device 110 according to the first embodiment described with reference to FIG. 2, and therefore detailed description thereof will be omitted.

[0049] An example of the functional configuration of the imaging device 100 and image analysis device 110 according to this embodiment will be described with reference to Fig. 8. As shown in Fig. 8, the imaging device 100 includes an image analysis device 110 and a tracking processing unit 880. The image analysis device 110 includes an image acquisition unit 810, a feature extraction unit 820, a detection region estimation unit 830, an angle estimation unit 840, and an association determination unit 870. Note that the components labeled with reference numerals 810 to 840 and 880 in Fig. 8 are substantially similar to the components labeled with reference numerals 310 to 340 and 380 in Fig. 3, and therefore detailed description thereof will be omitted. Association determination section 870 performs association determination using the angle estimated by angle estimation section 840 for a series of areas corresponding to the detection target or part of the detection target estimated by detection area estimation section 830 .

[0050] An example of the functional configuration of a learning device 150 according to this embodiment will be described with reference to Fig. 9. The learning device 150 includes a learning data storage unit 901, a learning data acquisition unit 902, an image acquisition unit 903, a feature extraction unit 904, a detection region estimation unit 905, and an angle estimation unit 906. The learning device 150 also includes a region error calculation unit 908, an angle error calculation unit 909, and a parameter learning unit 911. The components denoted by reference numerals 901 to 906 and 908 to 909 in Fig. 9 are substantially similar to the components denoted by reference numerals 601 to 606 and 608 to 609 in Fig. 6, and therefore detailed description thereof will be omitted. The parameter learning unit 611 updates the parameters of the feature extraction unit 904, the detection area estimation unit 905, and the angle estimation unit 906 so as to reduce the area error and the angle error.

[0051] As described above, the method according to this embodiment differs from the method according to the first embodiment in that learning and inference of the association likelihood map are not performed. With this configuration, the method of calculating the association score between two parts becomes simpler than the method of calculating the association score in the first embodiment. Therefore, according to the method according to this embodiment, it is expected to have an effect of further reducing the cost of learning the neural network and the cost of inference using the neural network, compared to the first embodiment.

[0052] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0053] The disclosure of the present embodiment also includes the following configurations, methods, and programs. (Configuration 1) An image analyzing device comprising: an extraction means for extracting features of an object from an image in which the object is captured; a detection means for detecting multiple parts of the object from the features extracted by the extraction means; an angle estimation means for estimating, with each of the multiple parts detected by the detection means as a target part, an angle formed between a half line extending in a direction in which the object extends from the target part as a base point and a line extending in a reference direction in the image; and a correspondence means for matching two of the multiple parts of the object that are linked to each other based on the detection results of the multiple parts by the detection means and the estimation results of the angle by the angle estimation means. (Configuration 2) The image analyzing device according to configuration 1, wherein the extraction means is realized by a neural network trained to receive the image as input and output features of the object in the image. (Configuration 3) The image analyzing device according to configuration 1 or 2, further comprising: a likelihood estimation means for estimating the likelihood that each pixel in the image is present in an area connecting two parts that are linked to each other based on the feature amount extracted by the extraction means; and a determination means for determining an area corresponding to a combination of two parts that is to be subject to correspondence determination by the correspondence means, based on the multiple parts detected by the detection means and the angle estimated for each of the multiple parts by the angle estimation means, wherein the correspondence means determines, for each area corresponding to the combination of two parts determined by the determination means, whether or not to perform correspondence between the two parts based on the likelihood estimated for each pixel in the area by the likelihood estimation means. (Configuration 4) The image analysis device described in Configuration 3, characterized in that the matching means determines whether or not to match the two parts for each region corresponding to the combination of the two parts determined by the determination means, based on the likelihood estimated for each of a plurality of pixels located on a series of line segments formed by half lines extended from a reference point in the region corresponding to each of the two parts in a direction corresponding to the angle estimated for each of the parts. (Configuration 5) The image analysis device described in configuration 1 or 2, characterized in that the matching means determines whether or not to match the two parts for each combination of two parts among the multiple parts detected by the detection means, based on a matching score calculated based on the distance between the intersection of a half line extended from a reference point in an area corresponding to each of the two parts in a direction corresponding to the angle estimated for each of the parts, and the midpoint of a line connecting the reference points corresponding to each of the two parts. (Configuration 6) The image analysis device described in Configuration 5, characterized in that the correspondence score is calculated so that the shorter the distance between the intersection and the midpoint, the higher the value that indicates that the two target parts are in a linked relationship with each other. (Method 1) A control method for an image analyzing device, comprising: an extraction step of extracting features of an object from an image in which the object is captured; a detection step of detecting multiple parts of the object from the features extracted in the extraction step; an angle estimation step of estimating, with each of the multiple parts detected in the detection step as a target part, an angle formed between a half line extending in a direction in which the object extends from the target part as a base point and a line extending in a reference direction in the image; and a matching means for matching two parts of the object that are linked to each other based on the detection results of the multiple parts in the detection step and the estimation results of the angle in the angle estimation step. 13. A method for controlling an image analysis apparatus, comprising: (Program 1) A program for causing a computer to function as an image analysis device, comprising: an extraction means for extracting features of an object from an image of the object; a detection means for detecting multiple parts of the object from the features extracted by the extraction means; an angle estimation means for estimating, with each of the multiple parts detected by the detection means as a target part, an angle formed between a half line extending in the direction in which the object extends from the target part as a base point and a line extending in a reference direction in the image; and a correspondence means for matching two of the multiple parts of the object that are linked to each other based on the detection results of the multiple parts by the detection means and the estimation results of the angle by the angle estimation means. [Explanation of symbols]

[0054] 100 Imaging device 110 Image analysis device 150 Learning Device 320 Feature Extraction Unit 330 Detection area estimation unit 340 Angle estimation part 370 Matching Determination Unit

Claims

1. An extraction means for extracting a feature amount of an object from an image of the object; a detection means for detecting a plurality of parts of the object from the feature amount extracted by the extraction means; An angle estimation means for estimating an angle between a half line extending in a direction in which the object extends, starting from the target portion as a base point, and a line extending in a reference direction in the image, with each of the plurality of portions detected by the detection means as a target portion; a matching means for matching two parts of the object that are linked to each other based on a detection result of the plurality of parts by the detection means and a result of estimating the angle by the angle estimation means; An image analysis device comprising:

2. 2. The image analyzing device according to claim 1, wherein the extraction means is realized by a neural network trained to receive the image as an input and output a feature amount of the object in the image.

3. a likelihood estimation means for estimating a likelihood that each pixel in the image is present in a region connecting the two parts that are linked to each other, based on the feature amount extracted by the extraction means; a determination means for determining an area corresponding to a combination of two parts to be referenced for each combination of two parts that is to be subjected to the correspondence determination by the correspondence determination means, based on the plurality of parts detected by the detection means and the angle estimated for each of the plurality of parts by the angle estimation means; having The association means determines, for each region corresponding to the combination of the two regions determined by the determination means, whether or not to associate the two regions based on the likelihood estimated for each pixel in the region by the likelihood estimation means.

2. The image analysis device according to claim 1 .

4. The image analysis device according to claim 3, characterized in that the matching means determines whether or not to match the two parts for each region corresponding to the combination of the two parts determined by the determination means, based on the likelihood estimated for each of a plurality of pixels located on a series of line segments formed by half lines extended from a reference point in each region corresponding to the two parts in a direction corresponding to the angle estimated for each of the parts.

5. The association means comprises: an intersection point of a half line extending from a reference point in an area corresponding to each of the two parts in a direction corresponding to the angle estimated for each part for each combination of two parts out of the plurality of parts detected by the detection means; A midpoint of a line connecting the reference points corresponding to the two parts; Based on the correspondence score calculated based on the distance between the two parts, it is determined whether or not to perform correspondence between the two parts.

2. The image analysis device according to claim 1 .

6. 6. The image analysis device according to claim 5, wherein the correspondence score is calculated so that the shorter the distance between the intersection and the midpoint, the higher the value that indicates that the two target parts are in a linked relationship with each other.

7. A method for controlling an image analysis apparatus, comprising: An extraction step of extracting a feature amount of an object from an image of the object; a detection step of detecting a plurality of parts of the object from the feature amount extracted in the extraction step; an angle estimation step of estimating an angle between a half line extending in a direction in which the object extends, starting from the target portion as a base point, and a line extending in a reference direction in the image, with each of the plurality of portions detected in the detection step as a target portion; a matching means for matching two parts of the object that are linked to each other based on a result of detecting the parts in the detection step and a result of estimating the angle in the angle estimation step; 13. A method for controlling an image analysis apparatus, comprising:

8. Computer, An extraction means for extracting a feature amount of an object from an image of the object; a detection means for detecting a plurality of parts of the object from the feature amount extracted by the extraction means; An angle estimation means for estimating an angle between a half line extending in a direction in which the object extends, starting from the target portion as a base point, and a line extending in a reference direction in the image, with each of the plurality of portions detected by the detection means as a target portion; a matching means for matching two parts of the object that are linked to each other based on a detection result of the plurality of parts by the detection means and a result of estimating the angle by the angle estimation means; A program for causing the image analyzer to function as an image analyzer, comprising:

Images