Image processing apparatus, and image processing method

Abstract

To provide a technique to more accurately specify a collection of objects that are seemingly related to each other.SOLUTION: An image processing apparatus, on the basis of a first collection of objects detected from a first frame, and a second collection corresponding to the first collection, of collections of objects detected from a second frame that is a frame earlier than the first frame, accumulates and adds values according to the degree of a change from the second collection to the first collection. The image processing apparatus performs output based on a result of the accumulation and addition.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to image processing technology. [Background technology]

[0002] In recent years, devices have been proposed that analyze crowd behavior in a shooting area based on images captured by a camera, etc. It is expected that analyzing crowd behavior will improve the effectiveness of marketing to people who make up crowds in commercial facilities, event venues, theme parks, etc.

[0003] People who make up a crowd may form groups of multiple people. These groups may include groups of unrelated people who happen to be there by chance, and groups of people who are related to each other, such as family members, friends, or coworkers.

[0004] A group of unrelated people is simply a group of people who are close to each other, and it is thought that events such as communication between people within the group do not occur. Therefore, it is thought that there is no essential difference in the behavior of each person, whether they are in a group or not.

[0005] On the other hand, in a group of people who are related to each other, the behavior of each person is likely to be significantly influenced by various phenomena such as communication within the group, which would not occur if the people were simply in close proximity to each other.

[0006] Therefore, it is believed that by identifying groups of people who are related to each other from among the people who make up a crowd, it is possible to improve the accuracy of crowd behavior analysis, which is expected to lead to more effective marketing, etc.

[0007] Patent document 1 discloses a device that determines whether people in an input image are intimate with each other, i.e., whether the people in the image are close to each other, based on the frequency with which they appear together in an image and the distance between them. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2012-79354 A Summary of the Invention [Problem to be solved by the invention]

[0009] When determining whether a group of people is related to each other, there is a problem of erroneous determination due to a group of unrelated people who simply exist nearby each other. In Patent Document 1, the determination is made under the condition that the more frequently people appear together, the closer the distance between the people is, but this method may erroneously determine that a group of unrelated people who simply happen to exist nearby each other is a group of people who are related to each other.

[0010] In particular, in a situation where people move relatively freely in a facility where many people gather, there is a strong tendency for unrelated people who happen to be there by chance to be close to each other, which increases the possibility of erroneously determining that they are a group of people who are related to each other. The present invention provides a technology for identifying a group of objects that are thought to be related to each other with a high degree of accuracy. [Means for solving the problem]

[0011] One aspect of the present invention is characterized in that it comprises a calculation means for accumulating a value corresponding to the degree of change from a first set of objects detected from a first frame and a second set of objects corresponding to the first set among sets of objects detected from a second frame that is an earlier frame than the first frame, and an output means for outputting an output based on the result of the cumulative addition. Effect of the Invention

[0012] According to the present invention, a set of objects that are thought to be related to each other can be identified with a high degree of accuracy. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing an example of the hardware configuration of an image processing device 10. [Diagram 2] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing device 10. [Diagram 3] 4 is a flowchart showing the operation of the image processing device 10. [Figure 4] 4 is a diagram showing an example of the operation of a position estimation unit 202. [Diagram 5] FIG. 1 is a diagram showing an example of the configuration of a neural network for estimating the position of a person in an input image. [Figure 6] FIG. 13 is a diagram showing an example of a cluster and a sub-cluster. [Figure 7] 13 is a diagram showing an example of a determination process performed by a reassembly determination unit 205. FIG. [Figure 8] FIG. 11 is a diagram showing an example of a screen display. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0015] [First embodiment] First, a hardware configuration example of an image processing device 10 according to the present embodiment will be described with reference to the block diagram of Fig. 1. The image processing device 10 according to the present embodiment can be applied to computer devices such as PCs (personal computers), smartphones, and tablet terminal devices.

[0016] The control device 11 has a processor such as a CPU, a volatile memory such as a RAM, and the processor executes various processes using computer programs and data stored in the volatile memory. As a result, the control device 11 controls the operation of the entire image processing device 10, and executes or controls various processes described as processes performed by the image processing device 10.

[0017] The storage device 12 is a large-capacity non-volatile memory such as a hard disk drive device, etc. The storage device 12 stores an OS (operating system), computer programs and data for causing the control device 11 to execute or control various processes described as processes performed by the image processing device 10, and the like.

[0018] The computer programs and data stored in the storage device 12 are loaded into a volatile memory of the control device 11 as appropriate under the control of the control device 11, and are processed by a processor of the control device 11. The arithmetic device 13 is a processing circuit for arithmetic operations such as a GPU, and is capable of executing various types of arithmetic operations under the control of the control device 11.

[0019] The input device 14 is a user interface such as a keyboard, a mouse, a touch panel screen, and buttons, and the user can input various information and instructions to the image processing device 10 by operating it.

[0020] The output device 15 is a display device having a liquid crystal screen or a touch panel screen, and can display the processing results by the control device 11 as images, characters, etc. The output device 15 may be a projection device such as a projector that projects images and characters.

[0021] The I / F device 16 is an interface device for performing data communication with an external device. For example, a wired interface such as a universal serial bus, Ethernet, or optical cable, or a wireless interface such as Wi-Fi or Bluetooth can be applied to the I / F device 16.

[0022] The control device 11, the storage device 12, the arithmetic device 13, the input device 14, the output device 15, and the I / F device 16 are all connected to a system bus 17. Note that the configuration shown in Fig. 1 is an example of a hardware configuration of a computer device capable of executing the various processes described below, and can be modified / altered as appropriate.

[0023] An example of the functional configuration of the image processing device 10 is shown in the block diagram of FIG. 2. In this embodiment, a case will be described in which all of the functional units shown in FIG. 2 are implemented by computer programs. In this embodiment, the functional units shown in FIG. 2 will be described as the subject of processing, but in reality, the functions of the functional units are realized by a processor of the control device 11 executing a computer program corresponding to the functional units. Note that the arithmetic device 13 may execute part or all of the computer program corresponding to the functional units shown in FIG. 2. Also, one or more of the functional units shown in FIG. 2 may be implemented by hardware.

[0024] The operation of the image processing device 10 according to this embodiment will be described with reference to the flowchart of FIG. 3. In step S101, the image acquisition unit 201 acquires a frame to be processed as an input image. For example, when an imaging device capable of capturing moving images is connected to the I / F device 16, the image acquisition unit 201 acquires each frame of a moving image captured by the imaging device as an input image. For example, when an imaging device that captures still images periodically or irregularly is connected to the I / F device 16, the image acquisition unit 201 acquires a still image (frame) captured by the imaging device as an input image. The image acquisition unit 201 may acquire a frame from one imaging device, or may acquire a frame from each of a plurality of imaging devices. The plurality of imaging devices may be provided so as to capture different imaging areas (two or more imaging areas may partially overlap).

[0025] It should be noted that the image acquisition unit 201 is not limited to acquiring frames from an imaging device, but may acquire frames from a device such as a server connected to the I / F device 16, or may acquire frames stored in the storage device 12.

[0026] That is, the input image is an image obtained in various forms, such as streaming, a moving image file, a series of image files stored frame by frame, moving images or still images stored on a medium, and the like.

[0027] In step S102, the position estimation unit 202 estimates (detects) the position of an object in the input image acquired in step S101. In this embodiment, a person is taken as an example of the object, but the object is not limited to a person and may be, for example, a vehicle such as a bicycle or a motorbike, a car or a truck, or an animal such as livestock. In addition, there are various techniques for estimating the position of an object in an input image, and the position estimation unit 202 may use any of these techniques.

[0028] As a method for estimating the position of a person in an input image, for example, there is a method of obtaining a density distribution expressing the position where a person is estimated to exist in the input image. According to such a method, for example, as shown in Fig. 4(a), for an input image 401 including people 402a, 402b, and 402c, the head positions of the people are set as "positions where people are estimated to exist", and a map (density map) 403 including density distributions 404a, 404b, and 404c of the head positions of the people 402a, 402b, and 402c, respectively, is obtained.

[0029] The density distribution may be, for example, a uniform distribution according to the size of a person, or an arbitrary distribution having a maximum at a position where a person is likely to exist. An example of the latter is a Gaussian distribution. When the heads of people overlap in the input image, the position estimation unit 202 obtains a density distribution as a distribution obtained by adding up the density distributions corresponding to each person. The density distribution may be normalized so that the sum of the density distributions for one person is 1.

[0030] As a method for estimating the position of a person in an input image, for example, there is a method for obtaining the coordinates of a position where a person is estimated to exist in the input image. According to such a method, for example, as shown in Fig. 4(b), for an input image 401, the head position of a person is set as the "position where a person is estimated to exist", and a map (coordinate map) 405 including coordinates 406a, 406b, and 406c of the head positions of people 402a, 402b, and 402c, respectively, is obtained.

[0031] The coordinates 406a, 406b, and 406c in the map 405 can be obtained as, for example, the coordinates of the respective maximum points of the density distributions 404a, 404b, and 404c in the map 403 shown in FIG.

[0032] For example, for the lattice points constituting each element of the density distribution, eight adjacent lattice points surrounding the lattice point of interest are searched for, and if the density values ​​of all the adjacent lattice points are lower than the density value of the lattice point of interest, then this lattice point of interest is a maximum point, and the coordinates of this lattice point of interest are output.

[0033] As a method for estimating the position of a person in an input image, for example, there is a method of obtaining a frame including a position where a person is estimated to exist in the input image as a detection frame. According to such a method, for example, as shown in Fig. 4(c), for an input image 401, the head position of the person is set as the "position where a person is estimated to exist", and a map 407 including detection frames 408a, 408b, and 408c including the head positions of people 402a, 402b, and 402c, respectively, is obtained.

[0034] The detection frame is not limited to a frame including the head position, and may be, for example, a frame including the head and shoulders, a frame including the head and torso, a frame including the torso, a frame including the entire human body, etc. Furthermore, by using the coordinates of a representative point calculated from the detection frame, for example, the coordinates of the center of the detection frame, map 407 can be converted into a map such as map 405.

[0035] Also, examples of methods for estimating the position of a person in an input image include machine learning, neural networks, etc. Examples of machine learning methods include decision trees, random forests, LightGBM, etc. Examples of neural networks include convolutional neural networks, deconvolutional neural networks, and autoencoders that combine both of them.

[0036] An example of the configuration of a neural network for estimating the position of a person in an input image is shown in Fig. 5. In Fig. 5, Conv stands for convolution layer, Pooling stands for pooling layer, Upsample stands for upsampling layer, Concat stands for concatenation layer, and Output stands for output layer.

[0037] 5, the position estimation unit 202 inputs an input image 502 acquired by the image acquisition unit 201 to a neural network 501. When the input image 502 is input to Conv101 in the neural network 501, calculation processing is performed according to the flow in FIG. 5, and a map 503 indicating the position of a person in the input image is output from OutPut101 of the neural network 501.

[0038] The method of Fig. 5 can be applied to all of the following cases: when map 503 is map 403 in Fig. 4(a), when map 405 in Fig. 4(b), and when map 407 in Fig. 4(c). If neural network 501 is trained using a density map as training data, it becomes a neural network that outputs a density map.

[0039] If neural network 501 is trained using a coordinate map as training data, a neural network that outputs a coordinate map can be obtained. Also, if neural network 501 is trained using a detection frame as training data, a neural network that outputs a detection frame can be obtained.

[0040] An example of a neural network learning method is to use a loss function that determines the difference between the output of the neural network and the training data. Examples of the loss function include the mean squared error, the mean absolute error, and the binary cross entropy. The neural network can be trained while modifying the weights so that the value of the loss function becomes smaller, that is, the difference between the output of the neural network and the training data becomes smaller. An example of the learning method is the error backpropagation method.

[0041] Returning to FIG. 3, next, in step S103, the detection unit 203 detects a cluster of people in the input image based on the positions of the people estimated in step S103. A "cluster of people" is a collection of people located at a certain distance (reference distance) or less from each other. For example, in FIG. 6(a), among the people represented by black circles, a collection of people surrounded by a dashed line is a "cluster of people". There are various techniques for detecting clusters of people in an input image, and the detection unit 203 may use any of these techniques.

[0042] For example, the detection unit 203 may use hierarchical clustering to detect clusters of people in the input image. In hierarchical clustering, the distance between all people detected from the input image is calculated, and people close to each other are grouped together in order to perform clustering. This makes it possible to obtain clusters such as those shown in FIG. 6(a).

[0043] The reference distance may be obtained by actual measurement using an image in which clusters can be distinguished, or may be the close distance or individual distance in the concept of personal space. Another example of the reference distance is the social distance in the concept of personal space, that is, the distance known as the social distance. Here, the individual distance is longer than the close distance. Also, the social distance is longer than the close distance and the individual distance.

[0044] It should be noted that a cluster may have sub-clusters within it. For example, in FIG. 6(b), within the cluster of five people represented by solid ellipses, there are two sub-clusters represented by dotted ellipses and one person who does not form a sub-cluster. Specific examples of the sub-clusters in FIG. 6(b) include acquaintances, friends, parents and children, and married couples within a group that is acting as a group. Such sub-clusters are highly likely to behave uniquely differently from higher-level clusters, such as individuals that do not form sub-clusters or groups that are acting as a group, so by detecting sub-clusters, the accuracy of behavior analysis can be improved.

[0045] A method for determining sub-clusters such as that shown in Figure 6(b) is, for example, to define a group of people who are located within a first reference distance from each other as a cluster, and for each cluster, to define a group of people who are located within a second reference distance from each other as a sub-cluster.

[0046] When determining a sub-cluster, the first and second reference distances may be, for example, set as the social distance in the concept of personal space, and the second reference distance as the close distance or individual distance in the concept of personal space.

[0047] When hierarchical clustering is used for clustering, people with short distances between them are grouped together in order, so that sub-clusters are determined until the distance between people exceeds a second reference distance, and clusters are determined until the distance between people exceeds a first reference distance.

[0048] In step S104, the history holding unit 206 acquires the history (total history) stored in the storage device 12. The total history includes the history of people detected from past frames (person history) and the history of clusters / sub-clusters detected from past frames (cluster / sub-cluster history).

[0049] The person history includes an identifier of a person detected from a past frame and a feature amount of the person. For example, an image of the person, statistics of pixel values ​​of the image of the person (e.g., a histogram of pixel values), an image feature vector of the image of the person, etc. can be applied to the feature amount of the person. Various methods can be applied to acquire the image feature vector of the image of the person, and for example, the image feature vector of the image of the person may be acquired using machine learning, neural network, etc. Examples of the machine learning method include a decision tree, a random forest, and LightGBM. Examples of the neural network include a convolutional neural network, a deconvolutional neural network, and an autoencoder and a transformer that are formed by connecting both of them.

[0050] In addition, the person history may include, for each frame, time information of the frame (e.g., the frame number and the time corresponding to the frame), an identifier of the person detected from the frame, and features of the person detected from the frame.

[0051] The cluster / subcluster history includes the identifiers of the clusters / subclusters detected from past frames, and the identifiers (set of identifiers) of the people included in the clusters / subclusters. Note that the cluster / subcluster history may include, for each frame, time information of the frame (for example, the frame number or the time corresponding to the frame), the identifiers of the clusters / subclusters detected from the frame, and the identifiers (set of identifiers) of the people included in the clusters / subclusters detected from the frame.

[0052] Then, the identifier assigning unit 204 assigns an identifier to each person in the input image and each cluster / sub-cluster in the input image, using the comprehensive history acquired by the history holding unit 206.

[0053] For example, for each person in the input image, the identifier assigning unit 204 identifies from the person history a feature amount having a similarity to the feature amount of the person acquired when estimating the position of the person in step S102 that is equal to or greater than a threshold, and assigns an identifier corresponding to the identified feature amount to the person. Note that if the identifier assigning unit 204 cannot identify from the person history a feature amount having a similarity to the feature amount of the person acquired in step S102 that is equal to or greater than a threshold, it assigns a new identifier to the person.

[0054] Furthermore, for each cluster / subcluster in the input image, the identifier assigning unit 204 identifies a set that is the same as the set of identifiers of the people included in the cluster / subcluster from the cluster / subcluster history, and assigns an identifier corresponding to the identified set to the cluster / subcluster. Note that if the identifier assigning unit 204 cannot identify a set that is the same as the set of identifiers of the people included in the cluster / subcluster from the cluster / subcluster history, it assigns a new identifier to the cluster / subcluster.

[0055] As the similarity between features, for example, the Euclidean distance or the cosine similarity between features can be applied. Here, a set previously detected as a cluster may be detected as a subcluster, or a set previously detected as a subcluster may be detected as a cluster. In such a case, by making the naming rules of the identifiers of the cluster and the subcluster common, both can be handled transparently.

[0056] The identifier assigning unit 204 may assign, to a person, the identifier of the cluster / sub-cluster to which the person belongs. Also, the identifier assigning unit 204 may assign, to a sub-cluster, the identifier of the cluster to which the sub-cluster belongs.

[0057] In step S104a, the control unit 200 judges whether a predetermined re-assembly condition is satisfied. If the re-assembly condition is satisfied as a result of this judgment, the process proceeds to step S105, and if the re-assembly condition is not satisfied, the process proceeds to step S106. The re-assembly condition is a condition that serves as an index for judging whether or not to perform the process of step S105, and various conditions can be applied as such a condition.

[0058] An applicable reassembly condition is, for example, "the number of clusters is greater than the number of clusters detected in the past." An applicable reassembly condition is "the number of person identifiers included in a cluster / subcluster is greater than the number of person identifiers included in a cluster / subcluster detected in the past." An applicable reassembly condition is "one or more people are detected from the input image." The process of step S105 may be performed periodically, in which case the reassembly condition may be "the current timing is the timing for performing the process of step S105."

[0059] In step S105, the re-assembly determination unit 205 determines the re-assembly state of clusters, sub-clusters, and people who do not belong to either clusters or sub-clusters. The clusters / sub-clusters detected by the detection unit 203 include a set of people who are related to each other, such as close people, and a set of unrelated people who happen to be there. In the following description, a set of close people will be used as an example of a set of people who are related to each other. However, this embodiment can also be applied to a set of people who have some kind of relationship or connection with each other.

[0060] In a group of close people, people who temporarily disperse tend to reunite (reassemble) later. On the other hand, in the case of a group of unrelated people who happen to be present, it is difficult for them to reunite in the same group once they have disperse. Therefore, by detecting the behavior of reunion, a group of close people can be detected. In addition, since it is difficult for a group of unrelated people who happen to be present to reunite in the same group once they have disperse, it is possible to prevent a group of unrelated people who happen to be present as a group of close people by mistake.

[0061] The determination process by regrouping determination section 205 will be described with reference to Fig. 7. Although the determination process for a cluster is described in Fig. 7, this determination process can be similarly applied to a sub-cluster.

[0062] A reassembly of a cluster refers to, for example, an event in which a group of people (first cluster) detected at time t1 disperses after time t1, the first cluster does not exist at time t2 after time t1, and the first cluster is reconstituted at time t3 after time t2. In the case of Fig. 7, if the same identifier as the identifier of the first cluster detected at time t1 is not obtained from the input image corresponding to time t2, and the same identifier as the identifier of the first cluster is assigned to the cluster detected at time t3, the reassembly determination unit 205 can determine that a reassembly of the first cluster has occurred at time t3.

[0063] However, even if a group of people disbands and then reconstitutes a group, there are cases where the group before disbanding and the group formed after disbanding are not the same. For example, there are cases where a new group is formed when some people from the group before disbanding leave and other people join instead. There are also cases where a new group is formed when some people from the group before disbanding leave. There are also cases where a new group is formed when new people join the group before disbanding.

[0064] If the cluster detected in step S103 (target cluster) is the same as a cluster detected in the past (cluster before dissolution), reassembly determination unit 205 adds 1 to the reassembly frequency (initial value is 0) managed in association with the identifier of the cluster before dissolution. More specifically, if there is a set (same set) among the "sets of identifiers of persons belonging to the cluster" of each cluster registered in the cluster / sub-cluster history that is the same as the set of identifiers of persons belonging to the target cluster, reassembly determination unit 205 adds 1 to the reassembly frequency managed in association with the identifier of the cluster corresponding to the same set.

[0065] Furthermore, if the target cluster is different from a cluster detected in the past (a cluster before dissolution), the reassembly determination unit 205 identifies a cluster (similar cluster) that is most similar to the target cluster among the clusters before dissolution.Then, the reassembly determination unit 205 determines the reassembly frequency of the target cluster as the result of adding the "similarity between the similar cluster and the target cluster" to the reassembly frequency managed in association with the identifier of the similar cluster.

[0066] More specifically, the re-set determination unit 205 identifies a set (similar set) that is most similar to the set of identifiers of persons belonging to the target cluster among the "sets of identifiers of persons belonging to the cluster" of each cluster registered in the cluster / sub-cluster history. The re-set determination unit 205 then obtains the similarity between the set of identifiers of persons belonging to the target cluster and the similar set. There are various methods for obtaining the similarity between the set of identifiers of persons belonging to the target cluster and the similar set, and the method is not limited to a specific method.

[0067] For example, it is assumed that the cluster / sub-cluster history includes the identifiers of the first cluster to the Nth cluster (N is an integer equal to or greater than 2) and the identifiers of the people belonging to the cluster. Here, the number of the same identifiers between the identifiers of the people belonging to the i-th cluster (1≦i≦N) and the identifiers of the people included in the target cluster is Ai, and the number of the identifiers of the people belonging to the target cluster is B. In this case, the re-settlement determination unit 205 performs a process of finding (Ai / B) as the similarity Ri between the i-th cluster and the target cluster for i=1 to N, and specifies the maximum value RM among R1 to RN. This similarity is the ratio of the number of objects commonly included in the target cluster and the i-th cluster to the number of people included in the target cluster. Note that examples of the similarity include the Jaccard coefficient, the Dice coefficient, and the Simpson coefficient. The closer to 1 these coefficients are, the higher the similarity is.

[0068] Then, reassembly determination unit 205 adds the similarity RM to the reassembly frequency managed in association with the identifier of the cluster (similar cluster) for which the similarity RM was found among the first cluster to the Nth cluster, and determines the result as the reassembly frequency of the target cluster.

[0069] "When the target cluster is different from the cluster detected in the past (the cluster before dissolution)" refers to, for example, the case where the target cluster is a cluster formed when some people in the cluster before dissolution leave and other people join instead. Also, the case where the target cluster is a cluster formed when new people join the set before dissolution. Also, the case where the target cluster is a cluster formed when people leave the set before dissolution.

[0070] In other words, the re-assembly determination unit 205 calculates the re-assembly frequency of the target cluster as described above when the identifiers of some of the persons in the similar cluster have been replaced with different identifiers, when the identifiers of new persons different from the identifiers of the persons belonging to the similar cluster have been added, or when some of the identifiers of the persons who belonged to the similar cluster have disappeared.

[0071] When the target cluster is a cluster formed by removing some people from a similar cluster, the re-assembly determination unit 205 may obtain the re-assembly frequency of the target cluster by the following process.

[0072] For example, the reassembly determination unit 205 determines the reassembly frequency of the target cluster as a result of adding the maximum value of the similarity, for example, 1, to the reassembly frequency of the similar cluster. Also, for example, if the reassembly frequency of the similar cluster is equal to or greater than a threshold value, the reassembly determination unit 205 determines the reassembly frequency of the target cluster as a result of adding the maximum value of the similarity, for example, 1, to the reassembly frequency of the similar cluster. Also, for example, if the time during which an identifier of a person who has disappeared from a similar cluster has been included in the similar cluster in the target cluster is equal to or greater than a specified time, the reassembly determination unit 205 determines the reassembly frequency of the target cluster as a result of adding the maximum value of the similarity, for example, 1, to the reassembly frequency of the similar cluster.

[0073] The re-assembly determination unit 205 performs such processing for each cluster detected from the input image to obtain the re-assembly frequency of the cluster. Note that the re-assembly frequency is obtained not only for clusters consisting of multiple people, but also for clusters consisting of one person.

[0074] A cluster with a high reassortment frequency obtained by such processing has a high probability of being a cluster made up of close people. In other words, the reassortment frequency can be regarded as a probability representing the closeness of a cluster made up of close people.

[0075] The re-assembly determination unit 205 may manage the re-assembly frequency calculated for a cluster in association with the identifiers of the people who belong to the cluster. This allows each person who makes up the cluster to be determined as having belonged to a cluster made up of close people even if the cluster is temporarily disbanded and the people who belonged to the cluster behave individually.

[0076] When a cluster made up of close people temporarily disbands and reassembles, it is expected that the information obtained by each person's individual actions will be aggregated and transmitted to other people when the cluster reassembles. This effect can indirectly affect a second person in a cluster who is interested in an event that a first person in the cluster is not directly interested in when the cluster reassembles. This effect is not expected to occur in a cluster formed temporarily by unrelated people who happen to be present, because the cluster will not reassemble once it disbands.

[0077] When a cluster reassembles, the frequency of reassembly is recorded for the people who belong to the cluster, making it possible to determine whether the above-mentioned effects can be expected for a certain person. In other words, the higher the frequency of reassembly, the higher the possibility that the interests of each person will spread to other people via the cluster.

[0078] In this way, the re-grouping determination unit 205 performs a calculation to cumulatively add values ​​corresponding to the degree of change from the second set to the first set, based on a first set of objects detected from a first frame and a second set of objects corresponding to the first set among sets of objects detected from a second frame, which is an earlier frame than the first frame.

[0079] Then, the history holding unit 206 registers the identifier of the person detected from the input image and the feature amount of the person in the person history. Furthermore, the history holding unit 206 registers the identifier of the cluster / subcluster detected from the input image and the identifier of the person belonging to the cluster / subcluster in the cluster / subcluster history. The history holding unit 206 also stores the re-assembly frequency of the person in a memory device such as the storage device 12 in association with the identifier of the person. Furthermore, the history holding unit 206 also stores the re-assembly frequency of the cluster / subcluster in a memory device such as the storage device 12 in association with the identifier of the cluster / subcluster.

[0080] In step S106, the information presenter 207 presents the result of the process in step S105. The method of presenting the result of the process in step S105 is not limited to a specific method.

[0081] For example, the information presenting unit 207 may display a screen 801 shown in Fig. 8 on the output device 15. In the screen 801, a person 802 is a person who has never belonged to a cluster (a person whose identifier is not registered in the cluster / sub-cluster history). Note that, although no information is displayed about such a person 802 in Fig. 8, for example, information indicating that the person 802 has never belonged to a cluster may be displayed.

[0082] A person 803 is a person who has belonged to a cluster (a person whose identifier is registered in the cluster / subcluster history). In Fig. 8, an ellipse 803a surrounding the person 803 is displayed, and a re-assembly frequency 804 corresponding to the person 803 is also displayed.

[0083] A dotted frame 805 is a frame surrounding a cluster whose reassortment frequency is 0, and is displayed together with the cluster's reassortment frequency 806. A solid frame 807 is a frame surrounding a cluster whose reassortment frequency is greater than 0, and is displayed together with the cluster's reassortment frequency 807a.

[0084] A solid line box 808 encloses a cluster that has a repopulation frequency greater than 0 and has subclusters, along with the cluster's repopulation frequency 808a. A solid line box 809 encloses a subcluster, along with the subcluster's repopulation frequency 810.

[0085] The display method of each piece of information shown in Fig. 8 is not limited to a specific display method. For example, the thickness, line type, shape, color, background color of the frame, etc. may be changed depending on the object to which the frame is attached. Also, for example, the thickness or color of the characters representing the reunion frequency may be changed depending on the magnitude of the reunion frequency, and a reunion frequency equal to or greater than a threshold may be displayed in a different display method (such as character size or color) from a reunion frequency below the threshold.

[0086] Also, if necessary, some of the displays shown in Fig. 8 may be hidden or new displays may be added. For example, person identifiers and cluster / sub-cluster identifiers may be displayed. Also, attributes and positions of people belonging to clusters / sub-clusters may be displayed.

[0087] Furthermore, by presenting information on individuals whose reunion frequency is equal to or greater than a threshold, it is possible, for example, to narrow down individuals on whom actions should be taken and to appropriately select actions to be taken against those individuals.

[0088] If an event that a first person who belongs to a cluster with a reunion frequency equal to or greater than a threshold does not directly interest is an event that a second person who belongs to the same cluster and is interested in, it can be determined that causing the event to the first person can have an indirect effect on the second person. Therefore, by presenting the reunion frequency, it is possible to perform behavioral analysis that takes into account the effects of individual behavior, which cannot be known by analyzing group behavior alone.

[0089] In addition, the information presentation unit 207 may display the presented information on an external display device connected to the I / F device 16, or may transmit the information to a device on a network that can communicate via the I / F device 16.

[0090] Thus, according to this embodiment, by presenting the reunion frequency, the accuracy of person behavior analysis can be improved, and by referring to the presented reunion frequency, it is possible to improve the effectiveness of marketing, etc.

[0091] [Second embodiment] In the first embodiment, a form in which the reassembly frequency is presented has been described. However, other information derived from the reassembly frequency may be presented in place of or in addition to the reassembly frequency.

[0092] As described above, a cluster with a high reunion frequency is highly likely to be a cluster made up of close people, so the reunion frequency can be regarded as a degree of certainty that indicates the closeness of a cluster made up of close people. Therefore, for such a cluster, the relationships of people belonging to the cluster (students at the same school, family members, members of the same team, etc.) may be displayed.

[0093] For example, when the image processing device 10 is applied to a school, attributes such as the class and club to which each student belongs are registered in the storage device 12 of the image processing device 10. Then, for a cluster whose re-assembly frequency is equal to or greater than a threshold, the information presentation unit 207 acquires attributes of people belonging to that cluster from the storage device 12 and displays the acquired attributes.

[0094] Moreover, the numerical values ​​such as the reassembly frequency are not limited to being displayed as text, but may be displayed as an image according to the magnitude of the numerical value. Furthermore, the image processing device 10 may control the imaging direction, the angle of view, etc. of one or more imaging devices in order to track and capture clusters whose reassembly frequency is equal to or greater than a threshold value.

[0095] In addition, the image processing device 10 may display a frame or other information about a person whose re-gathering frequency remains below a threshold for a specified period of time or longer, and may control the imaging direction, angle of view, etc. of one or more imaging devices in order to track and photograph such a person.

[0096] The image processing device 10 may be a device separate from the imaging device, or may be incorporated into the imaging device. The image processing device 10 may be implemented in one computer device, or in multiple computer devices.

[0097] The numerical values, processing timing, processing order, processing subject, data (information) acquisition method / destination / source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0098] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0099] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0100] The invention of this specification includes the following image processing device, image processing method, and computer program. (Item 1) a calculation means for cumulatively adding a value according to a degree of change from a first set of objects detected from a first frame and a second set of objects corresponding to the first set among sets of objects detected from a second frame that is an earlier frame than the first frame; an output means for outputting an output based on the result of the cumulative addition; An image processing device comprising: (Item 2) 2. The image processing device according to item 1, wherein the calculation means cumulatively adds a larger value as the change is smaller. (Item 3) The calculation means cumulatively adds up a ratio of the number of objects commonly included in the first set and the second set to the number of objects included in the first set. 3. The image processing device according to item 1 or 2. (Item 4) The image processing device according to any one of items 1 to 3, characterized in that the calculation means cumulatively adds the maximum value of the values ​​when there is an object included in the second set that is not included in the first set. (Item 5) The image processing device according to any one of items 1 to 4, characterized in that the calculation means cumulatively adds the maximum value of the values ​​when an object that has been included in the second set for a specified time or more is not included in the first set. (Item 6) The image processing device according to any one of items 1 to 5, characterized in that, when there is an object included in the second set that is not included in the first set, if the result of the cumulative addition for the second set is equal to or greater than a threshold, the calculation means cumulatively adds the maximum value of the values. (Item 7) The image processing device according to any one of items 1 to 6, characterized in that the calculation means performs the cumulative addition calculation when the number of object sets detected from the first frame is greater than the number of object sets detected from the second frame. (Item 8) The image processing device according to any one of items 1 to 6, characterized in that the calculation means performs the cumulative addition calculation when the number of objects included in the first set is greater than the number of objects included in the second set. (Item 9) 7. The image processing device according to any one of items 1 to 6, wherein the calculation means performs the cumulative addition calculation when an object is detected from the first frame. (Item 10) 7. The image processing device according to any one of items 1 to 6, wherein the calculation means periodically performs the cumulative addition calculation. (Item 11) 11. The image processing device according to any one of items 1 to 10, wherein the output means displays the result of the cumulative addition together with an indication representing the first set. (Item 12) 12. The image processing device according to any one of items 1 to 11, wherein the output means outputs information derived from a result of the cumulative addition. (Item 13) moreover, 13. The image processing device according to any one of items 1 to 12, further comprising a means for controlling one or more imaging devices to track and photograph the first set according to a result of the cumulative addition. (Item 14) 14. The image processing device according to any one of items 1 to 13, wherein the set is a cluster of objects. (Item 15) 14. The image processing device according to any one of items 1 to 13, wherein the set is a sub-cluster in a cluster of objects. (Item 16) An image processing method performed by an image processing device, comprising: a calculation step in which a calculation means of the image processing device cumulatively adds a value according to a degree of change from the second set to the first set, based on a first set of objects detected from a first frame and a second set of objects corresponding to the first set among sets of objects detected from a second frame that is an earlier frame than the first frame; an output step in which an output means of the image processing device outputs an output based on the result of the cumulative addition; An image processing method comprising: (Item 17) 16. A computer program for causing a computer to function as the image processing device according to any one of items 1 to 15.

[0101] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0102] 200: Control unit 201: Image acquisition unit 202: Position estimation unit 203: Detection unit 204: Identifier assignment unit 205: Reassembly determination unit 206: History storage unit 207: Information presentation unit

Claims

1. a calculation means for cumulatively adding a value according to a degree of change from a first set of objects detected from a first frame and a second set of objects corresponding to the first set among sets of objects detected from a second frame that is an earlier frame than the first frame; an output means for outputting an output based on the result of the cumulative addition; An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein the calculation means cumulatively adds a larger value as the change is smaller.

3. The calculation means cumulatively adds up a ratio of the number of objects commonly included in the first set and the second set to the number of objects included in the first set.

2. The image processing device according to claim 1,

4. 2 . The image processing device according to claim 1 , wherein the calculation means accumulates the maximum value of the values ​​when there is an object that is not included in the first set among the objects included in the second set.

5. 2 . The image processing apparatus according to claim 1 , wherein the calculation means cumulatively adds the maximum value of the values ​​when an object that has been included in the second set for a specified time or longer is not included in the first set.

6. The image processing device according to claim 1, characterized in that, when there is an object included in the second set that is not included in the first set, the calculation means accumulates the maximum value of the values ​​if the result of the cumulative addition for the second set is equal to or greater than a threshold value.

7. 2. The image processing device according to claim 1, wherein the calculation means performs the cumulative addition calculation when the number of object sets detected from the first frame is greater than the number of object sets detected from the second frame.

8. 2. The image processing device according to claim 1, wherein the calculation means performs the cumulative addition calculation when the number of objects included in the first set is greater than the number of objects included in the second set.

9. 2. The image processing apparatus according to claim 1, wherein said calculation means performs said cumulative addition calculation when an object is detected from said first frame.

10. 2. The image processing apparatus according to claim 1, wherein said calculation means periodically performs the cumulative addition calculation.

11. 2. The image processing apparatus according to claim 1, wherein said output means displays the result of said cumulative addition together with an indication representing said first set.

12. 2. The image processing apparatus according to claim 1, wherein the output means outputs information derived from the result of the cumulative addition.

13. moreover, 2 . The image processing apparatus according to claim 1 , further comprising: a unit for controlling one or more image capturing devices to track and capture the first set according to a result of the cumulative addition.

14. 2. The image processing apparatus according to claim 1, wherein the collection is a cluster of objects.

15. 2. The image processing apparatus according to claim 1, wherein the set is a sub-cluster in a cluster of objects.

16. An image processing method performed by an image processing device, comprising: a calculation step in which a calculation means of the image processing device cumulatively adds a value according to a degree of change from the second set to the first set, based on a first set of objects detected from a first frame and a second set of objects corresponding to the first set among sets of objects detected from a second frame that is an earlier frame than the first frame; an output step in which an output means of the image processing device outputs an output based on the result of the cumulative addition; An image processing method comprising:

17. A computer program for causing a computer to function as the image processing device according to any one of claims 1 to 15.

Images