Object analysis device and object analysis method

Abstract

To sufficiently maintain an identification accuracy of an object in which a photographing environment is changed momentarily.SOLUTION: In an object analysis device 100, a feature quantity calculation unit 130 calculates a first feature quantity expressing a feature quantity of an object from a RGB image, and calculates a second feature quantity expressing a feature quantity of an object from an IR image. A degree-of-importance calculation unit 140 calculates a first degree-of-importance expressing a degree-of-importance of the object in the RGB image, and a second degree-of-importance expressing the degree-of-importance of the object in the IR image. A similarity calculation unit 150 calculates a similarity level of the object in a RGB video image 203 and an IR video image 204 on the basis of the first and second feature quantity and the first and second degree-of-importance. An identity determination unit 160 determines whether or not the object in the RGB video image 203 and the object in the IR video image 204 are same on the basis of the similarity level calculated by the similarity calculation unit 150.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an apparatus and method for tracking objects from images. [Background technology]

[0002] In recent years, growing public interest in public safety has led to the widespread use of video surveillance systems using surveillance cameras. For example, there is a high demand for such video surveillance systems in public places where ensuring safety is essential, such as airports, train stations, schools, and office buildings. However, manually analyzing the large amounts of video information obtained by video surveillance systems requires a significant amount of effort. Therefore, there is a demand for technology that uses computers to automatically perform video analysis.

[0003] In video analysis for video surveillance systems, it is important to accurately identify objects present in the video for each frame in order to track moving objects, such as people, in the video captured by the surveillance camera over time. However, the environment in which the object is captured by the surveillance camera may change from moment to moment due to changes in the brightness of the shooting location or changes in the positional relationship between the object and its surroundings. In such cases, the accuracy of object identification decreases, resulting in the problem of being unable to track the object correctly.

[0004] For example, Patent Documents 1, 2, and 3 propose technologies for improving the accuracy of identifying objects photographed by a camera. Patent Document 1 discloses an item identification method that acquires auxiliary information, such as the item's depth, identification code, gravity, and odor, in addition to the item's position and type information in each frame of an image, and identifies the item by performing multimodal fusion on the position information and auxiliary information. Patent Document 2 discloses an image recognition method that performs image recognition by combining multiple classifiers that each distinguish various image features. Patent Document 3 discloses a person tracking method that photographs the same object with a visible light camera and an infrared camera, tracks a person from the image captured by the visible light camera, detects the person's area from the image captured by the infrared camera, and integrates these results to track the person. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] US Patent Application Publication No. 2021 / 0397844 [Patent Document 2] U.S. Patent No. 10,956,778 [Patent Document 3] U.S. Patent No. 9,245,196 Summary of the Invention [Problem to be solved by the invention]

[0006] The techniques of Patent Documents 1 to 3 all aim to improve the accuracy of identifying objects, but they are not based on the premise that the shooting environment of the object changes during shooting. Therefore, it is difficult to maintain sufficient accuracy in identifying objects whose shooting environment changes from moment to moment. [Means for solving the problem]

[0007] An object analysis device according to a first aspect of the present invention includes an image acquisition unit that acquires a visible light image, which is an image of an object included in a visible light video captured by a first camera capable of capturing visible light, and acquires an invisible light image, which is an image of the object included in an invisible light video captured by a second camera capable of capturing invisible light and captured at the same time as the visible light image; a feature calculation unit that calculates a first feature representing a feature of the object from the visible light image and calculates a second feature representing a feature of the object from the invisible light image; a value corresponding to the brightness of the visible light image, a first importance level representing the importance level of the object in the visible light image; a value corresponding to the brightness of the invisible light image, The video system includes an importance calculation unit that calculates a second importance representing a feature of the object in the invisible light image, a similarity calculation unit that calculates a similarity of the object in the visible light image and the invisible light image based on the first feature, the second feature, the first importance, and the second importance, and an identity determination unit that determines whether the object in the visible light image and the object in the invisible light image are the same based on the similarity. An object analysis device according to a second aspect of the present invention includes an image acquisition unit that acquires a visible light image, which is an image of an object included in a visible light video captured by a first camera capable of capturing visible light, and acquires an invisible light image, which is an image of the object included in an invisible light video captured by a second camera capable of capturing invisible light, captured at the same time as the visible light image; a value corresponding to the brightness of the visible light image, a first importance level representing the importance level of the object in the visible light image; a value corresponding to the brightness of the invisible light image, The image processing device includes an importance calculation unit that calculates a second importance that represents a feature amount of the object in the invisible light image; an image selection unit that selects either the visible light image or the invisible light image for the object for each time period based on the first importance and the second importance; and a display control unit that arranges the selection results of the visible light image or the invisible light image at each time period by the image selection unit in chronological order and displays them on a display device. An object analysis method according to the present invention is a method for analyzing an object using a computer, the method including the steps of: acquiring, by the computer, a visible light image that is an image of the object included in a visible light video captured by a first camera capable of capturing visible light; acquiring an invisible light image that is an image of the object included in an invisible light video captured by a second camera capable of capturing invisible light and taken at the same time as the visible light image; calculating a first feature amount representing a feature amount of the object from the visible light image; and calculating a second feature amount representing a feature amount of the object from the invisible light image; a value corresponding to the brightness of the visible light image, calculating a first importance level representing an importance level of the object in the visible light image; a value corresponding to the brightness of the invisible light image, A second importance level representing the feature of the object in the invisible light image is calculated, and a similarity level of the object in the visible light image and the invisible light image is calculated based on the first feature level, the second feature level, the first importance level, and the second importance level, and a determination is made based on the similarity level as to whether the object in the visible light image and the object in the invisible light image are the same. [Effects of the Invention]

[0008] According to the present invention, it is possible to maintain sufficient accuracy in identifying an object even when the shooting environment changes from moment to moment. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of an object analysis device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating details of an image acquisition unit. [Figure 3] FIG. 2 is a diagram illustrating details of a feature amount calculation unit. [Figure 4] FIG. 2 is a diagram illustrating details of an importance calculation unit. [Figure 5] FIG. 2 is a diagram illustrating details of a similarity calculation unit. [Figure 6] FIG. 10 is a diagram illustrating details of a matching score calculation unit. [Figure 7] 10 is a flowchart showing a series of processing steps for tracking an object. [Figure 8] 10 is a flowchart showing details of an MMMS calculation process. [Figure 9] FIG. 2 is a diagram illustrating details of an image selection unit. [Figure 10] 10 is a flowchart showing the flow of a learning data generation process. [Figure 11] FIG. 1 illustrates an example of the configuration of a learning device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. For clarity of explanation, the following description and drawings have been omitted or simplified as appropriate. The present invention is not limited to the present embodiment, and any application example that conforms to the concept of the present invention is included in the technical scope of the present invention. Unless otherwise specified, each component may be plural or singular.

[0011] In the following explanation, processing may be described using a "program" or its process as the subject, but a program is executed by a processor (e.g., a CPU (Central Processing Unit)) to perform a predetermined process using storage resources (e.g., memory) and / or communication interface devices (e.g., communication ports) as appropriate, so the subject of the processing may also be the processor. A processor operates as a functional unit that realizes a predetermined function by operating in accordance with a program. Devices and systems that include a processor are devices and systems that include these functional units.

[0012] An embodiment of the present invention will be described below.

[0013] 1 is a block diagram showing the configuration of an object analysis device according to one embodiment of the present invention. The object analysis device 100 of this embodiment is a device that detects and tracks objects, such as people, that appear in an image captured by an image capture device 200 of a predetermined monitoring area, thereby monitoring the behavior of the objects.

[0014] As shown in FIG. 1, object analysis device 100 is connected to an imaging device 200, an input device 300, and a display device 400. Object analysis device 100 includes an image acquisition unit 110, a tracking database 120, a feature calculation unit 130, an importance calculation unit 140, a similarity calculation unit 150, an identity determination unit 160, an image selection unit 170, and a display control unit 180. In object analysis device 100, the functional blocks of image acquisition unit 110, feature calculation unit 130, importance calculation unit 140, similarity calculation unit 150, identity determination unit 160, image selection unit 170, and display control unit 180 are implemented, for example, by a computer executing a predetermined program, and tracking database 120 is implemented using a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). Note that some or all of these functional blocks may be implemented using a graphics processing unit (GPU) or a field programmable gate array (FPGA).

[0015] The image capturing device 200 is configured with an RGB camera 201 and an IR camera 202. The RGB camera 201 is installed so that its imaging range includes a predetermined monitoring area, and captures an RGB image 203, which is an image captured using visible light within the imaging range. The IR camera 202 is installed so that its imaging range includes the same monitoring area as the RGB camera 201, and captures an IR image 204, which is an image captured using infrared light (invisible light) within the imaging range. The RGB image 203 and the IR image 204 are each composed of a combination of multiple images (frames) arranged in chronological order, and each of these images is acquired by the RGB camera 201 or the IR camera 202 capturing an image of the monitoring area at a predetermined frame rate. The RGB image 203 and the IR image 204 captured by the RGB camera 201 and the IR camera 202, respectively, are transmitted from the image capturing device 200 to the object analysis device 100 and input to the object analysis device 100.

[0016] Image acquisition unit 110 acquires an image of an object such as a person based on RGB image 203 and IR image 204 input from image capture device 200. Image acquisition unit 110 extracts an image portion including the object from each frame of RGB image 203 captured by RGB camera 201, outputs the image to feature amount calculation unit 130 and importance calculation unit 140 as an RGB image (visible light image) of the object, and stores the image in tracking database 120. Similarly, image acquisition unit 110 extracts an image portion including the object from each frame of IR image 204 captured by IR camera 202, and outputs the image to feature amount calculation unit 130 and importance calculation unit 140 as an IR image (invisible light image) of the object, and stores the image in tracking database 120.

[0017] The feature amount calculation unit 130 calculates feature amounts that indicate the degree of characteristics of the object in each image from the RGB image and the IR image acquired from the RGB image 203 and the IR image 204, respectively, by the image acquisition unit 110. The feature amounts calculated by the feature amount calculation unit 130 are stored in the tracking database 120 in association with the images used to calculate the feature amounts, and are also output to the similarity calculation unit 150.

[0018] The importance calculation unit 140 calculates importance, which indicates how important an object included in each image is, from the RGB image and the IR image acquired from the RGB image 203 and the IR image 204, respectively, by the image acquisition unit 110. The importance calculated by the importance calculation unit 140 is stored in the tracking database 120 in association with the image used to calculate the importance, and is also output to the similarity calculation unit 150.

[0019] The similarity calculation unit 150 calculates a similarity that indicates the degree of similarity between objects in the RGB video 203 and the IR video 204, based on the feature amounts and importance calculated from the RGB image and the IR image by the feature amount calculation unit 130 and the importance calculation unit 140, respectively. The similarity calculation unit 150 pairs an RGB image and an IR image extracted from frames captured at the same time in the RGB video 203 and the IR video 204, respectively, and calculates the similarity based on the feature amounts and importance of two pairs that are consecutive in time series. Note that the method of calculating the similarity by the similarity calculation unit 150 will be described in detail later.

[0020] The identity determination unit 160 determines whether an object in the RGB image 203 and an object in the IR image 204 are the same, based on the similarity calculated by the similarity calculation unit 150. For each pair of two chronologically consecutive RGB images and IR images whose similarities have been calculated by the similarity calculation unit 150, the identity determination unit 160 calculates an identity score that indicates the identity of the object in these images, and determines whether each object in the RGB image 203 and each object in the IR image 204 are the same, based on the value of this identity score. The identity determination unit 160 then tracks objects determined to be the same in the RGB image 203 and the IR image 204, respectively, to monitor the behavior of the objects. The results of object tracking by the identity determination unit 160 are stored in the tracking database 120 in association with the RGB image 203 and the IR image 204.

[0021] In response to a user instruction input via the input device 300, the image selection unit 170 acquires the RGB image and IR image of a specified object from the RGB images and IR images of each object stored in the tracking database 120 as tracking images of the object in the RGB video 203 and the IR video 204. Then, for each acquired image, the image selection unit 170 selects either the RGB image or the IR image for each time and outputs the selection result to the display control unit 180. At this time, the image selection unit 170 can determine whether to select the RGB image or the IR image based on the importance of each image calculated by the importance calculation unit 140.

[0022] The display control unit 180 arranges the RGB images or IR images selected by the image selection unit 170 in chronological order and displays them on the display device 400. By checking the screen displayed on the display device 400, the user can check the images of the object shown in the RGB image 203 or the IR image 204 in chronological order and understand the behavior of the object.

[0023] The input device 300 is configured with, for example, a keyboard, a mouse, a touch panel, etc., and detects user operations and transmits the details of the operations to the object analyzing device 100. The display device 400 is configured with, for example, a liquid crystal display, etc., and provides information to the user by displaying various screens under the control of the display control unit 180. Note that a computer connected to the object analyzing device 100 via a communication network may be used as the input device 300 or the display device 400.

[0024] Next, the image acquisition section 110, feature amount calculation section 130, importance calculation section 140, similarity calculation section 150, identity determination section 160 and image selection section 170 will be described in detail below with reference to FIGS.

[0025] 2 is a diagram showing details of the image acquisition unit 110. As shown in FIG. 2, the image acquisition unit 110 includes an object detection unit 111 and an association unit 112.

[0026] The object detection unit 111 detects objects such as people from each image constituting the RGB image 203 and IR image 204 input from the imaging device 200, and extracts the image portion surrounding the object to obtain the aforementioned RGB image and IR image, respectively.

[0027] The association unit 112 associates RGB images and IR images acquired by the object detection unit 111 that are assumed to represent the same object with each other. Here, for example, among RGB images and IR images extracted from the RGB video 203 and IR video 204 taken at the same time, a combination of an RGB image and an IR image that are assumed to exist in the same position based on their relative positions is identified by performing perspective transformation using a predetermined homography matrix, thereby associating the RGB images with the IR images. The example of FIG. 2 shows how RGB images 203a to 203f extracted from the RGB video 203 and IR images 204a to 204f extracted from the IR video 204 are associated with each other by the association unit 112. In the example of FIG. 2, an IR image corresponding to RGB image 203g does not exist in IR video 204, and therefore no IR image is associated with RGB image 203g.

[0028] The result of associating the RGB image and the IR image by the associating unit 112 is output from the image acquiring unit 110 together with the RGB image and the IR image, and is stored in the tracking database 120 .

[0029] Fig. 3 is a diagram showing details of the feature calculation unit 130. As shown in Fig. 3, the feature calculation unit 130 includes an intra-modality feature calculation unit 131 and a cross-modality feature calculation unit 132. Note that Fig. 3 illustrates two blocks for each of the intra-modality feature calculation unit 131 and the cross-modality feature calculation unit 132 in order to explain the operations of these units for an RGB image and an IR image, respectively, but in reality, the feature calculation unit 130 may include one intra-modality feature calculation unit 131 and one cross-modality feature calculation unit 132, or one for each of the RGB image and the IR image.

[0030] The intra-modality feature calculation unit 131 calculates a feature for each of the RGB image and the IR image to obtain a similarity between images of the same type. For example, the intra-modality feature calculation unit 131 calculates a feature f RGB (DRGB ) and calculate the feature value f IR (D IR ) is calculated.

[0031] The cross-modality feature calculation unit 132 calculates a feature for each of the RGB image and the IR image to obtain a similarity between different types of images. For example, the cross-modality feature calculation unit 132 calculates a feature f CM (D RGB ) and calculate the feature value f CM (D IR ) is calculated.

[0032] The feature quantity f calculated for the RGB image and the IR image by the intra-modality feature quantity calculation unit 131 and the cross-modality feature quantity calculation unit 132, respectively, is RGB (D RGB ), f CM (D RGB ), f IR (D IR ) and f CM (D IR ) is stored in the tracking database 120 in association with the RGB image and the IR image.

[0033] The intra-modality feature calculation unit 131 and the cross-modality feature calculation unit 132 can each be realized by, for example, artificial intelligence (AI) using a trained neural network.

[0034] Fig. 4 is a diagram showing details of the importance calculation unit 140. As shown in Fig. 4, the importance calculation unit 140 includes an RGB image importance calculation unit 141 and an IR image importance calculation unit 142.

[0035] The RGB image importance calculation unit 141 calculates the importance IS, which is expressed as a score value between 0 and 1, for the RGB image. RGBR The IR image importance calculation unit 142 calculates the importance IS, which is expressed as a score value between 0 and 1, for the IR image. IRCalculate the importance IS RGB , IS IR is information indicating how much the RGB image and the IR image each contain the information necessary for the similarity calculation unit 150 to accurately calculate the similarity.

[0036] The importance IS calculated for the RGB image and the IR image by the RGB image importance calculation unit 141 and the IR image importance calculation unit 142 respectively RGB and IS IR are stored in the tracking database 120 in association with the RGB image and the IR image.

[0037] Note that the RGB image importance calculation unit 141 and the IR image importance calculation unit 142 can be respectively realized by, for example, artificial intelligence (AI) using a learned neural network.

[0038] FIG. 5 is a diagram showing details of the similarity calculation unit 150. In FIG. 5, for each pair of the RGB image and the IR image extracted from the frames respectively captured at consecutive times t1 and t2 (t1 < t 2) in the RGB video 203 and the IR video 204, an example is shown in which the feature amount calculation unit 130 and the importance calculation unit 140 calculate the feature amount and the importance respectively.

[0039] In FIG. 5, the feature amounts f RGB (D 1 RGB ), f CM (D 1 RGB ), f IR (D 1 IR ) and f CM (D 1 IR ) represent the feature amounts calculated by the feature amount calculation unit 130 by the intramodality feature amount calculation unit 131 and the cross-modality feature amount calculation unit 132 respectively for the RGB image and the IR image extracted from the frame at time t1. Similarly, the feature amount f RGB (D 2 RGB ), f CM (D2 RGB ), f IR (D 2 IR ) and f CM (D 2 IR ) represent the feature amounts calculated by the feature amount calculation unit 130 using the intra-modality feature amount calculation unit 131 and the cross-modality feature amount calculation unit 132 for the RGB image and the IR image extracted from the frame at time t2, respectively.

[0040] Also, importance IS 1 RGB and IS 1 IR represents the importance calculated by the importance calculation unit 140 using the RGB image importance calculation unit 141 and the IR image importance calculation unit 142 for the RGB image and the IR image extracted from the frame at time t1. Similarly, the importance IS 2 RGB and IS 2 IR represents the importance calculated by the importance calculation unit 140 using the RGB image importance calculation unit 141 and the IR image importance calculation unit 142 for the RGB image and the IR image extracted from the frame at time t2, respectively.

[0041] As shown in FIG. 5, the similarity calculation unit 150 includes a matching score calculation unit 151, weighting multiplication units 153a to 153d, and a summing unit 154.

[0042] The matching score calculation unit 151 calculates an RGB-RGB matching score 152a, an RGB-IR matching score 152b, an IR-RGB matching score 152c, and an IR-IR matching score 152d based on the above feature amounts calculated by the feature calculation unit 130. The RGB-RGB matching score 152a represents the similarity between the RGB image at time t1 and the RGB image at time t2, and the RGB-IR matching score 152b represents the similarity between the RGB image at time t1 and the IR image at time t2. The IR-RGB matching score 152c represents the similarity between the IR image at time t1 and the RGB image at time t2, and the IR-IR matching score 152d represents the similarity between the IR image at time t1 and the IR image at time t2. Details of the matching score calculation unit 151 will be described later with reference to FIG. 6.

[0043] The weighting multiplication unit 153a applies the importance IS calculated by the importance calculation unit 140 for the RGB image at time t1 and the RGB image at time t2 to the RGB-RGB matching score 152a. 1 RGB and IS 2 RGB The weighting multiplication unit 153b multiplies the RGB-IR matching score 152b by the importance IS calculated by the importance calculation unit 140 for the RGB image at time t1 and the IR image at time t2, and performs weighting according to the importance. 1 RGB and IS 2 IR The weighting multiplication unit 153c multiplies the IR-RGB matching score 152c by the importance IS calculated by the importance calculation unit 140 for the IR image at time t1 and the RGB image at time t2, and performs weighting according to the importance. 1 IR and IS 2 RGBThe weighting multiplication unit 153d multiplies the IR-IR matching score 152d by the importance calculation unit 140 for the IR image at time t1 and the IR image at time t2, and weights the IR-IR matching score 152d according to the importance. 1 IR and IS 2 IR and weighting is performed according to their importance.

[0044] The summing unit 154 sums up the matching scores 152a to 152d weighted by the weighting multiplication units 153a to 153d, respectively, to calculate a multi-modality matching score (hereinafter referred to as "MMMS") 155. That is, the MMMS 155 calculated by the summing unit 154 is calculated by multiplying the above-mentioned feature value f RGB (D 1 RGB ), f CM (D 1 RGB ), f IR (D 1 IR ), f CM (D 1 IR ), f RGB (D 2 RGB ), f CM (D 2 RGB ), f IR (D 2 IR ) and f CM (D 2 IR ) and importance IS 1 RGB , IS 1 IR , IS 2 RGB and IS 2 IR Using these, it can be expressed by the following equation (1). MMMS = (IS 1 RGB * IS 2 RGB ) * MF(f RGB (D 1RGB ),f RGB (D 2 RGB )) + (IS 1 RGB * IS 2 IR ) * MF(f CM (D 1 RGB ),f CM (D 2 IR )) + (IS 1 IR * IS 2 RGB ) * MF(f CM (D 1 IR ),f CM (D 2 RGB )) + (IS 1 IR * IS 2 IR ) * MF(f IR (D 1 IR ),f IR (D 2 IR )) ···(1)

[0045] In addition, on the right side of equation (1), MF(f RGB (D 1 RGB ),f RGB (D 2 RGB )) is the RGB-RGB matching score 152a, and MF(f CM (D 1 RGB ),f CM (D 2 IR )) is the RGB-IR matching score 152b, and MF(f CM (D 1 IR ),f CM (D 2 RGB )) achieves an IR-RGB matching score of 152c, while MF(f IR (D 1IR ),f IR (D 2 IR )) represent the IR-IR matching score 152d. That is, on the right side of the formula (1), the first term is the importance IS 1 RGB and IS 2 RGB The RGB-RGB matching score 152a after weighting by 1 RGB and IS 2 IR The RGB-IR matching score 152b after weighting by the weighting factor is the importance factor IS 1 IR and IS 2 RGB The IR-RGB matching score 152c after weighting by 1 IR and IS 2 IR 15 and 16, respectively, represent the IR-IR matching scores 152d after weighting by

[0046] As described above, the similarity calculation unit 150 calculates matching scores 152a to 152d that respectively represent the similarity between objects for each combination of RGB images and IR images at times t1 and t2, and can calculate an MMMS 155 that represents the similarity between objects between the RGB images and IR images at times t1 and t2 based on these matching scores 152a to 152d.

[0047] Fig. 6 is a diagram showing details of the matching score calculation unit 151. As shown in Fig. 6, the matching score calculation unit 151 includes an RGB-RGB matching score calculation unit 151a, an RGB-IR matching score calculation unit 151b, an IR-RGB matching score calculation unit 151c, and an IR-IR matching score calculation unit 151d.

[0048] 6, the feature amount calculation unit 130 calculates the feature amount f of the object in the RGB image 101 for the RGB image 101 and the IR image 102 extracted from the frames at time t1 of the RGB image 203 and the IR image 204, respectively. RGB (D 1 RGB ) and f CM (D 1 RGB ) and the feature value f of the object in the IR image 102 IR (D 1 IR ) and f CM (D 1 IR ) are calculated, and the feature amount f of the object in the RGB image 103 is calculated for the RGB image 103 and the IR image 104 extracted from the frame at time t2. RGB (D 2 RGB ) and f CM (D 2 RGB ) and the feature value f of the object in the IR image 104 IR (D 2 IR ) and f CM (D 2 IR ) and shall be calculated respectively.

[0049] The RGB-RGB matching score calculation unit 151a calculates the feature value f of the RGB image 101 from among the above feature values. RGB (D 1 RGB ) and the feature value f of RGB image 103 RGB (D 2 RGB ) are input. The RGB-RGB matching score calculation unit 151a calculates the RGB-RGB matching score 152a based on these feature amounts.

[0050] The RGB-IR matching score calculation unit 151b calculates the feature value f of the RGB image 101 from among the above feature values. CM (D 1 RGB ) and the feature value f of the IR image 104 CM (D2 IR ) are input. The RGB-IR matching score calculation unit 151b calculates the RGB-IR matching score 152b based on these feature amounts.

[0051] The IR-RGB matching score calculation unit 151c calculates the feature value f of the IR image 102 from among the above feature values. CM (D 1 IR ) and the feature value f of RGB image 103 CM (D 2 RGB ) are input. The IR-RGB matching score calculation unit 151c calculates the IR-RGB matching score 152c based on these feature amounts.

[0052] The IR-IR matching score calculation unit 151d calculates the feature value f of the IR image 102 from among the above feature values. IR (D 1 IR ) and the feature value f of the IR image 104 IR (D 2 IR ) are input. The IR-IR matching score calculation unit 151d calculates the IR-IR matching score 152d based on these feature amounts.

[0053] Each of the above matching score calculation units 151a to 151d can determine the degree of similarity between a combination of two feature amounts using a well-known calculation method such as chi-square distribution, Euclidean distance, or cosine distance metrics, and calculate a matching score according to the determination result.

[0054] 7 is a flowchart showing the flow of a series of processes for tracking an object by the image acquisition unit 110, the feature amount calculation unit 130, the importance calculation unit 140, the similarity calculation unit 150, and the identity determination unit 160. In the object analysis device 100 of this embodiment, the processes shown in the flowchart in FIG. 7 are executed at predetermined time intervals, thereby tracking the object in the video acquired by the image capture device 200 and monitoring the behavior of the object.

[0055] In step S101, the image acquisition unit 110 acquires an RGB image 203 and an IR image 204, each composed of a plurality of images arranged in chronological order, from the RGB camera 201 and the IR camera 202 of the image capturing device 200. Then, objects are detected in the acquired RGB image 203 and IR image 204, respectively.

[0056] In step S102, the image acquisition unit 110 associates the same objects detected from the RGB image 203 and the IR image 204 in step S101 with each other.

[0057] In step S103, it is determined whether or not an object was detected in both the RGB image 203 and the IR image 204 in step S101. If an object was detected in both images, the process proceeds to step S104, and if an object was detected in only one of the images, the process proceeds to step S105. Note that if an object was not detected in both the RGB image 203 and the IR image 204, the process from step S104 onwards may be skipped and the process shown in the flowchart in FIG. 7 may end.

[0058] In step S104, the image acquisition unit 110 extracts pairs of an RGB image and an IR image corresponding to the object detected in step S101 from the RGB image 203 and the IR image 204, respectively.

[0059] In step S105, the image acquisition unit 110 extracts an RGB image or an IR image corresponding to the object detected in step S101 from either the RGB image 203 or the IR image 204.

[0060] In step S106, the pair of RGB image and IR image extracted by the image acquisition unit 110 in step S104 or S105, or either one of these images, is stored in the tracking database 120.

[0061] In step S107, the feature calculation unit 130 and the importance calculation unit 140 calculate the feature and importance, respectively, for the pair of RGB image and IR image extracted by the image acquisition unit 110 in step S104 or S105, or for either one of these images.

[0062] In step S108, the feature amount and importance calculated by the feature amount calculation unit 130 and importance calculation unit 140 in step S107 are stored in the tracking database 120 in association with the RGB image or IR image for which they were calculated.

[0063] In step S109, the similarity calculation unit 150 performs an MMMS calculation process to calculate the aforementioned MMMS 155. Here, two pairs of RGB and IR images that are consecutive in time series, i.e., the pair of RGB and IR images at time t1 and the pair of RGB and IR images at the next time t2, are identified in the tracking database 120, and the feature amounts and importance for each of these pairs are read from the tracking database 120. Then, the MMMS 155 is calculated based on these combinations. Details of the MMMS calculation process performed in step S109 will be described later with reference to the flowchart in FIG. 8.

[0064] In step S110, the value of MMMS 155 calculated by the MMMS calculation process in step S109 is stored in the tracking database 120 in association with each pair of RGB image and IR image at times t1 and t2 used in the calculation.

[0065] In step S111, identity determination unit 160 tracks the object based on the value of MMMS 155 calculated by the MMMS calculation process in step S109. Here, for example, based on the value of MMMS 155, it is determined whether the object in the RGB image and IR image at time t1 and the object in the RGB image and IR image at time t2 are the same. If it is determined that they are the same object, the object is tracked in RGB video 203 and IR video 204 during the period from time t1 to time t2.

[0066] In step S112, it is determined whether or not the object was successfully tracked in step S111. If it is determined that the object is the same at times t1 and t2 and the object was successfully tracked in the RGB image 203 and the IR image 204, the process proceeds to step S113; if it was not successfully tracked, the process proceeds to step S114.

[0067] In step S113, the identity determination unit 160 adds the tracking result of step S111 to the tracking results of the object obtained so far, thereby updating the tracking results of the object to the latest content.

[0068] In step S114, the identity determination unit 160 treats the object tracked in step S111 as a new object and starts tracking it.

[0069] In step S115, the processing contents of step S113 or S114 are reflected in the tracking database 120, and the tracking database 120 is updated. After the processing of step S115 is performed, the processing shown in the flowchart of FIG. 7 ends.

[0070] FIG. 8 is a flowchart showing the details of the MMMS calculation process.

[0071] In step S201, the similarity calculation unit 150 identifies pairs of RGB images and IR images at different times, i.e., a pair of RGB images and IR images at time t1 and a pair of RGB images and IR images at the next time t2, in the tracking database 120, and obtains the features and importance for each of these pairs from the tracking database 120.

[0072] In step S202, the similarity calculation unit 150 obtains the feature amounts of the object calculated by the feature amount calculation unit 130 for each of the RGB image and the IR image obtained in step S201. Specifically, the feature amount f RGB (D 1 RGB ) and f CM (D 1 RGB ) and the feature value f for the IR image at time t1 IR (D 1 IR ) and f CM (D 1 IR ) and the feature value f for the RGB image at time t2 RGB (D 2 RGB ) and f CM (D 2 RGB ) and the feature value f for the IR image at time t2 IR (D 2 IR ) and f CM (D 2 IR ) and get.

[0073] In step S203, the similarity calculation unit 150 calculates four types of matching scores that represent the similarity of the objects between the images based on the eight types of feature amounts acquired in step S202. Specifically, based on the above-mentioned feature amounts, the matching score calculation unit 151 of the similarity calculation unit 150 calculates an RGB-RGB matching score 152a, an RGB-IR matching score 152b, an IR-RGB matching score 152c, and an IR-IR matching score 152d by combining the feature amounts described in FIG.

[0074] In step S204, in parallel with steps S202 and S203, the similarity calculation unit 150 obtains the importance of the object calculated by the importance calculation unit 140 for each of the RGB image and the IR image obtained in step S201. Specifically, the importance IS 1 RGB and the importance IS for the IR image at time t1 1 IR and the importance IS for the RGB image at time t2 2 RGB and the importance IS for the IR image at time t2 2 IR and get.

[0075] In step S205, similarity calculation unit 150 calculates an MMMS by weighting each matching score calculated in step S203 according to the importance acquired in step S204. Specifically, weighting multiplication units 153a to 153d and summing unit 154 of similarity calculation unit 150 calculate an MMMS 155 representing the similarity of the object between the RGB image and the IR image at times t1 and t2 using the above-mentioned equation (1).

[0076] If MMMS can be calculated in step S205, the process shown in the flowchart of FIG. 8 ends, and the process proceeds from step S109 to step S110 in FIG.

[0077] Fig. 9 is a diagram showing details of the image selection unit 170. Fig. 9 shows an example of a user interface when the image selection unit 170 selects an image stored in the tracking database 120 in response to a user instruction and displays it on the display device 400 via the display control unit 180. At this time, a screen including a display target specification window 401, an image display window 402, and a condition setting window 403 is displayed on the display device 400, for example, as shown in Fig. 9.

[0078] The user can specify one of the people (objects) appearing in at least one of the RGB image 203 and the IR image 204 as the display object by selecting one of the ID numbers previously set for each person in the display object specification window 401. When the user specifies the person (object) to be displayed in this manner, the image selection unit 170 acquires, as tracking images of the person, the RGB images and IR images used to track the person from the tracking database 120. The example in Fig. 9 shows an example in which a series of RGB image group 902 including RGB images 902a to 902e and a series of IR image group 903 including IR images 903a to 903e are acquired as tracking images 901.

[0079] When the tracking image 901 is acquired from the tracking database 120, the image selection unit 170 selects either an RGB image or an IR image from the tracking image 901 for each time in accordance with the conditions set by the user in the condition setting window 403. For example, in the example of Fig. 9, the checkbox for "Image with high importance" is selected in the condition setting window 403, thereby setting the condition to select the image with higher importance from the pair of RGB image and IR image. Therefore, the image selection unit 170 selects the image with higher importance from the pair of RGB image and IR image for each time.

[0080] Here, among the RGB images 902a to 902e, the RGB images 902b and 902c are dark overall, and therefore the importance values ​​calculated for these images by the importance calculation unit 140 are lower than those of the paired IR images 903b and 903c. In this case, the RGB images 902a, 902d, and 902e, and the IR images 903b and 903c are selected, respectively.

[0081] After selecting either an RGB image or an IR image for each time period as described above, image selection unit 170 transmits each selected image to display device 400 via display control unit 180. In display device 400, each transmitted image is displayed in image display window 402 and presented to the user.

[0082] Although the above describes an example in which the image selection unit 170 selects the image with the higher importance from a pair of an RGB image and an IR image, it is also possible to select images according to other conditions. For example, both an RGB image and an IR image may be selected for each time, or a specified one of the RGB image and the IR image may be selected for each time. In addition, the image selection unit 170 may select at least one of each pair of an RGB image and an IR image included in the tracking image for each time according to any condition specified by the user, and display it on the display device 400.

[0083] Next, the learning of the neural networks in the feature amount calculation section 130 and importance calculation section 140 will be described below with reference to FIGS.

[0084] Fig. 10 is a flowchart showing the flow of a training data generation process. The process shown in the flowchart in Fig. 10 is performed, for example, in a training data generation device (not shown) at a timing instructed by a user or at regular intervals.

[0085] In step S301, an RGB image 203 and an IR image 204, each consisting of a plurality of images arranged in chronological order, are acquired from the RGB camera 201 and the IR camera 202 of the image capturing device 200. Note that instead of the RGB camera 201 and the IR camera 202, images captured by cameras having equivalent functions may be acquired as the RGB image 203 and the IR image 204.

[0086] In step S302, objects are detected in the RGB image 203 and the IR image 204 acquired in step S301.

[0087] In step S303, it is determined whether or not an object was detected in both the RGB image 203 and the IR image 204 in step S302. If an object was detected in both images, the process proceeds to step S304; if an object was not detected in at least one of the images, the learning data generation process shown in the flowchart in FIG. 10 ends.

[0088] In step S304, for the object detected in step S302, pairs of RGB images and IR images corresponding to the object are extracted from the RGB video 203 and the IR video 204 for each time.

[0089] In step S305, the object detected in step S02 is labeled for each pair of RGB image and IR image extracted in step S304. For example, if the object is a person, the ID number of the person is labeled for each pair of RGB image and IR image. This labeling process may be performed by a human or automatically by the learning data generation device using a predetermined algorithm.

[0090] In step S306, information about each pair of RGB image and IR image labeled in step S305 is stored as training data in training database 500. Training database 500 is a database of training data held by the training data generation device or another device, and is realized using a storage device such as an HDD or SSD. After the processing of step S306 is performed, the processing shown in the flowchart in FIG. 10 ends.

[0091] In this embodiment, training data is generated by the above-described processing and stored in the training database 500. Fig. 10 shows pairs 501a to 501f of an RGB image and an IR image as examples of training data stored in the training database 500. These image pairs 501a to 501f are each assigned one of ID numbers #1 to #5 as an example of labeling performed in step S305.

[0092] FIG. 11 is a diagram showing an example of the configuration of a learning device that performs learning of the importance calculation unit 140 using the learning data stored in the learning database 500 by the process of FIG.

[0093] The learning device 600 shown in FIG. 11 includes a learning data acquisition unit 601, a weighted matching score calculation unit 603, a loss calculation unit 605, and a network parameter calculation unit 606.

[0094] The training data acquisition unit 601 acquires three types of training data, namely, a reference pair 602a, a correct pair 602b, and an incorrect pair 602c, from the training database 500. The reference pair 602a and the correct pair 602b are pairs of an RGB image and an IR image labeled to indicate the same object among the training data stored in the training database 500. The incorrect pair 602c is a pair of an RGB image and an IR image labeled to indicate that they are not the same object among the training data stored in the training database 500. These training data are input to the importance calculation unit 140 and the weighted matching score calculation unit 603, respectively. Note that in FIG. 11, three importance calculation units 140 and two weighted matching score calculation units 603 are illustrated to explain the operation of the importance calculation unit 140 and the weighted matching score calculation unit 603 for each of the reference pair 602a, the correct pair 602b, and the incorrect pair 602c, but in reality, these are the same unit.

[0095] As described in FIG. 4, the importance calculation unit 140 has two neural networks that function as an RGB image importance calculation unit 141 and an IR image importance calculation unit 142, respectively, and uses these neural networks to calculate the importance of the reference pair 602a, the correct pair 602b, and the incorrect pair 602c, respectively.

[0096] The weighted matching score calculation unit 603 calculates a reference-correct matching score 604a for the combination of the reference pair 602a and the correct pair 602b by weighting the degree of similarity between these learning data in accordance with the importance calculated by the importance calculation unit 140. Similarly, the weighted matching score calculation unit 603 calculates a reference-incorrect matching score 604b for the combination of the reference pair 602a and the incorrect pair 602c by weighting the degree of similarity between these learning data in accordance with the importance calculated by the importance calculation unit 140.

[0097] The loss calculation unit 605 calculates the loss in each neural network of the importance calculation unit 140 based on the reference-correct matching score 604a and the reference-incorrect matching score 604b calculated by the weighted matching score calculation unit 603. Specifically, the loss value Loss of the neural network can be calculated, for example, by the following equation (2).

number

[0098] In equation (2), A i , P i , N i respectively represent the reference pair 602a, the correct pair 602b, and the incorrect pair 602c in the i-th training data. i , P i ) is A i and P i represents the criterion-correct matching score 604a for the combination of MMMMS(A i , N i ) is A i and N i represents the reference-incorrect matching score 604b for the combination of

[0099] The network parameter calculation unit 606 calculates the parameters of each neural network of the importance calculation unit 140 based on the loss calculated by the loss calculation unit 605. Then, the calculated parameters are reflected in each neural network, and the importance calculation unit 140 performs learning.

[0100] Although an example of a learning device that performs learning on the importance calculation unit 140 has been described with reference to FIG. 11, the feature calculation unit 130 can also be trained using a similar method.

[0101] According to the embodiment of the present invention described above, the following advantageous effects are achieved.

[0102] (1) The object analysis device 100 includes an image acquisition unit 110, a feature calculation unit 130, an importance calculation unit 140, a similarity calculation unit 150, and an identity determination unit 160. The image acquisition unit 110 acquires an RGB image (visible light image) that is an image of an object included in an RGB video 203 captured by an RGB camera 201 capable of capturing visible light, and also acquires an IR image (invisible light image) that is an image of the object included in an IR video 204 captured by an IR camera 202 capable of capturing invisible light and captured at the same time as the RGB image. The feature calculation unit 130 calculates a first feature representing the feature of the object from the RGB image, and calculates a second feature representing the feature of the object from the IR image. The importance calculation unit 140 calculates a first importance representing the importance of the object in the RGB image and a second importance representing the importance of the object in the IR image. The similarity calculation unit 150 calculates the similarity of the objects in the RGB image 203 and the IR image 204 based on the first feature amount, the second feature amount, the first importance, and the second importance. The identity determination unit 160 determines whether the object in the RGB image 203 and the object in the IR image 204 are the same based on the similarity calculated by the similarity calculation unit 150. In this manner, the first importance and the second importance are calculated according to the image quality of the RGB image and the IR image, which are images of the object acquired from the RGB image 203 and the IR image 204, respectively, and the similarity of the objects can be calculated by weighting the first feature amount and the second feature amount using these. Then, based on the calculated similarity, it can be determined whether the object in the RGB image 203 and the object in the IR image 204 are the same. Therefore, it is possible to maintain sufficient accuracy in identifying objects even when the shooting environment changes from moment to moment.

[0103] (2) The image acquisition unit 110 acquires an RGB image and an IR image at time t1, and an RGB image and an IR image at time t2, which is different from time t1. The feature amount calculation unit 130 calculates the feature amount f of the object in the RGB image and the IR image at time t1. RGB (D 1 RGB ), f CM (D1 RGB ), f IR (D 1 IR ) and f CM (D 1 IR ) and the feature value f of the object in the RGB image and the IR image at time t2 RGB (D 2 RGB ), f CM (D 2 RGB ), f IR (D 2 IR ) and f CM (D 2 IR The importance calculation unit 140 calculates the importance IS of the object in the RGB image and the IR image at time t1. 1 RGB and IS 1 IR and the importance of the object in the RGB image and the IR image at time t2, IS 2 RGB and IS 2 IR The similarity calculation unit 150 calculates (a) the feature value f at time t1. RGB (D 1 RGB ) and the feature value f at time t2 RGB (D 2 RGB ) and the importance at time t1 IS 1 RGB and the importance IS at time t2 2 RGB (b) a feature value f CM (D 1 RGB ) and the feature value f at time t2 CM (D 2 IR ) and the importance at time t1 IS 1 RGB and the importance IS at time t2 2IR (c) a feature value f CM (D 1 IR ) and the feature value f at time t2 CM (D 2 RGB ) and the importance at time t1 IS 1 IR and the importance IS at time t2 2 RGB (d) a feature value f IR (D 1 IR ) and the feature value f at time t2 IR (D 2 IR ) and the importance at time t1 IS 1 IR and the importance IS at time t2 2 IR (d) a fourth similarity (weighted IR-IR matching score 152d) representing the similarity between the object in the IR image at time t1 and the object in the IR image at time t2 is calculated based on these similarities, and an MMMS 155 representing the similarity of the object in the RGB image 203 and the IR image 204 at time t1 and time t2 is calculated based on these similarities. In this way, even if the shooting environment changes from moment to moment, it is possible to accurately calculate the similarity of the object between the RGB image 203 and the IR image 204 at different times.

[0104] (3) The object analysis device 100 includes an image selection unit 170 that selects either an RGB image or an IR image of the object for each time based on the first and second importance levels, and a display control unit 180 that causes the RGB image or IR image selection results for each time level by the image selection unit 170 to be arranged in chronological order and displayed on the display device 400. As a result, even if one of the RGB image or the IR image is dark and difficult to see, the object tracking results can be presented to the user in an easy-to-understand manner.

[0105] The present invention is not limited to the above-described embodiments, and can be implemented using any components within the scope of the present invention. The above-described embodiments and modifications are merely examples, and the present invention is not limited to these contents as long as the features of the invention are not impaired. Furthermore, although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other aspects conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention. [Explanation of symbols]

[0106] 100... object analysis device, 110... image acquisition unit, 120... tracking database, 130... feature calculation unit, 140... importance calculation unit, 150... similarity calculation unit, 160... identity determination unit, 170... image selection unit, 180... display control unit, 200... imaging device, 201... RGB camera, 202... IR camera, 203... RGB image, 204... IR image, 300... input device, 400... display device

Claims

1. an image acquisition unit that acquires a visible light image, which is an image of an object included in a visible light video captured by a first camera capable of capturing visible light, and acquires an invisible light image, which is an image of the object included in an invisible light video captured by a second camera capable of capturing invisible light, captured at the same time as the visible light image; a feature amount calculation unit that calculates a first feature amount representing a feature amount of the object from the visible light image and calculates a second feature amount representing a feature amount of the object from the invisible light image; an importance calculation unit that calculates a first importance level that is a value according to the brightness of the visible light image and represents the importance of the object in the visible light image, and a second importance level that is a value according to the brightness of the invisible light image and represents the importance of the object in the invisible light image; a similarity calculation unit that calculates a similarity of the object between the visible light image and the invisible light image based on the first feature amount, the second feature amount, the first importance, and the second importance; an identity determination unit that determines whether the object in the visible light image and the object in the invisible light image are identical based on the similarity; An object analysis device comprising:

2. 2. The object analysis device according to claim 1, the image acquisition unit acquires the visible light image and the invisible light image at a first time point, and the visible light image and the invisible light image at a second time point different from the first time point; the feature amount calculation unit calculates the first feature amount and the second feature amount at the first time point, and the first feature amount and the second feature amount at the second time point; the importance calculation unit calculates the first importance and the second importance at the first time, and the first importance and the second importance at the second time; The similarity calculation unit calculating a first similarity representing a similarity between the object in the visible light image at the first time and the object in the visible light image at the second time, based on the first feature amount at the first time, the first feature amount at the second time, the first importance at the first time, and the first importance at the second time; calculating a second similarity representing a similarity between the object in the visible light image at the first time and the object in the invisible light image at the second time, based on the first feature amount at the first time, the second feature amount at the second time, the first importance at the first time, and the second importance at the second time; calculating a third similarity representing a similarity between the object in the invisible light image at the first time and the object in the visible light image at the second time, based on the second feature amount at the first time, the first feature amount at the second time, the second importance at the first time, and the first importance at the second time; calculating a fourth similarity representing a similarity between the object in the invisible light image at the first time and the object in the invisible light image at the second time based on the second feature amount at the first time, the second feature amount at the second time, the second importance at the first time, and the second importance at the second time; calculating similarities of the object between the visible light image and the invisible light image at the first time point and the second time point based on the first similarity, the second similarity, the third similarity, and the fourth similarity; Object analysis device.

3. 3. The object analyzing apparatus according to claim 1, an image selection unit that selects either the visible light image or the invisible light image of the object for each time period based on the first importance and the second importance; a display control unit that displays the results of the selection of the visible light images or the invisible light images by the image selection unit at each time on a display device in chronological order; and An object analysis device comprising:

4. an image acquisition unit that acquires a visible light image, which is an image of an object included in a visible light video captured by a first camera capable of capturing visible light, and acquires an invisible light image, which is an image of the object included in an invisible light video captured by a second camera capable of capturing invisible light, captured at the same time as the visible light image; an importance calculation unit that calculates a first importance level that is a value according to the brightness of the visible light image and represents the importance of the object in the visible light image, and a second importance level that is a value according to the brightness of the invisible light image and represents the importance of the object in the invisible light image; an image selection unit that selects either the visible light image or the invisible light image of the object for each time period based on the first importance and the second importance; a display control unit that displays the results of the selection of the visible light images or the invisible light images by the image selection unit at each time on a display device in chronological order; and An object analysis device comprising:

5. A method for analyzing an object using a computer, comprising: The computer acquiring a visible light image of the object included in a visible light video captured by a first camera capable of capturing visible light; acquiring an invisible light image that is included in an invisible light video captured by a second camera capable of capturing invisible light, the invisible light image being an image of the object captured at the same time as the visible light image; calculating a first feature amount representing a feature amount of the object from the visible light image; calculating a second feature amount representing a feature amount of the object from the invisible light image; calculating a first importance level that is a value according to brightness of the visible light image and that represents an importance level of the object in the visible light image; calculating a second importance level that is a value according to the brightness of the invisible light image and that represents the importance of the object in the invisible light image; calculating a similarity of the object between the visible light image and the invisible light image based on the first feature amount, the second feature amount, the first importance level, and the second importance level; determining whether the object in the visible light image and the object in the invisible light image are the same based on the similarity; Object analysis methods.

Images