Determination device, determination method, and program

Abstract

To achieve automatic monitoring of equipment with a small amount of resources.SOLUTION: A determination device that determines a state of a lamp mounted on a device includes a positioning unit that specifies the device in each target image of a moving image by performing tracking on the moving image of an area including the device using a template image acquired by capturing the device in advance, and a state determination unit that executes color classification of the lamp on an image of the area of the lamp in the device and determines the state of the lamp on the basis of a result of the color classification.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a technique for determining the state of a device. [Background technology]

[0002] Data centers are equipped with many racks, each containing equipment such as servers. In data centers, equipment is monitored to ensure it is operating normally. There are two main methods known for determining the status of equipment in data center racks.

[0003] The first method is to install software that monitors the status of the CPU and disk on the device and issue an alert if an abnormality occurs in the monitored item. The second method is to have someone look at the LED lamp on the device and check whether the device is operating normally based on the appearance information.

[0004] When monitoring equipment, both of the above methods are often used. The reason for using information on the appearance of the LED lamp is that software monitoring can be difficult in some cases. In other words, there are cases where the equipment goes down silently without being detected by software.

[0005] Currently, when monitoring equipment such as servers in data centers, monitors make regular rounds and visually determine whether the status of LED lamps matches the pre-determined monitoring items.

[0006] In the above-mentioned equipment monitoring, the timing of monitoring is generally decided in advance. Therefore, the more devices that need to be monitored, the more time is required to complete the monitoring items per session, which increases labor costs and man-hours. Therefore, there is a demand for automating the monitoring of devices installed in data centers in order to reduce labor costs and man-hours. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Redmon, Joseph and Farhadi, Ali. YOLOv3: An Incremental Improvement.In: arXiv 2018. [Non-patent document 2] Philipp Lindenberger and Paul-Edouard Sarlin, Marc Pollefeys. LightGlue: Local Feature Matching at Light Speed.In: ICCV 2023. [Non-patent document 3] Uma Gopalakrishnan and P. Venkat Rangan, Ramkumar Narayanan. Spatio-Temporal Compositing of Video Elements for Immersive eLearning Classrooms. In: ISM 2017. [Non-patent document 4] Zhu, Jun-Yan and Park, Taesung and Isola, Phillip and Efros, Alexei A. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkss.In: ICCV 2017. [Non-Patent Document 5] Bewley, Alex and Ge, Zongyuan and Ott, Lionel and Ramos, Fabio and Upcroft, Ben.Simple online and realtime tracking.In: ICIP 2016. Summary of the Invention [Problem to be solved by the invention]

[0008] One possible method for automating equipment monitoring is to use a method for automatically determining the lighting status of LED lamps by using images of the equipment.

[0009] When implementing the above method based on conventional technology, it is possible to detect LED lamps using object detection technology, which is a deep learning method, but when using this object detection technology, there is a problem that it requires large resources (manpower, cost, etc.) to train the model.

[0010] The present invention has been made in view of the above points, and has an object to provide a technique for realizing automatic monitoring of devices with limited resources. [Means for solving the problem]

[0011] According to the disclosed technology, there is provided a determination device for determining the state of a lamp mounted on a device, the determination device comprising: a positioning unit that identifies the device in each target image of the video by tracking a region including the device using a template image acquired by photographing the device in advance; a state determination unit that performs color classification of the lamp on an image of an area of ​​the lamp in the device and determines a state of the lamp based on a result of the color classification; A determination device is provided, comprising: [Effects of the Invention]

[0012] The disclosed technology provides a technology for realizing automatic monitoring of devices with limited resources. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 10 is a diagram showing an example of attaching markers to the four corners of a server. [Figure 2] FIG. 10 is a diagram illustrating an example of a case where tracking is lost. [Figure 3] FIG. 10 is a diagram illustrating an example of alignment. [Figure 4] FIG. 1 illustrates an example of the configuration of a determination device 100. [Figure 5] 1 is a flowchart of an embodiment. [Figure 6] FIG. 10 is a diagram showing an example (image) of a template image. [Figure 7] FIG. 10 is a diagram showing examples (images) of an image with and without an obstacle. [Figure 8] FIG. 2 is a diagram illustrating an example of the hardware configuration of a determination device 100. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0015] In the following, we will first provide a detailed description of a technology that is expected based on conventional technology and its problems, and then provide a detailed description of the technology according to this embodiment. Note that the "object detection technology" and the "technology for estimating feature points" described below are publicly known, but the description of the expected technology using them and the description of the problems are not publicly known.

[0016] (About anticipated technologies and challenges) One possible method for automating equipment monitoring is to automatically grasp the lighting status of LED lamps. When this method is realized using conventional technology, it is possible to detect LED lamps using object detection technology (Non-Patent Document 1).

[0017] Also, if the ID of the LED lamp (information on whether it is the jth lamp on the i-th server) is known in advance from appearance information based on an image captured by a camera, there is a method to determine the lighting status by performing object detection or color judgment using the coordinates of the LED lamp.

[0018] However, with the above technology, when the lamp ID is not known, it is difficult to accurately estimate the lamp ID because determining which LED lamp a detected LED is depends on whether the LED lamp is detected successfully.

[0019] Another possible approach is to use an image matching-based method. In this method, first, using the technology described in Non-Patent Document 2, the corresponding points between a reference image annotated with the LED lamp positions and the image to be judged are estimated. Next, a transformation matrix for the server position is calculated from the corresponding points, and the reference image is projectively transformed onto the target image. As a result of the transformation, the positions of the LED lamps in the target image are estimated from the annotations in the reference image, and the positions of the LED lamps are extracted and subjected to image classification, which is then integrated and output as a result.

[0020] The prerequisite for the above technology is to take an image in advance that captures the entire rack, and then annotate that image with the positions of the servers and other devices and the LED lamps. With this method, one image is taken that captures the entire rack as a reference image. However, if the aisles where the racks and other devices are placed are narrow, or depending on the shooting conditions such as the shooting equipment, it may not be possible to fit the entire rack into the field of view, which can create the problem of not being able to prepare a reference image.

[0021] Furthermore, when analyzing a scene with an obstacle such as a rack net in front of a server, this method is unable to obtain corresponding points properly, resulting in a significant drop in accuracy in estimating the position of the LED lamp after projective transformation. This is because when matching feature points between a reference image without obstacles and a target image with obstacles, the matching ends up with a different location, such as an obstacle, rather than the server front panel, which is the desired location to be matched.

[0022] To solve this problem, it is conceivable to use a method in which markers are attached to the four corners of the server as shown in FIG. 1, and the reference image and the target image are aligned based on these markers (Non-Patent Document 3).

[0023] However, when performing automatic monitoring of servers in data centers, the ceiling lights are usually turned off, and the camera is often operated with the lights off during monitoring, making it difficult to adopt the above method. Even if the camera's built-in light is illuminated, it is difficult to brighten the entire field of view. Also, when using the method of attaching markers, the markers must be attached so as not to interfere with the front panel of the server, but in dark places there are insufficient feature points, making marker detection difficult.

[0024] (Overview of the Technology Relating to the Embodiments) In this embodiment, in order to provide a technology for realizing automatic monitoring of equipment with limited resources, and more specifically, to solve the problems with the above-mentioned assumed technology, a determination device 100 described below performs operations including the following first and second technologies.

[0025] <First Technology> The first technology maintains the accuracy of the server's positioning even when there are obstacles in the target image.

[0026] In the first technique, first, an image with an obstacle in front of the server and an image without the obstacle are prepared in advance. Then, an adversarial network (hereinafter referred to as GAN) such as Cycle GAN (Non-Patent Document 4) is used to generate an image without the obstacle from the image with the obstacle, and matching and alignment are performed on the generated images. By including images taken in dark places in the dataset, it is possible to generate images that are roughly free of obstacles even under dark shooting conditions, so there is no significant decrease in the accuracy of aligning the server and LED lamp. Therefore, it is possible to reduce the decrease in accuracy caused by obstacles and dark shooting conditions.

[0027] The technology used to generate an image without obstacles from an image with obstacles is not limited to GAN, and a neural network model other than GAN may be used to generate an image without obstacles from an image with obstacles.

[0028] <Second Technology> In this embodiment, it is assumed that the aisle where the racks are arranged is narrow, or that the entire rack cannot be included in the angle of view due to the shooting conditions of the shooting equipment, etc. However, the present invention is not limited to such a situation.

[0029] The second technology makes it possible to determine the status of equipment even when it is not possible to capture the entire rack in the field of view. The premise is that when photographing the rack, video of the rack is shot from top to bottom, and the server panel (either the front panel or the back panel) is captured from end to end in the horizontal direction. However, the method of capturing the video is not limited to this.

[0030] In the second technique, the determination device 100 prepares template images of servers of the same type as the types to be determined, and performs matching between the template images and the target image.

[0031] During matching, if the server associated with a specific template image differs between frames, it becomes impossible to properly determine the state of the LED lamp.

[0032] Therefore, the determination device 100 calculates a transformation matrix for transforming the template image into the target image based on matching of feature points between the template image and the target image of a specific server in the target image (e.g., an image of the server on the first level), and performs projective transformation on the template image. The determination device 100 also identifies the position of the specific server in the target image that appears in the projectively transformed template image (performs positioning).

[0033] Furthermore, the determination device 100 performs tracking such as SORT (Non-Patent Document 5) based on the centroid of the aligned specific server area (rectangle) in each subsequent frame, thereby matching the ID of the server (i.e., what type of server is located at which stage) that corresponds to a specific template image within the video.

[0034] In other words, the second technique involves tracking to match individual servers in a video sequence.

[0035] However, in this embodiment, since only one template image (which may also be called a reference image) is prepared for each model, even when the above tracking is performed, there may be cases where the server corresponding to a specific template image differs between frames, as shown in Figure 2.

[0036] Figure 2 shows an example of matching when using a template image of model a and a template image of model b. In the example of Figure 2, in frame t, the template image of model a is associated with the server of "model a first tier," and the template image of model b is associated with the server of "model b first tier." Assume that this association is successful up to frame t+(n-1).

[0037] However, at the time of frame t+n, the template image of model a is associated with the "model a second row" server, not the "model a first row." As a result, tracking is successful and status detection is only possible during the time when matching (alignment) is being performed.

[0038] Therefore, the second technique makes it possible to track a server that has been removed from the matching list in frame t+n, even in frame t+n.

[0039] Specifically, as shown in Fig. 3, the determination device 100 uses an image of frame t in a moving image as a reference image, matches the reference image (frame t) with the target image (frame t+n), and aligns the reference image (frame t) with the target image (frame t+n). For example, in the example of Fig. 3, the server of "model a, first stage" in the reference image (frame t) is associated with the server of "model a, first stage" in the target image (frame t+n).

[0040] Since the approximate server position in the reference image (frame t) is known, it is possible to detect the state even in frame t+n after the server has lost alignment with the template image.

[0041] In this embodiment, the server position is aligned in each frame by matching, but the accuracy of matching may be affected depending on the frame, potentially resulting in a large deviation in the estimated server position. Therefore, in the second technique, the bbox (rectangular region) of the server in each frame is tracked using SORT or the like based on the result of determining the server position in the target image using a reference image, and frames in which the movement distance of the server's centroid in each frame exceeds a threshold are then detected as outliers. The "centroid movement distance" for a given frame is, for example, the difference between the position of the centroid in the frame immediately before that frame and the position of the centroid in that frame.

[0042] The determination device 100 may interpolate the position of a server in a frame detected as an outlier by averaging the coordinates of the bounding boxes of the frames before and after the frame in which the server is determined to be in a normal position. This makes it possible to estimate the position of the server in each frame, enabling determination even in shooting conditions where the passage is narrow.

[0043] The device configuration and operation of this embodiment will be described in detail below.

[0044] (Device configuration example) Fig. 4 is a diagram showing an example of the configuration of a determination device 100 according to this embodiment. As shown in Fig. 4, the determination device 100 includes an image generation unit 110, an alignment unit 120, a state determination unit 130, and a storage unit 140. The determination device 100 may be a single physical device, or may be comprised of multiple devices. The determination device 100 may also be called a determination system.

[0045] An example of the operation of the determination device 100 will be described below as an example. In this embodiment, it is assumed that a data center or the like is equipped with one or more racks, and each rack is equipped with one or more devices. The devices are the monitoring targets. For example, one or more LED lamps are visible on the front or back panel of the device, and the state of the device can be determined based on the state of the LED lamps. Note that the type of lamp is not limited to LEDs. The technology according to the present invention can also be applied to lamps other than LEDs.

[0046] The equipment mounted in the rack is not limited to a specific equipment. The equipment mounted in the rack may be a server, a switch, or other equipment. Furthermore, the technology according to the present invention can be applied even when the equipment is not mounted in a rack. However, in this embodiment, the target is a server mounted in a rack. Furthermore, the term "front" used below means the side that is monitored, and may be the front or rear of the server.

[0047] In the following, basically, in each step, the data of the processing result is stored as output in storage unit 140, and in the next step, the data is read from storage unit 140 and used as input. In the final step, the processing result is output to the outside.

[0048] (Example) The processing according to this embodiment will be explained along the steps of the flowchart in FIG.

[0049] <S101(ステップ101> In S101, as a preliminary step, images of the front panel of each server model are taken with a camera, and annotations are added to the images for each model. The servers in the images are the servers whose states are to be estimated.

[0050] Specifically, the server information and the LED lamp position information are added to the image. The annotated image is called a template image.

[0051] Specifically, for example, if there are multiple LED lamps on server 1, the annotations include information such as "information about server 1, location information of LED lamp 1 in server 1, location information of LED lamp 2 in server 1, ... location information of LED lamp n in server 1."

[0052] The position information of the LED lamp is, for example, the x and y coordinates that indicate the rectangular area that includes the LED lamp. An example (image) of the template image is shown in Fig. 6. In Fig. 6, an image is shown in which three LED lamps are provided on the right side of each server.

[0053] If there is a net-like obstacle in front of the server whose status you want to determine, in advance preparation, you will acquire an image of the front of the server with the obstacle and an image of the server with the obstacle removed from the ``server with the obstacle.''

[0054] The template image, the annotations therefor, the image of the server with an obstacle, and the image of the server without an obstacle, which are obtained in advance, are stored in the storage unit 140 of the determination device 100.

[0055] <s102> In S102, a rack (server area) on which a server, the status of which is to be determined, is mounted is captured as a video by a camera, and the captured video (video file) is stored in the storage unit 140. The alignment unit 120 also extracts frames from the video stored in the storage unit 140. Each extracted frame (image of the server area whose status is to be determined) is stored in the storage unit 140. The image of the frame is called a "target image."

[0056] <s103> In S103, if there is a mesh-like obstacle in front of the server whose status is to be determined, the image generation unit 110 generates an image from each target image acquired in S102 in which the mesh-like obstacle has been removed. The image from which the mesh-like obstacle has been removed is used as the target image thereafter.

[0057] The image generation unit 110 corresponds to the GAN described above. The image generation unit 110 has been trained in advance using images with obstacles (a large number of images) and images without obstacles. In other words, the image generation unit 110 has been trained to generate an image without obstacles from an image with obstacles.

[0058] An image of the above process is shown in Figure 7. As shown in Figure 7(a), an image with an obstacle is input to the image generation unit 110, and the image generation unit 110 generates an image without the obstacle, as shown in Figure 7(b).

[0059] <s104> In S104, the alignment unit 120 reads the template image (and annotations) and the target image (e.g., the first frame) from the storage unit 140, and calculates a transformation matrix based on matching of feature points of the template image with feature points of the target image. This transformation matrix is ​​used to perform projective transformation of the template image so that the appearance of the server in the template image is the same as the appearance of the server in the target image.

[0060] The positioning unit 120 performs a projective transformation on the template image. As a result, the coordinates indicating the area of ​​the server in the template image and the coordinates indicating the area of ​​each LED lamp in the area of ​​the server are projectively transformed. The positioning unit 120 matches (aligns) the projectively transformed template image with the target image, thereby acquiring the position of the server in the template image and the positions of each LED lamp of the server in the target image. In S104, for each server in the target image, the server position and the LED lamp position are identified according to the annotations in the template image. For example, the position of Server 1 (e.g., model a, first row) is identified as Server Position 1 (e.g., four coordinates indicating a rectangular area), and LED Lamp 1 is identified as LED Position 1 (e.g., four coordinates indicating a rectangular area).

[0061] <s105> The alignment unit 120 performs tracking of each target image (each frame) using a rectangle representing the server in the projectively transformed template image (or a rectangle representing the server in the target image acquired by matching), and identifies the server ID in each target image. The tracking method is not limited to a specific method, but in this embodiment, a tracking method is used in which the centroid (center of gravity) of the rectangle representing the server is obtained.

[0062] For example, the rectangle of "model a 1st stage" (assuming it is server 1) in the projectively transformed template image (or the matched target image) is tracked, and a certain rectangular area A in frame 2 is identified as server 1, and a certain rectangular area B in frame 3 is identified as server 1. This process is performed for each frame.

[0063] By identifying the server (server ID) in the target image, the positioning unit 120 can identify the area of ​​the LED lamp in the area of ​​the server using the annotation in the template image and the coordinates of the LED lamp in the projectively transformed template image.

[0064] <s106> S106 corresponds to the second technique described above. Here, the target image for which tracking has been performed correctly is called the "reference image." Also, here, it is assumed that tracking has been lost in the target image (frame t+n). Note that the method for determining that tracking has been lost is not limited to a specific method. For example, as described above, it can be determined that tracking has been lost when the centroid has moved by more than a threshold value.

[0065] In S106, the combination unit 120 obtains a transformation matrix based on matching of frame feature points between the reference image (frame t) and the target image (frame t+n), and performs projective transformation on the reference image (frame t) using the transformation matrix. This enables the combination unit 120 to superimpose (match) the reference image (frame t) on the target image (frame t+n), and obtains the position of the server and the position of the LED lamp in the target image (frame t+n) based on the server ID in the reference image (frame t).

[0066] For the target image of each frame from frame t+n onwards, matching based on the reference image (frame t) may be performed, or tracking may be performed.

[0067] <s107> Through the processes of S104 to S106, the positioning unit 120 obtains the estimated position of the LED lamp of each server in each target image.

[0068] In S107, the alignment unit 120 receives the estimation result of the LED lamp area in the target image, cuts out the image of the LED lamp area from the target image, and outputs the cut-out result. This area cut-out is performed for each LED lamp. However, one area may include multiple LED lamps.

[0069] The cut-out results (cut-out areas) are obtained in the form of, for example, "LED lamp 1: (x1, y1, x2, y2)" and "LED lamp 2: (x3, y3, x4, y4)", where (x1, y1, x2, y2) is an example of coordinate information indicating the area, and indicates the x and y coordinates of both ends of the diagonal line of the rectangle.

[0070] <s108> In S108, the state determination unit 130 uses the cut-out result (image of the cut-out area) obtained in S107 as input and performs color classification on the image of the cut-out area.

[0071] As more detailed processing, there are the following Example 1 and Example 2, and each will be described. In any of the examples, one cut-out area will be taken as an example for explanation.

[0072] <S108: Example 1> In Example 1, the state determination unit 130 performs binarization of the image of the cut-out area based on, for example, the color ranges of four colors (e.g., green, red, orange, blue), and classifies the color based on whether the ratio of a certain color in the cut-out area is greater than a threshold value. For example, in the cut-out area, if the area of the part determined to be "green" has a ratio equal to or greater than the threshold value with respect to the area of the cut-out area, the color of the LED lamp corresponding to the area is classified as "green".

[0073] The above processing may be performed for each single image (image of one frame), or may be performed for a predetermined number of images. When performing for a predetermined number of images, the above processing is performed on the image obtained by averaging the predetermined number of images.

[0074] <S108: Example 2> In Example 2, it is assumed that the state determination unit 130 has a deep learning-based object detector. For each frame, the state determination unit 130 uses the object detector to acquire the rectangle of the LED lamp and detect each color of the LED lamp.

[0075] Also, for each frame, the state determination unit 130 calculates the distance between the center point of the area of the LED lamp acquired in S107 and the center point of the area of the LED lamp obtained by the object detector, and performs matching between the bounding boxes (areas of the LED lamps). Based on this result, the ID of the LED lamp and the detection result of its color are associated.

[0076] In S108, the state determination unit 130 obtains the cut-out region and the color classification result for the cut-out region.

[0077] <s109> In S109, the state determination unit 130 determines the lighting state (steady on / flashing / off) of the LED lamp for each target image (frame) and each cutout area using as input the color classification result for the cutout area (e.g., LED lamp 1 is lit green) and the coordinates of the cutout area (e.g., (x1, y1, x2, y2)), and outputs the color classification result for the cutout area and the lighting state determination result of the LED lamp. Specifically, this is as follows. The following process is performed for each cutout area.

[0078] Since the LED lamp blinks periodically, if the state determination unit 130 detects that the area of ​​the LED lamp (=extracted area) for a predetermined number of frames out of a preset number of frames is classified into one of the colors, the state determination unit 130 determines that the LED lamp is blinking. For example, if 50 frames out of 100 frames are classified as green and the remaining 50 frames are not classified into any color (or are classified as not lit), the determination unit 180 determines that the LED lamp is blinking.

[0079] Furthermore, if the state determination unit 130 detects that the cutout region is classified into any color in all of the frames of a preset number of frames, it determines that the corresponding LED lamp is constantly lit.

[0080] Furthermore, when the state determination unit 130 detects that there is a cutout area (= the coordinates of the LED lamp have been estimated) but there is no color classification result for that area, it determines that the LED lamp corresponding to that area is off.

[0081] <s110> In S110, the status determination unit 130 receives the color classification results and the lighting state determination results of the LED lamps for each cut-out area (or for each LED lamp), determines the state of each LED lamp based on the input information, and outputs the determination result. The state of the LED lamp here means, for example, that the LED lamp is "flashing orange." The status determination unit 130 may also determine whether or not the server is abnormal based on the state of each LED lamp. The status determination unit 130 outputs the determination result.

[0082] (Example of device hardware configuration) The determination device 100 described in this embodiment can be realized, for example, by causing a computer to execute a program.

[0083] That is, the determination device 100 can be realized by using hardware resources such as a CPU and memory built into a computer to execute a program corresponding to the processing performed by the determination device 100. The program can be recorded on a computer-readable recording medium (such as a portable memory) and can be saved or distributed. The program can also be provided via a network such as the Internet or email.

[0084] Fig. 8 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 8 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B.

[0085] A program for realizing processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0086] The memory device 1003 reads and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes the functions related to the determination device 100 in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network or the like. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, or the like, and is used to input various operation instructions. The output device 1008 outputs the calculation results.

[0087] (Effects of the embodiment) As described above, the technology according to the present embodiment makes it possible to realize automatic monitoring of equipment with fewer resources. More specifically, it becomes possible to realize more accurate status monitoring regardless of the presence or absence of obstacles. This is expected to advance automation of monitoring, reduce labor costs, and improve monitoring efficiency.

[0088] The following additional notes are provided regarding the above-described embodiments.

[0089] <Additional Notes> (Additional note 1) A determination device for determining the state of a lamp mounted on a device, a positioning unit that identifies the device in each target image of the video by tracking a region including the device using a template image acquired by photographing the device in advance; a state determination unit that performs color classification of the lamp on an image of an area of ​​the lamp in the device and determines a state of the lamp based on a result of the color classification; A determination device comprising: (Additional note 2) The apparatus further includes an image generating unit that generates a target image without the obstacle from each target image of the video when an obstacle is present in front of the device, and the positioning unit identifies the device in each target image without the obstacle. Item 1. The determination device according to item 1. (Additional note 3) When the tracking fails in a specific target image, the positioning unit identifies the device in the specific target image by using a target image for which tracking has been successful. Item 1. The determination device according to item 1. (Additional note 4) The positioning unit generates a transformation matrix based on feature points of the template image and feature points of a certain target image, performs projective transformation on the template image using the transformation matrix, and performs the tracking using the template image after projective transformation. Item 1. The determination device according to item 1. (Additional note 5) The state determination unit performs color classification for each of one or more frames constituting the moving image, and determines the state of the lamp as one of a constant light, a flashing light, and an off light based on the result of the color classification. Item 1. The determination device according to item 1. (Additional note 6) A determination method executed by a determination device that determines the state of a lamp mounted on a device, comprising: a positioning step of identifying the device in each target image of the video by tracking the video of an area including the device using a template image acquired by photographing the device in advance; a state determination step of performing color classification of the lamp on an image of the area of ​​the lamp in the device and determining a state of the lamp based on a result of the color classification; A determination method comprising: (Additional note 7) A program for causing a computer to function as each unit in the determination device according to any one of appended claims 1 to 5.

[0090] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0091] 100 Judgment device 110 Image generation unit 120 Alignment section 130 Status determination unit 140 Storage section 1000 Drive Device 1001 Recording media 1002 Auxiliary storage 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output Device

Claims

1. A determination device for determining the state of a lamp mounted on a device, a positioning unit that identifies the device in each target image of the video by tracking a region including the device using a template image acquired by photographing the device in advance; a state determination unit that performs color classification of the lamp on an image of an area of ​​the lamp in the device and determines a state of the lamp based on a result of the color classification; A determination device comprising:

2. The apparatus further includes an image generating unit that generates a target image without the obstacle from each target image of the video when an obstacle is present in front of the device, and the positioning unit identifies the device in each target image without the obstacle. The determination device according to claim 1 .

3. When the tracking fails in a specific target image, the positioning unit identifies the device in the specific target image by using a target image for which tracking has been successful. The determination device according to claim 1 .

4. The positioning unit generates a transformation matrix based on feature points of the template image and feature points of a certain target image, performs projective transformation on the template image using the transformation matrix, and performs the tracking using the template image after projective transformation. The determination device according to claim 1 .

5. The state determination unit performs color classification for each of one or more frames constituting the moving image, and determines the state of the lamp as one of a constant light, a flashing light, and an off light based on the result of the color classification. The determination device according to claim 1 .

6. A determination method executed by a determination device that determines the state of a lamp mounted on a device, comprising: a positioning step of identifying the device in each target image of the video by tracking the video of an area including the device using a template image acquired by photographing the device in advance; a state determination step of performing color classification of the lamp on an image of the area of ​​the lamp in the device and determining a state of the lamp based on a result of the color classification; A determination method comprising:

7. A program for causing a computer to function as each unit in the determination device according to any one of claims 1 to 5.

Images