Abnormality detection device, abnormality detection method, and computer-readable recording medium

Abstract

The program causes the abnormality detection device (100) to function as the following units: an acquisition unit (11) that acquires continuous image data representing images that are continuously photographed; a parameter determination unit (12) that determines, based on the continuous image data, parameters for dividing the continuous image data in at least one of the width direction and the height direction of the image and in the time direction; a feature quantity calculation unit (14) that calculates a feature quantity from each of the fragmented data obtained by dividing the continuous image data using the parameters determined by the parameter determination unit (12); and a detection unit (15) that detects an abnormality related to the subject of the photographed image based on a comparison between the feature quantity calculated by the feature quantity calculation unit (14) and a reference value.

Description

Technical Field

[0001] The present invention relates to a computer-readable recording medium, an abnormality detection device, and an abnormality detection method. Background Art

[0002] In FA (Factory Automation) sites, when controlling instruments, the presence or absence of abnormalities is often monitored (for example, see Patent Document 1). Patent Document 1 describes a technique for detecting abnormalities of an object of inspection from image data of an object of inspection such as instruments and equipment on a production line that is continuously photographed. In this technique, each image is divided into a plurality of blocks by a grid, and abnormalities are detected using feature quantities extracted in block units. According to this technique, abnormalities of an object of inspection such as instruments and equipment on a production line can be detected from image data as abnormalities different from the quality of manufactured products.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2013-191185 Summary of the invention

[0004] In the technology of Patent Document 1, the user needs to set the size of the block divided from the image as a parameter. Therefore, when a user with little knowledge and experience sets the size of the block, the size of the block may be inappropriate, and there is a risk that anomalies cannot be accurately detected. In addition, even if the user has knowledge or experience, when the shooting environment has changed from the previous environment, it is unclear about the appropriate block size. In these cases, it is a heavy burden to investigate the appropriate block size through repeated trials, and to set parameters for the device for detecting anomalies from images.

[0005] The present invention has been made in view of the above-mentioned circumstances, and an object of the present invention is to reduce the burden of work of setting parameters for an apparatus for detecting abnormalities from images.

[0006] In order to achieve the above-mentioned purpose, the abnormality detection program of the present invention enables a computer to function as the following units: an acquisition unit, which acquires continuous image data representing continuously captured images; a determination unit, which determines, based on the continuous image data, parameters for dividing the continuous image data in at least one of the width direction and the height direction of the image and in the time direction; a feature quantity calculation unit, which calculates a first feature quantity from each of the fragmented data obtained by dividing the continuous image data using the parameters determined by the determination unit; and a detection unit, which detects abnormalities related to the subject of the captured image based on a comparison between the first feature quantity calculated by the feature quantity calculation unit and a reference value.

[0007] Effects of the Invention

[0008] According to the present invention, the determination unit determines the parameters for dividing the continuous image data in at least one of the width direction and the height direction of the image and the time direction based on the continuous image data. Therefore, it is not necessary for the user to conduct an operation of investigating the parameters for dividing the continuous image data by repeated experiments. Thus, it is possible to reduce the burden of the operation of setting parameters for the device for detecting abnormalities from images. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is a diagram showing a system including the abnormality detection device and the imaging device involved in the first embodiment.

[0010] Figure 2 This is a diagram showing the hardware configuration of the abnormality detection device according to the first embodiment.

[0011] Figure 3 This is a diagram showing an overview of the division of continuous image data achieved by the abnormality detection device according to the first embodiment.

[0012] Figure 4 This is a diagram showing the functional structure of the abnormality detection device involved in the first embodiment.

[0013] Figure 5 This is a diagram for explaining the determination of the fragment size involved in Implementation Example 1.

[0014] Figure 6 This is a diagram for explaining the determination of the fragmentation time length involved in the first embodiment.

[0015] Figure 7 This is a flowchart showing the abnormality detection process involved in the first embodiment.

[0016] Figure 8 This is a diagram showing an example of a screen showing an abnormality detection result according to the first embodiment.

[0017] Fig. 9 This is a diagram showing the functional structure of the abnormality detection device involved in the second embodiment.

[0018] Fig.10 This is a diagram showing compression of continuous image data according to the second embodiment.

[0019] Fig.11 This is a diagram showing the functional structure of the abnormality detection device involved in the third embodiment.

[0020] Fig.12 This is a diagram used to illustrate the determination of the fragment size based on the result of object detection involved in the third embodiment.

[0021] Fig.13 This is a diagram showing the functional structure of the abnormality detection device involved in the fourth embodiment.

[0022] Fig.14 This is a diagram showing the functional structure of the abnormality detection device involved in the fifth embodiment.

[0023] Fig.15 This is a diagram showing the functional structure of the abnormality detection device involved in the sixth embodiment.

[0024] Fig.16 This is a diagram used to illustrate the determination of fragment size involved in Implementation Example 7.

[0025] Fig.17 It is a diagram showing the division of continuous image data according to the first modification.

[0026] Fig.18 It is a diagram showing the division of continuous image data according to the second modification. DETAILED DESCRIPTION

[0027] Hereinafter, an abnormality detection device for executing an abnormality detection program according to an embodiment of the present invention will be described in detail with reference to the drawings.

[0028] Implementation Method 1

[0029] like Figure 1 As shown, the abnormality detection device 100 involved in this embodiment is a device that obtains images of the factory continuously photographed by the camera 200 and detects abnormalities from a series of images. The abnormality detection device 100 is an industrial PC (Personal Computer) configured in the factory that becomes the object of photography together with the camera 200. However, the abnormality detection device 100 is not limited to such an industrial PC, and can be a control device represented by a PLC (Programmable Logic Controller) in the factory or other FA devices, or a management computer or server device configured outside the factory.

[0030] The abnormality detection device 100 and the photographing device 200 are connected via a communication path capable of transmitting data representing an image. The communication path may be, for example, a dedicated line, an industrial network within a factory, a common information network, or a communication network represented by the Internet. In addition, the communication path may also implement either wired communication or wireless communication.

[0031] The object photographed by the photographing device 200 is a part of the area in the factory. Figure 1In the example, the belt conveyor, the workpiece conveyed on the belt conveyor, the robot arm that controls the conveyance of the workpiece, and the inspection machine and inspection table that inspect the workpiece are taken as the shooting objects, and the shooting instrument 200 with the angle of view adjusted is fixed directly above these shooting objects. The shooting instrument 200 periodically sends the images of the shooting object directly below to the abnormality detection device 100 in sequence. In addition, the shooting instrument 200 can also shoot outside the factory. For example, the shooting instrument 200 can also be a surveillance camera that shoots the entrance and exit of the factory from the outside, or a surveillance camera that shoots the management room outside the factory for managing the operation of the factory.

[0032] The abnormality detected from the image refers to a state that deviates from the range of normal operating state of the photographed object as imagined by the workers involved in the operation of the factory. Examples of such abnormalities include damage to the workpiece and failure of the belt conveyor, robot arm, and inspection machine. In addition, the workers may be, for example, managers, operators, and operators of the factory, or manufacturers of FA equipment represented by robot arms and inspection machines.

[0033] exist Figure 2 The hardware structure of the abnormality detection device 100 is schematically shown in FIG. Figure 2 As shown, the abnormality detection device 100 is a computer having a processor 101, a main storage unit 102, an auxiliary storage unit 103, an input unit 104, an output unit 105, and a communication unit 106. The main storage unit 102, the auxiliary storage unit 103, the input unit 104, the output unit 105, and the communication unit 106 are connected to the processor 101 via an internal bus 107.

[0034] The processor 101 includes an integrated circuit represented by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The processor 101 implements various functions and performs the processing described below by executing a program P1 stored in the auxiliary storage unit 103. The program P1 corresponds to an example of an abnormality detection program.

[0035] The main storage unit 102 includes a RAM (Random Access Memory). The program P1 is loaded from the auxiliary storage unit 103 into the main storage unit 102. The main storage unit 102 is used as a work area for the processor 101.

[0036] The auxiliary storage unit 103 includes a nonvolatile memory represented by an EEPROM (Electrically Erasable Programmable Read-Only Memory) and an HDD (Hard Disk Drive). The auxiliary storage unit 103 stores various data used for processing by the processor 101 in addition to the program P1. The auxiliary storage unit 103 supplies the data used by the processor 101 to the processor 101 according to the instruction of the processor 101. In addition, the auxiliary storage unit 103 stores the data supplied from the processor 101.

[0037] The input unit 104 includes input devices represented by hardware switches, input keys, a keyboard, and a pointing device. The input unit 104 acquires information input by an operator using the abnormality detection device 100 and notifies the processor 101 of the acquired information.

[0038] The output unit 105 includes a display device represented by an LED (Light Emitting Diode) and an LCD (Liquid Crystal Display), and an output device represented by a buzzer and a speaker. The output unit 105 presents various information to the user according to the instruction of the processor 101.

[0039] The communication unit 106 includes an interface circuit for communicating with an external device. The communication unit 106 receives a signal from an external device and outputs data indicated by the signal to the processor 101. The communication unit 106 may also send a signal indicating data output from the processor 101 to an external device.

[0040] By cooperating with the above hardware structure, the abnormality detection device 100 divides the time-series image, that is, the continuous image data, into a plurality of fragmented data, and detects the presence or absence of abnormality for each of the fragmented data. Figure 3 The following describes an overview of the division of continuous image data by the abnormality detection device 100 .

[0041] like Figure 3 As shown, a series of image data including images 31, 32, and 33 captured continuously are collected and referred to as continuous image data 30. Each of the images 31 to 33 is a rectangular image having a width and a height. The continuous image data 30 obtained by arranging these images 31 to 33 in a time series order is three-dimensional data having a length in the time direction in addition to the width direction and the height direction.

[0042] Parameters for dividing the continuous image data 30 in the width direction, height direction, and time direction of the images 31 to 33 are determined. Figure 3In the example of , the parameters for dividing the continuous image data 30 into 4 in the width direction, 2 in the height direction, and 2 in the time direction are determined. The parameters represent the values ​​of the width, height, and length in the time direction of each of the fragmented data 40 obtained by dividing the continuous image data 30.

[0043] For example, in Figure 3 In the example, if the width of each of the images 31 to 33 is 960 pixels, the value of the parameter representing the width of the fragment data 40 is determined to be 240 pixels. Similarly, if the height of each of the images 31 to 33 is 720 pixels, the value of the parameter representing the height of the fragment data 40 is determined to be 360 ​​pixels. In addition, if the length of the continuous image data 30 in the time direction is 240 frames, the value of the parameter representing the length of the fragment data 40 in the time direction is determined to be 120 frames. However, the unit of the length of the fragment data 40 in the time direction may also be milliseconds.

[0044] The fragment data 40 divided from the continuous image data 30 is three-dimensional data like the continuous image data 30 , but corresponds to data that is smaller than the continuous image data 30 in any of the width direction, the height direction, and the time direction.

[0045] The abnormality detection device 100 has a Figure 3 The segmentation shown in FIG. 1 and the function of detecting anomalies based on the fragmented data obtained by segmentation. Figure 4 As shown, the abnormality detection device 100 has, as its functions: an acquisition unit 11, which acquires continuous image data from the photographing instrument 200; a parameter determination unit 12, which determines parameters for segmenting the continuous image data based on the continuous image data; a segmentation unit 13, which segments the continuous image data using the determined parameters; a feature value calculation unit 14, which calculates a feature value from each of the fragmented data obtained by segmenting the continuous image data; a detection unit 15, which detects an abnormality using the calculated feature value; and an output unit 16, which outputs the detection result of the abnormality to the outside.

[0046] The acquisition unit 11 is mainly implemented by the cooperation of the processor 101, the main storage unit 102 and the communication unit 106. The acquisition unit 11 receives continuous image data representing images continuously captured by the camera 200 from the camera 200, and outputs the received continuous image data to the parameter determination unit 12 and the segmentation unit 13. In addition, the continuous image data acquired by the acquisition unit 11 may be one block of data, or a collection of image data representing images sent sequentially from the camera 200, or video data sent from the camera 200 in a streaming format. In addition, the acquisition unit 11 may also perform image resizing or noise removal processing as needed. The acquisition unit 11 is equivalent to an example of an acquisition unit that acquires continuous image data in the abnormality detection device 100.

[0047] The parameter determination unit 12 is mainly implemented by the cooperation of the processor 101 and the main storage unit 102. The parameter determination unit 12 determines the values ​​of the parameters representing the sizes of the fragment data in the width direction, the height direction, and the time direction based on the information indicated by the continuous image data. Here, the sizes of the fragment data in the width direction and the height direction are referred to as the fragment size, and the length of the fragment data in the time direction is referred to as the fragment time length.

[0048] The fragment size is determined as a rectangular area including a minimum block area among block areas detected by comparing image feature quantities of an image constituting continuous image data with a threshold value. As the image feature quantity, for example, at least one of a pixel value, a gradient quantity, an optical flow, and a HOG (Histograms of Oriented Gradients) feature quantity is used.

[0049] Specifically, if Figure 5 As shown, the parameter determination unit 12 calculates the image feature quantity for each local area constituting each image constituting the continuous image data. The local area can be one pixel or a unit area composed of a plurality of adjacent pixels. Moreover, the parameter determination unit 12 calculates the image feature quantities having values ​​similar to each other, and detects the block area as a collection of local areas adjacent to each other. The detection of image feature quantities similar to each other is performed by using a predetermined threshold. In addition, since the detection of image feature quantities similar to each other can be called the classification of image feature quantities, it can also be performed by clustering the image feature quantities. The parameter determination unit 12 determines the fragment size in a manner connected to the block area of ​​the smallest size among the block areas detected from each image. However, instead of the smallest block area, a block area located in the lower 10% when the block areas are arranged in order from large to small in size can be used, and the fragment size can also be determined by setting a margin relative to the smallest block area.

[0050] Here, the image feature quantity corresponds to an example of the second feature quantity. In addition, the parameter determination unit 12 corresponds to an example of a determination unit that classifies the second feature quantity by calculating the second feature quantity for each of the local areas constituting the image, thereby detecting a set of local areas having the second feature quantity classified into the same group and adjacent to each other, that is, a block area, and determines parameters representing the width and height of the fragment data in such a manner that the fragment data includes a block area selected from the detected block areas.

[0051] like Figure 6 As shown, the fragment time length is determined by selecting a representative period from the frequency domain representation of the transition of the image feature quantity that changes along the time axis. The image feature quantity used to determine the fragment time length may be the same as or different from the image feature quantity used to determine the fragment size. In addition, the image feature quantity used to determine the fragment time length may have only one value for each image. The frequency domain representation may also be obtained by other transformations such as Fourier transform and wavelet transform. The representative period may be a period corresponding to the peak of the frequency domain representation, or a period selected by other methods.

[0052] The image feature quantity used to determine the fragment time length corresponds to an example of the third feature quantity. In addition, the parameter determination unit 12 corresponds to an example of a determination unit that determines a parameter representing the length of the fragment data in the time direction by selecting a period from the frequency domain representation of the transition of the third feature quantity calculated from the image.

[0053] The segmentation unit 13 is mainly realized by the cooperation of the processor 101 and the main storage unit 102. The segmentation unit 13 generates a fragmentation data group including a plurality of fragmentation data by segmenting the continuous image data using the parameters determined by the parameter determination unit 12. The generated fragmentation data group is output to the feature quantity calculation unit 14.

[0054] The feature quantity calculation unit 14 is mainly implemented by the cooperation of the processor 101 and the main storage unit 102. The feature quantity calculation unit 14 calculates the feature quantity for each fragment data, thereby generating a feature quantity data group including a plurality of calculated feature quantities. The feature quantity is, for example, at least one of optical flow, HOG feature quantity, KAZE feature quantity, and learning-based feature quantity using deep learning. As a feature quantity, only one type can be used, or a plurality of feature quantities can be combined. In addition, the type of feature quantity or its combination can be different for each fragment data. The feature quantity calculated by the feature quantity calculation unit 14 is equivalent to an example of the first feature quantity. In addition, the feature quantity calculation unit 14 is equivalent to an example of a feature quantity calculation unit in the abnormality detection device 100, which calculates the first feature quantity from each of the fragment data obtained by segmenting the continuous image data using the parameters determined by the determination unit.

[0055] The detection unit 15 is mainly realized by the cooperation of the processor 101, the main storage unit 102, and the auxiliary storage unit 103. The detection unit 15 corresponds to an example of a detection unit in the abnormality detection device 100, and the detection unit detects an abnormality related to the photographed object of the photographed image based on the comparison between the first feature quantity calculated by the feature quantity calculation unit and the reference value. The detection unit 15 has: a difference calculation unit 151, which compares the feature quantity and the reference value to calculate the difference; a reference feature quantity storage unit 152, which stores the reference feature quantity corresponding to the reference value; a diagnosis unit 153, which diagnoses the presence or absence of abnormality by determining whether the difference exceeds a threshold value; and a threshold setting unit 154, which is used to set the threshold used for diagnosis.

[0056] The difference calculation unit 151 compares each of the feature quantities constituting the feature quantity data group with the reference feature quantity, and calculates the difference representing the degree of difference between each of the fragment data, thereby generating a difference data group including a plurality of calculated differences. The difference may be a similarity represented by histogram similarity and cosine similarity, or a distance between feature quantities represented by Euclidean distance, Manhattan distance, Chebyshev distance, and Mahalanobis distance, or a feature quantity calculated by deep learning represented by a twin network. In addition, it is preferred that the reference feature quantity differs depending on the position on the image of the fragment data. The difference calculation unit 151 is equivalent to an example of a calculation unit in the abnormality detection device 100, which calculates the difference between the first feature quantity of each of the fragment data and the reference value.

[0057] In the baseline feature storage unit 152, a collection of baseline feature values, i.e., a baseline feature value data group, is pre-stored. The baseline feature values ​​are calculated in advance from fragmented data obtained by segmenting continuous image data of a normal operating state of a photographed object. The baseline feature value data group generated for at least one combination of fragment size and fragment time length may be stored in the baseline feature storage unit 152, and the baseline feature value data group generated for multiple combinations of different fragment sizes and fragment time lengths may be stored. Preferably, the baseline feature value data group supplied from the baseline feature storage unit 152 to the difference calculation unit 151 is pre-generated for the fragment size and fragment time length of the fragment data segmented by the segmentation unit 13.

[0058] The diagnosis unit 153 compares each of the differences constituting the difference data group with the threshold value supplied from the threshold setting unit 154, and determines whether the difference exceeds the threshold value, thereby diagnosing whether there is an abnormality. Then, the diagnosis unit 153 generates a diagnosis result data group representing the diagnosis result for each fragment data. It is predetermined that when the difference is greater than the threshold value, it is determined to be abnormal, or when the similarity equivalent to the difference is less than the threshold value, it is determined to be abnormal. Preferably, the threshold value compared with the difference is different depending on the position of the fragment data on the image, and preferably, the comparison of the difference and the threshold is performed for each fragment data with different positions on the image. The diagnosis unit 153 is equivalent to an example of a determination unit in the abnormality detection device 100 that determines whether the difference calculated by the calculation unit exceeds the threshold value.

[0059] The threshold setting unit 154 is used to set a threshold data set, which is a set of thresholds predefined for each fragment data. The setting of the threshold data set can be performed by the user, or it can be calculated as a statistic of the feature quantity of local data at a position or time on the image of the continuous image data that is the object of abnormal detection. The feature quantity can also be, for example, an optical flow, a HOG feature quantity, a KAZE feature quantity, or a feature quantity based on deep learning. In addition, the statistic of the feature quantity can also be calculated from, for example, the average, central value, most frequent value, variance, standard deviation, maximum value, and minimum value of the feature quantity.

[0060] The output unit 16 is mainly realized by the output unit 105. The output unit 16 reports the abnormality detection result obtained by the detection unit 15 to the user.

[0061] Next, use Figure 7, the abnormality detection process performed by the abnormality detection device 100 is described. The abnormality detection process is started by a specific operation of the user on the abnormality detection device 100. The specific operation can be, for example, the operation of a hardware switch or the execution of a specific application software. The abnormality detection process is equivalent to an example of an abnormality detection method performed by the abnormality detection device 100.

[0062] In the abnormality detection process, the acquisition unit 11 acquires continuous image data (step S1). The acquisition unit 11 may acquire the continuous image data from the imaging device 200, or may acquire the continuous image data by referring to the address of the auxiliary storage unit 103, a removable recording medium represented by a memory card, or an external server device specified by the user and reading the continuous image data 30.

[0063] Next, the parameter determination unit 12 uses the continuous image data to determine the parameters for segmenting the continuous image data in the width direction, height direction, and time direction (step S2). Here, the parameters may represent the size of the fragment data in the width direction, height direction, and time direction. For example, the parameters may represent the size of the fragment data in the width direction, height direction, and time direction. Figure 3 The positions of the boundaries corresponding to the bold lines for dividing the continuous image data 30 shown may also indicate the number of divisions when the continuous image data 30 is divided equally. The determined parameters are used for the division of the continuous image data by the division unit 13. Thus, a plurality of fragment data are obtained.

[0064] Next, the feature amount calculation unit 14 calculates the feature amount from each of the fragment data obtained by dividing the continuous image data using the parameters (step S3 ). Thus, the feature amounts equal in number to the number of the fragment data are calculated.

[0065] Next, the detection unit 15 selects one unselected fragment data (step S4). Here, the method of selecting the fragment data is arbitrary, for example, Figure 3 The coordinate values ​​in the predetermined width direction, height direction, and time direction shown are preferentially selected in this order for the fragment data having the smaller coordinate values.

[0066] Next, the difference calculation unit 151 of the detection unit 15 calculates the difference between the feature amount calculated in step S3 and the reference feature amount read from the reference feature amount storage unit 152 for the fragment data selected in step S4 (step S5 ).

[0067] Next, the diagnosis unit 153 of the detection unit 15 determines whether the difference calculated in step S5 exceeds the threshold value set by the threshold value setting unit 154 (step S6). However, when the similarity indicating the degree of similarity of the feature quantities is used as the difference, in step S6, it may be determined whether the similarity is lower than the threshold value.

[0068] When it is determined that the difference exceeds the threshold value (step S6; Yes), the detection unit 15 detects the abnormality, and the output unit 16 displays the detection result (step S7). Figure 8 As shown, the output unit 16 displays a screen in which the area corresponding to the fragmented data in which the abnormality is detected is emphasized.

[0069] When it is determined that the difference does not exceed the threshold value (step S6; No), or after step S7 is completed, the detection unit 15 determines whether all the fragment data divided from the continuous image data are selected (step S8). When it is determined that all the fragment data are not selected (step S8; No), the processing of step S4 and thereafter is repeated. On the other hand, when it is determined that all the fragment data are selected (step S8; Yes), the abnormality detection processing is completed.

[0070] Furthermore, the abnormality detection process may not be terminated after the process related to one continuous image data is completed, but the process related to the next continuous image data may be executed repeatedly. Figure 7 The abnormality detection process shown is only an example, and the order of the steps constituting the abnormality detection process may be changed arbitrarily. For example, instead of executing steps S5 to S7 for a selected piece of fragment data, after calculating the difference for all the fragment data and comparing all the differences with the threshold, the abnormality detection result may be displayed.

[0071] As described above, the parameter determination unit 12 according to the present embodiment determines the parameters for dividing the continuous image data in the width direction, height direction and time direction of the image based on the continuous image data. Therefore, it is not necessary for the user to conduct an operation of investigating the parameters for dividing the continuous image data by repeated experiments. Thus, it is possible to reduce the burden of the operation of setting parameters for the device for detecting abnormalities from images.

[0072] In addition, since appropriate parameters can be set in advance, the preparation time for operating the abnormality detection device 100 can be shortened. In addition, there is a risk that the abnormality cannot be accurately detected if the parameters set by the user are inappropriate. According to the abnormality detection device 100 involved in this embodiment, it is expected that the abnormality can be accurately detected. Thus, even users without knowledge and experience can easily and accurately detect abnormalities.

[0073] The parameter determination unit 12 detects the block area and determines the fragment size so that the fragment data includes one block area selected from the detected block area. By setting the fragment size as a group of local areas having similar feature values, it is possible to avoid making the fragment size smaller than necessary.

[0074] Furthermore, the parameter determination unit 12 determines the fragmentation time length by selecting a period from the frequency domain representation of the transition of the feature quantity, thereby avoiding determining a fragmentation time length that is longer than necessary.

[0075] Implementation Method 2

[0076] Next, Embodiment 2 will be described with a focus on differences from Embodiment 1. In addition, the same reference numerals are used for the same or equivalent configurations as those of Embodiment 1. This embodiment differs from Embodiment 1 in that the amount of calculation is reduced by compressing continuous image data.

[0077] like Fig. 9 As shown in FIG. 1 , the abnormality detection device 100 according to the present embodiment includes a compression unit 17, which compresses continuous image data by reducing the image size and lowering the frame rate. The compression unit 17 is mainly implemented by the cooperation of the processor 101 and the main storage unit 102. The compression unit 17 calculates compression parameters indicating how to compress the continuous image data based on the continuous image data acquired by the acquisition unit 11 and the parameters determined by the parameter determination unit 12. The compression parameters include an image reduction parameter related to the reduction of the image size and a time reduction parameter for shortening the length in the time direction.

[0078] Specifically, the compression unit 17 performs a reduction process on the feature amount of the local area used when calculating the fragment size, and calculates an image reduction parameter for reducing the image size within a range that does not affect the change of the block shape calculated from the feature amount, the disappearance of the block, etc. For example, the compression unit 17 repeatedly increases the reduction rate of the image size and detects the block area, and determines the reduction rate when the block area is about to disappear as the image reduction parameter.

[0079] In addition, the compression unit 17 calculates the maximum speed in the continuous image data from the optical flow, and sets the time reduction parameter so as to obtain a frame rate greater than or equal to the speed. For example, the compression unit 17 selects the maximum speed among the speeds indicated by the optical flow, and calculates the time reduction parameter for reducing the length of the continuous image data in the time direction within a range in which the optical flow corresponding to the maximum speed does not disappear.

[0080] Then, the compression unit 17 compresses the continuous image data using the calculated compression parameter, and outputs the compressed continuous image data to the segmentation unit 13. The segmentation unit 13 segments the compressed continuous image data into fragment data, and the feature amount calculation unit 14 calculates the feature amount from the fragment data obtained by segmenting the compressed continuous image data.

[0081] As described above, the compression unit 17 compresses the continuous image data based on the size of the block area, and the feature amount calculation unit 14 calculates the feature amount from the fragmented data obtained by dividing the continuous image data compressed by the compression unit 17. Therefore, if the continuous image data output from the acquisition unit 11 is processed as it is, the processing time becomes huge. In contrast to this, the image size is reduced to a degree that does not affect the detection of abnormalities and the frame rate is reduced. Fig.10 As shown, continuous image data is compressed, thereby shortening the processing time.

[0082] In addition, an example in which the compression unit 17 reduces the image size and reduces the frame rate has been described, but the compression unit 17 may perform only one of the image size reduction and frame rate reduction. In addition, the compression unit 17 may compress the image size only in one of the width direction and the height direction of the image. The compression unit 17 may compress the continuous image data in at least one of the width direction and the height direction of the image based on the size of the block area. The compression unit 17 in the abnormality detection device 100 corresponds to an example of a compression unit that compresses the continuous image data in at least one of the width direction and the height direction of the image based on the size of the block area.

[0083] Furthermore, the example in which the compression unit 17 is inserted between the acquisition unit 11 and the division unit 13 according to the first embodiment has been described, but the present invention is not limited thereto. For example, the compression unit 17 may be inserted between the division unit 13 and the feature amount calculation unit 14 to compress the fragmented data.

[0084] Implementation 3

[0085] Next, Embodiment 3 will be described with a focus on the differences from Embodiment 1. In addition, the same reference numerals are used for the same or equivalent structures as those in Embodiment 1. This embodiment differs from Embodiment 1 in that the parameters for segmenting the continuous image data are determined based on the detection results of the objects in the captured images instead of the image feature quantities.

[0086] like Fig.11As shown in FIG. 1 , the abnormality detection device 100 according to the present embodiment includes an object detection unit 18 for detecting an object in a captured image. The object detection unit 18 is mainly implemented by the cooperation of the processor 101 and the main storage unit 102. The object detection unit 18 detects each of the images constituting the continuous image data, such as Fig.12 Objects are detected as shown by the bold frame in FIG. 1 , and a detection result data group representing all detected objects is output to the parameter determination unit 12 .

[0087] The detection of the object is achieved by a method using a feature represented by a Haar-like feature and a HOG feature, or a deep learning-based method represented by a YOLO (You Look Only Once) algorithm and an SSD (Single Shot multi-box Detector) algorithm. The detection result indicates the coordinates of the detected object in the image, and the width and height of the area of ​​the photographed object. The detection result may further include other information such as classification level information and reliability. The object detection unit 18 in the abnormality detection device 100 is equivalent to an example of an object detection unit that detects an object in a photographed image.

[0088] The parameter determination unit 12 acquires the continuous image data and the detection result data group and determines the parameters for segmenting the continuous image data. For example, the parameter determination unit 12 determines the size of the smallest object detected or the average size of all detected objects as the fragment size.

[0089] As described above, the abnormality detection device 100 includes the object detection unit 18, and the parameter determination unit 12 determines the parameters representing the width and height of the fragment data based on the size of the region in the image in which the object detected by the object detection unit 18 is captured. Thus, the fragment size corresponding to the size of the object in the captured image is determined, and accurate abnormality detection is expected.

[0090] Implementation 4

[0091] Next, Embodiment 4 will be described with a focus on differences from Embodiment 3. In addition, the same reference numerals are used for the same or equivalent configurations as those in Embodiment 3. This embodiment differs from Embodiment 3 in that the amount of calculation is reduced by compressing continuous image data.

[0092] like Fig.13As shown in FIG. 1 , the abnormality detection device 100 according to the present embodiment includes a compression unit 17a for compressing continuous image data by reducing the image size. The compression unit 17a is mainly implemented by the cooperation of the processor 101 and the main storage unit 102. The compression unit 17a calculates a compression parameter indicating how to reduce the image constituting the continuous image data based on the detection result obtained by the object detection unit 18.

[0093] Specifically, the compression unit 17a calculates a compression parameter from a corrected image size obtained by multiplying a predetermined coefficient by the inverse of the number of pixels of the region where the smallest object among the objects detected by the object detection unit 18 is captured. Here, the ratio of the image size before compression to the corrected image size is the compression parameter.

[0094] The compression unit 17a corresponds to an example of a compression unit in the abnormality detection device 100, and the compression unit compresses the continuous image data based on the size of the area in the image in which the object detected by the object detection unit is captured. The compression unit 17a outputs the compressed continuous image data to the segmentation unit 13 using the determined compression parameter. The segmentation unit 13 segments the compressed continuous image data, and the feature amount calculation unit 14 calculates the feature amount from the fragmented data obtained by segmenting the compressed continuous image data.

[0095] As described above, the abnormality detection device 100 includes the compression unit 17a, and the feature amount calculation unit 14 calculates the feature amount from each of the fragmented data obtained by dividing the continuous image data compressed by the compression unit 17a. This can reduce the amount of calculation of the abnormality detection device 100.

[0096] Furthermore, the compression unit 17a may reduce one or both of the width and height of the image. The compression unit 17a may compress the continuous image data in at least one of the width direction and the height direction of the image.

[0097] Implementation method 5

[0098] Next, Embodiment 5 will be described with a focus on the differences from Embodiment 3. In addition, the same reference numerals are used for the same or equivalent structures as those in Embodiment 3. This embodiment differs from Embodiment 3 in that an object moving over time is detected, tracked on an image, and parameters are determined based on the tracking results.

[0099] like Fig.14As shown, the abnormality detection device 100 according to the present embodiment has an object tracking unit 19 for tracking an object captured in an image. The object tracking unit 19 is mainly implemented by the cooperation of the processor 101 and the main storage unit 102. The object tracking unit 19 tracks the object based on the continuous image data and the result of object detection, and outputs a tracking result data group indicating the tracking results of all objects.

[0100] The object tracking may be performed by a detection-based tracking method using the k-nearest neighbor method, or by using a particle filter or template matching method using the position of the object detected initially as the initial position, or by other methods. The object tracking unit 19 in the abnormality detection device 100 corresponds to an example of an object tracking unit that tracks an object captured by continuously captured images.

[0101] The parameter determination unit 12 obtains the continuous image data, the object detection results and the object tracking results, and determines the parameters for segmenting the continuous image data. Specifically, the parameter determination unit 12 determines the length of time that the object tracked after the longest time is captured as the fragmentation time length.

[0102] In addition, the parameter determination unit 12 may also set the time length from the moment when the object enters the image block corresponding to the fragment data to the moment when the object leaves the image block as the fragment time length. Here, the image block is equivalent to a local image divided according to the fragment size from the image constituting the continuous image data. In addition, the parameter determination unit 12 may set the minimum speed of the tracked object as Vmin, the preset maximum shooting distance as Mmax, and α as a predetermined coefficient, and calculate Pt as the fragment time length using a formula such as Pt=α·|Vmin| / Mmax.

[0103] As described above, the abnormality detection device 100 includes the object tracking unit 19, and the parameter determination unit 12 determines a parameter including a fragmentation time length, which indicates the length of the fragmentation data in the time direction, based on the object tracking result obtained by the object tracking unit 19. As a result, the fragmentation data includes an image block in which the photographed object is captured, and as a result, it is expected that an abnormality related to the object can be accurately detected. The parameter determination unit 12 is equivalent to an example of a determination unit, which determines a parameter indicating the length of the fragmentation data in the time direction based on the speed at which the object tracked by the object tracking unit moves in the image.

[0104] Implementation 6

[0105] Next, Embodiment 6 will be described with a focus on differences from Embodiment 5. In addition, the same reference numerals are used for the same or equivalent configurations as those of Embodiment 5. This embodiment differs from Embodiment 5 in that the amount of calculation is reduced by compressing continuous image data.

[0106] like Fig.15 As shown in FIG. 1 , the abnormality detection device 100 according to the present embodiment includes a compression unit 17b for compressing continuous image data by reducing the frame rate. The compression unit 17b is mainly implemented by the cooperation of the processor 101 and the main storage unit 102. The compression unit 17b calculates a compression parameter indicating how to shorten the length of the continuous image data in the time direction based on the object tracking result obtained by the object tracking unit 19.

[0107] Specifically, the compression unit 17b calculates the correction frame rate from the correction time length, which is obtained by multiplying the shortest time length among the time lengths during which the object being tracked is captured in each image by a predetermined coefficient, and calculates the compression parameter from the frame rate of the continuous image data before compression and the correction frame rate. The correction frame rate is the reciprocal of the correction time. Alternatively, the correction time length may be obtained by multiplying the time length from the moment when the object enters the image block corresponding to the fragment data to the moment when the object leaves the image block by a predetermined coefficient.

[0108] The compression unit 17b corresponds to an example of a compression unit in the abnormality detection device 100, and the compression unit compresses the continuous image data in the time direction based on the length of time when the object tracked by the object tracking unit is captured. The compression unit 17b outputs the compressed continuous image data to the segmentation unit 13 using the determined compression parameter. The segmentation unit 13 segments the compressed continuous image data, and the feature amount calculation unit 14 calculates the feature amount from the fragmented data obtained by segmenting the compressed continuous image data.

[0109] As described above, the abnormality detection device 100 includes the compression unit 17b, and the feature amount calculation unit 14 calculates the feature amount from each of the fragmented data obtained by dividing the continuous image data compressed by the compression unit 17b. This can reduce the amount of calculation of the abnormality detection device 100.

[0110] Implementation 7

[0111] Next, Embodiment 7 will be described with a focus on the differences from Embodiment 1. In addition, the same reference numerals are used for the same or equivalent structures as those in Embodiment 1. In Embodiment 1, each fragment data has a common fragment size. That is, the parameter determination unit 12 determines a set of parameters representing the width and height of the fragment data. In contrast, this embodiment is different from Embodiment 1 in that the parameter determination unit 12 determines the fragment size that varies depending on the position on the image as a parameter.

[0112] like Fig.16 As shown, the parameter determination unit 12 involved in this embodiment determines the parameters in a manner that the fragmented data has different fragment sizes depending on the position on the image, and outputs a parameter data set representing the fragment sizes of all the fragment data. The parameter data set represents the fragment sizes of the fragment data in association with the position on the image of the fragment data. In addition, the fragment size may have different values ​​corresponding to one or both of the horizontal direction corresponding to the width of the image and the vertical direction corresponding to the height. That is, the parameter may be determined so that it has a common value for each fragment data with respect to either the horizontal direction or the vertical direction.

[0113] As described above, the parameter indicating at least one of the width and height of one of the fragment data among the parameters determined by the parameter determination unit 12 has a different value from that of the other fragment data, and the position of the other fragment data in the image is different from that of the one fragment data. Thus, the fragment data is generated corresponding to the size on the image of the photographed object, and it is expected that the abnormality can be accurately detected.

[0114] For example, it is considered that the above-mentioned embodiment 3 is modified from the embodiment 1 to the present embodiment, and the fragment size is determined based on the position and size of the object detected by the object detection unit 18. In addition, the above-mentioned embodiment 5 may be modified in the same way, and the fragment size may be determined so that the fragment data includes as many objects tracked by the object tracking unit 19 as possible. Furthermore, the position on the image of the fragment data having the determined fragment size may be set by the user.

[0115] As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment.

[0116] For example, the example in which the continuous image data is divided in two directions, the width direction and the height direction of the image, is described, but the present invention is not limited thereto. The parameter determination unit 12 may determine the parameters for dividing the continuous image data in at least one of the width direction and the height direction of the image and the time direction based on the continuous image data.

[0117] Specifically, if Fig.17As shown, the abnormality detection device 100 may also divide the continuous image data in the width direction and the time direction instead of dividing the continuous image data in the height direction, thereby obtaining the fragmented data 40. Fig.17 As illustrated, when the segmentation is not performed in the height direction, the determination of the parameters for segmentation in the height direction and the segmentation in the height direction based on the parameters can be omitted from the above-mentioned embodiments. For example, when the subject of the captured image is arranged in the width direction but not in the height direction, or when the subject moves in the width direction but not in the height direction, the calculation load of the abnormality detection device 100 can be reduced by omitting the segmentation in the height direction.

[0118] In addition, if Fig.18 As shown, the abnormality detection device 100 may also divide the continuous image data in the height direction and the time direction instead of the width direction, thereby obtaining the fragmented data 40. In the case where the segmentation is not performed in the width direction, the determination of the parameters for segmentation in the width direction and the segmentation in the width direction based on the parameters may be omitted from the above-mentioned embodiments. For example, in the case where the subject of the captured image is arranged in the height direction but not in the width direction, by omitting the segmentation in the width direction, the calculation load of the abnormality detection device 100 can be reduced.

[0119] In addition, the method of determining the parameters implemented by the parameter determination unit 12 is not limited to the examples described in the above-mentioned embodiments, and any changes may be made.

[0120] In addition, for each fragment data Figure 3 Although the example of non-repetition as shown in FIG. 1 is described, a part of the fragmented data may be repeated. For example, Figure 3 The local data obtained by the division is divided into fragment data by setting a margin. At least a part of each fragment data may be located at a position different from other fragment data in at least one of the image and the time direction.

[0121] In addition, it is conceivable to provide a compression unit for compressing continuous image data in a manner different from the compression units 17, 17a, and 17b described in the above-mentioned embodiments 2, 4, and 6. For example, after the abnormality detection process is tried without compressing the continuous image data, the continuous image data may be compressed, and the abnormality detection process may be further tried, and the parameters for compressing the continuous image data may be adopted to such an extent that the detection accuracy is not affected. Specifically, the compression parameters may be determined in such a manner that the change in the detection accuracy falls within the range of a preset threshold value.

[0122] In addition, the above-mentioned embodiments may be arbitrarily combined. For example, the abnormality detection device 100 may include both the compression unit 17 according to the second embodiment and the compression unit 17 b according to the sixth embodiment.

[0123] The functions of the abnormality detection device 100 according to the above-described embodiment can also be realized by dedicated hardware or a general computer system.

[0124] For example, the program P1 may be stored in a computer-readable recording medium such as a floppy disk, a CD-ROM (Compact Disk Read-Only Memory), a DVD (Digital Versatile Disk), or an MO (Magneto-Optical Disk) and distributed, and installed in a computer, thereby forming a device that executes the above-mentioned processing.

[0125] Alternatively, the program P1 may be stored in advance in a disk device of a server device on a communication network such as the Internet, for example, superimposed on a carrier wave, and downloaded to a computer.

[0126] Furthermore, the above-mentioned processing can also be achieved by forwarding the program P1 via a network represented by the Internet and starting its execution.

[0127] Furthermore, the above-described processing can also be achieved by executing all or part of the program P1 on a server device, and by causing the computer to transmit and receive information related to the processing via a communication network and execute the program P1.

[0128] Furthermore, when the above functions are realized by sharing with the OS (Operating System) or by cooperation between the OS and the application, only the portion other than the OS may be stored in a medium and distributed, or may be downloaded to a computer.

[0129] In addition, the method of realizing the function of the abnormality detection device 100 is not limited to software, and a part or all of it may be realized by dedicated hardware or circuit.

[0130] The present invention can be set to various embodiments and variations without departing from the broad spirit and scope of the present invention. In addition, the above-mentioned embodiments are used to illustrate the present invention and are not intended to limit the scope of the present invention. That is, the scope of the present invention is not the embodiments, but is represented by the scope of the claims. Moreover, various variations applied within the scope of the claims and within the scope of the disclosure equivalent thereto are considered to be within the scope of the present invention.

[0131] Industrial Applicability

[0132] The present invention is suitable for abnormality detection using images.

[0133] Description of the label

[0134] 100 abnormality detection device, 11 acquisition unit, 12 parameter determination unit, 13 segmentation unit, 14 feature value calculation unit, 15 detection unit, 151 difference calculation unit, 152 reference feature value storage unit, 153 diagnosis unit, 154 threshold setting unit, 16 output unit, 17, 17a, 17b compression unit, 18 object detection unit, 19 object tracking unit, 101 processor, 102 main storage unit, 103 auxiliary storage unit, 104 input unit, 105 output unit, 106 communication unit, 107 internal bus, 30 continuous image data, 31 to 33 images, 40 fragmented data, 200 imaging instrument, P1 program.

Claims

1. A computer-readable recording medium storing an abnormality detection program for causing a computer to function as the following means: an acquisition unit that acquires continuous image data representing images that are continuously captured; a determination unit that determines, based on the continuous image data, parameters for dividing the continuous image data in at least one of a width direction and a height direction of the image and in a time direction; a feature quantity calculation unit that calculates a first feature quantity from each of fragmented data obtained by dividing the continuous image data using the parameter determined by the determination unit; as well as A detection unit detects an abnormality related to a subject of the image based on a comparison between the first feature amount calculated by the feature amount calculation unit and a reference value.

2. The computer-readable recording medium according to claim 1, wherein: The determination unit calculates the second feature value for each of the local areas constituting the image and classifies the second feature value, thereby detecting a set of the local areas that have the second feature value classified into the same group and are adjacent to each other, i.e., block areas, and determines the parameters representing the width and height of the fragment data in such a way that the fragment data includes one of the block areas selected from the detected block areas.

3. The computer-readable recording medium according to claim 1, wherein: The abnormality detection program further causes the computer to function as an object detection unit that detects an object captured in the image. The determination unit determines the parameters indicating the width and height of the fragment data based on the size of a region in the image in which the object detected by the object detection unit is captured.

4. The computer-readable recording medium according to any one of claims 1 to 3, wherein: The determination unit determines the parameter indicating the length of the fragment data in the time direction by selecting a period from a frequency domain expression of a transition of a third feature amount calculated from the image.

5. The computer-readable recording medium according to any one of claims 1 to 3, wherein: The abnormality detection program further causes the computer to function as an object tracking unit that tracks an object captured by the continuously captured images. The determination unit determines the parameter indicating the length of the fragment data in the time direction based on the speed at which the object tracked by the object tracking unit moves in the image.

6. The computer-readable recording medium according to any one of claims 1 to 3, wherein: The parameter representing at least one of the width and the height of one fragment data among the parameters determined by the determination unit has a value different from that of the parameter of other fragment data whose position in the image is different from that of the one fragment data.

7. The computer-readable recording medium according to claim 2, wherein: The abnormality detection program further causes the computer to function as a compression unit that compresses the continuous image data in at least one of a width direction and a height direction of the image based on the size of the block area. The feature amount calculation unit calculates the first feature amount from each of the fragmented data obtained by dividing the continuous image data compressed by the compression unit.

8. The computer-readable recording medium according to claim 3, wherein: The abnormality detection program further causes the computer to function as a compression unit that compresses the continuous image data in at least one of a width direction and a height direction of the image based on a size of an area in the image in which the object detected by the object detection unit is captured, The feature amount calculation unit calculates the first feature amount from each of the fragmented data obtained by dividing the continuous image data compressed by the compression unit.

9. The computer-readable recording medium according to claim 5, wherein: The abnormality detection program further causes the computer to function as a compression unit that compresses the continuous image data in a time direction based on the length of time when the object tracked by the object tracking unit is captured in the image, The feature amount calculation unit calculates the first feature amount from each of the fragmented data obtained by dividing the continuous image data compressed by the compression unit.

10. The computer-readable recording medium according to any one of claims 1 to 3, wherein: The detection unit has: a calculation unit that calculates a difference between the first feature value and the reference value of each of the fragment data; and a determination unit, configured to determine whether the difference calculated by the calculation unit exceeds a threshold value, When it is determined that the degree of difference exceeds the threshold, an abnormality is detected.

11. An abnormality detection device, comprising: an acquisition unit that acquires continuous image data representing images that are continuously captured; a determination unit that determines, based on the continuous image data, parameters for dividing the continuous image data in at least one of a width direction and a height direction of the image and in a time direction; a feature amount calculation unit that calculates a feature amount from each of fragmented data obtained by dividing the continuous image data using the parameter determined by the determination unit; as well as A detection unit detects an abnormality related to a subject of the image based on a comparison between the feature amount calculated by the feature amount calculation unit and a reference value.

12. A method for detecting anomalies, comprising the following steps: determining, based on the continuous image data, parameters for dividing the continuous image data representing the continuously captured images in at least one of the width direction and the height direction of the image and in the time direction; calculating a feature amount from each piece of fragmented data obtained by dividing the continuous image data using the determined parameter; and Based on the comparison between the feature amount and the reference value, an abnormality related to the object of the image is detected.

Images