Status detection device, status detection method, and status detection program

Abstract

To simply suppress the frequency at which a state of an observation object is erroneously recognized due to difference between a learning environment and an operation environment.SOLUTION: A storage part stores observation data inputted to an input part and a recognition result of a state of the observation object recognized, on the basis of a feature amount of the observation data, by a recognition model generated by machine learning in association with each other. When the recognition result is designated to be erroneous recognition, an extraction part extracts, on the basis of a feature amount of observation data having acquired the designated recognition result, other observation data having a possibility that the recognition result is erroneous recognition. When receiving an update permission of the recognition model, an update part updates a parameter to be used to determine a state of the observation object by the recognition model.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a technology for recognizing the state of an object to be observed based on observation data of the object to be observed. [Background technology]

[0002] Conventionally, there has been a device that recognizes the identity of a detection target and a known detection target (see Patent Document 1).

[0003] The device of Patent Document 1 uses machine learning to store, in a storage unit, information (spatial information) that constitutes a reference space (Mahalanobis space) generated from the feature quantities of each known detected object for each type of known detected object. The device acquires the feature quantities of the inputted detectable object. For each type of known detected object, the device calculates the distance (Mahalanobis distance) between the reference space of that type and the acquired feature quantities of the detectable object. The device recognizes which of the known detected objects the detectable object is based on the distance calculated for each type of known detected object.

[0004] The device of Patent Document 1 also cumulatively stores feature amounts acquired from detectable objects that have not been recognized as being one of the known detectable objects. This device also performs re-learning at appropriate times using the stored feature amounts of detectable objects that have not been recognized as being one of the known detectable objects, thereby improving the accuracy of detecting detectable objects. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-214682 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the device of Patent Document 1 does not consider the possibility of erroneous recognition of a detectable object due to a difference between the environment in which the learning data used in machine learning is acquired (learning environment) and the environment in which observation data of the detectable object to be recognized is acquired (operational environment). In other words, the device of Patent Document 1 cannot suppress erroneous recognition of a detectable object due to a difference between the learning environment and the operational environment.

[0007] Furthermore, the operating environment may change over time, and even if sufficient recognition accuracy is achieved at the start of operation, the recognition accuracy may decrease over time (the frequency of misrecognition increases).

[0008] An object of the present invention is to provide a technology that can easily reduce the frequency with which the state of an observation target is erroneously recognized due to differences between the learning environment and the operational environment. [Means for solving the problem]

[0009] In order to achieve the above object, the status detection device of the present invention is configured as follows.

[0010] Observation data of the observation target is input to the input unit. The observation target is an object whose state is to be recognized, such as a space, a person, an object, or a substance. The observation data is sensing data obtained by sensing the observation target with a sensor. The observation data may be, for example, a frame image of the observation target captured with a camera, audio data around the observation target collected with a microphone, or the temperature of the observation target measured with a temperature sensor. Furthermore, the observation data may be multiple pieces of observation data, rather than just one piece.

[0011] The storage unit stores, in association with each other, the observation data input to the input unit and a recognition result of the state of the observation target, which is recognized by a recognition model generated by machine learning based on the feature quantities of the observation data. The recognition model may have, for example, a variational autoencoder (VAE) trained using training data including observation data of the observation target when it is normal, or may be another type of recognition model. The recognition model may, for example, recognize whether the state of the observation target is a normal state or an abnormal state, or may recognize whether the state is one of three or more predetermined states.

[0012] When a recognition result is designated as an erroneous recognition, the extraction unit extracts from the storage unit other observation data whose recognition result may have been an erroneous recognition, based on the feature quantities of the observation data from which the designated recognition result was obtained. For example, the extraction unit extracts observation data whose feature quantities are similar to the feature quantities of the observation data from which the recognition result indicated as an erroneous recognition was obtained.

[0013] When permission to update the recognition model is received after the observation data extracted by the extraction unit is output, the update unit updates the parameters used by the recognition model to determine the state of the observation target. For example, the observation data extracted by the extraction unit may be displayed on a display device so that it can be visually recognized by a user.

[0014] This configuration allows the user to check other observation data for which the recognition result may have been erroneous. The user also checks the extracted observation data for which the recognition result may have been erroneous, and if the recognition results for all output observation data are erroneous, performs an input operation to allow updating of the recognition model. This causes the update unit to update the parameters used by the recognition model to determine the state of the observation target.

[0015] Therefore, the user can suppress the frequency of erroneous recognition of the observation target, which occurs due to differences between the learning environment and the operational environment, by simply selecting and specifying the erroneous recognition result.

[0016] Furthermore, for example, if the extraction unit receives a specification of observation data for which the recognition result is not to be corrected after outputting the extracted observation data, the configuration may include an instruction unit that instructs the extraction unit to narrow down other observation data for which the previously extracted recognition result may have been an error, based on the features of the specified observation data.

[0017] With this configuration, if the observation data extracted by the extraction unit includes observation data for which the recognition result is not erroneous, the recognition result can be excluded and the parameters used by the recognition model to determine the state of the observation object can be updated, thereby preventing the parameters used by the recognition model to determine the state of the observation object from being updated to inappropriate values.

[0018] Furthermore, if the recognition model is, for example, a VAE, the VAE may be configured to acquire the distribution of latent variables of the observation data input to the input unit as a feature, which is a judgment distribution, and recognize the state of the object to be observed based on the distance between the judgment distribution and the reference distribution.

[0019] The distance between the judgment distribution and the reference distribution may be determined using, for example, the Mahalanobis distance.

[0020] The observation data may also be a differential image between a first average image of n frame images captured consecutively in time and a second average image of m frame images captured consecutively in time, where n is greater than m, and m is equal to or greater than 1. The frame image captured latest in time and used to generate the first average image is the same frame image.

[0021] With this configuration, an image (difference image) of an object that appeared in the imaging area immediately before the imaging time of the latest frame image used to generate the first average image and the second average image can be acquired as observation data. Therefore, in the case of a recognition model that recognizes the state of the observation target based on the type, size, position, etc. of the object located in the observation target area, observation data suitable for recognizing the state of the observation target can be used. [Effects of the Invention]

[0022] According to this invention, it is possible to easily reduce the frequency with which the state of the observation target is erroneously recognized due to differences between the learning environment and the operational environment. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 is a diagram illustrating an abnormality recognition system to which the state detection device of this example is applied. [Figure 2] 10 is an example of a frame image captured by a video camera of the station premises, which is the observation target. [Figure 3] 10 is an example of a frame image of a road as an observation target captured by a video camera. [Figure 4] 1 is a diagram showing the configuration of a main part of a status detection device according to this example. [Figure 5] FIG. 2 is a block diagram showing the functional configuration of a recognition unit of the status detection device of this example. [Figure 6] 10 is a flowchart showing a state recognition process of the state detection device of this example. [Figure 7] 10 is a flowchart showing an update process of the status detection device in this example. DETAILED DESCRIPTION OF THE INVENTION

[0024] Hereinafter, an embodiment of the present invention will be described.

[0025] <1. Application Examples> 1 is a diagram showing an anomaly recognition system to which the condition detection device of this example is applied. The anomaly recognition system 100 of this example includes a condition detection device 1, a video camera 2, a display 3, and an input device 4.

[0026] The condition detection device 1 of this example recognizes whether the condition of the observed object is normal or abnormal, and outputs the recognition result. The observed object may be the inside of a railway station or a road on which vehicles travel. When the observed object is the inside of a railway station, the condition detection device 1 of this example recognizes that an abnormal state exists if there is an abandoned suspicious object 105. Furthermore, when the observed object is a road on which vehicles travel, the condition detection device 1 of this example recognizes that an abnormal state exists if there is an obstacle on the road that obstructs the travel of the vehicle 110. The obstacle may be, for example, a fallen object (baggage) that has fallen onto the road from the loading platform of the vehicle 110, or a vehicle 110 that has stopped on the road due to a malfunction such as engine trouble.

[0027] Video camera 2 captures an image of the observation target and outputs the captured frame images to condition detection device 1. Video camera 2 is mounted at an angle such that the observation target fits within the imaging area. In other words, all or part of the imaging area of ​​video camera 2 is the observation target. FIGS. 2 and 3 are schematic diagrams showing frame images captured by the video camera. FIG. 2 is an example of a frame image captured by video camera 2 of a station premises, which is the observation target, and FIG. 3 is an example of a frame image captured by video camera 2 of a road, which is the observation target. The frame rate of video camera 2 is, for example, several tens of frames / sec (e.g., 10 to 30 frames / sec). The observation data referred to in this invention is acquired from the frame images of the observation target captured by video camera 2.

[0028] The video camera 2 may be a digital still camera that captures still images in response to a release signal input from an external device.

[0029] The display device 3 displays the recognition results of the state of the observation target in the state detection device 1. The display device 3 displays a screen based on screen display data input from the state detection device 1. In other words, the state detection device 1 controls the screen displayed on the display device 3. The display device 3 also displays a screen related to a GUI (Graphical User Interface).

[0030] The input device 4 is a keyboard, a mouse, or the like, through which a user performs an input operation on the status detection device 1.

[0031] The condition detection device 1 recognizes whether the condition of the observed object is normal using a recognition model generated by machine learning. This recognition model recognizes whether the condition of the observed object is normal based on the feature quantities of the observation data obtained from frame images of the observed object captured by the video camera 2. The condition detection device 1 stores the observation data and the recognition results in association with each other.

[0032] Furthermore, when the status detection device 1 receives an input designation that the recognition result of the recognition model is an error, it extracts and outputs other observation data that may have resulted in an error in the recognition result, based on the feature quantities of the observation data from which the specified recognition result was obtained. For example, the status detection device 1 displays on the screen of the display device 3 some or all of the extracted observation data that may have resulted in an error in the recognition result.

[0033] When the user confirms that all of the observation data extracted because the recognition result was possibly an erroneous recognition is an erroneous recognition, the user operates the input device 4 to input permission to update the recognition model to the state detection device 1. Upon receiving the input of permission to update, the state detection device 1 updates the values ​​of the parameters used to determine the state of the observation target to values ​​that will obtain corrected recognition results for all of the observation data extracted because the recognition result was possibly an erroneous recognition.

[0034] Furthermore, when the user confirms that there is observation data for which the recognition result is correct (not incorrect) among the observation data extracted as having a possibility that the recognition result was incorrect, the user operates the input device 4 to input a request to not allow the update of the recognition model to the status detection device 1. At this time, the user performs an input operation to specify one or more observation data for which the recognition result is correct for the observation data extracted as having a possibility that the recognition result was incorrect. Upon receiving the input to not allow the update, the status detection device 1 narrows down the observation data extracted as having a possibility that the previous recognition result was incorrect, excluding observation data for which the recognition result is estimated to be correct, based on the feature quantities of the observation data for which the recognition result is correct that has been specified this time.

[0035] The condition detection device 1 displays some or all of the narrowed-down observation data on the screen of the display device 3, and waits for the user to input permission or denial of permission to update.

[0036] Therefore, the user can update the parameter values ​​used to determine the state of the observed object by performing an input operation to specify the recognition result that has been confirmed to be an erroneous recognition. Therefore, the state detection device 1 of this example can easily reduce the frequency of erroneous recognition of the observed object caused by differences between the learning environment and the operational environment.

[0037] <2.Configuration example> 4 is a diagram showing the configuration of the main parts of the condition detection device 1 of this example. The condition detection device 1 of this example includes a control unit 11, an input unit 12, an output unit 13, a memory unit 14, a display output unit 15, and an input reception unit 16.

[0038] The control unit 11 controls the operation of each part of the main body of the status detection device 1. The control unit 11 also has a recognition unit 11a, an extraction unit 11b, an update unit 11c, an instruction unit 11d, and a display control unit 11e. The recognition unit 11a, extraction unit 11b, update unit 11c, instruction unit 11d, and display control unit 11e of the control unit 11 will be described later.

[0039] A video camera 2 attached at an angle to capture an image of the observation target is connected to the input unit 12. Frame images of the observation target captured by the video camera 2 are input to the input unit 12.

[0040] The output unit 13 outputs the recognition result of the state of the observation target to a higher-level device.

[0041] The storage unit 14 stores observation data of the observation target and a recognition result of the state of the observation target for the observation data in association with each other. The storage unit 14 is a storage medium such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an SD memory card.

[0042] The display device 3 is connected to the display output unit 15. The display output unit 15 outputs screen display data to the display device 3. The display device 3 displays a screen based on this screen display data. The display output unit 15 is an interface circuit with the display device 3.

[0043] The input device 4 is connected to the input reception unit 16. Input operation data corresponding to an input operation of the input device 4 by a user is input to the input reception unit 16. The input reception unit 16 is an interface circuit with the input device 4.

[0044] Next, the recognition unit 11a, extraction unit 11b, update unit 11c, instruction unit 11d, and display control unit 11e included in the control unit 11 will be described.

[0045] The recognition unit 11a recognizes the state of the observation target (in this example, normal or abnormal). FIG. 5 is a block diagram showing the functional configuration of the recognition unit of the state detection device of this example. In this example, the recognition unit 11a has a variational autoencoder 20 (hereinafter referred to as VAE 20), a feature calculation unit 23, a distance calculation unit 24, a distribution storage unit 25, and a determination unit 26.

[0046] The VAE 20 is a recognition model generated by deep learning using learning data including observation data of the object under normal conditions. The VAE 20 has an encoder 21 and a decoder 22. The encoder 21 and the decoder 22 are neural networks. The encoder 21 extracts a latent variable z from the input observation data. The decoder 22 receives the latent variable z as input and restores the observation data. Here, the observation data restored by the decoder 22 from the latent variable z is referred to as restored data.

[0047] The feature calculation unit 23 calculates the feature of the latent variable z. The distribution storage unit 25 stores the distribution of the feature of the latent variable z obtained for the observation data when the state of the observation target is normal (hereinafter referred to as normal distribution). The distribution storage unit 25 stores the distribution of the feature of the latent variable z in a reference space. In this example, the reference space is described as a Mahalanobis space, but it may be a space generated by other statistical methods.

[0048] The distribution storage unit 25 may store the distribution of the features of the latent variable z obtained for the observed data when the state of the observation target is abnormal (hereinafter referred to as the abnormal distribution), or may store both the normal distribution and the abnormal distribution.

[0049] The distance calculation unit 24 applies the feature amount of the latent variable z calculated by the feature amount calculation unit 23 for the observed data to the Mahalanobis space, which is a reference space, and calculates the distance (Mahalanobis distance) to the normal distribution.

[0050] The determination unit 26 compares the Mahalanobis distance calculated by the distance calculation unit 24 with a predetermined judgment distance and determines whether the state of the observation target is normal or abnormal. For example, if the Mahalanobis distance to the normal distribution calculated by the distance calculation unit 24 is equal to or less than the judgment distance, the determination unit 26 determines that the state of the observation target is normal, and if the Mahalanobis distance to the normal distribution calculated by the distance calculation unit 24 is not equal to or less than the judgment distance, the determination unit 26 determines that the state of the observation target is abnormal.

[0051] If the distribution storage unit 25 stores an abnormal distribution of the feature amount of the latent variable z, the distance calculation unit 24 calculates the Mahalanobis distance for the abnormal normal distribution when the feature amount of the latent variable z calculated for the observation data by the feature amount calculation unit 23 is assigned to the Mahalanobis space, which is the reference space. Furthermore, the determination unit 26 may determine that the state of the observation target is normal if the Mahalanobis distance for the abnormal distribution calculated by the distance calculation unit 24 is equal to or greater than the judgment distance, and may determine that the state of the observation target is abnormal if the Mahalanobis distance for the abnormal distribution calculated by the distance calculation unit 24 is not equal to or greater than the judgment distance.

[0052] Furthermore, when the distribution storage unit 25 stores a normal distribution and an abnormal distribution of the feature of the latent variable z, the distance calculation unit 24 may calculate a Mahalanobis distance for the normal distribution and a Mahalanobis distance for the abnormal normal distribution when the feature of the latent variable z calculated for the observed data by the feature calculation unit 23 is assigned to a Mahalanobis space, which is a reference space. In this case, the determination unit 26 may determine the state of the observed object by comparing the Mahalanobis distance for the normal distribution calculated by the distance calculation unit 24 (first distance) with the Mahalanobis distance for the abnormal distribution calculated by the distance calculation unit 24 (second distance). Specifically, the determination unit 26 determines the state of the observed object as normal if the first distance calculated by the distance calculation unit 24 is shorter than the second distance, and determines the state of the observed object as abnormal if the first distance is longer than the second distance.

[0053] When it is specified that the recognition result of the state of the observation object by the recognition unit 11a is an erroneous recognition, the extraction unit 11b extracts, based on the feature amount of the observation data from which the specified recognition result was obtained, other observation data that may have been an erroneous recognition result of the state of the observation object by the recognition unit 11a from the storage unit 14. The number of other observation data extracted by the extraction unit 11b may be one or more.

[0054] When the update unit 11c determines that other observation data extracted by the extraction unit 11b (other observation data for which the recognition unit 11a may have misrecognized the state of the observation target) is also misrecognized and instructs updating of the parameters used for state recognition, the update unit 11c updates the parameters used for state recognition. In this example, the determination unit 26 updates the determination distance used for determining the state of the observation target.

[0055] The update unit 11c may update the distribution of the feature amount of the latent variable z in the reference space, which is stored in the distribution storage unit 25.

[0056] When the instruction unit 11d receives a designation that any of the other observation data extracted by the extraction unit 11b (other observation data for which the recognition result of the state of the observation target by the recognition unit 11a may have been an erroneous recognition) is not an erroneous recognition (the recognition result is correct), the instruction unit 11d excludes the observation data designated as not an erroneous recognition, and instructs the extraction unit 11b to narrow down the other observation data for which the recognition result of the state of the observation target may have been an erroneous recognition in accordance with the instruction from the instruction unit 11d, and outputs the result (other observation data for which the narrowed down recognition result of the state of the observation target may have been an erroneous recognition).

[0057] The display control unit 11e generates screen display data to be displayed on the screen of the display device 3.

[0058] When the hardware CPU constituting the control unit 11 of the condition detection device 1 executes the condition detection program according to the present invention, it operates as a recognition unit 11a, an extraction unit 11b, an update unit 11c, an instruction unit 11d, and a display control unit 11e. The memory also has an area for expanding the condition detection program according to the present invention and an area for temporarily storing data generated when the condition detection program is executed. The control unit 11 may be an LSI that integrates the hardware CPU, memory, etc. The hardware CPU is also a computer that executes the condition detection method according to the present invention.

[0059] <3. Example of operation> A description will be given of a state recognition process in which the state detection device 1 of this example recognizes the state of an observation target. Fig. 6 is a flowchart showing the state recognition process of the state detection device of this example.

[0060] The condition detection device 1 has an observation data storage unit (not shown) that temporarily stores observation data of the observation target input to the input unit 12. In this example, the observation data is frame images of the observation target captured by the video camera 2. The condition detection device 1 selects observation data to be processed from the observation data temporarily stored in the observation data storage unit (s1). For example, the condition detection device 1 may select frame images of a moving image of the observation target captured by the video camera 2 in order as the observation data to be processed, or may select the mth frame image from the frame image previously selected as the observation data to be processed as the observation data to be processed.

[0061] The above-described method for selecting observation data to be processed is merely an example, and other methods may be used for selection.

[0062] The state detection device 1 encodes the observation data to be processed selected in s1 using the encoder 21 of the VAE 20 and extracts the latent variable z of this observation data (s2). At this time, the VAE 20 may or may not restore the observation data from the latent variable z extracted by the decoder 22.

[0063] The feature calculation unit 23 calculates the feature of the latent variable z extracted from the observation data (s3). The feature of the latent variable z calculated in s3 may be calculated for each pixel of the observation data (frame image) to be processed, or may be calculated for each divided region obtained by dividing the frame image into a plurality of predetermined regions. The frame image may be divided according to the surrounding environment of the observation target, etc. For example, the frame image may be divided into n × m rectangular regions, that is, n vertically and m horizontally, or may be divided into multiple regions surrounded by curved lines, or may be divided by other methods.

[0064] In s3, the feature amount of the calculated latent variable z is a feature amount of a type for which a normal distribution is stored in the distribution storage unit 25.

[0065] The distance calculation unit 24 applies the feature calculated in s3 to the Mahalanobis space, which is the reference space, and calculates the distance (Mahalanobis distance) to the normal distribution (s4). In this example, the Mahalanobis distance to the normal distribution is calculated, but if the distribution storage unit 25 stores an abnormal distribution of the feature of the latent variable z, the Mahalanobis distance to the abnormal distribution is calculated. If the distribution storage unit 25 stores the normal distribution and the abnormal distribution of the feature of the latent variable z, the Mahalanobis distance to the normal distribution and the Mahalanobis distance to the abnormal distribution are calculated.

[0066] The determination unit 26 performs a determination process to determine whether the state of the observation target is normal or abnormal based on the Mahalanobis distance calculated in s4 (s5). The determination result by the determination unit 26 is the recognition result of the recognition unit 11a.

[0067] The condition detection device 1 outputs the determination result by the determination unit 26, and stores the current observation data, the feature calculated in s3, and the recognition result in the storage unit 14 in association with each other (s6, s7), and then returns to s1. In s6, for example, the recognition result may be output to a higher-level device only if it is determined that the state of the observation target is abnormal. Also, in s6, the display control unit 11e may generate screen display data for displaying a screen showing the current observation data and the recognition result on the display device 3, and output the screen display data to the display device 3. By looking at the screen of the display device 3, the user can confirm the recognition result along with the frame image of the observation target.

[0068] Next, an update process will be described in which the status detection device 1 of this example updates the parameters used when recognizing the status of the observed object. Fig. 7 is a flowchart showing the update process of the status detection device of this example. The status detection device 1 executes this update process when the user operates the input device 4 to indicate that the recognition result of the status of the observed object is an error.

[0069] The status detection device 1 waits for the input receiving unit 16 to receive an input indicating that the recognition result of the status of the observation target is an erroneous recognition (s11).

[0070] When the input receiving unit 16 of the condition detection device 1 receives an input indicating that the recognition result of the state of the observation target is an error, the extraction unit 11b extracts other observation data from the recognition results stored in the storage unit 14, based on the observation data designated as the current error (s12). In s12, for example, based on the feature of the latent variable z extracted from the observation data designated as the current error, other observation data from the recognition results stored in the storage unit 14, which may have been the recognition result an error. Specifically, the extraction unit 11b extracts, as the other observation data, observation data whose similarity in the feature of the latent variable z is similar to that of the observation data designated as the current error (i.e., whose similarity exceeds a predetermined threshold). Furthermore, for example, the extraction unit 11b extracts, as the other observation data whose recognition result may have been the error, observation data whose difference from the Mahalanobis distance to the normal distribution calculated for the observation data designated as the current error falls within a predetermined range.

[0071] The display control unit 11e generates screen display data for displaying on the screen of the display device 3 other observation data that may have been erroneous recognition results extracted in s12, and outputs the generated screen display data to the display device 3 (s13). As a result, the display device 3 displays on the screen other observation data that may have been erroneous recognition results extracted this time.

[0072] The user checks the screen of the display device 3 to confirm whether all of the other observation data for which the currently extracted recognition result may have been an error is an error. In other words, the user checks the screen of the display device 3 to confirm whether the other observation data for which the currently extracted recognition result may have been an error includes any observation data for which the recognition result is not an error.

[0073] If all of the other observation data for which the currently extracted recognition result may have been an erroneous recognition are erroneous recognitions, the user performs an input operation to permit the update on the input device 4. Furthermore, if the other observation data for which the currently extracted recognition result may have been an erroneous recognition includes observation data for which the recognition result was not an erroneous recognition, the user specifies that observation data (the observation data for which the recognition result was not an erroneous recognition) and performs an input operation to instruct not to permit the update.

[0074] The status detection device 1 waits for an input regarding permission or denial of permission to update (s14, s15). When the user performs an input operation regarding permission to update on the input device 4, the status detection device 1 causes the update unit 11c to instruct the recognition unit 11a to update the parameters used to determine the status of the observation target. In accordance with this instruction, the recognition unit 11a updates the parameters used to determine the status of the observation target (s16) and returns to s11. In s16, for example, the judgment distance may be updated, or the normal distribution, the anomalous distribution, or both the normal distribution and the anomalous distribution of the feature of the latent variable z stored in the distribution storage unit 25 may be updated. In s16, the update unit 11c updates the parameters so as to change the recognition result of the observation data that the user confirmed to be an erroneous recognition.

[0075] Furthermore, when the user performs an input operation to prohibit updating on the input device 4, the status detection device 1 narrows down the observation data extracted in s12 (other observation data whose recognition result may have been an error) (s17). In s17, the instruction unit 11d instructs the extraction unit 11b to divide the observation data extracted in s12 into two groups: an error recognition group whose recognition result has been determined to be an error recognition, and a correct recognition group whose recognition result has been determined to be correct, based on the observation data whose recognition result has not been an error recognition, which was specified along with the input to prohibit updating. For example, the observation data extracted in s12 is divided into two groups based on the feature amount of the latent variable z of the observation data specified to be an error recognition this time and the feature amount of the latent variable z of the observation data specified to be a non-error recognition this time. For example, for each piece of observation data extracted in s12 (excluding observation data whose current recognition result is specified as not being an error), the extraction unit 11b calculates the distance (first feature distance) between the feature of the latent variable z of that observation data and the feature of the latent variable z of the observation data whose current recognition result is specified as not being an error in s11. Furthermore, for each piece of observation data extracted in s12 (excluding observation data whose current recognition result is specified as not being an error), the extraction unit 11b calculates the distance (second feature distance) between the feature of the latent variable z of that observation data and the feature of the latent variable z of the observation data whose current recognition result is specified as not being an error in s15. The extraction unit 11b determines whether the observation data belongs to the error recognition group or the correct recognition group based on the size of the first feature distance and the second feature distance. Specifically, the extraction unit 11b determines that observation data whose first feature distance is shorter than the second feature distance belongs to the error recognition group. Conversely, the extraction unit 11b determines that observation data in which the first feature amount distance is longer than the second feature amount distance belongs to an appropriate recognition group.

[0076] The display control unit 11e generates screen display data for displaying the observation data on the screen of the display device 3 for each group divided in s17 as a result of the narrowing down, and outputs the data to the display device 3 (s18). As a result, the display device 3 displays a screen that distinguishes between observation data for which the recognition result has been determined to be erroneous recognition and observation data for which the recognition result has been determined to be proper recognition, among the observation data extracted in s12. Therefore, when the recognition result of the observation data specified this time is determined to be proper, the condition detection device 1 allows the user to confirm which observation data's recognition result has been determined to be erroneous recognition and which observation data's recognition result has been determined to be proper recognition.

[0077] When the status detection device 1 completes the process in s18, it returns to s14 and repeats the above-mentioned process.

[0078] In this way, the user can update the parameters used by the recognition unit 11a (recognition model) to recognize the state of the observed object by simply specifying the observation data whose recognition result is determined to be erroneous. Therefore, the state detection device 1 of this example can easily reduce the frequency of erroneous recognition of the state of the observed object caused by differences between the learning environment and the operational environment.

[0079] It should be noted that the parameter referred to here is not limited to the judgment distance used to judge normality or abnormality, as described above, but may also be a normal distribution of the feature values ​​of the observation data when the state of the observation object is normal, an abnormal distribution of the feature values ​​of the observation data when the state of the observation object is abnormal, or the like.

[0080] <4. Modifications> In the above example, the recognition unit 11a uses the frame images captured by the video camera 2 as observation data for the object to be observed, but it may also be configured to generate a differential image between a first average image of n frames of frame images captured by the video camera 2 that are consecutive in time and a second average image of m frames (n>m) of frame images captured by the video camera 2 that are consecutive in time, and use this differential image as observation data.

[0081] In this case, n is 2 or more, and m is 1 or more. Also, the m frame images used for generating the second average image may or may not be included in the n frame images used for generating the first average image.

[0082] For example, the first average image may be generated using n temporally consecutive frame images captured by the video camera 2, and the second average image may be generated using m frame images with a later imaging time among these n frame images. In this case, the m frame images used for generating the second average image are included in the n frame images used for generating the first average image.

[0083] Also, for example, the first average image may be generated using n frame images with an earlier imaging time among p (p < n + m) temporally consecutive frame images captured by the video camera 2, and the second average image may be generated using m frame images with a later imaging time among these p frame images. In this case, some of the frame images used for generating the second average image are included in the n frame images used for generating the first average image.

[0084] Also, for example, the first average image may be generated using n frame images with an earlier imaging time among n + m temporally consecutive frame images captured by the video camera 2, and the second average image may be generated using m frame images with a later imaging time among these n + m frame images. In this case, the second average image is generated by m temporally consecutive frame images with respect to the n frame images used for generating the first average image.

[0085] Furthermore, for example, among q frames (q>n+m) of consecutive frame images captured by the video camera 2, a first average image may be generated using a frame image from an n frame captured earlier, and a second average image may be generated using a frame image from m frames captured later than the q frames. In this case, there may be frame images among the q frames that are not used to generate the first and second average images. In this case, the number of frame images among the q frames that are not used to generate the first and second average images may be several to several tens of frames.

[0086] When configured in this manner, the recognition unit 11a uses an image of an object (e.g., a suspicious object 105 or an object that fell from a vehicle 110) that appeared in the imaging area of ​​the video camera 2 (within the area to be observed) just before the second time as observation data (a differential image between the first average image and the second average image), thereby making it possible to obtain observation data suitable for recognizing the state of the object to be observed.

[0087] Furthermore, in the above example, the recognition unit 11a is not limited to the VAE 20 described above, and may be configured to have a recognition model of another type, as long as it has a configuration having a recognition model generated by machine learning.

[0088] In the above example, the frame images or difference images of the observation target are used, but the data may be audio data collected by a microphone around the observation target, the temperature of the observation target or its surroundings measured by a temperature sensor, or sensing data of the observation target or its surroundings measured by a sensor other than a temperature sensor. The observation data may be of multiple types, rather than just one.

[0089] Furthermore, in the above example, the recognition unit 11a is configured to recognize whether the state of the observed object is normal or abnormal, but it may also be configured to recognize whether the state of the observed object is one of three or more states (for example, a first state, a second state, and an intermediate state (a state between the first state and the second state)).

[0090] It should be noted that this invention is not limited to the above-described embodiments, and that the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. Furthermore, the order of the processes shown in Figures 6 and 7 is not limited to the order shown, and may be appropriately changed.

[0091] Furthermore, the correspondence between the configuration according to the present invention and the configuration according to the above-described embodiment can be described as follows: <Additional Notes> an input unit (12) into which observation data of an observation target is input; a storage unit (14) that stores the observation data input to the input unit (12) in association with a recognition result of the state of the observation target that is recognized by a recognition model (20) generated by machine learning based on the feature amount of the observation data; an extraction unit (11b) that, when the recognition result is designated as being erroneous, extracts from the storage unit (14) other observation data that may have been recognized as being erroneous based on the feature amount of the observation data from which the designated recognition result was obtained; and an update unit (11c) that updates parameters used by the recognition model to determine the state of the observed object when permission to update the recognition model is received after the observation data extracted by the extraction unit (11b) is output. [Explanation of symbols]

[0092] 1...Status detection device 2. Video camera 3...Indicator 4. Input device 11...Control unit 11a...Recognition part 11b...Extraction part 11c…Update section 11d...Instruction section 11e...Display control unit 12...Input section 13...Output section 14...Storage section 15...Display output section 16...Input reception section 20...Variational autoencoder (VAE) 21...Encoder 22...Decoder 23...Feature calculation unit 24...Distance calculation unit 25...Distribution storage section 26…Judgment section

Claims

1. an input unit to which observation data of an observation target is input; a storage unit that stores, in association with each other, observation data input to the input unit and a recognition result of the state of the observation target that is recognized by a recognition model generated by machine learning based on the feature amount of the observation data; an extraction unit that, when the recognition result is designated as being erroneous, extracts from the storage unit other observation data that may have been recognized as being erroneous based on the feature amount of the observation data from which the designated recognition result was obtained; an instruction unit that, when receiving designation of observation data for which the recognition result is not to be corrected after the extraction unit outputs the extracted observation data, instructs the extraction unit to narrow down other observation data for which the previously extracted recognition result may have been an erroneous recognition, based on the feature amount of the designated observation data; and an update unit that updates parameters used by the recognition model to determine the state of the observed object when permission to update the recognition model is received after the observation data extracted by the extraction unit is output.

2. An input unit to which observation data of an observation target is input; a storage unit that stores, in association with each other, observation data input to the input unit and a recognition result of the state of the observation target that is recognized by a recognition model generated by machine learning based on the feature amount of the observation data; an extraction unit that, when the recognition result is designated as being erroneous, extracts from the storage unit other observation data that may have been recognized as being erroneous based on the feature amount of the observation data from which the designated recognition result was obtained; an update unit that updates parameters used by the recognition model to determine the state of the observation target when permission to update the recognition model is received after the observation data extracted by the extraction unit is output, the observation data is a difference image between a first average image of n frames of frame images captured successively in time and a second average image of m frames of frame images captured successively in time, n is greater than m, A state detection device, wherein m is 1 or greater.

3. the observation data is a difference image between a first average image of n frames of frame images captured successively in time and a second average image of m frames of frame images captured successively in time, n is greater than m, The condition detection device according to claim 1 , wherein m is 1 or greater.

4. The recognition model is machine learning using learning data including observation data of the object under normal conditions, and a variational autoencoder that acquires a distribution of latent variables extracted from the observation data under normal conditions as a reference distribution; the variational autoencoder acquires, as the feature, a distribution of latent variables of the observation data input to the input unit as a determination distribution; Recognizing the state of the observation target based on the distance between the judgment distribution and the reference distribution. The condition detection device according to any one of claims 1 to 3.

5. The condition detection device according to claim 4 , wherein the distance between the judgment distribution and the reference distribution is a Mahalanobis distance.

6. 6. The condition detection device according to claim 1, wherein the observation data is a frame image of an object to be observed.

7. The condition detection device according to any one of claims 1 to 6, further comprising a display control unit that displays the observation data extracted by the extraction unit on a display.

8. 8. The condition detection device according to claim 1, wherein the recognition model recognizes whether the condition of the observed object is normal or abnormal.

9. a recognition result storage step of storing, in a storage unit, observation data of the observation object input to an input unit and a recognition result of the state of the observation object recognized by a recognition model generated by machine learning based on the feature quantities of the observation data, in association with each other; an extraction step of extracting, when the recognition result is designated as being erroneous, other observation data that may have been recognized as being erroneous from the storage unit based on the feature amount of the observation data from which the designated recognition result was obtained; an instruction step of instructing the extraction step, when receiving designation of observation data for which the recognition result is not to be corrected after outputting the observation data extracted in the extraction step, to narrow down other observation data for which the recognition result extracted previously may have been erroneous, based on the feature amount of the designated observation data; an update step of updating parameters used by the recognition model to determine the state of the observed object if permission to update the recognition model is received after outputting the observation data extracted in the extraction step.

10. A recognition result storage step of storing, in a storage unit, observation data of an observation object input to an input unit and a recognition result of a state of the observation object recognized by a recognition model generated by machine learning based on features of the observation data, in association with each other. an extraction step of extracting, when the recognition result is designated as being erroneous, other observation data that may have been recognized as being erroneous from the storage unit based on the feature amount of the observation data from which the designated recognition result was obtained; an updating step of updating a parameter used by the recognition model to determine a state of the observation target when permission to update the recognition model is received after outputting the observation data extracted in the extraction step, the observation data is a difference image between a first average image of n frames of frame images captured successively in time and a second average image of m frames of frame images captured successively in time, n is greater than m, A state detection method, wherein m is 1 or greater.

11. a recognition result storage step of storing, in a storage unit, observation data of the observation object input to an input unit and a recognition result of the state of the observation object recognized by a recognition model generated by machine learning based on the feature quantities of the observation data, in association with each other; an extraction step of extracting, when the recognition result is designated as being erroneous, other observation data that may have been recognized as being erroneous from the storage unit based on the feature amount of the observation data from which the designated recognition result was obtained; an instruction step of instructing the extraction step, when receiving designation of observation data for which the recognition result is not to be corrected after outputting the observation data extracted in the extraction step, to narrow down other observation data for which the recognition result extracted previously may have been erroneous, based on the feature amount of the designated observation data; and an update step of updating parameters used by the recognition model to determine the state of the observed object if permission to update the recognition model is received after outputting the observation data extracted in the extraction step.

12. A recognition result storage step of storing, in a storage unit, observation data of an observation object input to an input unit and a recognition result of the state of the observation object recognized by a recognition model generated by machine learning based on the feature quantities of the observation data, in association with each other. an extraction step of extracting, when the recognition result is designated as being erroneous, other observation data that may have been recognized as being erroneous from the storage unit based on the feature amount of the observation data from which the designated recognition result was obtained; an updating step of updating a parameter used by the recognition model to determine a state of the observation target when permission to update the recognition model is received after outputting the observation data extracted in the extraction step; the observation data is a difference image between a first average image of n frames of frame images captured successively in time and a second average image of m frames of frame images captured successively in time, n is greater than m, m is 1 or greater; and the condition detection program.

Images