Monitoring device, learning apparatus, method executed by monitoring device, and recording medium

Abstract

The present invention provides a monitoring device, a learning apparatus, a method and a recording medium. The monitoring device includes: an image acquisition unit that acquires an image captured by a monitoring camera; a monitoring logic execution unit that uses a learned monitoring logic to generate a monitoring result corresponding to the image captured by the monitoring camera; an image sending unit that performs interval elimination on the image captured by the monitoring camera and sends it to a learning device that performs learning processing on other monitoring logics of the same type as the monitoring logic; a monitoring logic receiving unit that receives from the learning device other monitoring logics that have been additionally learned by the learning device using the images from the image sending unit; and an updating unit that updates the monitoring logic executed by the monitoring logic execution unit to the other monitoring logic received by the monitoring logic receiving unit.

Description

Technical Field

[0001] The present invention relates to a monitoring device, a learning apparatus, a method and a recording medium. Background Art

[0002] Conventionally, various security systems using surveillance cameras have been proposed (see, for example, Patent Document 1).

[0003] Patent Document 1: Japanese Patent Publication No. 2015-162232 Summary of the invention

[0004] In a first aspect of the present invention, a monitoring device is provided. The monitoring device may include a captured image acquisition unit that acquires an image captured by a monitoring camera. The monitoring device may include a monitoring logic execution unit that uses a learned monitoring logic to generate a monitoring result corresponding to the image captured by the monitoring camera. The monitoring device may include an image sending unit that performs interval elimination on the image captured by the monitoring camera and sends it to a learning device that performs a learning process on other monitoring logics of the same type as the monitoring logic. The monitoring device may include a monitoring logic receiving unit that receives from the learning device other monitoring logics that have been additionally learned by the learning device using images from the image sending unit. The monitoring device may include an updating unit that updates the monitoring logic executed by the monitoring logic execution unit to other monitoring logics received by the monitoring logic receiving unit.

[0005] The image transmitting unit may transmit at least one image generating a monitoring result indicating abnormality.

[0006] The image transmitting unit may further transmit at least one image before or after the at least one image indicating the abnormal monitoring result is generated.

[0007] The image transmitting unit may transmit at least one image generating a monitoring result with a certainty less than a threshold value.

[0008] The image transmitting unit may adjust the threshold value according to the amount of images to be transmitted.

[0009] The monitoring device may include a monitoring result output unit that outputs the monitoring result.

[0010] In a second aspect of the present invention, a learning device is provided. The learning device may include an image receiving unit that receives an image that has been throttled after being captured from a monitoring device that uses monitoring logic to generate a monitoring result corresponding to an image captured by a monitoring camera. The learning device may include a learning processing unit that performs additional learning on other monitoring logics of the same type as the monitoring logic of the monitoring device by using learning data of the image received by the image receiving unit. The learning device may include a monitoring logic sending unit that sends the other monitoring logics that have been additionally learned to the monitoring device.

[0011] The monitoring logic transmitting unit may transmit the other monitoring logic after the additional learning to a plurality of monitoring devices that use the monitoring logic before the additional learning.

[0012] The learning device may include a learning data generating unit that generates learning data by performing image processing for protecting the privacy of a person captured in the image on an image received from the monitoring device.

[0013] The learning device may include a learning data generating unit that generates learning data by performing at least one of angle of view correction, brightness correction, and color tone correction on an image received from the monitoring device.

[0014] In a third aspect of the present invention, a method is provided. The method may include a captured image acquisition phase for acquiring an image captured by a surveillance camera. The method may include a monitoring logic execution phase, which generates a monitoring result corresponding to the image captured by the surveillance camera using the learned monitoring logic. The method may include an image sending phase, which performs interval elimination on the image captured by the surveillance camera and sends it to a learning device that performs a learning process on other monitoring logics of the same type as the monitoring logic. The method may include a monitoring logic receiving phase, which receives from the learning device other monitoring logics that have been additionally learned by the learning device using the images sent by the image sending phase. The method may include an updating phase, which updates the monitoring logic executed by the monitoring logic execution phase to other monitoring logics received by the monitoring logic receiving phase.

[0015] In a fourth aspect of the present invention, a method is provided. The method may include an image receiving stage, which receives an image that has been throttled after being captured from a monitoring device that uses monitoring logic to generate a monitoring result corresponding to an image captured by a monitoring camera. The method may include a learning processing stage, which performs additional learning on other monitoring logics of the same type as the monitoring logic of the monitoring device by using learning data of the image received by the image receiving stage. The method may include a monitoring logic sending stage, which sends the other monitoring logics after additional learning to the monitoring device.

[0016] In a fifth aspect of the present invention, a recording medium having a program recorded thereon is provided. The computer can function as an image acquisition unit by executing the program, and the image acquisition unit acquires an image captured by a surveillance camera. The computer can function as a monitoring logic execution unit by executing the program, and the monitoring logic execution unit generates a monitoring result corresponding to the image captured by the surveillance camera using the learned monitoring logic. The computer can function as an image sending unit by executing the program, and the image sending unit thins out the image captured by the surveillance camera and sends it to a learning device that performs a learning process on other monitoring logics of the same type as the monitoring logic. The computer can function as a monitoring logic receiving unit by executing the program, and the monitoring logic receiving unit receives from the learning device other monitoring logics that have been additionally learned by the learning device using the images from the image sending unit. The computer can function as an updating unit by executing the program, and the updating unit updates the monitoring logic executed by the monitoring logic execution unit to other monitoring logics received by the monitoring logic receiving unit.

[0017] In a sixth aspect of the present invention, a recording medium having a program recorded thereon is provided. The computer can function as an image receiving unit by executing the program, and the image receiving unit receives an image that has been interpolated after being captured from a monitoring device that uses monitoring logic to generate a monitoring result corresponding to an image captured by a monitoring camera. The computer can function as a learning processing unit by executing the program, and the learning processing unit performs additional learning on other monitoring logics of the same type as the monitoring logic of the monitoring device by using learning data of the image received by the image receiving unit. The computer can function as a monitoring logic sending unit by executing the program, and the monitoring logic sending unit sends other monitoring logics after additional learning to the monitoring device.

[0018] In addition, the above summary of the invention does not list all the essential features of the present invention. In addition, sub-combinations of these feature groups may also constitute inventions. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A system 1 according to this embodiment is shown.

[0020] Figure 2 Indicates the storage content of the correspondence table 501A.

[0021] Figure 3 Indicates the storage content of the correspondence table 501B.

[0022] Figure 4 Indicates actions related to monitoring of the monitoring area.

[0023] Figure 5 Indicates actions related to updating of the monitoring logic 41.

[0024] Figure 6 An example of a computer 2200 is shown in which various aspects of the present invention may be implemented in whole or in part.

[0025] Description of Reference Numerals

[0026] 1 system, 3 surveillance camera, 4 surveillance equipment, 5 learning device, 7 surveillance terminal, 10 communication network, 40 captured image acquisition unit, 41 surveillance logic, 42 surveillance logic execution unit, 43 communication unit, 44 update unit, 50 storage unit, 51 communication unit, 52 learning data generation unit, 53 learning processing unit, 70 communication unit, 71 display unit, 431 image sending unit, 432 surveillance result output unit, 433 surveillance logic receiving unit, 501 correspondence table, 510 image receiving unit, 511 surveillance logic sending unit, 2200 computer, 2201 DVD-ROM, 2210 main controller, 2212 CPU, 2214 RAM, 2216 graphic controller, 2218 display device, 2220 input / output controller, 2222 communication interface, 2224 hard disk drive, 2226 DVD-ROM drive, 2230 ROM, 2240 input / output chip, 2242 keyboard. DETAILED DESCRIPTION

[0027] The present invention will be described below by way of the embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all combinations of features described in the embodiments are essential to the solution means of the invention.

[0028] [1. System 1]

[0029] Figure 1 A system 1 according to this embodiment is shown.

[0030] The system 1 performs remote monitoring of a monitoring area, and includes: one or more monitoring cameras 3 , one or more monitoring equipment 4 , a learning device 5 and a monitoring terminal 7 .

[0031] The monitoring area may be an area monitored by the monitoring terminal 7, and may include an area photographed by each monitoring camera 3 (also referred to as a photographing area). In the present embodiment, as an example, the monitoring area is the land of a facility, but it may also be a driveway or a sidewalk, etc. The facility may be a factory, or it may be a school, a residence, a station, an airport, a museum, a hospital, a store (as an example, a restaurant, a convenience store), etc., or it may be a theme park or an amusement park, their amusement facilities, etc. As a factory, in addition to chemical or biological industrial factories, there may be listed factories that manage and control wellheads such as gas fields or oil fields and their surroundings, factories that manage and control power generation such as hydropower, thermal power, and atomic energy, factories that manage and control environmental power generation such as solar power or wind power, and factories that manage and control water supply and drainage or dams, etc. Various security sensors (not shown) may be provided separately from the monitoring camera 3 in the facility.

[0032] Here, in each configuration of the system 1, the monitoring device 4, the learning apparatus 5, and the monitoring terminal 7 can be connected to each other via a communication network 10. In the figure, the monitoring device 4, the learning apparatus 5, and the monitoring terminal 7 are connected via a single communication network 10, but they may be connected via different communication networks.

[0033] The communication network 10 may include various networks such as the Internet, a wide area network (WAN), a local area network, or a combination thereof. The communication network 10 may include connection points based on at least one of wired and wireless. The communication network 10 may also be implemented by a dedicated line separated from a public line such as the Internet.

[0034] [1.1. Surveillance camera 3]

[0035] Each surveillance camera 3 shoots a shooting area. Each surveillance camera 3 can supply the image data obtained by shooting to the monitoring device 4. The image data can be dynamic image data, and the surveillance camera 3 can always shoot to generate dynamic image (also called video) data. For example, each surveillance camera 3 can shoot at 30f / s. In addition, the image data can also be still image data. In this case, the surveillance camera 3 can shoot at regular intervals to generate still image data. The surveillance camera 3 can be a visible light camera, or an infrared or ultraviolet (as an example, X-ray) camera.

[0036] [1.2. Monitoring equipment 4]

[0037] Each monitoring device 4 is connected to at least one monitoring camera 3. Each monitoring device 4 includes a captured image acquisition unit 40, a monitoring logic 41, a monitoring logic execution unit 42, a communication unit 43, and an update unit 44.

[0038] [1.2.(1) Photographed image acquisition unit 40]

[0039] The captured image acquisition unit 40 acquires an image captured by the monitoring camera 3. The captured image acquisition unit 40 can acquire image data from each monitoring camera 3 connected to the monitoring device 4. The captured image acquisition unit 40 can supply the acquired image data to the monitoring logic execution unit 42 and the communication unit 43. For example, the captured image acquisition unit 40 can supply the image data to the monitoring logic execution unit 42 in correspondence with the identification information (also referred to as the camera ID) of each monitoring camera 3. In addition, the captured image acquisition unit 40 can also store the acquired image data in a storage area (not shown) of the monitoring device 4. The storage area can store image data for at least a reference period (five days as an example), and can sequentially overwrite and store new image data.

[0040] [1.2.(2) Monitoring logic 41]

[0041] The monitoring logic 41 can perform image analysis on the image data according to the input of the image data of the analysis object, and output the judgment result of whether the shooting area is normal or abnormal. The judgment result can also include its confidence. The confidence is the degree of correctness of the judgment result, and as an example, it can be a percentage such as 40%, or a proportion such as 0.4.

[0042] When the detection object is detected, the monitoring logic 41 may output a judgment result indicating that the shooting area is abnormal. The detection object may be an object to be detected (also referred to as a target object) that may be included in the objects of the image data. The monitoring logic 41 may further output feature data indicating features of the target object based on the detection of the target object.

[0043] The target subject may be a subject that should be noted in terms of security among the subjects that can be detected by image analysis. For example, the target subject may be at least one of a person's face, a person's (for example, an adult or a child) whole body, clothing, a vehicle (for example, a moving vehicle or a parked vehicle), a vehicle type, a vehicle color, a license plate, and a specific object (for example, a bag or a weapon). In addition, the target subject may also be a specific person's face or a specific vehicle.

[0044] When the object subject is a human face, the feature data may represent the features of the face in the image. The features of the face in the image may be the positional relationship of the feature points of the face, or the color of the pupil, skin, hair, the color or shape of accessories (for example, glasses or earrings). In addition, when the object subject is a vehicle, the feature data may represent the type of vehicle in the image. In addition, when the object subject is a license plate, the feature data may represent the number of the vehicle in the image.

[0045] The image of the analysis object for the monitoring logic 41 to perform image analysis can be a still image, or, as an example, a frame extracted from a dynamic image. In addition, the detection object of the monitoring logic 41 can also be an action performed over multiple frames (as an example, the action of stealing goods). In this case, the image data of the analysis object for the monitoring logic 41 to perform image analysis can be dynamic image data of a reference time width (as an example, 10 seconds or 1 minute). The monitoring logic 41 can perform action analysis on the dynamic image data of the reference time width, and output a judgment result indicating that the shooting area is abnormal or feature data indicating the characteristics of the subject performing the action based on the detection of a specific action.

[0046] In addition, in the present embodiment, as an example, the monitoring device 4 is described as having a single monitoring logic 41, but it may also have a plurality of monitoring logics 41. In this case, at least one of the detection object or the shooting condition applicable to the analysis may be different between the plurality of monitoring logics 41. The shooting condition may be a condition determined by the monitoring camera 3 that performs shooting, or a condition determined by the installation status of the monitoring camera 3 or the shooting timing. For example, the shooting condition of the image data may be at least one of the performance (number of pixels, frame rate, etc.) of the monitoring camera 3 that performs shooting, the purpose (indoor use, outdoor use, etc.), the shooting direction (direction, direction relative to the light source, direction relative to the target subject that may be included in the image data, etc.), the installation location (address, GPS data, relative position relative to the target subject that may be included in the image data, indoor or outdoor, etc.), the number of target subjects that may be included in the image data (single or multiple, etc.), the status of the target subject that may be included in the image data (moving or stationary, etc.), and the shooting time period and the status of the shooting area at the time of shooting (lighting color, brightness, weather, etc.).

[0047] The monitoring logic 41 may be generated by a learning process. In the present embodiment, as an example, the monitoring logic 41 may be a model obtained by machine learning such as deep learning, but the present invention is not limited thereto.

[0048] [1.2.(3) Monitoring logic execution unit 42]

[0049] The monitoring logic execution unit 42 generates a monitoring result corresponding to the image captured by the monitoring camera 3 using the learned monitoring logic 41 .

[0050] The monitoring logic execution unit 42 can generate a monitoring result including the camera ID of the monitoring camera 3 and the judgment result based on the monitoring logic 41 (in this embodiment, as an example, a judgment result indicating whether the shooting area is normal or abnormal, the certainty of the judgment, etc.). The monitoring logic execution unit 42 can generate a monitoring result including at least one of the image data of the detected target object and the feature data of the detected target object based on the target object detected by the monitoring logic 41. In the case where the monitoring logic execution unit 42 uses multiple monitoring logics 41, the monitoring logic execution unit 42 can cause the multiple monitoring logics 41 to operate in parallel or at staggered timings, and generate monitoring results according to each monitoring logic 41.

[0051] The monitoring logic execution unit 42 may supply the monitoring result to the communication unit 43. In the present embodiment, as an example, the monitoring logic execution unit 42 may supply the monitoring result to an image transmission unit 431 and a monitoring result output unit 432 in the communication unit 43, which will be described later.

[0052] [1.2.(4) Communication Department 43]

[0053] The communication unit 43 communicates with the learning device 5 and the monitoring terminal 7 via the communication network 10. The communication unit 43 includes an image transmission unit 431, a monitoring result output unit 432, and a monitoring logic reception unit 433.

[0054] [1.2.(4-1) Image sending unit 431]

[0055] The image transmission unit 431 thins out the images captured by the monitoring camera 3 and transmits the thinned images to the learning device 5 .

[0056] For example, the image sending unit 431 may determine a portion of the plurality of images per unit amount analyzed by the monitoring logic 41 as the image to be sent. The image sending unit 431 may determine the image to be sent for each monitoring camera 3 and give the image a tag indicating the camera ID of the monitoring camera 3 that took the image.

[0057] The image sending unit 431 may randomly determine the image data to be sent, or may regularly thin out the images according to the arrangement order of the image data of each analysis unit of the monitoring logic 41, or may thin out the images based on the content of the image data or the analysis result. In this embodiment, as an example, the image sending unit 431 may thin out the images based on the monitoring result supplied from the monitoring logic execution unit 42.

[0058] [1.2. (4-2) Monitoring result output unit 432]

[0059] The monitoring result output unit 432 outputs the monitoring result generated by the monitoring logic execution unit 42. The monitoring result output unit 432 can send the monitoring result to a preset sending destination. In this embodiment, as an example, the monitoring result can be sent to the monitoring terminal 7. In addition, the sending destination is not limited to this, and can be the owner of the facility, a police station or a hospital in the area including the facility. In addition, the monitoring result output unit 432 can also display the monitoring result on a display (not shown) of the monitoring device 4.

[0060] [1.2.(4-3) Monitoring logic receiving unit 433]

[0061] The monitoring logic receiving unit 433 receives another monitoring logic 41A from the learning device 5. The monitoring logic 41A may be the same type of monitoring logic as the monitoring logic 41, or may be a monitoring logic of a different version from the monitoring logic 41. As an example, the monitoring logic 41A may be a monitoring logic of a newer version than the monitoring logic 41. The monitoring logic receiving unit 433 may supply the received monitoring logic 41A to the updating unit 44.

[0062] [1.2.(5) Update section 44]

[0063] The updating unit 44 updates the monitoring logic 41 executed by the monitoring logic executing unit 42 to the monitoring logic 41A received by the monitoring logic receiving unit 433. For example, the updating unit 44 may replace the monitoring logic 41 with the monitoring logic 41A.

[0064] [1.3. Learning device 5]

[0065] The learning device 5 performs a learning process on one or more monitoring logics 41A. The learning device 5 includes a storage unit 50 , a communication unit 51 , a learning data generation unit 52 , and a learning processing unit 53 .

[0066] [1.3.(1) Storage unit 50]

[0067] The storage unit 50 stores a plurality of monitoring logics 41A and two correspondence tables 501A and 501B.

[0068] The plurality of monitoring logics 41A may be monitoring logics of different types. For example, at least one of the detection object and the shooting condition suitable for analysis may be different between the monitoring logics 41A. The plurality of monitoring logics 41A may be the latest version. Each monitoring logic 41A may be given a label indicating a logic ID and version information. The version information may indicate the number of times the learning process is performed on the monitoring logic 41A.

[0069] The correspondence table 501A stores information about each monitoring logic 41A. The correspondence table 501A can store the logic ID, the latest version information (in this embodiment, as an example, the version information of the monitoring logic 41A in the storage unit 50), the shooting conditions of the image data suitable for analysis, and the device ID of the monitoring device 4 using the same type of monitoring logic 41 as the monitoring logic 41A in correspondence with each other.

[0070] The correspondence table 501B stores information about each monitoring device 4. The correspondence table 501B can store the device ID of the monitoring device 4, the camera ID of each monitoring camera 3 connected to the monitoring device 4, the shooting conditions of the image data of the monitoring camera 3, the logic ID of each monitoring logic 41 used in the monitoring device 4, and its version information in a corresponding manner.

[0071] [1.3.(2) Communications Department 51]

[0072] The communication unit 51 communicates with the monitoring device 4 and the monitoring terminal 7 via the communication network 10. The communication unit 51 includes an image receiving unit 510 and a monitoring logic transmitting unit 511.

[0073] [1.3.(2-1) Image receiving unit 510]

[0074] The image receiving unit 510 receives the image after thinning (also referred to as a thinned-out image) from the monitoring device 4 . The image receiving unit 510 may supply the received thinned-out image to the learning data generating unit 52 .

[0075] [1.3.(2-2) Monitoring logic transmission unit 511]

[0076] The monitoring logic sending unit 511 sends the monitoring logic 41A that has been learned by the learning processing unit 53 to the monitoring device 4. The monitoring logic sending unit 511 can send the monitoring logic 41A to the monitoring devices 4 of each device ID corresponding to the logical ID of the monitoring logic 41A that has been learned in the correspondence table 501A. The monitoring logic sending unit 511 can refer to the version information corresponding to each logical ID in the correspondence tables 501A and 501B, and send the monitoring logic 41A in the storage unit 50, that is, the latest version of the monitoring logic 41A, to the monitoring device 4 that uses a monitoring logic 41 that is not the latest version. Thus, the monitoring logic 41A after additional learning is sent to the monitoring device 4 that uses the same type of monitoring logic 41 before additional learning. The monitoring logic sending unit 511 can send the monitoring logic 41A after additional learning to a plurality of monitoring devices 4 that use the monitoring logic 41 before additional learning.

[0077] [1.3.(3) Learning data generation unit 52]

[0078] The learning data generating unit 52 generates learning data based on the thinned-out image received by the image receiving unit 510 .

[0079] The learning data generating unit 52 may generate learning data by performing image processing on the thinned-out image to protect the privacy of persons captured in the image, or may generate learning data by performing at least one of perspective correction, brightness correction, or hue correction.

[0080] Image processing to protect the privacy of a person may be, for example, non-recognizable processing of a facial part (masking as an example), or processing to represent the joints / bones of the body as points / lines through skeleton detection. View correction, brightness correction, and hue correction may be corrections to convert the image after thinning out to any view, brightness, or hue through the operation of the operator. These corrections may make the object subject in the image easy to identify or difficult to identify. The generated learning data may be given a label indicating the camera ID of the surveillance camera 3 that captured the image after thinning out.

[0081] [1.3.(4) Learning Processing Unit 53]

[0082] The learning processing unit 53 performs additional learning on one or more monitoring logics 41A by using the learning data of the thinned-out image received by the image receiving unit 510. The learning processing unit 53 may perform additional learning on other monitoring logics 41A of the same type as the monitoring logic 41 of the monitoring device 4. Performing additional learning on the monitoring logic 41A may be further performing learning processing on the monitoring logic 41A that has already been learned.

[0083] The learning processing unit 53 can perform additional learning on the monitoring logic 41A using the learning data generated by the learning data generating unit 52. The learning processing unit 53 can perform additional learning on the monitoring logic 41A in the storage unit 50 and update the learning data.

[0084] The learning processing unit 53 can detect the camera ID based on the label given to the learning data, and read the shooting condition corresponding to the camera ID from the correspondence table 501B. In addition, the learning processing unit 53 can read the logic ID of each monitoring logic 41 corresponding to the shooting condition read from the correspondence table 501B from the correspondence table 501A. The learning processing unit 53 can additionally learn the monitoring logic 41A of each logic ID read out based on the learning data.

[0085] [1.4. Monitoring terminal 7]

[0086] The monitoring terminal 7 monitors the monitoring area. The monitoring terminal 7 includes a communication unit 70 and a display unit 71. The communication unit 70 communicates with each monitoring device 4 via the communication network 10. The communication unit 70 can receive monitoring results from each monitoring device 4 and supply them to the display unit 71. The display unit 71 displays the monitoring results received by the communication unit 70. Thus, the monitoring area is monitored using the monitoring results.

[0087] In addition, the monitoring terminal 7 may also communicate with the learning device 5. For example, the monitoring terminal 7 may obtain the learning status of each monitoring logic 41A in the learning device 5 and the correspondence between the monitoring logic 41A and the monitoring device 4 from the learning device 5 and display them on the display unit 71. In this embodiment, as an example, the monitoring terminal 7 may display the version information of each monitoring logic 41A stored in the correspondence table 501A and the device ID of the monitoring device 4 using each monitoring logic 41A.

[0088] [1.5. Effects obtained from system 1]

[0089] [1.5. (1) Effects obtained from the monitoring device 4]

[0090] According to the monitoring device 4 of the above system 1, the images supplied to the monitoring logic 41 for generating monitoring results are thinned out and sent to the learning device 5, and used for additional learning of the monitoring logic 41A, so that the monitoring results can be generated sequentially based on the captured images, and on the other hand, the images can be thinned out to suppress the communication load and perform additional learning of the monitoring logic 41A. In addition, since the internal monitoring logic 41 is updated to the monitoring logic 41A after additional learning, monitoring can be performed by the monitoring logic 41A after additional learning without performing additional learning internally.

[0091] Furthermore, since the monitoring results are output to the monitoring terminal 7 or the like, monitoring can be performed using the output monitoring results.

[0092] [1.5.(2) Effects obtained from the learning device 5]

[0093] Furthermore, according to the learning device 5, the images supplied to the monitoring logic 41 for generating monitoring results are thinned out and received from the monitoring device 4 for additional learning of the monitoring logic 41A, so that the monitoring device 4 can sequentially generate monitoring results based on the images, and on the other hand, the communication load can be suppressed by thinning out the images and additional learning of the monitoring logic 41A can be performed. Furthermore, since the monitoring logic 41 of the monitoring device 4 can be updated to the monitoring logic 41A after additional learning, monitoring can be performed by the monitoring logic 41 after additional learning without performing additional learning inside the monitoring device 4.

[0094] Furthermore, since the monitoring logic 41A after additional learning is transmitted to the plurality of monitoring devices 4 that use the monitoring logic 41 before additional learning, monitoring can be performed using the additionally learned monitoring logic without causing each monitoring device 4 to perform a learning process.

[0095] Furthermore, since learning data is generated by performing image processing for protecting the privacy of a person captured in the image on the image received from the monitoring device 4 , it is possible to prevent the privacy of the subject from being violated by the image received from the monitoring device 4 .

[0096] Furthermore, since learning data is generated by performing at least one of angle of view correction, brightness correction, and color tone correction on the image received from the monitoring device 4, the received image can be made suitable for learning and the learning effect can be improved.

[0097] [2. Correspondence Table 501A, 501B]

[0098] Figure 2 Indicates the storage content of the correspondence table 501A. In the present embodiment, as an example, the correspondence table 501A can store, for each monitoring logic 41A, a logic ID (e.g., "L001", etc.), the latest version information (e.g., "ver.4", etc.), shooting conditions of image data suitable for analysis (e.g., the purpose of the monitoring camera 3 is indoor use, the installation location is indoors, etc.), and the device ID (e.g., "aaa1", etc.) of the monitoring device 4 using the same type of monitoring logic 41.

[0099] The contents of the correspondence table 501A can be arbitrarily set by the operator. In addition, the learning processing unit 53 can update the version information of the monitoring logic 41A every time additional learning is performed.

[0100] Figure 3 Indicates the storage content of the correspondence table 501B. As an example in the present embodiment, the correspondence table 501B may store, for each monitoring device 4, the device ID (e.g., "aaa1", etc.), the camera ID of each monitoring camera 3 connected to the monitoring device 4 (e.g., "c-01", etc.), the shooting conditions of the image data of the monitoring camera 3 (e.g., the purpose of the monitoring camera 3 is indoor use, the installation location is indoors, etc.), the logic ID of each monitoring logic 41 used in the monitoring device 4 (e.g., "L001", etc.) and its version information (e.g., "ver.3", etc.) in correspondence.

[0101] The contents of the correspondence table 501B can be set arbitrarily by the operator. For example, the camera ID and the shooting conditions can be updated every time the monitoring camera 3 is replaced or installed. In addition, the monitoring logic sending unit 511 can update the version information of the monitoring logic 41 every time the monitoring logic 41A is sent.

[0102] [3. Action of System 1]

[0103] [3.1. Monitoring of monitoring area]

[0104] Figure 4 The monitoring device 4 of the system 1 supports the monitoring of the monitoring area by performing the processes of steps S11 to S15.

[0105] In step S11, the captured image acquisition unit 40 acquires images captured by the monitoring camera 3. In the present embodiment, the captured image acquisition unit 40 acquires images from each monitoring camera 3 periodically (for example, every 5 seconds), but may acquire images all the time.

[0106] In step S13, the monitoring logic execution unit 42 uses the monitoring logic 41 to generate a monitoring result corresponding to the image captured by the monitoring camera 3. As an example in the present embodiment, the monitoring logic execution unit 42 can generate a monitoring result including the camera ID of the monitoring camera 3 and the judgment result of whether the shooting area is normal or abnormal for each monitoring camera 3. The judgment result can also include its confidence. The monitoring logic execution unit 42 can generate a monitoring result further including at least one of the image data of the detected target subject and the feature data of the detected target subject based on the detection of the target subject. In addition, in the case where the captured image acquisition unit 40 always acquires an image during the processing of step S11, the processing of step S13 and the processing of step S15 described later can also be performed in parallel with the processing of step S11.

[0107] In step S15, the monitoring result output unit 432 outputs the generated monitoring result. In this embodiment, as an example, the monitoring result output unit 432 may send the monitoring result to the monitoring terminal 7. Thereby, the monitoring area is monitored in the monitoring terminal 7.

[0108] [3.2. Update of monitoring logic 41]

[0109] Figure 5 This shows the operation related to the update of the monitoring logic 41. The system 1 updates the monitoring logic 41 by performing the processing of steps S21 to S25 and S31 to S37. This processing may be executed in parallel with the processing of steps S11 to S15.

[0110] In step S21, the image transmission unit 431 thins out the images captured by the monitoring camera 3 and transmits the thinned images to the learning device 5. The image transmission unit 431 may determine a portion of the plurality of images per unit amount analyzed by the monitoring logic 41 as the image to be transmitted. The image of the analysis unit may be still image data (for example, a frame included in dynamic image data) or a dynamic image of a reference time width.

[0111] The image sending unit 431 may determine at least one image that generates a monitoring result indicating an abnormality as a sending target. The image sending unit 431 may further determine at least one image before or after the at least one image that generates a monitoring result indicating an abnormality as a sending target. The image sending unit 431 may detect an image that generates a monitoring result indicating an abnormality based on the monitoring result supplied from the monitoring logic execution unit 42.

[0112] On this basis or in lieu thereof, the image sending unit 431 may determine as a sending target at least one image that generates a monitoring result whose confidence level of the judgment result of the monitoring logic 41 is less than a threshold. The image sending unit 431 may detect an image that generates a monitoring result whose confidence level is less than a threshold based on the monitoring result supplied from the monitoring logic execution unit 42.

[0113] In this embodiment, as an example, the threshold value may be variable. The threshold value may be arbitrarily set by the operator. On this basis or in lieu thereof, the threshold value may be adjusted by the image sending unit 431 according to the amount of images sent by the image sending unit 431. As an example, when the amount of images sent is larger than the reference amount, the threshold value may be changed to a higher value. Thus, the amount of images to be sent is reduced, and the amount of images sent is further reduced.

[0114] The image transmission unit 431 may transmit an image every time an image to be transmitted is determined, or may collectively transmit the images determined to be transmitted during the reference interval (for example, one day) at each reference interval.

[0115] In step S31 , the image receiving unit 510 of the learning device 5 receives the thinned-out image from the monitoring device 4 .

[0116] In step S33 , the learning data generating unit 52 generates learning data based on the thinned-out image received from the monitoring device 4 .

[0117] In step S35, the learning processing unit 53 performs additional learning on one or more monitoring logics 41A based on the learning data using the thinned-out image received by the image receiving unit 510. The learning processing unit 53 may perform additional learning on other monitoring logics 41A of the same type as the monitoring logic 41 of the monitoring device 4. The learning processing unit 53 may update the version information of the additionally learned monitoring logic 41A in the correspondence table 501A.

[0118] In step S37, the monitoring logic sending unit 511 sends the monitoring logic 41A that has been learned by the learning processing unit 53 to the monitoring device 4. As an example, the monitoring logic sending unit 511 may send the monitoring logic 41A to the monitoring device 4 that is the source of the image data. The monitoring logic sending unit 511 may update the version information of the monitoring logic 41 corresponding to the monitoring device 4 at the destination in the correspondence table 501B.

[0119] In step S23 , the monitoring logic receiving unit 433 of the monitoring device 4 receives the additionally learned monitoring logic 41A from the learning apparatus 5 .

[0120] Then, in step S25 , the updating unit 44 updates the monitoring logic 41 executed by the monitoring logic executing unit 42 to the monitoring logic 41A received by the monitoring logic receiving unit 433 .

[0121] According to the above operation, since at least one image generating a monitoring result indicating an abnormality is sent to the learning device 5, the image generating a monitoring result indicating an abnormality can be used for additional learning. Therefore, even in the case where the captured image is thinned out and used for additional learning, the learning efficiency of the additional learning can be improved.

[0122] Furthermore, in addition to the generated image showing the monitoring result of the abnormality, at least one of the previous and subsequent images is sent to the learning device 5, so that the learning efficiency of the additional learning can be further improved.

[0123] Furthermore, since at least one image that generates a monitoring result with a confidence level less than a threshold is sent to the learning device 5, images of insufficiently learned content can be used for additional learning. Therefore, even when the captured images are thinned out and used for additional learning, the learning efficiency of the additional learning can be improved.

[0124] Furthermore, since the threshold is adjusted according to the amount of images to be transmitted, the communication load of the images can be adjusted arbitrarily.

[0125] [4. Modifications]

[0126] In the above embodiment, the monitoring device 4 is described as having the monitoring logic 41 , but it is not necessary to have the monitoring logic 41 . In this case, the monitoring logic execution unit 42 may execute the monitoring logic 41 externally connected to the monitoring device 4 .

[0127] Furthermore, although it is described that the monitoring logic sending unit 511 sends the additionally learned monitoring logic 41A to the monitoring device 4 using the same type of monitoring logic 41 as the monitoring logic 41A, the monitoring logic sending unit 511 may send the additionally learned monitoring logic 41 to the monitoring device 4 using a different type of monitoring logic 41. For example, the monitoring logic sending unit 511 may send the additionally learned monitoring logic 41 to the monitoring device 4 using a different type of monitoring logic 41 based on the operator changing the correspondence between the monitoring logic 41 and the monitoring device 4 in the correspondence tables 501A and 501B.

[0128] In addition, various embodiments of the present invention may be described with reference to flow charts and block diagrams, where a module may represent (1) a stage of a process for performing an operation or (2) a portion of a device having the function of performing an operation. Specific stages and portions may be implemented by dedicated circuits, programmable circuits supplied together with computer-readable instructions stored on a computer-readable medium, and / or processors supplied together with computer-readable instructions stored on a computer-readable medium. Dedicated circuits may include digital and / or analog hardware circuits, and may also include integrated circuits (ICs) and / or discrete circuits. Programmable circuits may include reconfigurable hardware circuits, which include logical AND, logical OR, logical XOR, logical NAND, logical NOR and other logical operations, flip-flops, registers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs) and other memory elements, etc.

[0129] Computer-readable media may include any tangible device capable of storing instructions executed by an appropriate device, and as a result, a computer-readable medium having instructions stored therein includes a product containing instructions that can be executed in order to make a means for performing the operations specified by the flowchart or block diagram. Examples of computer-readable media may include: electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable media may include: floppy (registered trademark) disks, magnetic disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), electrically erasable programmable read-only memories (EEPROM), static random access memories (SRAM), compact disk read-only memories (CD-ROM), digital versatile disks (DVD), Blu-ray (RTM) disks, memory sticks, integrated circuit cards, etc.

[0130] Computer readable instructions include any of source code and object code described by any combination of one or more programming languages ​​including assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or object-oriented programming languages ​​such as Smalltalk, JAVA (registered trademark), C++, and existing procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0131] The computer readable instructions may be provided to a processor or programmable circuit of a general-purpose computer, a special-purpose computer or other programmable data processing device via a local or local area network (LAN), a wide area network (WAN) such as the Internet, and the computer readable instructions are executed to produce a means for performing the operations specified by the flowchart or block diagram. Examples of processors include: computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0132] Figure 6 The computer 2200 is an example of a computer that can implement various modes of the present invention in whole or in part. The computer 2200 can perform operations associated with the device of the embodiment of the present invention or functions of one or more parts of the device, or perform the operations or the one or more parts, and / or the computer 2200 can perform processes of the embodiment of the present invention or stages of the processes, through the program installed in the computer 2200. In order to make the computer 2200 perform specific operations associated with some or all of the modules in the flowcharts and block diagrams described in this specification, the CPU 2212 can execute such a program.

[0133] The computer 2200 of this embodiment includes a CPU 2212, a RAM 2214, a graphic controller 2216, and a display device 2218, which are connected to each other through a main controller 2210. The computer 2200 also includes an input / output unit such as a communication interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive, which are connected to the main controller 2210 via an input / output controller 2220. The computer also includes a conventional input / output unit such as a ROM 2230 and a keyboard 2242, which are connected to the input / output controller 2220 via an input / output chip 2240.

[0134] The CPU 2212 controls each unit by operating according to the program stored in the ROM 2230 and the RAM 2214. The graphic controller 2216 acquires image data generated by the CPU 2212 in a frame buffer or the like provided in the RAM 2214 or in itself, and displays the image data on the display device 2218.

[0135] The communication interface 2222 can communicate with other electronic devices via a network. The hard disk drive 2224 stores programs and data used by the CPU 2212 in the computer 2200. The DVD-ROM drive 2226 reads programs or data from the DVD-ROM 2201 and provides the programs or data to the hard disk drive 2224 via the RAM 2214. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0136] The ROM 2230 stores therein a boot program or the like executed by the computer 2200 upon activation and / or a program that depends on the hardware of the computer 2200. The input / output chip 2240 may also connect various input / output units to the input / output controller 2220 via a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0137] The program is provided by a computer-readable medium such as DVD-ROM 2201 or an IC card. The program is read from the computer-readable medium and installed in the hard disk drive 2224, RAM 2214 or ROM 2230, which is also an example of a computer-readable medium, and is executed by the CPU 2212. The information processing described in these programs is read to the computer 2200, thereby bringing about cooperation between the program and the various types of hardware resources described above. The device or method can be constituted by realizing the operation or processing of information with the use of the computer 2200.

[0138] For example, when communication is performed between the computer 2200 and an external device, the CPU 2212 can execute a communication program loaded in the RAM 2214, and instruct the communication interface 2222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 2212, the communication interface 2222 reads the transmission data stored in the transmission buffer processing area provided in the RAM 2214, the hard disk drive 2224, the DVD-ROM 2201, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes the reception data received from the network to the reception buffer processing area provided on the recording medium, etc.

[0139] Furthermore, the CPU 2212 can read all or a necessary part of a file or database stored in an external recording medium such as the hard disk drive 2224, the DVD-ROM drive 2226 (DVD-ROM 2201), or an IC card into the RAM 2214, and perform various types of processing on the data on the RAM 2214. Then, the CPU 2212 writes the processed data back to the external recording medium.

[0140] Various types of information such as various types of programs, data, tables, and databases can be stored in the recording medium and receive information processing. CPU2212 performs various types of processing described in various places of this disclosure on the data read from RAM2214 and writes the results back to RAM2214. The various types of processing include various types of operations specified by the instruction sequence of the program, information processing, conditional judgment, conditional branching, unconditional branching, information retrieval / replacement, etc. In addition, CPU2212 can retrieve information in files, databases, etc. in the recording medium. For example, in the case where multiple entries having attribute values ​​of the first attribute associated with the attribute values ​​of the second attribute are stored in the recording medium, CPU2212 can retrieve an entry consistent with the condition specifying the attribute value of the first attribute from the multiple entries, and read the attribute value of the second attribute stored in the entry, thereby obtaining the attribute value of the second attribute associated with the first attribute that meets the predetermined condition.

[0141] The program or software module described above may be stored in a computer-readable medium on or near the computer 2200. In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable medium, thereby providing the program to the computer 2200 via the network.

[0142] The present invention has been described above using the embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various changes or improvements can be made to the above embodiments. According to the description of the claims, the method of making such changes or improvements can also be included in the technical scope of the present invention.

[0143] The execution order of each process such as actions, processes, steps and stages in the apparatus, system, program and method shown in the claims, specifications and drawings is not specifically indicated as "earlier", "before", etc. In addition, it should be noted that as long as the output of the previous process is not used in the subsequent process, it can be implemented in any order. Even if the action flow in the claims, specifications and drawings is described using "first," "next," etc. for the sake of convenience, it does not mean that it must be implemented in this order.

Claims

1. A monitoring device, characterized in that include: A captured image acquisition unit that acquires images captured by a surveillance camera; a monitoring logic execution unit that generates a monitoring result corresponding to the image captured by the monitoring camera using the learned monitoring logic; an image sending unit that thins out the image captured by the monitoring camera and sends it to a learning device that performs a learning process on other monitoring logics of the same type as the monitoring logic; a monitoring logic receiving unit that receives, from the learning device, the other monitoring logic that has been additionally learned by the learning device using the image from the image sending unit; as well as an updating unit, which updates the monitoring logic executed by the monitoring logic executing unit to the other monitoring logic received by the monitoring logic receiving unit, The learning device stores a first table and a second table, wherein the first table establishes a correspondence between identification information of a monitoring logic and a shooting condition of an image suitable for analysis of the monitoring logic, and the second table establishes a correspondence between identification information of a monitoring camera and a shooting condition of an image of the monitoring camera, The learning device reads out from the second table the shooting conditions corresponding to the surveillance camera that shot the image sent from the image sending unit, reads out from the first table the identification information of the monitoring logic corresponding to the shooting conditions, and performs additional learning on the monitoring logic of the identification information by using the learning data of the image.

2. The monitoring device according to claim 1, characterized in that: The image transmitting unit transmits at least one image generating the monitoring result indicating abnormality.

3. The monitoring device according to claim 2, characterized in that: The image transmitting unit further transmits at least one image before or after the at least one image indicating the abnormal monitoring result is generated.

4. The monitoring device according to claim 1, characterized in that The image transmitting unit transmits at least one image that generates the monitoring result with a certainty lower than a threshold value.

5. The monitoring device according to claim 4, characterized in that: The image transmission unit adjusts the threshold value according to the amount of images to be transmitted.

6. The monitoring device according to any one of claims 1 to 5, characterized in that: A monitoring result output unit is included to output the monitoring result.

7. A learning device, characterized in that include: An image receiving unit receives an image that has been taken and thinned out from a monitoring device that generates a monitoring result corresponding to an image taken by a monitoring camera using monitoring logic; a learning processing unit that performs additional learning on another monitoring logic of the same type as the monitoring logic of the monitoring device by using the learning data of the image received by the image receiving unit; A monitoring logic sending unit, which sends the other monitoring logic after additional learning to the monitoring device; as well as a storage unit storing a first table and a second table, wherein the first table establishes a correspondence between identification information of a monitoring logic and a shooting condition of an image suitable for analysis of the monitoring logic, and the second table establishes a correspondence between identification information of a monitoring camera and a shooting condition of an image of the monitoring camera, The learning processing unit reads out from the second table the shooting conditions corresponding to the surveillance camera that shot the image received by the image receiving unit, reads out from the first table the identification information of the monitoring logic corresponding to the shooting conditions, and performs additional learning on the monitoring logic of the identification information by using the learning data of the image.

8. The learning device according to claim 7, characterized in that: The monitoring logic transmitting unit transmits the other monitoring logic after additional learning to a plurality of monitoring devices that use the monitoring logic before additional learning.

9. The learning device according to claim 7 or 8, characterized in that: The learning data generating unit is provided for generating the learning data by performing image processing for protecting the privacy of a person captured in the image on the image received from the monitoring device.

10. The learning device according to claim 7 or 8, characterized in that: The learning data generating unit is provided for generating the learning data by performing at least one of angle of view correction, brightness correction, and color tone correction on the image received from the monitoring device.

11. A method performed by a monitoring device, characterized in that include: In the captured image acquisition stage, images captured by the surveillance camera are acquired; In the monitoring logic execution phase, the learned monitoring logic is used to generate monitoring results corresponding to the images captured by the monitoring camera; In the image sending stage, the images taken by the monitoring camera are thinned out and sent to a learning device that performs learning processing on other monitoring logics of the same type as the monitoring logic; A monitoring logic receiving stage, receiving from the learning device the other monitoring logic additionally learned by the learning device using the image sent in the image sending stage; as well as an updating phase, updating the monitoring logic executed by the monitoring logic execution phase to the other monitoring logic received by the monitoring logic reception phase, The learning device stores a first table and a second table, wherein the first table establishes a correspondence between identification information of a monitoring logic and a shooting condition of an image suitable for analysis of the monitoring logic, and the second table establishes a correspondence between identification information of a monitoring camera and a shooting condition of an image of the monitoring camera, The learning device reads out from the second table the shooting conditions corresponding to the surveillance camera that shot the image sent from the image sending stage, reads out from the first table the identification information of the monitoring logic corresponding to the shooting conditions, and performs additional learning on the monitoring logic of the identification information by using the learning data of the image.

12. A method performed by a learning device, characterized in that include: In the image receiving stage, the image captured and thinned out is received from a monitoring device that uses monitoring logic to generate monitoring results corresponding to the image captured by the monitoring camera; In a learning processing stage, additional learning is performed on other monitoring logics of the same type as the monitoring logic of the monitoring device by using the learning data of the image received in the image receiving stage; In the monitoring logic sending stage, the other monitoring logic after additional learning is sent to the monitoring device; as well as a storage stage of storing a first table and a second table, wherein the first table establishes a correspondence between identification information of a monitoring logic and a shooting condition of an image suitable for analysis of the monitoring logic, and the second table establishes a correspondence between identification information of a monitoring camera and a shooting condition of an image of the monitoring camera, The learning processing stage reads out from the second table the shooting conditions corresponding to the surveillance camera that shot the image received by the image receiving stage, reads out from the first table the identification information of the monitoring logic corresponding to the shooting conditions, and performs additional learning on the monitoring logic of the identification information by using the learning data of the image.

13. A recording medium having a program recorded thereon, characterized in that: The computer executes the program to function as a captured image acquisition unit, a monitoring logic execution unit, an image transmission unit, a monitoring logic reception unit, and an update unit. The captured image acquisition unit acquires an image captured by a surveillance camera. The monitoring logic execution unit generates a monitoring result corresponding to the image captured by the monitoring camera using the learned monitoring logic. The image sending unit thins out the image captured by the monitoring camera and sends the image to a learning device that performs a learning process on another monitoring logic of the same type as the monitoring logic. The monitoring logic receiving unit receives from the learning device the other monitoring logic additionally learned by the learning device using the image from the image transmitting unit, The updating unit updates the monitoring logic executed by the monitoring logic executing unit to the other monitoring logic received by the monitoring logic receiving unit, The learning device stores a first table and a second table, wherein the first table establishes a correspondence between identification information of a monitoring logic and a shooting condition of an image suitable for analysis of the monitoring logic, and the second table establishes a correspondence between identification information of a monitoring camera and a shooting condition of an image of the monitoring camera, The learning device reads out from the second table the shooting conditions corresponding to the surveillance camera that shot the image sent from the image sending unit, reads out from the first table the identification information of the monitoring logic corresponding to the shooting conditions, and performs additional learning on the monitoring logic of the identification information by using the learning data of the image.

14. A recording medium having a program recorded thereon, characterized in that: The computer executes the program to function as an image receiving unit, a learning processing unit, a monitoring logic sending unit, and a storage unit. The image receiving unit receives an image that has been thinned out after being photographed from a monitoring device that generates a monitoring result corresponding to an image photographed by a monitoring camera using monitoring logic. The learning processing unit performs additional learning on another monitoring logic of the same type as the monitoring logic of the monitoring device by using the learning data of the image received by the image receiving unit. The monitoring logic sending unit sends the other monitoring logic after additional learning to the monitoring device, The storage unit stores a first table and a second table, wherein the first table establishes a correspondence between identification information of a monitoring logic and a shooting condition of an image suitable for analysis of the monitoring logic, and the second table establishes a correspondence between identification information of a monitoring camera and a shooting condition of an image of the monitoring camera, The learning processing unit reads out from the second table the shooting conditions corresponding to the surveillance camera that shot the image received by the image receiving unit, reads out from the first table the identification information of the monitoring logic corresponding to the shooting conditions, and performs additional learning on the monitoring logic of the identification information by using the learning data of the image.

Images