Information processing device, information processing system, information processing method, program and storage medium

Abstract

To specify a prescribed moving image section while reducing a processing load of image quality improvement.SOLUTION: An information processing device executes image processing for improving image quality to a plurality of images acquired prescribed frequently by an image acquisition part frequently less than the prescribed frequently. It executes first image analysis processing for determining whether each image undergoing the image processing is an image that satisfies a first condition. It executes image processing to at least one image of an image acquired before the image determined to satisfy the first condition and an image acquired after the image determined to satisfy the first condition, and executes second image analysis processing for determining whether the image undergoing the execution of the image processing and determined to satisfy the first condition is an image that satisfies a second condition. It specifies a moving image section that satisfies the second condition in a moving image composed of the plurality of images on the basis of a result of the second image analysis processing.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing system, an information processing method, a program, and a storage medium. [Background technology]

[0002] In surveillance systems, huge amounts of video footage captured by surveillance cameras are used to detect and analyze abnormal situations such as fires and crimes. For users conducting real-time surveillance and video analysis, it takes a lot of effort to visually check all of this video footage and identify video sections where abnormalities are occurring. In addition, it is difficult for users to check the video footage in environments with poor visibility, such as at night or when foggy. Therefore, it is desirable to support users' surveillance and analysis by automatically detecting abnormalities using anomaly detection processing that uses image recognition technology and presenting the images as high-quality, highly visible images.

[0003] Several methods have been proposed for compressing huge amounts of surveillance camera video in order to efficiently use recording media and communication resources and to make monitoring / analysis easier for users. In Patent Document 1, the compression rate of frames with low abnormality is increased to reduce the overall size of video data while maintaining high image quality and frame rate in abnormal video sections to be analyzed.

[0004] In addition, in Patent Document 2, the resolution of the images transmitted from the surveillance camera to the analysis device is controlled depending on whether or not an abnormality is detected, and low-quality compressed images are transmitted under normal circumstances, while uncompressed high-quality images are transmitted for the video sections before and after the abnormality. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 5190968 [Patent Document 2] Patent Publication No. 2006-93839 Summary of the Invention [Problem to be solved by the invention]

[0006] The problem to be solved by the present invention is to identify a predetermined moving image section while reducing the processing load for achieving high image quality. [Means for solving the problem]

[0007] In order to solve the above problem, an information processing device according to one embodiment of the present invention has an image quality improvement unit that performs image processing to improve image quality on a plurality of images acquired at a predetermined frequency by an image acquisition unit at a frequency lower than the predetermined frequency, and a first image analysis unit that performs a first image analysis process on the images on which the image processing has been performed to determine whether or not the images satisfy a first condition, wherein the image quality improvement unit further has a second image analysis unit that performs the image processing on at least one of an image acquired before the image determined to satisfy the first condition and an image acquired after the image determined to satisfy the first condition, and that performs a second image analysis process on the images on which the image processing has been performed and that determine whether or not the images determined to satisfy the first condition satisfy a second condition, and a section identification unit that identifies a video section that satisfies the second condition among the video consisting of the plurality of images based on a result of the second image analysis process. Effect of the Invention

[0008] According to the present invention, it is possible to identify a predetermined moving image section while reducing the processing load for improving image quality. [Brief description of the drawings]

[0009] [Figure 1] A diagram showing an example of the hardware configuration of a monitoring system. [Diagram 2] FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device; [Diagram 3] 1 is a flowchart showing a processing procedure executed by an information processing device; [Figure 4] Examples of time series images and images before and after image quality improvement processing [Diagram 5] FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device; [Figure 6] 1 is a flowchart showing a processing procedure executed by an information processing device; [Figure 7] Examples of time series images and images before and after image quality improvement processing [Figure 8] Hardware configuration diagram of a surveillance camera device [Figure 9] FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] With reference to the attached drawings, the embodiment for carrying out the present invention will be described in detail. The embodiment described below is an example of a means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiment. Note that the same reference numerals in the drawings perform the same operation, so repeated explanations will be omitted. Also, the components listed in this embodiment are merely examples, and are not intended to limit the scope of the present invention to only them.

[0011] The present invention relates to a process for identifying a video section by performing image analysis processing such as anomaly detection on video captured by a surveillance camera or the like in two stages, namely, low-load processing and high-load processing combined with image quality improvement processing. In addition to anomaly detection processing, image analysis processing includes object and person detection processing, tracking processing, and scene recognition processing. Anomaly detection processing includes processing for detecting anomalies based on primitive features such as color changes and movements, and semantic anomalies such as anomalies in human behavior. Image quality improvement processing includes, for example, image noise removal processing, brightness correction processing for captured images in low-illumination environments, high-resolution processing for low-resolution images, and mist removal processing from captured images when mist occurs.

[0012] <Embodiment 1> In this embodiment, a case will be described in which an anomaly detection process in which a person's fall is defined as an anomaly is used as the image analysis process, and a defogger process is used as the image quality improvement process. The information processing device according to this embodiment performs image processing (defogger process) for improving images at a frequency lower than the predetermined frequency for a plurality of images acquired at a predetermined frequency by an image acquisition unit. In addition, a first image analysis process is performed on the images that have been subjected to the image quality improvement process to determine whether or not the images satisfy a first condition. In this embodiment, the first condition is whether or not a person's fall is detected from the image.

[0013] Furthermore, image processing is also performed on at least one of the images acquired before the image determined to satisfy the first condition and the image acquired after the image determined to satisfy the first condition. Then, a second image analysis process is performed to determine whether the image on which the image processing has been performed satisfies the second condition. In this embodiment, the second condition is the same as the first condition, that is, whether a person's fall is detected. Finally, based on the processing result of the second image analysis process, a layer that satisfies the second condition is identified from among the video consisting of multiple images.

[0014] (Hardware configuration) In this embodiment, a remote monitoring system using a monitoring camera connected to a network is assumed. FIG. 1(a) shows the hardware configuration of this monitoring system. This monitoring system is composed of a monitoring camera device 101 and an analysis device 103 connected by a network 102. As shown in FIG. 1(a), the monitoring camera device 101 may be a single camera or a group of multiple monitoring cameras 100. In this embodiment, the user is described as monitoring / analyzing the video that the analysis device has received and stored in advance from the monitoring camera, but the video received from the monitoring camera may be directly monitored / analyzed. Note that a case where part of the processing performed by the analysis device is performed on the monitoring camera side will be described in the third embodiment described later. FIG. 1(b) is a hardware configuration diagram of the analysis device in this embodiment. The CPU 103-1 controls the entire device by executing a control program stored in the ROM 103-2. The RAM 103-3 temporarily stores various data from each component. In addition, the program is expanded and made executable by the CPU 103-1.

[0015] The storage unit 103-4 stores data to be processed in this embodiment, and is used to temporarily store camera images received from security cameras and data generated during information processing. As a medium for the storage unit 103-4, a HDD, a flash memory, various optical media, etc. can be used.

[0016] (Analysis processing) Fig. 2 shows an example of the functional configuration of the information processing device 1 according to this embodiment. The information processing device 1 corresponds to the analysis device in Fig. 1, and includes an image acquisition unit 201, an image quality improvement unit 202, a first image analysis unit 203, a second image analysis unit 204, a temporary recording unit 205, a section identification unit 206, and a storage unit 207. Each functional configuration unit will be briefly described.

[0017] The image acquisition unit 201 sequentially (at a predetermined frequency) acquires time-series images from the storage unit 207. The time-series images correspond to surveillance camera footage. The image acquisition unit 201 may acquire images directly from the imaging unit of the surveillance camera device 101.

[0018] The image quality improvement unit 202 performs image processing for improving image quality (improving image quality) on a plurality of images (time-series images) obtained from the image acquisition unit 201, and outputs high-image-quality images. At this time, the image quality improvement unit 202 performs the image quality improvement processing less frequently than the frequency at which the time-series images are acquired by the image acquisition unit 201. More specific operations will be described later. The first image analysis unit 203 performs a first image analysis process (anomaly detection process) on the high-image-quality images obtained from the image quality improvement unit 202 to determine whether or not the images satisfy a first condition.

[0019] The second image analysis unit 204, like the first image analysis unit 203, performs a second image analysis process (anomaly detection process) on the image quality improvement image obtained from the image quality improvement unit 202 to determine whether or not the image satisfies a second condition. Note that in this embodiment, the first condition and the second condition are whether or not the image is one in which a fall can be detected.

[0020] However, the process is performed when the first image analysis unit 203 detects an abnormality from the time-series images. At this time, the second image analysis process is sequentially performed on images acquired before or after the image in which the first image analysis unit 203 detects an abnormality among the acquired multiple time-series images. Furthermore, the abnormality detection processes performed by the first image analysis unit 203 and the second image analysis unit 203 may be the same or different processes.

[0021] The temporary recording unit 205 temporarily stores the high quality images analyzed by the first image analysis unit 203 and the second image analysis unit 204, and the first and second image analysis results obtained as outputs. The section identification unit 206 acquires the first and second image analysis results recorded in the temporary recording unit 205, and identifies an abnormal section based on these. The memory unit 207 records the surveillance camera video for which an abnormal section is to be identified, and information on the identified abnormal section.

[0022] 3 is a flowchart showing the flow of identifying an abnormal section in this embodiment. In the following description, each process (step) is represented by adding an S to the beginning, and the notation of the process (step) is omitted. However, the information processing device does not necessarily have to perform all of the processes described in this flowchart. The processes executed by CPU 103-1 are shown as functional blocks. Therefore, the series of processes shown in this flowchart are performed by CPU 103-1 loading a program stored in ROM 103-2 into RAM 103-3 and executing it line by line.

[0023] In S301, the image acquisition unit 201 acquires time-series images from the storage unit 207 in chronological order at intervals of T sheets. An example of the time-series images is shown in FIG. 4. FIG. 4(a) shows images at times t=0 to 9 arranged in a row, and a person in the image at times t=2 to 5 has fallen. The present embodiment aims to identify the falling section t=2 to 5 from the time-series images without excess or deficiency. In addition, in the present embodiment, fog occurs at the time and location of the image, making it difficult to see the person in the image, and the abnormality detection process performed thereafter does not assume a foggy image, and assumes a situation in which accuracy cannot be achieved without a fog removal process. Here, a case will be described in which images are acquired in sequence from an image at time t=0 at intervals of T=4 sheets. At this time, images at times t=0, 4, and 8 are acquired in sequence. Then, in the subsequent process, image quality improvement and abnormality detection process are performed on these images acquired at intervals of T sheets. This can reduce the processing load compared to the case in which the time-series images are processed every frame.

[0024] Next, in S302, the image quality improving unit 202 improves the image quality of the time-series images acquired by the image acquiring unit 201 in S301 to generate high-image quality images. Here, the subscript t is an index indicating time, and indicates that this is the t-th image in the time-series images. Note that images are acquired at intervals of T=1 image (a predetermined frequency), and the image processing (image quality improvement processing) described below may be performed at a frequency lower than the predetermined frequency (T=4 image intervals). The following describes a case where the image acquiring unit 201 acquires images at intervals of T=4 images, but this may also be achieved by acquiring images at intervals of T=1 image, and executing the image quality improvement processing at intervals of T=4 images.

[0025] In S303, the first image analysis unit 203 performs anomaly detection processing on the image quality improvement image obtained from the image quality improvement unit 202 in S302. In this description, a fall detection processing is used as the anomaly detection processing. For example, a process may be performed in which images of a person walking normally are learned and a deviation in gait is detected, or a judger may be used that judges whether or not a fall has occurred based on a posture detected by a posture detector. Then, the image quality improvement image and the anomaly detection result are recorded in the temporary recording unit 205 as the first image analysis result.

[0026] In S304, the first image analysis unit determines whether to proceed to the second image analysis process based on the abnormality detection result. Specifically, if no abnormality is detected, the process returns to S301, and if an abnormality is detected, the process proceeds to S305. In the example of FIG. 4, the image at time t=0 is determined to be normal, so the process returns to S301. In this embodiment, the first image analysis process is performed at intervals of T=4, so the image at time t=0 is processed next to the image at time t=4. Then, since the image at time t=4 is determined to be abnormal, the process does not return to S301, and the process from S305 onwards is performed. In S305 to S308, images before time t=4 are processed, and in S309 to S312, images after time t=4 are processed. The processes of S305 to S308 and S309 to S312 may be performed in parallel or in order, and are not limited to this.

[0027] In S305, the image acquisition unit 201 sequentially acquires images t' at time points before time t from the storage unit 207 at intervals of T'. That is, starting from the image in which an abnormality is detected in the first image analysis process, at least one image is acquired from among the images acquired before the starting image and the images acquired thereafter. Here, 0 ≤ T' < T, and the images are acquired at a higher frequency than the frequency at which the images for the first image analysis are acquired in S301. In the example of FIG. 4, when T' = 1, the images are acquired in the order of times t = 3, 2, 1 going back from time t = 4 at which an abnormality is detected in S304.

[0028] In S306, the image quality improvement unit 202 improves the image quality of the image t' acquired by the image acquisition unit 201 in S305 to generate a high image quality image t'. In other words, for the images acquired before the image determined to satisfy the first condition, an image process (image quality improvement process) for improving the image quality is executed. Examples of the images before and after the image quality improvement are shown in FIGS. 4(b-1) and 4(b-2). By applying the image quality improvement process by the image quality improvement unit 202 to the image shown in FIG. 4(b-1) of a scene in which haze has occurred and removing the haze as shown in FIG. 4(b-2), it becomes easier to perform the abnormality detection process and easier for the user to perform monitoring / analysis.

[0029] In S307, the second image analysis unit 204 performs an abnormality detection process on the high image quality image t' obtained from the image quality improvement unit 202 in S306. Then, the abnormality detection result is recorded in the temporary recording unit 205 as the second image analysis result.

[0030] In S308, the second image analysis unit 204 judges whether to end the second image analysis. The end judgment may be performed by sequentially tracing images before and after the image judged to satisfy the first condition, and judging the end at the point when no abnormality is detected in S307. That is, the second analysis process is executed until the end judgment is performed. In that case, in the example of FIG. 4, since the images at times t=3 and t=2 are judged to be abnormal in S307, the process returns to S305 and is executed. Then, since the image at time t=1 is judged to be normal in S307, it is judged to be ended at time t=1. That is, the image quality improvement process is executed on an image that was acquired before the image judged to satisfy the first condition and satisfies the second condition.

[0031] The method of determining the end is not limited to this, and the end may be determined when the second image analysis process of S307 is performed on a predetermined number of images. That is, the end may be determined based on whether the number of images determined to satisfy the second condition exceeds a predetermined number. When it is desired to specify the section where an abnormality is detected more precisely, the end may be determined based on whether the second condition (or the first condition) is satisfied. In addition, the second image analysis results of the images before and after the detection or at each time obtained in S307 may be saved, and the end may be determined when the number of images where an abnormality is detected in the most recent images becomes smaller than a predetermined threshold. In this way, it is possible to prevent the abnormality from being overlooked due to the influence of outliers such as whiteout of the image caused by changes in the lighting conditions, and the abnormal section can be specified more robustly. In addition, the monitoring target in the time series images may be tracked, and the end may be determined when the section where the same person as the person determined to be abnormal is shown in the video is exceeded. Furthermore, in parallel with S305 to S308, the tracking process may be performed going back in time, and the process may be determined to be completed when the same person goes out of frame.

[0032] In S309, the image acquisition unit 201 sequentially acquires images t' at times after time t at intervals of T' from the storage unit 207. In the example of Fig. 4, if T' = 1, images are acquired in the order of time t = 5, t = 6, starting from time t = 4 when an abnormality was detected in S304.

[0033] In S310, similarly to S306, the image quality improving unit 202 improves the image quality of the image t' acquired by the image acquiring unit 201 in S305 to generate a high image quality image t'. That is, image processing (image quality improvement processing) for improving the image quality is executed on an image acquired after the image determined to satisfy the first condition.

[0034] In S311, similarly to S307, the second image analysis unit 204 performs anomaly detection processing on the image quality improvement image t' obtained in S310 from the image quality improvement unit 202. Then, the anomaly detection result is recorded in the temporary recording unit 205 as the second image analysis result.

[0035] In S312, similar to S308, the second image analysis unit 204 judges whether to end the second image analysis. In the example of Fig. 4, first, the image at time t = 5 is judged to be normal in S311, so the process returns to S301 and is carried out. Then, the image at time t = 6 is judged to be normal in S311. If the end judgment in this process S312 is the point in time when even one image is found to be normal, it is judged to end at time t = 6.

[0036] In S313, the section identification unit 206 identifies an abnormal section (a moving image section that satisfies the second condition) in a moving image (time-series images) consisting of a plurality of images based on the second image analysis result recorded in the temporary recording unit 205. For example, in the example of FIG. 4, the abnormal section may be from the image with the earliest time determined to be abnormal by the second image analysis to the image with the latest time. In that case, t=2 to 5 is identified as the abnormal section. Alternatively, the section in which the second image analysis was performed may be identified as the abnormal section. In that case, the abnormal section is t=1 to 6.

[0037] The information of the identified abnormal section may be recorded in the storage unit 207 and used for the user's search and analysis of the abnormal section. Instead of recording the information of the abnormal section, it may be used for displaying it in the form of a thumbnail or the like. In addition, as the information of the abnormal section, the type of abnormality, the first image analysis result at each time, and the second image analysis result may be recorded and displayed, or used to control the display method. In addition, an image with high image quality at a time corresponding to the abnormal section may be acquired from the temporary recording unit 205 and stored in the storage unit 207 as an abnormal section video. In addition, it may be used to determine a section to be deleted from the storage unit 207. By deleting or compressing images from the storage unit 207 at a time earlier than the abnormal section, only the images of the parts required for analysis are left in the recording unit, improving the viewability for the user and making effective use of resources when the capacity of the recording unit is limited. After the process of S313 is completed, the information temporarily recorded in the temporary recording unit 205 may be discarded.

[0038] In this embodiment, an example has been shown in which the image quality improvement process is performed on both images acquired before and after the image determined to satisfy the first condition, but in order to further reduce the processing load, the image quality improvement process may be performed only on images acquired before or after the image determined to satisfy the first condition.

[0039] Also, as in a third embodiment described later, each camera in the surveillance camera group 100 may be configured to have the function of the first image analysis unit 203. In this case, an output unit (not shown) of the surveillance camera device 101 outputs the analysis result by the first image analysis unit 203, i.e., the time t of the image at which an abnormality was detected, to the analysis device (information processing device) 103.

[0040] (Effects of the First Embodiment) According to the present invention, since there is no need to constantly execute image quality improvement processing, it is possible to identify a predetermined moving image section (abnormal moving image section) while reducing the processing load by suppressing the number of times image quality improvement processing is executed.

[0041] <Embodiment 2> In this embodiment, an example will be described in which the first image analysis unit performs image analysis processing on the entire area of ​​the monitored object, and the second image analysis unit performs image analysis processing on a partial area of ​​the monitored object. In addition, in this embodiment, the high image quality processing is performed only on the image to which the second image analysis processing is applied. Specifically, the monitored object is a car, and the first image analysis processing detects a rapid change in the traveling speed of the car from an image that has not been subjected to high image quality processing, and the second image analysis processing detects an abnormality of a person inside the car from an image with high image quality. As the high image quality processing, a high resolution processing is assumed. Thus, this embodiment is characterized in that the first condition and the second condition are different from those of the first embodiment. Thereby, while reducing the processing load of the first image analysis processing, the partial area of ​​the car that is difficult to see in the second image analysis processing is made high resolution, thereby improving the accuracy of the abnormality detection processing. Then, a section in which the details of the abnormality can be grasped can be accurately identified as an abnormal section.

[0042] The hardware configuration of a monitoring system assumed in this embodiment is shown in Fig. 1, similar to that of embodiment 1. An example of the functional configuration of an information processing device 1 in this embodiment is shown in Fig. 5. The information processing device 1 corresponds to the analysis device in Fig. 1, and includes an image acquisition unit 501, an image quality improvement unit 502, a first image analysis unit 503, a second image analysis unit 504, a temporary recording unit 505, a section identification unit 506, and a storage unit 507. Each functional configuration unit will be briefly described.

[0043] The image acquisition unit 501 sequentially acquires time-series images from the storage unit 507. The time-series images correspond to surveillance camera images. The image quality improvement unit 502 improves the image quality of the time-series images acquired from the image acquisition unit 501 and outputs the high-image quality images. This process improves the image quality of a partial area of ​​a surveillance object such as a person riding in a car. For example, a person is detected from the area of ​​a car shown in a surveillance camera image, an image area including the person is cut out, and then a high-resolution image of the person is output by a super-resolution image process that has been learned to increase the resolution of an input image. This makes it easier to detect an abnormality in a person inside a car, which tends to be low-resolution, by an image analysis process described later, and generates an image that is easy for a user to analyze. The first image analysis unit 503 performs an abnormality detection process from the time-series images acquired from the image acquisition unit 501. Here, the abnormality detection process is performed on the entire area of ​​a surveillance object that has been determined in advance. For example, a process of detecting a car from a time-series image and detecting a sudden change in the speed of the car may be performed.

[0044] The second image analysis unit 504 performs anomaly detection processing on the image quality improvement image obtained from the image quality improvement unit 502. However, this processing is performed when the first image analysis unit 503 detects an anomaly from the time-series images. In this embodiment, the second image analysis processing is performed on a partial region of the monitored object, which is different from the image region on which the first image analysis processing was performed.

[0045] The temporary recording unit 505 temporarily stores the high quality images analyzed by the first image analysis unit 503 and the second image analysis unit 504, and the first and second image analysis results obtained as outputs. The section identification unit 506 acquires the first and second image analysis results recorded in the temporary recording unit 505, and identifies an abnormal section based on these. The memory unit 507 records the surveillance camera video for which an abnormal section is to be identified, and information on the identified abnormal section.

[0046] 6 is a flowchart showing the flow of identifying an abnormal section in this embodiment. In the following description, each process (step) is represented by adding an S to the beginning, and the description of the process (step) is omitted. However, the information processing device does not necessarily have to perform all of the processes described in this flowchart. The processes executed by the CPU H101 are each shown as a functional block.

[0047] In S601, the image acquisition unit 501 acquires time-series images from the storage unit 507 in chronological order. In this embodiment, it is assumed that every frame is acquired, but it may be acquired every few frames as in the first embodiment, and this is not limited to this. An example of time-series images is shown in FIG. 7. FIG. 7(a) shows images at times t=0 to 9 arranged in a row, in which the driver of the car is asleep at times t=2 to 5, and the driving speed of the car shown in the image at times t=3 to 4 suddenly increases due to this influence. This embodiment aims to identify the driver's dozing section t=2 to 5 from the time-series images without excess or deficiency. In this embodiment, a situation is considered in which the accuracy of dozing detection is low because the size of the driver of the car in the image is small. For example, it is assumed that a dozing detection process that has learned the appearance of a driver under normal circumstances using a huge amount of high-resolution images is already available. If a person in a surveillance camera image to which this process is to be applied is captured at a resolution lower than the resolution of the image used for learning, the accuracy of dozing detection tends to be low. Therefore, by increasing the resolution of the surveillance camera image to a resolution assumed for the drowsiness detection process and then applying the anomaly detection process, the accuracy can be improved. However, performing the high-resolution process and anomaly detection on all frames of the surveillance camera image increases the processing load. Therefore, in this embodiment, for the vehicle, which is an area larger than the driver, anomalies such as a sudden change in driving speed are detected using primitive features, and only for the time before and after the detection, super-resolution processing is performed to detect the driver's anomaly. In this way, the processing load can be reduced while the abnormal section can be accurately determined.

[0048] Next, in S602, the first image analysis unit 503 performs an anomaly detection process on the time-series images t and t-ΔT (here, ΔT=1) obtained from the image acquisition unit 501 in S601. Here, the subscript t is an index representing time, and indicates that it is the t-th image in the time-series images. In this description, the anomaly detection process is a process for detecting a sudden change in the speed of a car. For example, a car is detected from the time-series images t and t-ΔT, and the travel distance of the car is calculated. Then, if the difference with the travel distance in the image at the past time is equal to or greater than a threshold, it may be determined that an anomaly has occurred. After performing such an anomaly detection process, the first image analysis unit 503 records the time-series images t, the travel distance t of the car, and the anomaly detection result t in the temporary recording unit 505 as the first image analysis result t.

[0049] In S603, the first image analysis unit determines whether to proceed to the second image analysis process based on the abnormality detection result t. Specifically, if no abnormality is detected, the process returns to S601, and if an abnormality is detected, the process proceeds to S604. In the example of FIG. 7, first, the images at time t=0, 1, 2, and 3 are determined to be normal, and the process returns to S601. Then, the image at time t=4 is determined to be abnormal because the moving distance of the car from time t=3 is greater than the moving distance between the images at other adjacent times. Therefore, after performing S603 on the image at time t=4, the process does not return to S601, and the process from S604 onward is performed. In S604 to S608, the image before time t=4 is processed, and in S609 to S613, the image after time t=4 is processed. The processes of S604 to S608 and S609 to S613 may be performed in parallel or in order, and are not limited to this.

[0050] In S604, the image acquisition unit 501 sequentially acquires images t' at times before time t=4 at intervals of T' from the storage unit 507. In the example of Fig. 7, if T'=1, images are acquired in the order of times t=3, 2, and 1 going back from the time t=3 when the abnormality was detected in S304.

[0051] In S605, the image quality improving unit 502 cuts out a partial region of the car from the time-series image t acquired by the image acquiring unit 501 in S601, and outputs it as a partial region image t. In this description, the partial region is the region of a person in the car in which an abnormality was detected in S602. The person region may be detected from the time-series image t using a person detector trained to detect people. The region to be cut out may include an area that is a constant multiple of the area of ​​the person region.

[0052] In S606, the image quality improvement unit 502 improves the image quality of the partial region image t cut out in S605 to generate a high-image quality image t'. Examples of images before and after the application of image quality improvement processing for images in an abnormal state and in a normal state are shown in Fig. 7(b-1) and Fig. 7(b-2). In the image in Fig. 7(b-1), the driver's region, which is a partial region, is small, making it difficult to distinguish between an abnormal state and a normal state. In addition, if the image size is significantly smaller than the image size of a person assumed by the abnormality detection processing, it becomes difficult to achieve the accuracy of abnormality detection. Therefore, by applying super-resolution processing by the image quality improvement unit 502 to the image of the person region and increasing the image resolution as shown in Fig. 7(b-2), it becomes easier to perform the abnormality detection processing and to make it easier for the user to monitor / analyze. In addition, image quality improvement may be performed before the cutout of the partial region in S605, which may improve the detection accuracy of the partial region and cut out an image in which an abnormality is easily detected.

[0053] In S607, the second image analysis unit 504 performs anomaly detection processing on the image quality improvement image t' obtained in S606 from the image quality improvement unit 502. Then, the anomaly detection result is recorded in the temporary recording unit 505 as the second image analysis result.

[0054] In S608, the second image analysis unit 504 determines whether to end the second image analysis. The end determination may be made when no abnormality is detected in S607. In that case, in the example of FIG. 7, the images at times t=3 and t=2 are determined to be abnormal in S607, so the process returns to S604 and is performed. Then, the image at time t=1 is determined to be normal in S607, so it is determined to be ended at time t=1. The method of the end determination is not limited to this, and may be performed by another method shown in S308.

[0055] In S609, the image acquisition unit 501 sequentially acquires images t' after time t at intervals of T' from the storage unit 507. In the example of Fig. 7, if T' = 1, images are acquired in the order of time t = 5, t = 6, starting from time t = 4 when an abnormality was detected in S603.

[0056] In S610, similarly to S605, the image quality improving section 502 improves the image quality of the image t' acquired by the image acquiring section 501 in S609, to generate an image quality improved image t'.

[0057] In S611, similarly to S607, the second image analysis unit 504 performs anomaly detection processing on the image quality improvement image t' obtained in S610 from the image quality improvement unit 502. Then, the anomaly detection result is recorded in the temporary recording unit 505 as the second image analysis result.

[0058] In S613, similarly to S608, the second image analysis unit 504 judges whether or not to end the second image analysis. In the example of Fig. 7, first, the image at time t = 5 is judged to be normal in S611, so the process returns to S601 and is carried out. Then, the image at time t = 6 is judged to be normal in S611. If the end judgment in this process S612 is the point in time when even one image is found to be normal, it is judged to end at time t = 6.

[0059] In S614, the section identification unit 506 acquires the first image analysis result and the second image analysis result recorded in the temporary recording unit 505, and identifies the abnormal section based on these. For example, in the example of FIG. 7, the abnormal section may be from the image at the earliest time to the image at the latest time determined to be abnormal by the second image analysis. In that case, t=2 to 5 is identified as the abnormal section. Alternatively, the section in which the second image analysis was performed may be identified as the abnormal section. In that case, the abnormal section is t=1 to 6.

[0060] The information on the identified abnormal section may be recorded in the storage unit 507 and used for the user's search and analysis of the abnormal section. Instead of recording the information on the abnormal section, it may be used for displaying it in the form of a thumbnail or the like. In addition, as the information on the abnormal section, the type of abnormality, the first image analysis result at each time, and the second image analysis result may be recorded and displayed, or used to control the display method. In addition, a partial area image (the driver's area in the example of FIG. 7) obtained by improving the image quality of the image at the time corresponding to the abnormal section may be acquired from the temporary recording unit 505 and stored in the storage unit 507 as an abnormal section video. Alternatively, the entire area of ​​the monitored object (the car area in the example of FIG. 7) before the image quality improvement or the entire surveillance camera image may be stored in the storage unit 507 as an abnormal section video. In addition, it may be used to determine the section to be deleted from the storage unit 507. By deleting or compressing images from the storage unit 507 at a time earlier than the abnormal section, only the images of the parts required for analysis are left in the temporary recording unit 505. This can improve the user's viewability and can effectively utilize resources when the capacity of the temporary recording unit 505 is limited. After the process of S613 is completed, the information temporarily recorded in the temporary recording unit 505 may be discarded.

[0061] (Effects of the second embodiment) In this embodiment, the first image analysis process does not perform high-quality processing, but detects simple abnormalities (such as movement and color changes) that can be detected even with low image quality using low-load processing, and roughly identifies the abnormal time. Then, in the second image analysis process, images from previous and following times are combined with high-load high-quality processing to detect abnormalities using more advanced image information (such as semantic information such as whether the person is dozing or not). Compared to applying high-load high-quality processing to images at all times, it is possible to detect abnormalities that go beyond the content shown in the images with a low processing load and accurately identify abnormal video sections. In addition, by generating images that have been high-quality processed in the second image analysis process, it is possible to provide the user with videos that are easy to analyze.

[0062] In the description of this embodiment, as an example, the first image analysis process detects abnormalities in the movement speed for the entire area of ​​the monitored subject, and the second image analysis process performs super-resolution processing on a partial area of ​​the monitored subject to improve image quality and detect abnormalities in detail. The second image analysis process requires not only high image quality processing but also partial area detection processing, which increases the processing load, but the first image analysis process can omit these steps, resulting in a greater reduction in processing load.

[0063] <Embodiment 3> In the third embodiment, a case where the first image analysis process is performed by the surveillance camera and the second image analysis process is performed by the analysis device in a surveillance system similar to that in the first embodiment will be described. When analyzing the images of multiple surveillance cameras installed in different locations, it is desirable to optimize the image quality improvement process and image analysis process for each surveillance camera according to the situation of the installation location. However, with the limited computational resources of the surveillance camera, it is difficult to execute both the first image analysis process and the second image analysis process as described in the first embodiment. In addition, when the number of surveillance cameras is huge, it is inefficient to hold models for each of the multiple surveillance cameras on the analysis device side. Therefore, in this embodiment, a lightweight process optimized for each of the multiple cameras is used for the first image analysis process, and a general-purpose and high-load process that provides high performance on average for the multiple cameras is used for the second image analysis process. As a result, it is possible to apply the optimal process for each surveillance camera while keeping the load low, and to accurately identify abnormal sections.

[0064] (Hardware configuration) In the third embodiment, as in the first embodiment, a remote monitoring system using a monitoring camera connected to a network as shown in FIG. 1(a) will be described. This monitoring system is composed of a monitoring camera and an analysis device connected to a network. The hardware configuration of the analysis device is as shown in FIG. 1(b) as in the first embodiment. The hardware configuration of the monitoring camera device is shown in FIG. 8. FIG. 1(b) is a hardware configuration diagram of the analysis device in this embodiment. The CPU H101 controls the entire device by executing a control program stored in the ROM H102. The RAM H103 temporarily stores various data from each component. In addition, the program is expanded to make the CPU H101 executable. The storage unit H104 stores data to be processed in this embodiment, and is used to temporarily store camera images received from the monitoring camera and data generated during information processing. As a medium for the storage unit H104, a HDD, a flash memory, various optical media, etc. can be used.

[0065] (Functional configuration) An example of the functional configuration of the information processing device 1 of this embodiment is shown in Fig. 9. The information processing device 1 straddles the surveillance camera device and the analysis device in Fig. 1, and the functional configuration of the surveillance camera includes an imaging unit 901, a first image acquisition unit 902, a first image quality improvement unit 903, a first image analysis unit 904, a first communication unit 905, and a first storage unit 906. The functional configuration of the analysis device includes a second communication unit 907, a second image acquisition unit 908, a second image quality improvement unit 909, a second image analysis unit 910, a temporary recording unit 911, a section identification unit 912, and a second storage unit 913.

[0066] The imaging unit 901 captures images of the situation at the installation location of the surveillance camera device, and stores the images in a first storage unit 906 described later. The first image acquisition unit 902 sequentially acquires the time-series images captured by the imaging unit 901 from the first storage unit 906. The first image enhancement unit 903 enhances the image quality of the time-series images acquired by the first image acquisition unit 902, and outputs the enhanced image quality images. The first image analysis unit 904 performs anomaly detection processing on the enhanced image quality images obtained from the first image enhancement unit 903, and outputs a first image analysis result. The first communication unit 905 communicates with an analysis device, and an output unit (not shown) transmits (outputs) the time-series images captured by the imaging unit 901, the enhanced image quality images output by the first image enhancement unit 903, and the first image analysis result output by the first image analysis unit 904 to the analysis device. The first storage unit stores the time-series images captured by the imaging unit 901. For example, the time t of an image that is determined to satisfy a first condition, which is the first analysis result, is output.

[0067] The second communication unit 907 receives the time-series images, the high-quality images, and the first image analysis results received from the first communication unit, and stores the time-series images in the second storage unit 913 and the high-quality images and the first image analysis results in the temporary recording unit 911. The second image acquisition unit 908 acquires the time-series images from the second storage unit 913. However, the processing is performed only when the first image analysis unit 904 detects an abnormality from the time-series images. The second image improvement unit 909 improves the image quality of the time-series images acquired by the second image acquisition unit 908, and outputs the high-quality images. The second image analysis unit 910 performs an abnormality detection processing on the high-quality images obtained from the second image improvement unit 909. Alternatively, as in the first embodiment, the time t of the image determined to satisfy the first condition is referred to. Then, the image improvement processing and the second analysis processing are performed on at least one of the images before and after the time t from among the multiple images acquired by the second image acquisition unit 908. The temporary recording unit 911 temporarily stores the high quality images analyzed by the first image analysis unit 203 and the second image analysis unit 204, and the first and second image analysis results obtained as outputs. The section identification unit 912 acquires the first and second image analysis results recorded in the temporary recording unit 911, and identifies an abnormal section based on these. The second storage unit 913 records the surveillance camera video for which an abnormal section is to be identified, and information on the identified abnormal section.

[0068] In this embodiment, the image quality improvement process performed by the first image quality improvement unit 903 is lighter than the image quality improvement process performed by the second image quality improvement unit 909, and is optimized according to the installation location of the surveillance camera image. The optimization method may be switching according to the time period, such as performing noise removal processing after sunset and not performing noise removal processing before sunset, or adjusting the degree of processing. Alternatively, the strength of the mist removal processing may be changed for each installation location according to the likelihood of mist and haze occurring. In addition, parameters used in the processing may be optimized by learning using image data of the installation location. Learning may be performed on the surveillance camera, or parameters learned on an analysis device may be sent to the surveillance camera and used for processing. In this example, as in the first embodiment, mist removal processing is assumed as the image quality improvement process, and the strength of the mist removal processing is adjusted for each surveillance camera.

[0069] (Processing in surveillance camera device and analysis device) The process in the surveillance camera is basically the same as S301 to S304 in FIG. 3, and if no abnormality is determined in S304, only the time-series images are sent from the surveillance camera to the analysis device and stored in the analysis device. If an abnormality is determined, the high-quality image and the first image analysis result are also sent to the analysis device. Here, the abnormality detection process in this embodiment is the process of detecting a person's fall, as in the first embodiment, and a fall is considered to be an abnormality. The process in the analysis device is the same as S304 to S313. The processes of S301 to S304 are executed, and the processes of S304 to S313 are executed by the CPU H103-1. Only S301 and S304, which are different from the first embodiment, will be described below.

[0070] In S301, the imaging unit 901 captures images of the situation at the installation location of the surveillance camera device, and stores the time-series images in the first storage unit 906. Then, the first image acquisition unit 902 sequentially acquires the time-series images t captured by the imaging unit 901 from the first storage unit 906. Here, the subscript t is an index indicating time, and indicates that this is the t-th image in the time-series images.

[0071] In S304, the first image analysis unit determines whether to proceed to the second image analysis process based on the abnormality detection result t, and transmits data to the analysis device.

[0072] Specifically, if no abnormality is detected, the first communication unit 905 transmits the time-series images captured by the imaging unit 901 to the second communication unit 907 on the analysis device side. After that, the process returns to S301. On the analysis device side, the second communication unit 907 stores the high-quality images received from the first communication unit 905 in the second storage unit 913. Furthermore, the processes after S105 are not performed.

[0073] If an abnormality is detected, the first communication unit 905 transmits the time-series images captured by the imaging unit 901, the image quality improvement image output by the first image improvement unit 903, and the first image analysis result output by the first image analysis unit 904 to a second communication unit 907 on the analysis device side. The second communication unit 907 stores the image quality improvement image in a second storage unit 913 and the first image analysis result in a temporary storage unit 911. Then, the process proceeds to S305 and subsequent steps.

[0074] (Effects of the Third Embodiment) In this embodiment, the first image analysis process uses a lightweight process optimized for each of the multiple cameras, and the second image analysis process uses a general-purpose, high-load process that provides high performance on average for the multiple cameras. This makes it possible to apply optimal processing for each surveillance camera while keeping the load low, and to accurately identify abnormal sections.

[0075] <Other embodiments> The present invention can be realized by a process of reading and executing a program that realizes one or more functions of the above-mentioned embodiment 1. The program is supplied to a system or device via a network or a computer-readable storage medium, and is read and executed by one or more processors in the computer of the system or device. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0076] 201 Image Acquisition Unit 202 High-definition image processing section 203 First Image Analysis Department 204 Second Image Analysis Department 205 Temporary Recording Section 206 Section Identification Unit 207 Memory section

Claims

1. an image quality improving unit that performs image processing for improving image quality on a plurality of images acquired by an image acquiring unit at a predetermined frequency, the image processing unit performing image processing at a frequency lower than the predetermined frequency; a first image analysis unit that performs a first image analysis process on the image on which the image processing has been performed to determine whether the image satisfies a first condition; the image quality improving unit executes the image processing on at least one of an image acquired before the image determined to satisfy the first condition and an image acquired after the image determined to satisfy the first condition; a second image analysis unit that performs a second image analysis process on the image that has been subjected to the image processing and that has been determined to satisfy the first condition, to determine whether or not the image satisfies a second condition; a section identification unit that identifies a video section that satisfies the second condition from among the video including the plurality of images based on a result of the second image analysis process; 4. An information processing apparatus comprising:

2. a first image analysis unit that executes a first image analysis process for determining whether or not a plurality of images acquired by the image acquisition unit satisfy a first condition; an image quality improving unit that performs image processing to improve image quality on at least one of an image acquired before the image determined to satisfy the first condition and an image acquired after the image determined to satisfy the first condition; a second image analysis unit that performs a second image analysis process on the image that has been subjected to the image processing and that has been determined to satisfy the first condition, to determine whether or not the image satisfies a second condition that is different from the first condition; a section identification unit that identifies a video section that satisfies the second condition from a video including a plurality of images acquired by the image acquisition unit based on a result of the image analysis process of the second image analysis unit; An information processing device having the above configuration.

3. The information processing apparatus according to claim 1 , wherein the first image analysis process and the second image analysis process are different processes.

4. 3. The information processing apparatus according to claim 1, wherein the first image analysis process is lighter than the second image analysis process.

5. 3. The information processing device according to claim 1, wherein the image quality improvement processing applied to the image subjected to the second image analysis processing is a process with a higher load than the image quality improvement processing applied to the image subjected to the first image analysis processing.

6. The information processing device according to claim 1 or 2, characterized in that the second image analysis unit sequentially traces images before and after the image identified by the first image analysis unit, and applies the second image analysis process until an end determination is made.

7. 7. The information processing apparatus according to claim 6, wherein the determination of completion is based on whether or not the number of images that have been subjected to the second image analysis process exceeds a predetermined number.

8. 3. The information processing device according to claim 1, wherein the first image analysis process and the second image analysis process are anomaly detection processes, and the first condition and the second condition are information indicating whether or not an anomaly has been detected from an image.

9. a first image quality improving unit that performs image processing for improving image quality on a plurality of images captured by an imaging unit at a predetermined frequency at a frequency lower than the predetermined frequency; a first image analysis unit that executes a first image analysis process for determining whether the image on which the image processing has been executed satisfies a first condition; an output unit that outputs the images captured by the imaging unit and an analysis result by the first image analysis unit to an information processing device; a second image quality improving unit that performs image processing to improve image quality on at least one of an image acquired before the image determined to satisfy the first condition and an image acquired after the image determined to satisfy the first condition based on the analysis result output from the output unit; and a second image analysis unit that performs a second image analysis process on the image that has been subjected to the image processing and that has been determined to satisfy the first condition, to determine whether or not the image satisfies a second condition; an information processing device having a section identification unit that identifies a video section that satisfies the second condition from among the video including the plurality of images based on a result of the second image analysis process; An information processing system having the above configuration.

10. an image quality improvement step of performing image processing for improving image quality on a plurality of images acquired at a predetermined frequency at a frequency lower than the predetermined frequency; a first image analysis step of performing a first image analysis process on the image on which the image processing has been performed to determine whether the image satisfies a first condition; In the image quality improvement step, the image processing is performed on at least one of an image acquired before the image determined to satisfy the first condition and an image acquired after the image determined to satisfy the first condition, a second image analysis step of performing a second image analysis process for determining whether or not the image, which has been subjected to the image processing and determined to satisfy the first condition, satisfies a second condition; a section identification step of identifying a video section that satisfies the second condition from among the video including the plurality of images based on a result of the second image analysis process; The information processing method further comprising:

11. a first image analysis step of executing a first image analysis process for determining whether or not the acquired images satisfy a first condition for the plurality of images; an image quality improvement step of performing image processing to improve image quality on at least one of an image acquired before the image determined to satisfy the first condition and an image acquired after the image determined to satisfy the first condition; a second image analysis step of executing a second image analysis process for determining whether or not the image, which has been subjected to the image processing and determined to satisfy the first condition, satisfies a second condition different from the first condition; a section identification step of identifying a video section that satisfies the second condition from among the video including the plurality of acquired images based on a result of the image analysis process of the second image analysis step; An information processing method comprising the steps of:

12. 12. A program for causing a computer to execute the information processing method according to claim 10 or 11.

13. A computer-readable storage medium storing the program according to claim 12.

Images