Image processing apparatus, image processing method, and program

Abstract

To accurately detect a changed area in an imaging area without performing complex settings.SOLUTION: An image processing apparatus has: an image acquisition unit 401 that acquires a picked-up image obtained by picking up an imaging area; an extraction unit 404 that inputs the acquired picked-up image to an input layer of a neural network, to extract a first feature map that is output from a first feature extraction layer in the neural network and a second feature map that is output from a second feature extraction layer farther away from the input layer than the first feature extraction layer; and a detection unit 406 that, on the basis of the first feature map, estimates the boundary between a changed area that is the imaging area changed from a predetermined state and an area that is not the changes area in the picked-up image, and detects the changed area in the picked-up image on the basis of the second feature map and the estimated boundary.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a technique for detecting changes in an imaging area.

Background Art

[0002] As one of the techniques for analyzing an imaging image obtained by an imaging device such as a surveillance camera imaging an imaging area, there is a technique for detecting an area (change area) where the imaging area has changed from a reference state using a machine learning (deep learning) model such as a convolutional neural network. In this technique, a convolutional neural network having a plurality of feature extraction layers for extracting feature amounts of an image can be used to detect a change area from the feature amounts of the image.

[0003] Regarding the nature of detecting a change area using the feature amounts extracted by a convolutional neural network, when using the feature amounts of a layer closer to the input, while it is easy to detect the edge of the change area, there may be a situation where the inside of the change area is not detected and a hole occurs. Also, when using the feature amounts of a layer far from the input, the approximate position of the change area is detected and it is less likely for a hole to occur, but on the other hand, the edge of the change area becomes ambiguous and an area larger than the actual change area may be detected. Thus, in detecting a change area using the feature amounts extracted by a convolutional neural network, there are advantages and disadvantages depending on which layer's feature amounts are used, and there is a problem that the change area cannot be detected accurately.

[0004] Patent Document 1 describes a method for detecting an area that has changed from a reference good product image as an abnormal area. Patent Document 1 describes that by performing weighted addition of the feature amounts obtained from a plurality of layers in a convolutional neural network, an abnormal area can be detected accurately.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In the method of Patent Document 1, in order for the user to perform weighted addition of feature amounts, it is necessary to set a plurality of weight parameters. However, setting a plurality of weight parameters is complicated, and it has been difficult for the user to set them appropriately.

[0007] The present invention has been made in view of the above problems. Its object is to be able to accurately detect a change region in an imaging region without performing complicated settings.

Means for Solving the Problems

[0008] An image processing apparatus according to the present invention includes an image acquisition unit that acquires an imaging image in which an imaging region is imaged, and by inputting the imaging image acquired by the image acquisition unit into an input layer of a neural network, a first feature map output from a first feature extraction layer in the neural network, and a second feature map output from a second feature extraction layer farther from the input layer than the first feature extraction layer, an extraction unit that extracts the first feature map and the second feature map, an estimation unit that estimates a boundary between a change region in which the imaging region has changed from a predetermined state and a region that is not the change region in the imaging image based on the first feature map extracted by the extraction unit, and a detection unit that detects the change region in the imaging image based on the second feature map extracted by the extraction unit and the boundary estimated by the estimation unit.

Effects of the Invention

[0009] According to the present invention, it is possible to accurately detect a change region in an imaging region without performing complicated settings.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components described in the following embodiments are examples of embodiments of the present invention, and the present invention is not limited thereto.

[0012] In this embodiment, an image processing system that notifies a user of a change when an imaging area imaged by an imaging device such as a surveillance camera changes from a predetermined reference state (hereinafter referred to as the initial state) will be described. First, an overview of the processing performed by this image processing system will be described with reference to FIG. 1.

[0013] FIG. 1(a) shows an imaging image 10 obtained by imaging an imaging area with an imaging device. The imaging area in this embodiment is assumed to be, for example, inside a warehouse. The imaging area shown in the imaging image in FIG. 1(a) is in a state where no articles are placed as the initial state. In the imaging image 10, a determination area 11 is set as an area for determining that the state has changed. FIG. 1(b) shows an area 12 indicating an area where the state has changed in the determination area 11. The area 12 is an area where an article such as a cardboard box is placed and the state of the imaging area has changed from the initial state. The image processing system in this embodiment is a system that detects a changed area (hereinafter also referred to as a change area) such as the area 12 from the initial state and notifies the user.

[0014] This image processing system can be used in the following scenarios. For example, when the inside of a warehouse is imaged as described above, a notification is given when a predetermined number or more of articles are placed with the state of no articles as the initial state. Also, for example, when a shelf or table on which products are arranged is imaged, a notification is given when the products are purchased and the number decreases by a predetermined number or more with the state of the products being sufficiently arranged as the initial state. Thus, this image processing system is applicable to various imaging areas.

[0015] FIG. 2 is a diagram for explaining the configuration of the image processing system in this embodiment. The image processing system 100 of this embodiment includes an imaging device 110, an image processing device 120, a client device 130, and a network 140. The configuration of the system 100 shown in FIG. 2 is merely an example, and other components may be further added. Hereinafter, each component will be described.

[0016] The imaging device 110 is an imaging device such as a network camera, which captures the real space to generate a captured image. The imaging device 110 transmits the captured image to the image processing device 120 and the client device 130 via the network 140. There may be not only one imaging device 110 but also a plurality of them.

[0017] The image processing device 120 acquires the captured image from the imaging device 110 and executes various image processes. In the captured image acquired from the imaging device 110, the image processing device 120 identifies a change area where the state of the imaging area has changed and notifies the user. When notifying, the image processing device 120 outputs information for notifying the determination result or the like to the client device 130 via the network 140. Details of the image processing performed by the image processing device 120 will be described later.

[0018] The client device 130 receives inputs from the user and outputs information to the user (for example, displays information). The client device 130 can transmit an imaging instruction or an image processing execution instruction to the imaging device 110 and the image processing device 120 according to an input from the user. Further, when the client device 130 acquires information regarding the captured image and the notification from the imaging device 110 and the image processing device 120, the client device 130 displays the acquired captured image and information. The client device 130 in the present embodiment is a computer such as a PC, and inputs and outputs information by a browser or an application installed in the computer.

[0019] In the example shown in FIG. 2, the imaging device 110, the image processing device 120, and the client device 130 are shown as separate devices, but the present invention is not limited to such a configuration. For example, these devices may be integrated into one device. Further, for example, a configuration in which the image processing device 120 and the client device 130 are integrated may be used, or a configuration in which the image processing device 120 is included in the imaging device 110 may be used.

[0020] Network 140 is a communication network that connects imaging device 110, image processing device 120, and client device 130 in a mutually communicable state. Network 140 is composed of a plurality of routers, switches, cables, etc. that satisfy a communication standard such as Ethernet (registered trademark). In the present embodiment, network 140 only needs to be able to communicate between imaging device 110 and client device 130, regardless of its communication standard, scale, or configuration. For example, network 140 may be composed of the Internet, a wired LAN (Local Area Network), a wireless LAN (Wireless Lan), a WAN (Wide Area Network), or the like.

[0021] Next, with reference to FIG. 3, the hardware configuration of each device in image processing system 100 will be described. FIG. 3(a) is a diagram showing the hardware configuration of imaging device 110. Imaging device 110 includes CPU 201, RAM 202, ROM 203, HDD 204, input unit 205, output unit 206, network I / F unit 207, drive unit 208, imaging unit 209, and bus 210.

[0022] CPU 201 reads out the control computer programs stored in ROM 203 and HDD 204 and loads them into RAM 202, and executes various control processes. RAM 202 is used as a storage area for temporarily storing the programs executed by CPU 201 and work memories, etc.

[0023] ROM 203 stores data related to the settings of imaging device 110 and boot programs, etc. HDD 204 stores image data, setting parameters, and various programs, etc. It is assumed that HDD 204 can receive data input from an external device via network I / F unit 207.

[0024] The input unit 205 is an input device such as a button and a touch panel, for example. The output unit 206 is a device that outputs information, such as a display device such as a liquid crystal display and an audio output device such as a speaker. The input unit 205 and the output unit 206 are also connected to the bus 210. Note that the imaging device 110 may be configured not to have at least either the input unit 205 or the output unit 206.

[0025] The network I / F unit 207 transmits and receives various information between the imaging device 110 and other devices via the Internet. The data received from other devices via the network I / F unit 207 is transmitted and received to / from the CPU 204, the RAM 205, the ROM 206, etc. via the bus 210.

[0026] The drive unit 208 drives the imaging unit 209 to control the posture (imaging direction) and the angle of view, etc. of the imaging unit 209. The control target by the drive unit 208 is not limited to a specific target, and may be the posture and the angle of view of the imaging unit 209, or only one of them, or other targets (for example, the position of the imaging unit 209). Also, at least any one of the position, posture, and angle of view of the imaging unit 209 may be fixed. Also, when the position, posture, and angle of view of the imaging unit 209 are fixed and there is no need to control them, the imaging device 110 may be configured not to have the drive unit 208.

[0027] The imaging unit 209 has an image sensor and an optical system, and forms an image of a subject on the image sensor with the intersection of the optical axis of the optical system and the image sensor as the imaging center. Examples of the image sensor include a CMOS (Complementary metal-Oxide Semiconductor) and a CCD (Charged Coupled Device). The imaging unit 209 is controlled in terms of position, posture, angle of view, etc. by the drive unit 208.

[0028] FIG. 3(b) is a diagram showing the hardware configuration of the image processing apparatus 120. The image processing apparatus 120 includes a CPU 211, a RAM 212, a ROM 213, an HDD 214, an input unit 215, an output unit 216, a network I / F unit 217, and a bus 218. Note that since these functions are the same as those of the CPU 201, the RAM 202, the ROM 203, the HDD 204, the input unit 205, the output unit 206, the network I / F unit 207, and the bus 210 in the imaging apparatus 110, the description thereof will be omitted. Also, it is assumed that the hardware configuration of the client apparatus is the same as that in FIG. 3(b). Further, for example, the image processing apparatus 120 may be a server or the like that does not have the input unit 215 and the output unit 216.

[0029] Next, the functional configuration of the image processing apparatus 120 will be described with reference to FIG. 4. The image processing apparatus 120 includes an image acquisition unit 401, a storage unit 402, a calculation unit 403, an extraction unit 404, a setting unit 405, a detection unit 406, and an information output unit 407.

[0030] The image acquisition unit 401 acquires a captured image transmitted from the imaging apparatus 110. The image acquisition unit 401 outputs the acquired captured image to the storage unit 402, the calculation unit 403, and the extraction unit 404. In the present embodiment, the image acquisition unit 401 is assumed to acquire a captured image from the imaging apparatus 110, but the present invention is not limited thereto. The image acquisition unit 401 may be configured to acquire a captured image from, for example, the storage unit 402 or a storage device connected externally.

[0031] The storage unit 402 is connected to each processing unit of the image processing apparatus 120 and stores various types of information. Further, the storage unit 402 stores the captured image output from the image acquisition unit 401 as a captured image list. The captured image list is a list in which the captured image and the identifier are associated. Also, when the state of the imaging area is specified by the user via the client device 130 for the captured image, the captured image list further associates information indicating the state of the imaging area. The storage unit 402 associates information indicating that it is the initial state with the captured image of the imaging area captured in the initial state, for example, based on an instruction from the user. The process of specifying the state of the imaging area by the user will be described later. The identifier in the present embodiment is assumed to be the imaging date and time when the imaging was performed, but is not limited thereto, and any information that can uniquely identify the captured image may be used. For example, the identifier may be a hash value calculated from the captured image. Also, when the storage unit 402 receives an instruction to change or delete information included in the captured image list from the client device 130, it changes and deletes the information corresponding to the identifier included in the instruction.

[0032] Further, the storage unit 402 stores information indicating the determination area. The determination area is a predetermined area in which the change area detection process is executed by the extraction unit 404 and the detection unit 406, which will be described later. The information indicating the determination area is specified by the user via the client device 130. In the present embodiment, the determination area is assumed to be represented by the coordinates of each vertex of a rectangle, but is not limited thereto. The shape of the determination area may be, for example, an arbitrary polygon. Also, when the storage unit 402 receives an instruction to change the determination area from the client device 130, it changes the information indicating the determination area.

[0033] The calculation unit 403 acquires from the storage unit 402 the captured image associated with the information indicating the initial state and the information indicating the determination area. Further, the calculation unit 403 acquires the initial state parameters based on the determination area in the acquired captured image and outputs them to the detection unit 406. The initial state parameters will be described later.

[0034] The extraction unit 404 extracts a map (hereinafter referred to as a feature map) representing the feature amount in the determination region of the captured image from the captured image acquired from the storage unit 402 and the information indicating the determination region. The extraction unit 404 outputs the extracted feature map to the detection unit 406.

[0035] Here, the processing of the calculation unit 403 and the extraction unit 404 will be described. The calculation unit 403 and the extraction unit 404 use a convolutional neural network to acquire the initial state parameters and extract the feature map. As the neural network to be used, for example, a generally used convolutional neural network such as VGG16 is used. Also, it is assumed that the neural network to be used is a model pre-trained for object detection. Note that as the image used for pre-training, it is not necessarily required to use an image including an object that is a factor in the change of the imaging region. However, it is preferable that an image including an object having many features such as an edge is learned. A neural network that has learned such an image is suitable for the process of extracting the feature amount for change detection.

[0036] The structure of the neural network is shown in FIG. 6(a). The neural network model used in this embodiment has an input layer, an output layer, and a plurality of intermediate layers. The plurality of intermediate layers include, for example, a convolutional layer that performs a convolutional operation, and a pooling layer that performs a pooling process. When the model learned for object detection as described above is used, in the model shown in FIG. 6(a), when the input image is input to the input layer, the result of detecting the object included in the input image is output from the output layer as an estimation result. Also, from each intermediate layer, a feature map in which the feature amount in the input image is extracted is output.

[0037] The feature map extracted by the convolutional neural network in this embodiment is represented by a three-dimensional tensor of vertical × horizontal × number of channels. Specifically, by performing a convolution operation in each intermediate layer, a feature map represented by a three-dimensional tensor is output. An example of the feature map is shown in FIG. 6(b). FIG. 6(b) is assumed to be a feature map output from any intermediate layer of the neural network shown in FIG. 6(a). As shown in FIG. 6(b), the feature map is represented by the vertical and horizontal directions indicating the feature map plane and the number of channels corresponding to the number of kernels used in the convolution operation. At each position corresponding to the pixels of the feature map plane, a value representing a feature amount is shown. In the example of FIG. 6(b), the number of channels is 3, but the number of channels is not limited to this.

[0038] Also, by performing pooling in each intermediate layer, the image size is compressed in the vertical and horizontal directions (hereinafter, the plane direction). Therefore, the size of the feature map in the plane direction extracted becomes smaller as the neural network operations progress. If the pooling kernel size is, for example, 2×2, the vertical and horizontal sizes become 1 / 2 each for each pooling.

[0039] The calculation unit 403 inputs the imaging image in the initial state into the neural network described above and calculates the initial state parameters from the obtained feature map. Here, it is assumed that the calculation unit 403 acquires two or more imaging images. The calculation unit 403 calculates, as the initial state parameters, the average value of the feature amounts extracted from the imaging images in a plurality of initial states and the covariance matrix of the feature amounts. The average value of the feature amounts is obtained by calculating the average value of the feature amounts at the positions on the plane of the plurality of feature maps. The covariance matrix of the feature amounts is a matrix representing the covariance of the feature amounts at the positions on the plane of the plurality of feature maps.

[0040] Taking FIG. 6(b) as an example, the calculation unit 403 acquires the feature map shown in FIG. 6(b) from each of the plurality of captured images. Further, the calculation unit 403 acquires the feature amount corresponding to the position 601 in each feature map. The feature amount acquired at this time is represented by a vector composed of a numerical group in the channel direction at the same position of the feature map. The calculation unit 403 acquires the vector of the feature amount for each of the feature maps corresponding to the plurality of captured images, and calculates the average value and the covariance. The above processing is executed for each position of the feature map including the position 601. As a result, the average value and the covariance matrix of the feature amount are acquired as the initial state parameters. The calculation unit 403 outputs the acquired initial state parameters to the detection unit 406.

[0041] The extraction unit 404 inputs the acquired captured image into the neural network described above and acquires the feature map. The captured image input by the extraction unit 404 is, for example, a captured image obtained by capturing an imaging region at regular time intervals. The extraction unit 404 acquires, as an image for which a change region is to be detected, for example, an image frame in a video captured at a predetermined frame rate, extracts the feature map, and outputs it to the detection unit 406 described later.

[0042] Here, in order to accurately detect the change region, the detection unit 406 described later detects the change region using a plurality of feature maps acquired from two or more intermediate layers. Therefore, the calculation unit 403 and the extraction unit 404 acquire two or more feature maps for one captured image. In the present embodiment, it is described that two feature maps are acquired for one captured image. Further, the acquired feature maps are assumed to be the feature maps output from two pooling layers among the plurality of intermediate layers included in the neural network. Further, in the following description, among the two intermediate layers, the feature map output from the first intermediate layer closer to the input layer is represented as the first feature map, and the feature map output from the second intermediate layer farther from the input layer than the first intermediate layer is represented as the second feature map.

[0043] FIG. 6(c) is a diagram showing an example of a feature map obtained in the present embodiment. In the example of FIG. 6(c), it is assumed that the size of the input image 610 input to the neural network is 40×40, the planar size represented by the vertical×horizontal of the first feature map 620 is 20×20, and the planar size represented by the vertical×horizontal of the second feature map 630 is 10×10. The input image 610 is assumed to be an image corresponding to a determination region in the captured image. Since the first feature map and the second feature map are outputs from the pooling layer, respectively, they are compressed compared to the size of the input image. Note that the sizes of the input image and the feature map are not limited to the above. Also, for the subsequent processing, the size of the feature map is enlarged so as to match the size of the input image. Therefore, for example, a region 611 represented by 4 pixels in the input image 610 corresponds to a region 621 represented by 1 pixel in the first feature map 620. Also, a region 612 represented by 16 pixels in the input image 610 corresponds to a region 632 represented by 1 pixel in the second feature map 630.

[0044] The calculation unit 403 calculates a first initial state parameter and a second initial state parameter corresponding to each of the first feature map and the second feature map, and outputs them to the detection unit 406. Also, the extraction unit 404 outputs the first feature map and the second feature map extracted from the captured image to the detection unit 406.

[0045] Note that the neural network described above may be configured to be learned on the image processing apparatus 120, or the image processing apparatus 120 may have a learned model to which parameters learned externally are applied.

[0046] The setting unit 405 sets a boundary determination threshold, a change determination threshold, and an area threshold used by the detection unit 406 and the information output unit 407 described later. The description and setting method of each threshold will be described later in the description of each processing unit.

[0047] The detection unit 406 detects a changed area based on the initial state parameters output from the calculation unit 403 and the feature map output from the extraction unit 404. FIG. 5 shows the configuration of the detection unit 406. The detection unit 406 includes a change degree map generation unit 510, a boundary candidate generation unit 520, and a changed area creation unit 530. Hereinafter, the processes performed by each processing unit included in the detection unit 406 will be described. Also, an example of each process will be described using FIGS. 7 to 10. FIG. 7 shows a captured image 710 in an initial state of the imaging area and a captured image 720 that has changed from the initial state. It is assumed that an article is placed in an area 721 within the captured image 720. The captured images 710 and 720 both have a planar size of 40×40, and it is assumed that the size of the determination area is 40×40 and is set for the entire image. Also, although an article is placed in the area 711, since the article in the area 711 is placed in both of the captured images 710 and 720, it should not be detected as a change. Hereinafter, the process of detecting the area 721 as a changed area will be described.

[0048] The change degree map generation unit 510 calculates a change degree indicating the degree of change from the initial state for the captured image acquired by the extraction unit 404 based on the initial state parameters output from the calculation unit 403 and the feature map output from the extraction unit 404. Also, the change degree map generation unit 510 generates a change degree map in which the change degree is calculated for each position of the determination area in the captured image. Specifically, a change degree map is generated based on the first and second feature maps obtained by inputting the captured image 710 in FIG. 7 into a neural network and the first and second initial state parameters based thereon.

[0049] In the present embodiment, the change degree is calculated based on the Mahalanobis distance corresponding to each position of the feature map. The Mahalanobis distance DM at a certain position of the feature map is calculated based on the following formula (1).

[0050]

Equation

[0051] Here, x is a vector of feature amounts in the feature map extracted by the extraction unit 404. Also, μ and Σ are, respectively, a vector of the average value of the feature amounts output from the calculation unit 403 and a covariance matrix. Note that the Mahalanobis distance can take a value of 0 or more, but an upper limit of the value may be provided depending on the form of calculation or UI display. Also, the value of the Mahalanobis distance may be normalized. In this way, the degree-of-change map generation unit 510 calculates the Mahalanobis distance subjected to upper limit processing and normalization processing as needed, as the degree of change. In the present embodiment, the degree of change is an integer of 0 or more and 10 or less, but is not limited thereto.

[0052] The degree-of-change map generation unit 510 calculates the Mahalanobis distance for each position of the feature map using Expression (1). Thereby, a degree-of-change map having the same planar size as the feature map is generated. At this time, the degree-of-change map generation unit 510 generates a first degree-of-change map from the first feature map output by the extraction unit 404 and the first initial state parameter output by the calculation unit 403. Also, the degree-of-change map generation unit 510 generates a second degree-of-change map from the second feature map output by the extraction unit 404 and the second initial state parameter output by the calculation unit 403. Note that the degree of change based on the Mahalanobis distance is a scalar quantity. Therefore, when using a feature map having the same size as the example of FIG. 6(c), the size of the first degree-of-change map generated from the first feature map 620 is 20×20×1, and the size of the second degree-of-change map generated from the second feature map 630 is 10×10×1. However, in the following processing, it is assumed that the first and second feature maps are enlarged to have the same size as the original captured image and the processing is performed.

[0053] FIG. 8 shows a first change degree map 810 and a second change degree map 820 generated from the captured image 710. In FIG. 8, an area 721 to be detected as a change area is superimposed. As shown in FIG. 8, for both the change degree maps 810 and 820, a higher change degree is calculated around the area 721. On the other hand, a lower change degree is calculated around the area 711. The change degree map generation unit 510 outputs the first change degree map and the second change degree map to the boundary candidate generation unit 520 and the change map generation unit 530, respectively.

[0054] The boundary candidate generation unit 520 generates a boundary candidate map indicating a position estimated to be the boundary between the change area and the non-change area based on the first change degree map. Specifically, the boundary candidate generation unit 520 identifies positions in the first change degree map where the value of the change degree is greater than the boundary determination threshold. The boundary candidate generation unit 520 generates a boundary candidate map indicating the identified positions. Here, the boundary determination threshold is set by the setting unit 405. The boundary determination threshold may be specified by the user via the client device 130, or may be determined in advance based on characteristics of the extracted feature amounts, etc. Also, the boundary determination threshold may be a parameter indicating a range of values of the change degree determined to be a boundary candidate.

[0055] FIG. 9 is a boundary candidate map showing the boundary candidates when the boundary determination threshold is set to 7 in the first change degree map 810. Two boundary candidates 901 and 902 are identified in the first change degree map 810. The boundary candidate generation unit 520 outputs the generated boundary candidate map to the change map generation unit 530.

[0056] The change map generation unit 530 generates a change map indicating the change area in the captured image based on the second change degree map and the boundary candidate map. The change map generation unit 530 generates the change map by performing the following processing.

[0057] The change map generation unit 530 associates, in the boundary candidate map, a plurality of divided regions divided by the boundary candidates with the regions in the second change degree map. The change map generation unit 530 determines whether the change degree of the regions in the second change degree map corresponding to each of the plurality of divided regions is greater than the change determination threshold. The change map generation unit 530 determines the divided regions corresponding to the regions whose change degree is greater than the change determination threshold as changed regions. Here, the change determination threshold is set by the setting unit 405. The change determination threshold may be specified by the user via the client device 130, or may be determined in advance based on the characteristics of the extracted feature amounts, etc. Also, the change determination threshold may be a parameter indicating the range of the values of the change degree determined to be a changed region.

[0058] Note that, for the change degree of the second feature map used for determining whether it is a changed region, the change degree at any position within the region corresponding to the divided region may be used. For example, taking one position within the region as a representative point, the change degree corresponding to the position of the representative point may be used, or the average value of the change degrees corresponding to a plurality of positions may be used. As the representative point, for example, the position closest to the center of the region may be used, or any position may be used.

[0059] However, it is preferable to use the change degree corresponding to a position different from the position of the boundary candidate. However, when the divided region is narrow and it is difficult to set a representative point at a position different from the position of the boundary candidate, for example, a change degree map based on a feature map output from another intermediate layer in a neural network may be used as the second change degree map. By using the feature maps output from different intermediate layers, it may be possible to set a representative point at a position different from the position of the boundary candidate. Note that even if a change degree map based on a feature map output from another intermediate layer is used as the second change degree map and a representative point cannot be set at a position different from the position of the boundary candidate, the change map generation unit 530 determines the divided regions where the representative point cannot be set as changed regions.

[0060] The change map generation unit 530 finally detects, as the change region, a region that combines the positions of the boundary candidates and the regions determined to be change regions among the divided regions divided by the boundary candidates. FIG. 10 is a diagram for explaining the process of generating a change map and detecting a change region. FIG. 10(a) is a diagram in which boundary candidates 901 and 902 are superimposed on the second change degree map 820. As shown in FIG. 10(a), due to the boundary candidates 901 and 902, the second change degree map 820 is divided into a divided region 1010 outside the boundary candidate 901, a divided region 1020 between the boundary candidate 901 and the boundary candidate 902, and a divided region 1030 inside the boundary region 902.

[0061] For each divided region, the change map generation unit 530 sets a representative point at a position different from the position of the boundary candidate. Here, as shown in FIG. 10(b), representative points 1011, 1021, and 1031 are set in the divided regions 1010, 1020, and 1030, respectively. Also, the change map generation unit 530 determines whether the value of the change degree corresponding to each representative point is greater than the change determination threshold. Here, it is assumed that the change determination threshold is set to 8. In the example of FIG. 10(b), the value of the change degree corresponding to the representative point 1021 is greater than the change determination threshold, and the values of the change degrees corresponding to the representative points 1011 and 1031 are smaller than the change determination threshold. Therefore, the change map generation unit 530 determines that the divided region 1020 including the representative point 1021 is a change region.

[0062] The change map generation unit 530 generates a change map indicating, as the change region, a region including the divided region 1021 determined to be a change region and the boundary candidates 901 and 902. FIG. 10(c) is a diagram showing the generated change map. According to FIG. 10(c), it can be seen that the region corresponding to the region 721 in FIG. 7 is shown as the change region.

[0063] The above is the detection process of the change area by the detection unit 406. Note that the detection unit 406 outputs the first and second change degree maps generated in the process of the process to the client device 130 and presents them to the user by the client device 130. According to this configuration, the user can set the boundary determination threshold value and the change determination threshold value while visually observing the change degree map displayed on the client device 130. The change map generation unit 530 outputs the generated change map to the information output unit 407.

[0064] Based on the change map output from the detection unit 406, the information output unit 407 calculates the size of the change area. In the present embodiment, as the size of the change area, the ratio of the number of pixels in the area indicated as the change area on the change map to the total number of pixels in the change map is calculated. Note that the size of the change area is not limited to this, and the number of pixels in the area indicated as the change area on the change map itself may be used.

[0065] The information output unit 407 determines whether the size of the change area is larger than the area threshold value set by the setting unit 405. If it is larger, it is determined that the change in the imaging area is notified. When the determination result indicates that a notification is to be issued, the information output unit 407 outputs information for notifying the user of the change in the imaging area to the client device 130. Further, when the determination result indicates that no notification is to be issued, the information output unit 407 does not output information to the client device 130. Note that the process when the determination result indicates that no notification is to be issued is not limited to the above, and a configuration in which information indicating that there is no change in the imaging area is output may also be used. Further, a configuration in which information for notifying the change in the imaging area is output when the size of the change area continues to be larger than the area threshold value for a predetermined time may also be used.

[0066] Note that the area threshold is set by the setting unit 405 by calculating the size of the change area based on the captured image including the state of the imaging area for which a change is to be notified, and setting the calculated size of the change area as the area threshold, but it is not limited to this. It may also be a configuration set in advance by the user. Further, the area threshold may be a parameter indicating the range of the size of the change area to be notified.

[0067] Next, the functional configuration of the client device 130 will be described with reference to FIG. 4. The client device 130 includes a communication unit 411, a display unit 412, and an operation unit 413.

[0068] The communication unit 411 communicates information with the image processing device 120. Although not shown in FIG. 4, the communication unit 411 can communicate with the imaging device 110 via the image processing device 120 or directly. The communication unit 411 acquires the captured image and information related to the notification obtained from the information output unit 407 and outputs it to the display unit 412. Further, the communication unit 411 transmits a transmission instruction such as an imaging image list and information indicating the determination area to the image processing device 120 in response to an input from the user to the operation unit 413, and acquires the instructed information. Also, the communication unit 411 can transmit an instruction to change or delete the information included in the imaging image list and the information indicating the determination area in response to an input from the user to the operation unit 413.

[0069] In addition, the communication unit 411 can transmit an instruction to set the boundary determination threshold, the change determination threshold, and the area threshold to the image processing device 120 in response to an input from the user to the operation unit 413. Also, the communication unit 411 can transmit an imaging instruction and an instruction to change the imaging direction or the viewing angle, etc. to the imaging device 110 in response to an input from the user to the operation unit 413.

[0070] The display unit 412 corresponds to the output unit 216 and presents information to the user via, for example, a display. In the present embodiment, information is presented to the user by displaying, on the display, the result rendered by the browser based on the information acquired by the communication unit 411. Note that information may be presented by a method other than screen display, such as voice and vibration.

[0071] The display unit 412 displays information indicating a determination area, a captured image, and information indicating the imaging state of the imaging area. The display unit 412 also displays the captured image received from the communication unit 411 and information related to the notification.

[0072] The operation unit 413 corresponds to the input unit 215 and receives an operation from the user. In the present embodiment, the operation unit 413 is a mouse and a keyboard, and it is assumed that the user operates these to input a user operation to the browser. However, the present invention is not limited to this, and the operation unit 413 may be any device such as, for example, a touch panel and a microphone. The operation unit 413 transmits an instruction corresponding to the input from the user to the imaging device 110 and the image processing device 120 via the communication unit 411.

[0073] Next, with reference to FIG. 11, the threshold setting process performed by the image processing device 120 will be described. The process shown in FIG. 5 is realized by the CPU 211 of the image processing device 120 loading and executing a computer program stored in various storage units into the RAM 212. When the user instructs to acquire a captured image for setting a threshold, the process starts.

[0074] In S1101, the image acquisition unit 401 acquires a captured image from the imaging device 110. The storage unit 402 stores the acquired captured image as a captured image list. In S501, it is assumed that two captured images in the initial state are acquired, but this is not the limit. For example, three or more captured images in the initial state may be acquired. Further, the image acquisition unit 401 also acquires a captured image for which a change area is to be detected.

[0075] In S1102, the image processing apparatus 120 receives a specification of the state of an imaging region corresponding to the captured imaging image. The information output unit 407 receives an instruction for specifying the state of the imaging region output by the client device 130 in response to a user input, and outputs it to the storage unit 402. The storage unit 402 registers or changes information indicating the state of the imaging region associated with the imaging image in the imaging image list according to the instruction. Here, information indicating that it is the initial state is specified for the imaging image acquired in S1101.

[0076] FIG. 13 shows an example of a user interface presented to the user by the client device 130. FIG. 13 shows a user interface 1300. As an example, the user interface 1300 includes a result display unit 1310, a list display unit 1320, and a button display unit 1330.

[0077] An imaging image 1311, a determination region 1312, and a notification 1313 are displayed on the result display unit 1310. Also, the region 1314 indicates a region that has changed from the initial state. The imaging image 711 is an imaging image captured for the user to monitor whether there is a change in the imaging region. That is, in the region of the imaging image 1311, imaging images captured by the imaging device 110 at regular time intervals are displayed. Note that it is assumed that the determination region 1312 is superimposed and displayed on the imaging image 1311, but this is not the case in all situations. For example, whether to display the determination region 1312 may be switched based on a user input. Also, as the image displayed in the region of the imaging image 1311, an imaging image in the initial state, a feature map, a change degree map, etc. may be displayed.

[0078] The position and shape of the determination area 1312 can be edited, for example, by performing operations such as clicking and dragging on the sides, vertices, and interior of the determination area 1312. When the determination area 1312 is edited by the user, the client device 130 transmits information indicating the edited determination area to the image processing device 120 as an area change instruction. Note that the method of editing the determination area is not limited to the above. For example, it may be edited by inputting the coordinate values of the determination area.

[0079] In the process of S1102, the state of the imaging area is specified by operating the list display unit 1320. The list display unit 1320 is associated with and displays a summary display unit 1321, a selection unit 1322, and a delete button 1323. Information indicating the summary of the captured images included in the captured image list is displayed on the summary display unit 1321. In this embodiment, it is assumed that a thumbnail image of the captured image included in the captured image list is displayed on the summary display unit 721 as information indicating the summary, but this is not the only case. For example, identifiers such as the capture date and time of the captured images included in the captured image list may be displayed on the summary display unit 1321.

[0080] Information indicating the state of the imaging area included in the captured image list is displayed on the selection unit 1322. The selection unit 1322 is, for example, a drop-down list, and information such as "initial state", "state to be notified", and "unselected (blank)" can be selected. Note that the captured image in the state to be notified is a captured image including a change area to be notified. The user can register and change information indicating the state of the imaging area by operating the selection unit 1322 in the process of S1102. When an input for registering and changing information indicating the state of the imaging area is received from the user, the client device 130 transmits an instruction corresponding to the input to the image processing device 120.

[0081] Note that the form of the selection unit 1322 is not limited to the above. For example, the selection unit 1322 may be a radio button. Also, in this embodiment, the state of the imaging region is specified by the user after imaging with respect to the captured imaging image, but this is not the only case. For example, the state of the imaging region may be specified before imaging, and the state of the imaging region specified in advance may be registered with respect to the imaging images captured until a new state of the imaging region is specified. Also, in addition to the "initial state", "state to be notified", and "unselected", there may be other states for the selectable state of the imaging region. Also, for example, a configuration in which only the "initial state" is selectable may be used.

[0082] When the delete button 723 is pressed by the user, a delete instruction for the captured image is transmitted from the client device 130 to the image processing device 120. The delete button 723 in this embodiment is arranged corresponding to the outline display unit 721 and the selection unit 722, but this is not the only case. For example, in the list display unit 720, if a set of the outline display unit 721 and the selection unit 722 is selectable, the set of the selected outline display unit 721 and the selection unit 722 may be made deletable by pressing a separately provided delete button 723.

[0083] In this way, the user can view and edit the list of the set of the captured image and the imaging state by visually observing and operating the list display unit 720.

[0084] Returning to FIG. 11, in S1103, the calculation unit 403 calculates the initial state parameters using the captured image obtained in S501 and specified as the initial state in S502. Specifically, the calculation unit 403 inputs the captured image in the initial state into the neural network, and calculates the initial state parameters from the obtained feature map. Here, the calculation unit 403 calculates, as the initial state parameters, the average value of the feature amounts extracted from a plurality of captured images in the initial state, and the covariance matrix of the feature amounts. Note that the process of S1103 is executed according to an instruction from the user.

[0085] Using FIG. 13, the instruction for calculating the initial state parameters will be described. On the button display unit 1330, an imaging button 1331 and a calculation button 1332 are displayed. When the imaging button 1331 is pressed by the user, an imaging instruction is sent from the client device 130 to the imaging device 110. The user can obtain an arbitrary captured image, for example, by pressing the imaging button 1331 at a timing when the user wants to newly obtain a captured image of the initial state. Note that, when the imaging button 1331 is pressed, the process of FIG. 11 may be started.

[0086] When the calculation button 1332 is pressed by the user, an instruction for generating initial state information is sent from the client device 130 to the image processing device 120. The user can update the initial state parameters at an arbitrary timing, for example, by pressing the calculation button 1332 when a newly obtained captured image of the initial state is obtained. Note that, when the calculation button 1332 is pressed, it is assumed that the process of S1103 in FIG. 11 is executed, but it is not limited thereto. For example, a configuration may be such that only the imaging button 1331 is provided without providing the calculation button 1332. In this case, when the imaging button 1331 is pressed, the initial state parameters may be automatically updated.

[0087] Returning to FIG. 11, in S1104, the detection unit 406 generates a change degree map of the captured image obtained in S1101 using the initial state parameters calculated in S1103. Specifically, the detection unit 406 generates first and second change degree maps using the captured image designated as the state to be notified in S1102. Note that the change degree map may be generated using a captured image for which the state is not designated instead of the captured image designated as the state to be notified. The detection unit 406 transmits the generated change degree map to the client device 130.

[0088] In S1105, the setting unit 405 sets a boundary determination threshold value according to an instruction by the user. Also, in S1107, the setting unit 405 sets a change determination threshold value according to an instruction by the user. With reference to FIG. 13, the threshold value setting process will be described. On the button display unit 1330, a slider bar 1333 for setting the threshold value is displayed. Also, in the area of the captured image 1311, the first and second change degree maps generated by the detection unit 406 are displayed. The user sets the boundary determination threshold value by operating the slider bar 1333 while looking at the first change degree map. As the boundary determination threshold value, it is desirable to set a value at which the boundary between the changed area and the area that is not the changed area is specified without excess or deficiency.

[0089] Also, the user sets the change determination threshold value by operating the slider bar 1333 while looking at the second change degree map. As the boundary determination threshold value, it is desirable to set a value at which the area to be detected as the changed area is specified without excess or deficiency. In this way, the user can appropriately set the threshold value while visually observing the change degree map.

[0090] In S1107, the detection unit 406 generates a change map using the boundary determination threshold value and the change determination threshold value set in S1105 and S1106. The generated change map is output to the setting unit 405 and the information output unit 407. In S1108, the setting unit 405 sets an area threshold value based on the change map. Specifically, the setting unit 405 calculates the size of the changed area based on the captured image including the state of the imaging area for which a change is to be notified, and sets the calculated size of the changed area as the area threshold value. Note that the present invention is not limited to this, and a configuration may be adopted in which the change map is displayed in the area of the captured image 1311 shown in FIG. 13, and the user operates the slider bar 1333 to set the boundary determination threshold value.

[0091] The above is the description of the threshold setting process performed by the image processing apparatus 120. In the present embodiment, the threshold is set by operating the slider bar 1333 in FIG. 13, but the present invention is not limited to this configuration. The user may directly input the value of the threshold. Further, various thresholds may be automatically set by the image processing apparatus 120, or the user may be able to change the value of the automatically set threshold. Further, when the change degree map and the change map are displayed in the area of the captured image 1311, they may be displayed superimposed on the original captured image.

[0092] Next, with reference to FIG. 12, the process of detecting a change in the imaging area and notifying the user will be described. In the process of FIG. 12, information for notifying a change in the imaging area is output based on a comparison between the size of the change area in the captured image captured by the imaging apparatus 110 at regular time intervals and the area threshold set in the process shown in FIG. 11. This process is also realized by the CPU 211 of the image processing apparatus 120 loading and executing a computer program stored in various storage units into the RAM 212. When an imaging instruction for change detection is given by the user, the process is started.

[0093] In S1201, the image processing apparatus 120 acquires a captured image from the imaging apparatus 110 or the storage unit 402. The image acquisition unit 120 outputs the acquired captured image to the extraction unit 404. In S1201, one of the captured images captured at regular time intervals is acquired, and the processes of S1202 to S1207 are executed. Further, the captured images captured at regular time intervals are assumed to be, for example, image frames in a moving image captured at a predetermined frame rate.

[0094] In S1202, the extraction unit 404 obtains a feature map based on the acquired captured image. The extraction unit 404 outputs the acquired feature map to the detection unit 406. In S603, the detection unit 406 obtains a change map by using the feature map, the initial state parameters calculated in the process of FIG. 11, and the boundary determination threshold and change determination threshold set in the process of FIG. 11. The detection unit 406 outputs the acquired change map to the information output unit 407.

[0095] In S1204, the information output unit 407 calculates the size of the change region. As the size of the change region, the ratio of the number of pixels of the region indicated as the change region on the change map to the total number of pixels of the change map is calculated. In S1205, the information output unit 407 determines whether to issue a notification based on the size of the change region calculated in S1204 and the area threshold set in the process of FIG. 11. When the size of the change region is larger than the area threshold (YES in S1205), the determination unit 406 determines to issue a notification and proceeds to S1206. In S1206, the information output unit 407 outputs information for notifying the change of the imaging region.

[0096] On the other hand, when the size of the change region is equal to or less than the area threshold (NO in S1205), it is determined not to issue a notification and information for notifying the change of the imaging region is not output. In this case, only the captured image may be output, or information indicating that there is no change in the imaging region may be output.

[0097] Using FIG. 13, an example of a notification to the user will be described. In notification 1313, a display corresponding to the information output from the information output unit 407 of the image processing apparatus 120 in S1206 of FIG. 12 is shown. In the present embodiment, when it is determined that a notification is necessary in the image processing apparatus 120, "Change detected" is displayed in notification 1313 based on the information output from the information output unit 407. Note that when it is determined that a notification is not necessary in the image processing apparatus 120, it is assumed that nothing is displayed (not displayed) in notification 1313, but the present invention is not limited to this. For example, when it is determined that a notification is not necessary, "No change" may be displayed in notification 1313. Further, for example, when it is determined that a notification is necessary, the color of the determination area 1312 may be changed. Further, when it is determined that a notification is necessary, the notification may be made by voice. Thus, any form can be used for the notification.

[0098] Returning to FIG. 12, in S1207, the image processing apparatus 120 determines whether there is an instruction to end the change detection process by the client apparatus 130. If there is an end instruction (YES in S1207), the process ends. If there is no end instruction (NO in S1207), the process returns to S1201, and the processes of S1202 to S1207 are executed for the captured image to be acquired next in S1201. Note that the captured image to be acquired next in S1201 is, for example, the next frame image in a moving image.

[0099] As shown in FIG. 13, the output information is displayed on the display unit 412 of the client apparatus 130 and presented to the user. Thereby, the user can know that there has been a change in the imaging area.

[0100] As described above, according to the image processing system in the present embodiment, it is possible to accurately detect a change area using a plurality of feature maps output from the neural network. Further, according to the image processing system in the present embodiment, since the user can set various thresholds while visually observing the change degree map, it is possible to execute the change area detection process with an easy operation without performing complicated settings.

[0101] (Modification Example) As a first modification example, some or all of the storage unit 402, calculation unit 403, extraction unit 404, setting unit 405, detection unit 406, and information output unit 407 included in the image processing apparatus 120 may be configured to be executed by the imaging apparatus 110 or the client apparatus 130.

[0102] As a second modification example, the detection unit 406 may acquire a boundary candidate map without generating a first change degree map from the captured image. The detection unit 406 acquires the boundary candidate map using, for example, a learned model that has been learned so that a boundary candidate map is output when the captured image is input. The detection unit 406 generates a final change map based on the output of the learned model and the second change degree map. In this case, the configuration may be such that the setting unit 405 does not set a boundary determination threshold value, or does not use the change determination threshold value set by the setting unit 405.

[0103] As a third modification example, the detection unit 406 may acquire a change map without generating a second change degree map from the captured image. The detection unit 406 acquires the change map using, for example, a learned model that has been learned so that a change map is output when the captured image is input. The detection unit 406 generates a final change map by combining the output of the learned model and the boundary candidate map. In this case, the configuration may be such that the setting unit 405 does not set a change determination threshold value, or does not use the change determination threshold value set by the setting unit 405.

[0104] As a fourth modification example, the information output unit 407 may be configured to determine not to notify when the size (area) of the changed area is larger than an area threshold value, and to determine to notify that there is no change in the imaging area otherwise. According to such a configuration, it is possible to notify the user that there is no change when there is no change. In this case, the setting unit 405 shall adopt a threshold value indicating the maximum value among a plurality of area threshold values of the captured images in a plurality of states to be notified.

[0105] As a fifth modification example, the determination result output by the information output unit 407 may be a value representing the necessity of notification in three or more levels. For example, the determination result may be the notification necessity level 1, the notification necessity level 2, and the notification necessity level 3 in descending order of the necessity of notification. In this case, the area threshold may be configured such that two values, i.e., a first area threshold and a second area threshold, are set. The first area threshold and the second area threshold are set based on the captured images in the notification required states corresponding to the notification necessity level 1 and the notification necessity level 2, respectively.

[0106] When the size of the change area is larger than the first area threshold, the information output unit 407 sets the determination result to "notification necessity level 1", and when the size of the change area is less than or equal to the first area threshold and larger than the second threshold, the information output unit 407 sets the determination result to "notification necessity level 2". Further, when the size of the change area is less than or equal to the second area threshold, the information output unit 407 sets the determination result to "notification necessity level 3".

[0107] As a sixth modification example, there may be a plurality of determination areas. In this case, the client device 130 displays a plurality of determination areas 1312 on the result display unit 1310, and each of them is editable. The storage unit 402 stores information indicating a plurality of determination areas. The calculation unit 403 calculates initial state parameters for each of the plurality of determination areas. The detection unit 406 acquires a change map for each of the plurality of determination areas in the captured image. The information output unit 407 determines whether to issue a notification for each of the plurality of change maps corresponding to the plurality of determination areas. At this time, the setting unit 405 may be configured to set a boundary determination threshold, a change determination threshold, and an area threshold for each of the plurality of determination areas.

[0108] The information output unit 407 can output information such as the following according to the determination result of the determination unit 406. That is, the information output unit 407 outputs information indicating whether there is a change or no change for each of the plurality of determination areas.

[0109] As a seventh modification, in the above-described embodiment, an example in which the feature map output from the intermediate layer of the neural network is used has been described, but the present invention is not limited to this. The feature map may be an output from a feature extraction layer in which feature extraction is performed in the neural network. For example, a configuration in which the output layer outputs a feature map as a feature extraction layer may be used. Further, a configuration in which a feature map is output from a feature extraction layer branched from an intermediate layer may be used.

[0110] The disclosure of this specification includes the following image processing apparatus, image processing method, and program.

[0111] (Item 1) Image acquisition means for acquiring a captured image of the imaging area; Extraction means for extracting a first feature map output from a first feature extraction layer in the neural network and a second feature map output from a second feature extraction layer farther from the input layer than the first feature extraction layer by inputting the captured image acquired by the image acquisition means into the input layer of the neural network; Estimation means for estimating a boundary between a change area where the imaging area has changed from a predetermined state and an area that is not the change area in the captured image based on the first feature map extracted by the extraction means; Detection means for detecting the change area in the captured image based on the second feature map extracted by the extraction means and the boundary estimated by the estimation means An image processing apparatus comprising the above.

[0112] (Item 2) It has acquisition means for acquiring a degree of change indicating the degree of change from the predetermined state in the captured image based on the feature map, The estimation means estimates, as the boundary, a position in the captured image where the degree of change based on the first feature map acquired by the acquisition means is greater than a threshold value. The image processing apparatus according to Item 1, characterized by the above.

[0113] (Item 3) The detection means detects, as the change region, a divided region that includes a position where the degree of change based on the second feature map acquired by the acquisition means is greater than a threshold value among the divided regions divided by the boundary in the captured image. The image processing apparatus according to item 2, characterized in that.

[0114] (Item 4) The detection means specifies a position different from the position of the boundary in the divided region as a representative point, and detects, as the change region, a divided region that includes a representative point where the degree of change at the representative point is greater than a threshold value. The image processing apparatus according to item 3, characterized in that.

[0115] (Item 5) The acquisition means acquires the degree of change using a Mahalanobis distance calculated based on a feature map extracted based on a neural network based on a captured image of an imaging region in the predetermined state and the first feature map or the second feature map. The image processing apparatus according to any one of items 2 to 4, characterized in that.

[0116] (Item 6) The image processing apparatus according to any one of items 1 to 5, characterized in that it has output means for outputting information for notifying another device of a change in the imaging region when the size of the change region detected by the detection means is greater than a threshold value.

[0117] (Item 7) The image processing apparatus according to item 6, characterized in that the size of the change region is represented by the number of pixels of the change region detected by the detection means with respect to the captured image.

[0118] (Item 8) The image processing apparatus according to item 6, wherein the size of the change region is represented by the number of pixels of the change region with respect to the number of pixels of a predetermined region in the captured image where the detection process of the change region is executed by the detection means.

[0119] (Item 9) The image processing apparatus according to any one of items 1 to 8, wherein the captured image is an image frame in a moving image.

[0120] (Item 10) The image processing apparatus according to any one of items 1 to 9, wherein the first feature extraction layer and the second feature extraction layer are pooling layers.

[0121] (Item 11) An image processing apparatus according to any one of items 1 to 10, imaging means for imaging the imaging region An imaging apparatus characterized by comprising.

[0122] (Item 12) An image acquisition step of acquiring a captured image in which an imaging region is imaged, By inputting the captured image acquired in the image acquisition step into the input layer of the neural network, a first feature map output from a first feature extraction layer in the neural network and a second feature extraction layer farther from the input layer than the first feature extraction layer An extraction step of extracting a second feature map output from; An estimation step of estimating a boundary between a change region where the imaging region has changed from a predetermined state and a region that is not the change region in the captured image based on the first feature map extracted in the extraction step; A detection step of detecting the change region in the captured image based on the second feature map extracted in the extraction step and the boundary estimated in the estimation step An image processing method characterized by comprising.

[0123] (Item 13) An image acquisition step of acquiring a captured image of the imaging area; By inputting the captured image acquired in the image acquisition step into the input layer of a neural network, a first feature map output from a first feature extraction layer in the neural network and a second feature map output from a second feature extraction layer farther from the input layer than the first feature extraction layer are extracted; an extraction step; Based on the first feature map extracted in the extraction step, an estimation step of estimating a boundary between a change area where the imaging area has changed from a predetermined state and an area that is not the change area in the captured image; Based on the second feature map extracted in the extraction step and the boundary estimated in the estimation step, a detection step of detecting the change area in the captured image A program for causing a computer to execute an image processing method having the above.

[0124] (Other embodiments) Each of the above-described embodiments may be implemented in combination with any embodiment.

[0125] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium and causing one or more processors in a computer of the system or device to read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Explanation of reference numerals

[0126] 120 Image processing apparatus 401 Image acquisition unit 404 Extraction unit 406 Detection unit

Claims

1. An image acquisition means for acquiring a captured image of a captured area; By inputting the captured image acquired by the image acquisition means into the input layer of a neural network, a first feature map output from a first feature extraction layer in the neural network, and a second feature extraction layer farther from the input layer than the first feature extraction layer An extraction means for extracting a second feature map output from; An estimation means for estimating a boundary between a change area where the captured area has changed from a predetermined state and an area that is not the change area in the captured image based on the first feature map extracted by the extraction means; A detection means for detecting the change area in the captured image based on the second feature map extracted by the extraction means and the boundary estimated by the estimation means An image processing apparatus characterized by comprising:

2. It has an acquisition means for acquiring a degree of change indicating the degree of change from the predetermined state in the captured image based on the feature map, The estimation means estimates, as the boundary, a position in the captured image where the degree of change based on the first feature map acquired by the acquisition means is greater than a threshold value. The image processing apparatus according to claim 1, characterized in that:

3. The detection means detects, as the change area, a divided area including a position where the degree of change based on the second feature map acquired by the acquisition means is greater than a threshold value among the divided areas divided by the boundary in the captured image. The image processing apparatus according to claim 2, characterized in that:

4. The detection means Identifies a position different from the position of the boundary in the divided area as a representative point, Detects, as the change area, a divided area including a representative point where the degree of change at the representative point is greater than a threshold value. The image processing apparatus according to claim 3, characterized in that...

5. The acquisition means acquires the degree of change by using a Mahalanobis distance calculated based on a feature map extracted based on a neural network from a captured image in which an imaging region in the predetermined state is imaged, and the first feature map or the second feature map. The image processing apparatus according to any one of claims 2 to 4, characterized in that.

6. The image processing apparatus according to claim 1, further comprising output means for outputting information for notifying another device of a change in the imaging region when a size of a change region detected by the detection means is larger than a threshold value.

7. The image processing apparatus according to claim 6, characterized in that the size of the change region is represented by the number of pixels of the change region detected by the detection means with respect to the captured image.

8. The image processing apparatus according to claim 6, characterized in that the size of the change region is represented by the number of pixels of the change region with respect to the number of pixels of a predetermined region in which detection processing of the change region is executed by the detection means in the captured image.

9. The image processing apparatus according to claim 1, characterized in that the captured image is an image frame in a moving image.

10. The image processing apparatus according to claim 1, characterized in that the first feature extraction layer and the second feature extraction layer are pooling layers.

11. The image processing apparatus according to claim 1, and imaging means for imaging the imaging region An imaging apparatus, characterized in that it has.

12. An image acquisition step of acquiring a captured image in which an imaging region is imaged, By inputting the captured image obtained in the image acquisition step into the input layer of a neural network, a first feature map output from a first feature extraction layer in the neural network and a second feature map output from a second feature extraction layer farther from the input layer than the first feature extraction layer are extracted in an extraction step. Based on the first feature map extracted in the extraction step, an estimation step of estimating a boundary between a change region where the imaging region has changed from a predetermined state and a region that is not the change region in the captured image. Based on the second feature map extracted in the extraction step and the boundary estimated in the estimation step, a detection step of detecting the change region in the captured image. An image processing method characterized by comprising the above.

13. An image acquisition step of acquiring a captured image of an imaging region. By inputting the captured image obtained in the image acquisition step into the input layer of a neural network, a first feature map output from a first feature extraction layer in the neural network and a second feature map output from a second feature extraction layer farther from the input layer than the first feature extraction layer are extracted in an extraction step. Based on the first feature map extracted in the extraction step, an estimation step of estimating a boundary between a change region where the imaging region has changed from a predetermined state and a region that is not the change region in the captured image. Based on the second feature map extracted in the extraction step and the boundary estimated in the estimation step, a detection step of detecting the change region in the captured image. A program for causing a computer to execute an image processing method having the above.

Images