Image processing apparatus, image processing method, and program

Abstract

To enable a user to easily set a region where an object can be erroneously detected.SOLUTION: An image processing apparatus 104 includes: a number estimation unit 306 for acquiring analysis information based on the result of a first detection process for detecting an object in an image; a parameter acquisition unit 302 for setting a first region of an image in which the result of the first detection process is not included in the analysis information; and an information generation unit 307 for generating the analysis information and setting information for changing the setting according to the result of comparison between a second region and the first region in the image identified based on the result of the first detection process and the result of a second detection process for detecting a moving body in the image.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to a technique for detecting an object in an image.

Background Art

[0002] In recent years, systems for detecting an object (e.g., a person, etc.) from an image captured by a network camera have been proposed. As a method for detecting an arbitrary object in an image, pattern matching, machine learning, etc. are used. However, with these methods, when detecting a person, for example, an object such as a mannequin, a poster, or a flower pot may be erroneously detected as a person. Patent Document 1 describes a method for improving the detection accuracy of a target object by having a user specify a region in the image where the target object is likely to be erroneously detected.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the method described in Patent Document 1, every time the position of the erroneously detected object changes, the user needs to re-set the region, and the processing on the user side is complicated.

[0005] The present disclosure has been made in view of the above problems. The object is to enable a user to easily set a region where an object can be erroneously detected.

Means for Solving the Problems

[0006] The image processing apparatus according to the present disclosure includes an acquisition unit that acquires analysis information based on the result of a first detection process for detecting a target object in an image, a setting unit that performs a setting related to a first region in the image where the result of the first detection process is not included in the analysis information, analysis information acquired by the acquisition unit, and setting information for changing the setting according to a comparison result between a second region in the image specified based on the result of the first detection process and the result of a second detection process for detecting a moving object in the image and the first region set by the setting unit, and a generation unit that generates the setting information.

Effect of the Invention

[0007] According to the present disclosure, a user can easily set a region where an object may be erroneously detected.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the components described in the following embodiments are examples of embodiments of the present disclosure, and the present disclosure is not limited thereto.

[0010] In this embodiment, an image processing system that detects a person from an image captured by an imaging device and counts the number of people in the image will be described as an example. According to this system, based on the number of people in the image, the approximate distance between human bodies, the degree of congestion, etc. can be estimated. Therefore, this system can be used, for example, to grasp the degree of congestion and avoid crowded states in offices, commercial facilities, etc. Note that the embodiment described below may be applied to systems other than those that count the number of people, and the target object to be detected may be other than a person. Also, in the following description, an example of a system that detects a target object in a continuously captured image (video) will be described, but it is not limited to this. This system may be a system that detects a target object in a plurality of arbitrary images (still images) including images that are not continuously captured. Also, in the following description, each image constituting a plurality of images (videos) is also referred to as an image frame.

[0011] FIG. 1 is a diagram for explaining the outline of the image processing system in this embodiment. The image processing system 101 in FIG. 1 is constructed for facilities and institutions to be analyzed, such as offices and commercial facilities. The image processing system 101 includes an imaging device 102, an image storage device 103, an image processing device 104, and a terminal device 105.

[0012] The imaging device 102 captures an area to be analyzed and transmits the captured image to the image storage device 103 and the image processing device 104. Also, the imaging device 102 receives instructions from the image storage device 103, the image processing device 104, and the terminal device 105, and performs controls such as imaging direction and zoom, and changes in imaging parameters. The imaging device 102 associates the captured image with meta information indicating information related to the imaging. The meta information includes, for example, imaging setting values of the image, date and time information indicating the date and time of imaging, and audio information. The meta information is associated, for example, by being described in an area where metadata of the image file can be described. Note that the imaging device 102 used in the image processing system 101 may be one or a plurality. Also, the imaging device 102 may be configured to transmit the captured image to the terminal device 105.

[0013] The image storage device 103 is a device that acquires the image captured by the imaging device 102 and the meta information related to the image, and stores the acquired information. The image storage device 103 receives instructions from the image processing device 104 and the terminal device 105, and transmits the stored information to the image processing device 104 and the terminal device 105. Also, the image storage device 103 can instruct the imaging device 102 to start / stop imaging and transmit the image.

[0014] The image processing device 104 performs analysis processing on the image acquired from at least one of the imaging device 102 and the image storage device 103. Also, the image processing device 104 receives instructions from the terminal device 105, and performs execution of analysis and changes in setting values. Also, the image processing device 104 notifies the terminal device 105 of analysis information including the analysis result.

[0015] The analysis process in this embodiment includes person (human body) detection processing, moving object detection processing, and estimation processing of a count removal area in an image, etc. Details of the content of each process and the output value will be described later. Note that the count removal area in this embodiment is an area in the image where, when person detection is performed to count the number of people in the image, an object that is not a person can be erroneously detected as a person (a person can be erroneously detected). If an object that is not a person is erroneously detected as a person, the erroneously detected object will be counted as a person, so the correct count cannot be obtained. The image processing apparatus 104 in this embodiment estimates and sets an area where a person can be erroneously detected as a count removal area, and does not include the number of objects detected in the count removal area in the total number of people in the image. That is, the count removal area is an area in the image area where the result of the human body detection process is not included in the analysis information.

[0016] The terminal device 105 instructs at least one of the imaging device 102, the image storage device 103, and the image processing device 104 to execute a process in response to an input from the user. Further, the terminal device 105 can display an image acquired from the imaging device 102 or the image storage device 103 on a display. Further, the terminal device 105 receives the analysis result and the set parameters from the image processing device 104 and displays the information on the display.

[0017] It is assumed that the imaging device 102, the image storage device 103, the image processing device 104, and the terminal device 105 are each connected to a network and can communicate with each other. Note that in the example in FIG. 1, it is assumed that the imaging device 102, the image storage device 103, the image processing device 104, and the terminal device 105 are different devices, but it is not limited to this. For example, a configuration in which the image processing device 104 is included inside the imaging device 102 may be used, or the image storage device 103, the image processing device 104, and the terminal device 105 may be configured as one device. Thus, a configuration in which at least two or more of the imaging device 102, the image storage device 103, the image processing device 104, and the terminal device 105 are integrated in a single PC or edge device may also be used.

[0018] FIG. 2 is a block diagram showing an example of the hardware configuration of the image processing apparatus 104. The image processing apparatus 104 is realized by a personal computer (PC), an embedded system, a tablet terminal, or the like. The image processing apparatus 104 includes a CPU 201, a ROM 202, a RAM 203, an external storage device 204, a storage 205, an operation unit 206, a display 207, an interface (I / F) 208, and a bus 209.

[0019] The CPU 201 is a central processing unit, which cooperates with other components based on a computer program and controls the operation of the entire image processing apparatus 104. The ROM 202 is a read-only memory, which stores a basic program, data used for basic processing, and the like. The RAM 203 is a writable memory, which functions as a work area of the CPU 201 and the like.

[0020] The external storage device 204 is a medium (recording medium) such as a hard disk drive or a USB memory, and stores a computer program, data, and the like. The storage 205 is a device that functions as a large-capacity memory such as an SSD (Solid State Drive). Various computer programs and data are stored in the storage 205.

[0021] The operation unit 206 is a device that receives inputs such as instructions and commands from the user. The operation unit 206 is, for example, a pointing device such as a keyboard and a mouse, and a device such as a touch panel. The display 207 is a display device that displays inputs from the operation unit 206, responses of the information processing apparatus to the inputs, various analysis results, and the like. Note that a configuration in which the operation unit 206 and the display 207 are integrated, such as a touch panel, may be used.

[0022] The I / F 208 is a device that relays data exchange with an externally connected device. The bus 209 is a data bus that controls the flow of data within the device.

[0023] The above is the hardware configuration of the image processing apparatus 104. Note that the image storage apparatus 103 and the terminal apparatus 105 also have the hardware configuration shown in FIG. 2. On the other hand, the imaging apparatus 102 has an imaging element, a lens, an image processing circuit, etc. in addition to the configuration shown in FIG. 2, and often does not have an operation unit 206, a display 207, etc. Also, the image storage apparatus 103 and the image processing apparatus 104 may each have a configuration like a server that does not have an operation unit 206, a display 207, etc. Further, it may be configured as an alternative to the hardware device by software that realizes functions equivalent to those of each configuration shown in FIG. 2.

[0024] FIG. 3 is a block diagram showing the functional configuration of the apparatuses in the image processing system 101. The functional configurations of the image processing apparatus 104 and the terminal apparatus 105 are shown in FIG. 3. The image processing apparatus 104 includes an image acquisition unit 301, a parameter acquisition unit 302, a human body detection unit 303, a moving body detection unit 304, a count removal area estimation unit 305, a number of people estimation unit 306, an information generation unit 307, and a storage unit 308 and a display unit 311. Also, the terminal apparatus 105 includes a display unit 311 and an input unit 312. Each function shown in FIG. 3 is realized by the CPU 201 included in the image processing apparatus 104 and the terminal apparatus 105 executing a computer program. Image information analysis Parameter detection Human body information detection Moving body information Removal candidate area information Number of people information The following describes each function.

[0025] The image acquisition unit 301 acquires image information from the imaging apparatus 102 or the image storage apparatus 103. The image acquisition unit 301 transmits the acquired image information to the human body detection unit 303 and the moving body detection unit 304. Also, the image acquisition unit 301 stores the image information in the storage unit 308 or the image storage apparatus 103. Here, the image information is an image obtained by the imaging apparatus 102 and meta information of the image.

[0026] The parameter acquisition unit 302 acquires analysis parameters from the terminal device 105. Further, based on the acquired analysis parameters, the parameter acquisition unit 302 performs settings related to the processing performed by the image processing device 104. The analysis parameters refer to parameters related to the analysis process input via the terminal device 105, and include, for example, a threshold value for determining whether a detected object is a human body in human body detection, information on the count removal area, and the execution interval of the analysis.

[0027] Note that the information on the count removal area included in the analysis parameters includes, for example, information indicating the coordinates of the count removal area. Further, the information on the count removal area includes information indicating the count removal area specified in advance by the user and stored in the storage unit 308 or the image storage device 103. Based on the information on the count removal area, the parameter acquisition unit 302 performs settings for the count removal area used in the number estimation process performed by the image processing device 104.

[0028] As a first detection process, the human body detection unit 303 detects a human body from the images included in the image information and outputs detected human body information. Each time the human body detection unit 303 performs human body detection, it associates the detected human body information with the image information and stores it in the storage unit 308. Here, the detected human body information is a value obtained from the output of the human body detector included in the human body detection unit 303, and includes coordinate information of the detection area for each detected human body and likelihood information indicating the probability of being a human body. Note that when the human body detector can also perform attribute classification such as gender and person identification processing in addition to human body detection, the detected human body information includes IDs and labels indicating specific persons and attributes.

[0029] As a second detection process, the moving object detection unit 304 detects moving objects using a plurality of image frames included in the image information and outputs detected moving object information. Each time the moving object detection unit 304 performs moving object detection, it associates the detected moving object information with the image information and stores it in the storage unit 308. Here, the detected moving object information is information related to the moving objects detected by the moving object detection unit 304, and includes coordinate information of the detected moving object area and an index value indicating the magnitude of the movement. For example, a moving person or the like can be detected as a moving object.

[0030] Based on the detected human body information and the detected moving object information, the count removal area estimation unit 305 estimates the misdetection area in the image and outputs information indicating the removal candidate area. The count removal area estimation unit 305 estimates the misdetection candidate area where the human body can be misdetected by specifying the detection frequencies of the human body and the moving object based on the detected human body information and the detected moving object information within a predetermined period. Further, the count removal area estimation unit 305 outputs information indicating the removal candidate area by performing correction processing on the estimated misdetection area. Specific estimation methods and correction methods will be described later. Note that the method for estimating the misdetection candidate area performed by the count removal area estimation unit 305 is not limited to the method described in this embodiment, and any method can be used according to the characteristics of the system. The information indicating the removal candidate area includes the coordinate information of the removal candidate area output by the count removal area estimation unit 305, the likelihood information indicating the probability of the removal candidate area, and the like.

[0031] Based on the information of each human body included in the detected human body information and the detection threshold value and the count removal area included in the analysis parameters, the number of people estimation unit 306 counts the number of people in the image and outputs the number of people information as analysis information. The number of people information is the total number of human bodies in the image counted by the number of people estimation unit 306. At this time, when the count removal area is set, the number of human bodies detected in the count removal area is not included in the total number of human bodies in the image. Note that, in addition to the total number of human bodies, the number of human bodies included in the count removal area may be included in the number of people information as the number of people removed from the total number.

[0032] The information generation unit 307 generates information for presenting at least any one of image information, analysis parameters, detected human body information, information indicating a removal candidate region, and number information to the user, and outputs it to the terminal device 105. For example, when the result of human body detection as the output information is displayed on the terminal 105 side, the information generation unit 307 generates the result of number estimation and the breakdown information of the estimation result in accordance with the display format based on the image information, detected human body information, and number information. When the setting content of the count removal region is to be displayed on the terminal 105 side, the information generation unit 307 generates information for changing the setting of the count removal region set by the user in accordance with the display format based on the image information, analysis parameters, and information indicating the removal candidate region.

[0033] The generated information can take an arbitrary form in accordance with the display format of the display unit 311. Also, the generated information may be a structure or markup language that aggregates information necessary for display.

[0034] The storage unit 308 stores the image information, analysis parameters, detected human body information, detected moving body information, information indicating the removal candidate region, number information, and information generated by the information generation unit 307, which are output from each processing unit. The storage unit 308 corresponds to the RAM 203, external storage device 204, and storage 205 of the image processing apparatus 104.

[0035] The display unit 311 is a display unit on which the display information output from the image processing apparatus 104 is displayed, and corresponds to the display 207 of the terminal device 105. The display method varies depending on the type of terminal and the system configuration, and can take an arbitrary form. In addition, when the imaging device 102, image storage device 103, and image processing apparatus 104 have a display 207, the display information may be displayed on the display 207 of each device.

[0036] The input unit 312 receives input from the user. Further, the terminal device 105 transmits an instruction to the image processing device 104 according to the input from the user. The input unit 312 corresponds to the operation unit 206 of the terminal device 105. The instruction content is, for example, an instruction to set analysis parameters and an instruction to change the settings, or an instruction to execute or stop the processing. Further, the terminal device 105 can also give instructions such as execution or stop of processing to the imaging device 102 and the image storage device 103.

[0037] Hereinafter, the processing flow in the image processing system 101 will be described using the flowchart of FIG. 4. Here, based on the results of human body detection and moving object detection within a predetermined period, a candidate region for count removal is estimated, and by comparing the count removal region set in the image processing device 104 with the candidate region, a process of presenting a recommended setting of the count removal region to the user will be described. The processing shown in FIG. 4 is realized by the CPU 201 of the image processing device 104 executing a computer program stored in the storage unit.

[0038] In S401, the parameter acquisition unit 302 acquires information on the count removal region. The parameter acquisition unit 302 receives a designation of the count removal region by the user via the terminal device 105, or acquires information indicating the count removal region stored in advance as analysis parameters. The parameter acquisition unit 302 sets the count removal region in the image processing device 104 based on the designation by the user or the acquired information. If there is no count removal region stored in advance or there is no designation from the user, the setting of the image processing device 104 is completed with no count removal region, and the process proceeds to S402.

[0039] In S402, the count removal region estimation unit 305 estimates a misdetection region based on the detected human body information and detected moving object information of the human body detected by the human body detection unit 303 and the moving object detection unit 304 within a predetermined period, and outputs information indicating a removal candidate region. Specific estimation means will be described with reference to FIG. 5.

[0040] In S501, the image acquisition unit 301 acquires a plurality of images captured during a predetermined period and outputs image information. The image acquisition unit 301 acquires a plurality of images captured during a predetermined period based on the date and time information included in the meta information associated with the images. The method of acquiring the images may be realized by communication between devices or may be realized using the file input / output function. Also, the predetermined period during which a plurality of images are acquired may be a period preset in the system or a period specified by the user.

[0041] In S502, the human body detection unit 303 extracts an image frame from the plurality of images acquired in S501, performs human body detection, and creates list information of the detected human body information. In the list information, it is assumed that the information indicating the detected human body area is associated with the time of the image frame used for the detection. Note that the time of the image frame corresponds to the time when the image frame was captured, but the information associated with the information indicating the human body area is not limited to this and may be an identification number of the image frame or the like.

[0042] The human body detection unit 303 of the present embodiment performs human body detection by an object detection algorithm using machine learning. The method of human body detection is not limited to this, and a method of performing matching with a pre-registered human body model may be used. Also, the human body detection unit 303 may be configured to execute resizing processing of the image frame, noise removal, etc. as preprocessing for human body detection. The human body detection unit 303 of the present embodiment extracts image frames at regular intervals from a plurality of images to perform human body detection, but is not limited to this, and all image frames may be extracted, or image frames may be extracted according to predefined conditions.

[0043] In S503, the moving object detection unit 304 extracts image frames from the plurality of images acquired in S501 to perform moving object detection, and creates list information of the detected moving object information. It is assumed that in the list information, information indicating the detected moving object area and the time of the image frame used for the detection are associated with each other. Note that the time of the image frame corresponds to the time when the image frame was captured, but the information associated with the information indicating the moving object area is not limited to this, and may be an identification number of the image frame or the like.

[0044] The moving object detection unit 304 of the present embodiment compares pixel information between frames, and determines pixels with a pixel value difference equal to or greater than a certain value as moving objects. The method of moving object detection is not limited to this, and it may be a method of comparing the variance of pixel values within a specified block size between frames, or a method of realizing moving object determination using algorithms such as background subtraction method and optical flow. Also, the moving object detection unit 304 of the present embodiment extracts image frames at regular intervals from a plurality of images to perform moving object detection, but is not limited to this, and all image frames may be extracted, or image frames may be extracted according to predefined conditions. Also, the image frames extracted in S503 and the image frames extracted in S502 may be the same or different.

[0045] In S504, the count removal area estimation unit 305 generates a frequency map of human body detection in the image from the list of detected human body information output by S502. The frequency map of human body detection in the present embodiment is generated by comparing the detected human body information corresponding to different times included in the list of detected human body information output in S502, and weighting the detection areas where the human body is detected multiple times. The method of generating the frequency map is not limited to this, and different weightings may be performed according to the likelihood of the detected human body, or weighting may be performed on the area based on indicators such as the ratio of the overlapping area of a plurality of human body detection areas and the distance between the center points of the human body detection areas.

[0046] In S505, the count removal area estimation unit 305 generates a moving object detection frequency map within the image from the list of detected moving object information output in S503. The moving object detection frequency map in the present embodiment is generated by comparing the detected moving object information corresponding to different times included in the list of detected moving object information output in S503 and weighting the detection areas where the moving object is detected multiple times. The method for generating the frequency map is not limited to this, and different weightings may be performed according to the flow rate of the moving object, or the area ratio where a plurality of moving object detection areas overlap, and the distance between the center points of the moving object detection areas may be used as an index to weight the areas. Note that the areas weighted in S504 and S505 may be areas corresponding to the pixels of the image corresponding to the imaging range of the imaging device 102, or areas divided into a predetermined size.

[0047] In S506, the count removal area estimation unit 305 determines a false detection candidate area based on the human body detection frequency map generated in S504 and the moving object detection frequency map generated in S505. In the present embodiment, an area where the weight corresponding to the detection frequency in the human body detection frequency map is higher than a predetermined threshold and the weight corresponding to the detection frequency in the moving object detection frequency map is lower than the predetermined threshold is determined as the false detection candidate area. The method for determining the false detection candidate area is not limited to this, and an area that satisfies any one of the above conditions may be determined as the false detection area, or it may be determined using an area of either the human body detection frequency map or the moving object detection frequency map. Also, a likelihood indicating the probability of the false detection candidate area may be determined according to the weight corresponding to the detection frequency in the human body detection and moving object detection frequency maps.

[0048] An area that satisfies the condition in S506 is an area where the frequency of detecting a moving object is low but the frequency of detecting a human body is high. That is, there is a high possibility that there is an object that does not move, such as a mannequin, and is easily misdetected as a human body. In S506, such an area is determined as the false detection candidate area.

[0049] In S506, when the count removal area estimation unit 305 determines that there is an area that satisfies the condition (YES in S506), it proceeds to S507. When it is determined that there is no area that satisfies the condition (NO in S506), the estimation process of the count removal candidate area ends.

[0050] In S507, the count removal area estimation unit 305 corrects the false detection candidate area output in S506 and outputs information indicating the removal candidate area. In the present embodiment, a morphological operation is performed on the false detection candidate area, and correction is performed to remove excessively small areas, areas with a large number of vertices, and the like. Note that the correction method and whether to perform correction are arbitrarily determined according to the setting method of the count removal area. Also, the estimated removal candidate area may be one or a plurality.

[0051] Through the above flow, the count removal area estimation unit 305 estimates the false detection area based on the detected human body information and the detected moving object information, and outputs information indicating the removal candidate area.

[0052] Note that in the above flow, when a human body passes in front of an object that may be falsely detected, there is a problem that the corresponding area is determined as a moving object by moving object detection and is not correctly determined as a false detection candidate area. In view of this problem, in S504 and S505, the count removal area estimation unit 305 may extract the detection results used for generating the frequency map according to the following conditions. That is, the count removal area estimation unit 305 can generate a frequency map corresponding to a period in which a human body does not pass in front of the falsely detected object very much by extracting a period in which the displacement of the moving object detection count or the human body detection count is below a certain level. Also, the extraction conditions of the detection results are not limited to this, and other indicators such as the overlap degree between detection areas may be used, or the period may be extracted by an external input. Also, for example, the business hours of the store where the system is installed may be input, and a configuration may be adopted in which the analysis results outside the business hours are extracted and the count removal area is estimated.

[0053] Returning to FIG. 4, in S403, the count removal area estimation unit 305 executes the processes of S404 to S408 on the count removal area (hereinafter referred to as the set removal area) set in the image processing apparatus 104 in S401. When there are a plurality of set removal areas, one set removal area is set as the processing target, and the processes of S404 to S408 are executed. This is repeated for the number of set removal areas. The processing for the case where there is no set removal area will be described later.

[0054] In S404, the count removal area estimation unit 305 executes the processes of S405 to S408 on the removal candidate area estimated by the process shown in FIG. 5. When there are a plurality of removal candidate areas, one removal candidate area is set as the processing target, and the processes of S405 to S408 are executed. This is repeated for the number of removal candidate areas.

[0055] In S405, the count removal area estimation unit 305 compares the set removal area that was the processing target in S403 with the removal candidate area that was the processing target in S404, and determines whether at least a part of each area overlaps. Based on the comparison result, when the count removal area estimation unit 305 determines that there is an overlap (YES in S405), it proceeds to S406. On the other hand, when the count removal area estimation unit 305 determines that there is no overlap (NO in S405), it proceeds to S407. In the present embodiment, it is determined whether at least a part of each area overlaps, but it is not limited to this. A configuration may be adopted in which it is determined that there is an overlap when the area of the overlapping area is larger than a threshold value. That is, when at least a part of the set removal area and the removal candidate area overlaps, or when the area of the overlapping area is larger than a predetermined threshold value, the count removal area estimation unit 305 proceeds to S406. Also, when the set removal area and the removal candidate area do not overlap, or when the area of the overlapping area is smaller than a predetermined threshold value, the count removal area estimation unit 305 proceeds to S407. Also, a configuration may be adopted in which it is determined that there is an overlap when the distance between the center points of the areas is smaller than a threshold value.

[0056] In the case where there is no set removal area in S403, it is assumed that the processes of S404, S405, and S408 are executed. That is, in S405, it is determined as NO, and S408 described later is executed. This is repeated for the number of removal candidate areas.

[0057] In S406, the count removal area estimation unit 305 determines that the set removal area is a maintenance recommended area that recommends maintaining it as a count removal area. That is, the maintenance recommended area indicates that since the set removal area is an area where false detection is likely to occur, it should continue to be maintained as a count removal area. The information generation unit 307 generates information for presenting the maintenance recommended area to the user as display information. After executing the process of S406, the count removal area estimation unit 305 returns to S403 and sets the next removal candidate area to be processed.

[0058] In S407, the count removal area estimation unit 305 determines that the set removal area is a release recommended area that should be released from the setting of the count removal area. That is, the release recommended area indicates that since the set removal area is not an area where false detection is likely to occur, it can be determined that it may be released from the setting of the count removal area. The information generation unit 307 generates information for presenting the release recommended area to the user as display information.

[0059] Note that depending on the usage environment of the system, a count removal area may be set even if there are no objects that can be misdetected. For example, consider a case where one of two adjacent different stores uses the system, but both stores are reflected in the captured image used for analysis. In this case, the user of this system sets the other store area in the image as a count removal area in order to grasp the congestion situation only within their own store. The count removal area estimation unit 305 of the present embodiment determines whether the set removal area is a recommended removal area according to the detection result of the misdetection area. Therefore, for a removal area set regardless of the misdetection area, it is impossible to determine whether it is a recommended removal area. Thus, processing such as not making the count removal area set regardless of the misdetection area the subject of the determination in S407 or not selecting it as a processing target in S403 may be performed. Also, in S407, the count removal area estimation unit 305 may be configured to additionally determine whether the set removal area was set in consideration of misdetection or for other purposes. For example, when a certain number or more of human bodies are detected in the set removal area, the count removal area estimation unit 305 determines that it is not for the purpose of excluding specific misdetected objects and does not determine it as a recommended removal area. Note that as a determination criterion, indicators such as the area of the set removal area and the flow rate of people with respect to the set removal area may be used. Also, a configuration may be adopted in which information indicating the purpose of the pre - set in the set removal area is associated, and the count removal area estimation unit 305 makes a determination based on the associated information.

[0060] In S408, the count removal area estimation unit 305 determines the removal candidate area as an additional recommended area to be set as the count removal area. That is, the additional recommended area indicates that the removal candidate area is an area where misdetection is likely to occur and is not set as the count removal area, so it should be set as the count removal area. The information generation unit 307 generates information for presenting the additional recommended area to the user as display information.

[0061] By repeating the above-described process the number of times equal to the number of set removal areas and the number of candidate removal areas, display information for presenting to the user the recommended maintenance area, the recommended release area, and the recommended addition area is generated. In S409, the display information generated by the information generation unit 307 is output to the terminal device 105 in order to be displayed on the display unit 311. The display information displayed on the terminal device 105 includes setting information for changing the set count removal area according to the identified recommended maintenance area, recommended release area, and recommended addition area. The display format can take any form according to the system configuration. Here, an example of the display is shown using FIG. 6.

[0062] In FIG. 6, the display screen 601 is an example in which the image acquired by the image acquisition unit 301 is displayed. The image displayed on the display screen 601 may be a video (image) captured in real time, or may be the image used for the analysis in S401 to S408. Also, both the video captured in real time and the image used for the analysis may be displayed, or several frames before and after either image may be further displayed.

[0063] The display frame 602 is an example showing the area determined to be the recommended maintenance area in S406. The recommended maintenance area is displayed in a predetermined display format. The display frame 602 is represented, for example, by a frame line of a default color set in advance in the image processing apparatus 104 as information indicating the area. Note that, similar to the recommended maintenance area, the set removal area may be configured to be displayed in a predetermined display format.

[0064] The display frame 603 is an example showing the area determined as the additional recommended area in S406. In order to express that the area should be additionally set, the frame line of the area is displayed in a color different from the default color. Note that the display methods of the maintenance recommended area and the additional recommended area are not limited to this, and they may be displayed with any line thickness, color density, pattern, etc. That is, the additional recommended area is displayed in a display format different from the display formats of the maintenance recommended area and the set removal area. Also, when an index value or the like is used in the determination process of the misdetection area in S407, the index value may be associated with the line thickness and density of the frame line, etc., and the line thickness and density of the frame line may change according to the index value. Thereby, there is an effect that the importance of the additional recommended area according to the index value becomes easy for the user to understand.

[0065] The display frame 604 is an example showing the area determined as the removal recommended area in S407. In the display frame 604, in order to express that the set area should be removed, the frame line of the area is changed to a dotted line instead of a solid line and displayed. Note that the display method of the removal recommended area is not limited to this, and it may be displayed using any frame line color, thickness, density, pattern, etc. That is, the removal recommended area is displayed in a display format different from the display formats of the maintenance recommended area and the set removal area. As described above, the maintenance recommended area, the additional recommended area, and the removal recommended area are presented to the user in distinguishable different display formats.

[0066] The display frame 605 is an enlarged display of the display frame 603. The display frame 605 may execute processes such as display / non-display and zoom-in / zoom-out according to the user's input. Also, when the area where the image is displayed in the display frame 605 is enlarged, the image within the area corresponding to different times or periods may be displayed. The dialog 606 is a dialog that presents options for whether to apply the additional recommended area corresponding to the display frame 603 to the system. Note that the display content such as the explanatory text of the dialog is not limited to the example in FIG. 6.

[0067] The user can input to the dialog 606 using the input unit 312 of the terminal device 105. The parameter acquisition unit 302 branches the processing according to the input value for the display 606. When "Yes" is selected for the dialog 606, the parameter acquisition unit 302 applies the additional recommended area corresponding to the display frame 603 as a count removal area on the system. When "No" is selected for the dialog 606, the parameter acquisition unit 302 discards the additional recommended area corresponding to the display frame 603. The discarded area information may be deleted from the storage device or added to the additional removal list. The areas added to the additional removal list are controlled so as not to be detected as additional recommended areas in the subsequent estimation process of the count removal area, or are controlled so as not to be presented to the user even if they are redetected as additional recommended areas.

[0068] Also, when the number of additional recommended areas is large, the displays asking about the applicability of multiple areas may be aggregated into one dialog. Also, when the human body detection unit 303 has an attribute classification function, the dialogs asking about the applicability of areas for each attribute may be aggregated. The vertex 607 is the vertex of the display frame 603. The user can change the shape of the additional recommended area by moving the vertices of the display frame 603 including the vertex 607 by a drag operation. Note that the method of correcting the additional recommended area is not limited to this, and addition / deletion of vertices or enlargement / reduction of the area may be possible by user operations. By performing such processing, it becomes possible to easily perform the addition process of the count removal area.

[0069] The display frame 608 is an enlarged version of the removal recommended area displayed in the display frame 604. In response to a user instruction for the display frame 608, display / non-display of the image within the frame and enlargement / reduction processing may be executed. Also, when the image within the display frame 608 is enlarged and displayed, a configuration in which images within areas corresponding to different times or periods are displayed may be used. The dialog 609 is a dialog that presents options for whether to apply the removal recommended area corresponding to the display 604 to the system. Note that the display content such as the explanatory text of the dialog is not limited to the example of FIG. 6.

[0070] The user can input to the dialog 609 using the input unit 312 of the terminal device 105. The parameter acquisition unit 302 branches the process according to the input to the dialog 609. When "Yes" is selected for the dialog 609, the parameter acquisition unit 302 deletes the recommended release area corresponding to the display frame 604 from the count removal area. When "No" is selected for the dialog 609, the parameter acquisition unit 302 maintains the recommended release area corresponding to the display frame 604 as the count removal area on the system. Note that the maintained area information may be added to the deletion removal list. The area added to the deletion removal list is controlled so as not to be detected as the recommended release area in the subsequent estimation process of the count removal area, or not to be presented to the user even if it is redetected as the recommended release area.

[0071] Also, when the number of recommended release areas is large, the displays asking about the applicability of a plurality of areas may be aggregated into one dialog. Also, when the human body detection unit 303 has an attribute classification function, the dialogs asking about the applicability of areas for each attribute may be aggregated. The vertex 610 is the vertex of the frame indicating the recommended release area corresponding to the display 604. The user can change the shape of the recommended release area by moving the vertex of the frame including the vertex 610 by a drag operation. Note that the correction method of the recommended release area is not limited to this, and addition / deletion of vertices or enlargement / reduction of the area may be possible by user operations. In this way, the user can easily perform the release process of the count removal area.

[0072] As described above, the image processing apparatus 104 compares a removal candidate region, which is a second region specified based on the results of the human body detection process and the moving object detection process, with a count removal region set in the image processing apparatus 104 as the first region, by the process shown in FIG. 4. Further, the image processing apparatus 104 causes the display unit 311 to display setting information for changing the setting of the count removal region, according to the comparison result. The processes in FIGS. 4 and 5 are assumed to be executed at an arbitrary timing. For example, it may be executed at the timing when the interior of the store where the present system is installed is changed, or may be executed at a predetermined interval such as every hour. According to the processes in FIGS. 4 and 5, even when the positions of objects such as mannequins and flowerpots, which are likely to be erroneously detected as human bodies, are moved, it is possible to easily re-set the count removal region.

[0073] Further, the number estimation unit 306 of the image processing apparatus 104 performs number estimation in consideration of the count removal region set by the processes in FIGS. 4 and 5. The information generation unit 307 generates the result of the number estimation as analysis information to be displayed on the display unit 311. Note that the process of the number estimation unit 306 may be performed simultaneously with the processes in FIGS. 4 and 5, or may be independently executed at a different timing. When the process of the number estimation unit 306 is executed at a timing different from the processes in FIGS. 4 and 5, the human body detection unit 303 may be configured to perform a human body detection process for number estimation performed by the number estimation unit 306, separately from the process shown in S502 of FIG. 5. Further, the human body detection unit 303 may be configured not to perform human body detection in the count removal region in human body detection for number estimation. Thereby, the area of the region where the human body detection process is performed becomes smaller, and when calculating the total number of people in the image, it is not necessary to subtract the number of people in the count removal region, so the processing load is reduced.

[0074] As described above, according to the image processing system of the present embodiment, by comparing a removal candidate region based on a false detection candidate region where false detection may occur with a count removal region set in the system, a recommended setting of the count removal region is presented to the user. Thereby, the user can easily set the count removal region, which is a region where an object may be erroneously detected.

[0075] (Modification Example 1) In the first embodiment, a system for counting the number of people in an image was described as an example. However, the present invention is not limited to this. The system in the first embodiment is applicable to, for example, a system that detects a person, an animal, a moving object, etc. as a target object in an image and notifies that the target object has been detected as analysis information. Also in such a system, in order to prevent a different object from being erroneously detected from the target object to be detected, it is assumed that a detection removal area where no object is detected is set. Also in this case, the image processing apparatus 104 can estimate a detection removal candidate area by the process of FIG. 5 and present information on a recommended area for recommending to the user to cancel or add by comparing with a preset detection removal area.

[0076] Note that the above-described system is not limited to a configuration in which object detection is not performed in the detection removal area, and may be a configuration in which no notification is made even if a target object is detected in the detection removal area. Also, in addition to the above-described notification, a configuration in which information on the detected target object and analysis information indicating the number of detected people are displayed may be used.

[0077] (Modification Example 2) In the above-described embodiments, the configuration in which the image processing apparatus 104 executes analysis processes such as human body detection processing, moving body detection processing, and number estimation processing has been described, but the present invention is not limited to this. For example, the imaging apparatus 102 may be configured to execute at least any one of human body detection processing, moving body detection processing, and analysis processing. As an example, when the imaging apparatus 102 includes a human body detection unit 303 and a number estimation unit 306, the image processing apparatus 104 acquires detection human body information and analysis information from the imaging apparatus 102. Further, the count removal area estimation unit 306 of the image processing apparatus 104 estimates a removal candidate area based on the acquired detection human body information and the detected moving body information generated by the moving body detection unit 304. Thus, a configuration in which at least any one of human body detection processing, moving body detection processing, and analysis processing is performed by a different apparatus may be employed. Further, the imaging apparatus 102 and the image processing apparatus 104 may be the same apparatus. In this case, the imaging apparatus 102 performs human body detection processing, moving body detection processing, and analysis processing, and causes the display unit 311 of the terminal device 105 to display analysis information and setting information for changing settings related to the count removal area.

[0078] The disclosure of this specification includes the following image processing apparatus, image processing method, and program.

[0079] (Item 1) An acquisition unit that acquires analysis information based on a result of a first detection process for detecting a target object in an image; A setting unit that performs setting for a first area in the image, where a result of the first detection process is not included in the analysis information; A generation unit that generates the analysis information acquired by the acquisition unit, setting information for changing the setting according to a comparison result between a second area in the image specified based on the result of the first detection process and a result of a second detection process for detecting a moving body in the image, and the first area set by the setting unit An image processing apparatus, characterized by comprising the above.

[0080] (Item 2) The image processing apparatus according to item 1, wherein when an input for instructing to change the setting regarding the first region is performed, the setting means changes the setting regarding the first region.

[0081] (Item 3) The image processing apparatus according to item 1 or 2, wherein when the first region and the second region do not overlap or the area of the overlapping region is smaller than a predetermined threshold, the generation means generates information indicating that the second region should be added as the first region, as information for causing the display means to display it.

[0082] (Item 4) The image processing apparatus according to item 3, wherein the information indicating the first region and the information indicating the second region are information displayed in different display formats.

[0083] (Item 5) The image processing apparatus according to any one of items 1 to 3, wherein when the first region and the second region do not overlap or the area of the overlapping region is smaller than a predetermined threshold, the generation means generates information indicating that the first region set by the setting means should be released, as information for causing the display means to display it.

[0084] (Item 6) The image processing apparatus according to item 5, wherein when the information indicating the first region set by the setting means is displayed in a predetermined display format, the display format of the first region to be released is different from the predetermined display format.

[0085] (Item 7) The image processing apparatus according to any one of items 1 to 6, wherein when at least a part of the first region and the second region overlap or the area of the overlapping region between the first region and the second region is larger than a predetermined threshold, the generation means generates information indicating that the first region set by the setting means should be maintained, as information for causing the display means to display it.

[0086] (Item 8) The acquisition means acquires, as analysis information based on the result of the first detection process, information indicating the number of the target objects detected in the image, and is characterized in that it is the image processing apparatus according to any one of Items 1 to 7.

[0087] (Item 9) The number of the target objects indicated by the analysis information acquired by the acquisition means does not include the number of the target objects detected by the first detection process in the first region, and is characterized in that it is the image processing apparatus according to Item 8.

[0088] (Item 10) The acquisition means acquires, as analysis information based on the result of the first detection process, information indicating that the target object has been detected in the image, and is characterized in that it is the image processing apparatus according to any one of Items 1 to 9.

[0089] (Item 11) The analysis information means acquired by the acquisition means does not include information indicating that the target object has been detected by the first detection process in the first region, and is characterized in that it is the image processing apparatus according to Item 10.

[0090] (Item 12) The first detection process for acquiring the analysis information with respect to the first region set by the setting means is not performed, and is characterized in that it is the image processing apparatus according to any one of Items 1 to 11.

[0091] (Item 13) The target object is a person, and is characterized in that it is the image processing apparatus according to any one of Items 1 to 11.

[0092] (Item 14) An acquisition step of acquiring analysis information based on the result of a first detection process for detecting a target object in an image; A setting step of setting a first region among regions in the image, where the result of the first detection process is not included in the analysis information; A generation step of generating the analysis information acquired in the acquisition step, setting information for changing the setting according to a comparison result between a second region in the image specified based on the result of the first detection process and the result of a second detection process for detecting a moving object in the image and the first region set by the setting means; An image processing method characterized by comprising the above.

[0093] (Item 15) A program for causing a computer to function as the image processing apparatus according to any one of Items 1 to 13.

[0094] (Other embodiments) The present disclosure can also be realized by supplying a program for realizing one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium and causing one or more processors in a computer of the system or apparatus to read and execute the program. Further, it can also be realized by a circuit (for example, ASIC) for realizing one or more functions.

Explanation of reference numerals

[0095] 104 Image processing apparatus 302 Parameter acquisition unit 306 Number-of-people estimation unit 307 Information generation unit

Claims

1. An acquisition means for acquiring analysis information based on the result of a first detection process for detecting a target object in an image; A setting means for setting a first area among the areas in the image, where the result of the first detection process is not included in the analysis information; A generation means for generating the analysis information acquired by the acquisition means, setting information for changing the setting according to a comparison result between a second area in the image specified based on the result of the first detection process and the result of a second detection process for detecting a moving object in the image and the first area set by the setting means An image processing apparatus characterized by comprising the above.

2. The image processing apparatus according to claim 1, wherein the setting means changes the setting regarding the first area when an input for instructing to change the setting regarding the first area is performed.

3. The image processing apparatus according to claim 1, wherein when the first area and the second area do not overlap or the area of the overlapping region is smaller than a predetermined threshold, the generation means generates, as information for causing a display means to display, information indicating that the second area should be added as the first area.

4. The image processing apparatus according to claim 3, wherein the information indicating the first area and the information indicating the second area are information displayed in different display forms.

5. The image processing apparatus according to claim 1, wherein when the first area and the second area do not overlap or the area of the overlapping region is smaller than a predetermined threshold, the generation means generates, as information for causing a display means to display, information indicating that the first area set by the setting means should be released.

6. The image processing apparatus according to claim 5, wherein when the information indicating the first area set by the setting means is displayed in a predetermined display form, the display form of the first area to be released is different from the predetermined display form.

7. The image processing apparatus according to claim 1, wherein when at least a part of the first area and the second area overlaps or the area of the overlapping region between the first area and the second area is larger than a predetermined threshold, the generation means generates, as information for causing a display means to display, information indicating that the first area set by the setting means should be maintained.

8. The image processing apparatus according to claim 1, wherein the acquisition means acquires, as analysis information based on the result of the first detection process, information indicating the number of the target objects detected in the image.

9. The image processing apparatus according to claim 8, wherein the number of the target objects indicated by the analysis information acquired by the acquisition means does not include the number of the target objects detected by the first detection process in the first region.

10. The image processing apparatus according to claim 1, wherein the acquisition means acquires, as analysis information based on the result of the first detection process, information indicating that the target object has been detected in the image.

11. The image processing apparatus according to claim 10, wherein the analysis information means acquired by the acquisition means does not include information indicating that the target object has been detected by the first detection process in the first region.

12. The image processing apparatus according to claim 1, wherein the first detection process for acquiring the analysis information with respect to the first region set by the setting means is not performed.

13. The image processing apparatus according to claim 1, wherein the target object is a person.

14. An acquisition step of acquiring analysis information based on the result of a first detection process for detecting a target object in an image; A setting step of setting, among regions in the image, a first region whose result of the first detection process is not included in the analysis information; A generation step of generating the analysis information acquired in the acquisition step, setting information for changing the setting according to a comparison result between the first region set by the setting means and a second region in the image specified based on the result of the first detection process and the result of a second detection process for detecting a moving object in the image An image processing method characterized by comprising the steps.

15. A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 13.

Images