Control device, actuation method for control device, actuation program for control device, imaging apparatus, and display device

Abstract

To provide a control device capable of also detecting a subject outside of a region defined based on a specific region in which specific processing is performed, an actuation method for a control device, an actuation program for a control device, an imaging apparatus, and a display device.SOLUTION: A subject detection region setting section and a display / storage region image generation section of a CPU in a control section of a monitoring camera set to a picked-up image a subject detection region in which subject detection processing is performed and a display / storage region in which display / storage processing is performed. The subject detection region setting section switches the subject detection region between the display / storage region that is a first region and an entire region that is a second region of which the size and the position are different from the display / storage region.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a control device, an operation method for a control device, an operation program for a control device, an imaging device, and a display device.

[0002] The image processing device described in Patent Document 1 has an image input means for inputting image data, a subject detection means for detecting a specific subject from the image data input by the image input means, an area determination means for determining an area for detection by the subject detection means, and a recording means for recording the image data input by the image input means on a recording medium. The area determination means switches the method for determining the area for subject detection depending on whether or not the recording means is recording moving image data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-136204 Summary of the Invention

[0004] One embodiment of the technology disclosed herein provides a control device, an operating method for the control device, an operating program for the control device, an imaging device, and a display device that are capable of detecting subjects outside an area defined based on a specific area in which specific processing is performed. [Means for solving the problem]

[0005] The control device of the present disclosure is a control device that performs subject detection processing and identification processing on an image captured from an imaging element, and is equipped with a processor. The processor sets a subject detection area in which the subject detection processing is performed and a specific area in which the specific processing is performed on the image captured, and switches the subject detection area between a first area defined based on the specific area and a second area that differs from the first area in at least one of size, shape, and position.

[0006] The second region is preferably larger in size than the first region.

[0007] It is preferable that the first area is the specific area, and the second area is an effective area of ​​the imaging surface of the imaging element.

[0008] It is preferable that the processor repeatedly switches the subject detection area between the first area and the second area.

[0009] The processor: It is preferable that if a state in which a subject is not detected continues for a first threshold time after the subject detection area is set to the first area, the subject detection area is switched to the second area, and if a state in which a subject is not detected continues for a second threshold time after the subject detection area is set to the second area, the subject detection area is switched to the first area.

[0010] Preferably, the second threshold time is different from the first threshold time.

[0011] The second threshold time is preferably longer than the first threshold time.

[0012] It is preferable that the present invention be provided in a surveillance camera system having a rotation mechanism, and that when the rotation mechanism is operated while the subject detection area is the second area, the processor switches the subject detection area from the second area to the first area.

[0013] It is preferable that the processor acquires brightness information indicating brightness of the first area and the second area, and switches the subject detection area to one of the first area and the second area that is brighter.

[0014] When a subject is detected in the subject detection area, the processor preferably switches the subject detection area to a third area defined based on the subject detection result in the previous frame.

[0015] The third area is preferably an area obtained by enlarging the area of ​​the subject detected in the previous frame based on a set enlargement magnification, using the center of gravity of the area of ​​the subject detected in the previous frame as a reference.

[0016] It is preferable that the processor changes the set magnification ratio in accordance with the moving speed of the detected subject within the captured image.

[0017] It is preferable that the processor switches the subject detection area from the third area to the second area when the subject detection area is set to the third area and the state in which the subject is not detected continues for a third threshold time.

[0018] It is preferable that the processor performs, as the identification process, at least one of a process related to displaying an image of the specific area on a display unit and a process related to storing an image of the specific area in a storage unit.

[0019] The operation method of the control device disclosed herein is a method of operating a control device that performs subject detection processing and identification processing on an image obtained from an imaging element, and includes setting a subject detection area in which the subject detection processing is performed and a specific area in which the specific processing is performed on the image, and switching the subject detection area between a first area defined based on the specific area and a second area that differs from the first area in at least one of size, shape, and position.

[0020] The operating program of the control device disclosed herein is an operating program of the control device that performs subject detection processing and identification processing on an image obtained from an imaging element, and causes a computer to perform processing including setting a subject detection area in which the subject detection processing is performed and a specific area in which the specific processing is performed on the image, and switching the subject detection area between a first area defined based on the specific area and a second area that differs from the first area in at least one of size, shape, and position.

[0021] The imaging device of the present disclosure includes the control device described above.

[0022] Preferably it is a surveillance camera.

[0023] The display device of the present disclosure includes the control device described above.

[0024] It is preferable that the device is a management device for a surveillance camera system. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a diagram illustrating a surveillance camera system. [Figure 2] FIG. 2 is a block diagram showing a computer that constitutes a management device. [Figure 3] FIG. 1 is a diagram showing the internal configuration of a surveillance camera. [Figure 4] FIG. 2 is a diagram illustrating an imaging element. [Figure 5] FIG. 2 is a block diagram showing a computer that constitutes a control unit. [Figure 6] FIG. 1 is a block diagram of a CPU. [Figure 7] FIG. 10 is a diagram illustrating a group of threshold times. [Figure 8] FIG. 10 is a diagram showing subject detection area setting information. [Figure 9] 10 is a diagram showing a state in which a subject detection region image is input to a subject detection model and subject detection information is output from the subject detection model. FIG. [Figure 10]FIG. 10 is a diagram illustrating a process of generating a delivery image. [Figure 11] 10A and 10B are diagrams illustrating transitions in the setting of the subject detection area when a subject has not been detected. [Figure 12] 10A and 10B are diagrams illustrating transitions in the setting of the subject detection area when a subject is detected. [Figure 13] FIG. 10 is a diagram showing an expanded detection frame region. [Figure 14] 10A and 10B are diagrams illustrating changes in the setting of the subject detection area when a subject non-detection state continues for a third threshold time. [Figure 15] FIG. 10 is a diagram showing a case where a plurality of people are detected. [Figure 16] FIG. 10 is a diagram showing a case where a person and part of the detection frame are outside the display and storage area. [Figure 17] 10 is a flowchart showing an operation procedure of the surveillance camera. [Figure 18] 10 is a flowchart showing an operation procedure of the surveillance camera. [Figure 19] 10 is a flowchart showing an operation procedure of the surveillance camera. [Figure 20] 10 is a flowchart showing an operation procedure of the surveillance camera. [Figure 21] 10 is a flowchart showing an operation procedure of the surveillance camera. [Figure 22] 10A and 10B are diagrams illustrating transitions in the setting of a subject detection area in the second embodiment. [Figure 23] FIG. 10 is a diagram illustrating a processing unit of a third embodiment. [Figure 24] 13A to 13C are diagrams illustrating transitions in the setting of a subject detection area in the third embodiment. [Figure 25] FIG. 10 is a diagram illustrating a processing unit of a fourth embodiment. [Figure 26] 10A and 10B show how the set magnification ratio is changed depending on the subject's movement speed, where (A) shows the case where the subject's movement speed is less than the speed threshold, and (B) shows the case where the subject's movement speed is greater than or equal to the speed threshold. [Figure 27] FIG. 1 is a diagram illustrating a management device incorporating a control device. DETAILED DESCRIPTION OF THE INVENTION

[0026] [First embodiment] As an example, as shown in FIG. 1, a surveillance camera system 2 includes a surveillance camera unit 10 and a management device 11. The surveillance camera unit 10 is installed at various monitoring locations, such as streets, private homes, apartment complexes, public facilities, company buildings, factories, construction sites, and other construction sites. The surveillance camera unit 10 includes a surveillance camera 12, a case 13, a swivel mechanism 14, and a support pole 15. The surveillance camera 12 captures video of the monitoring location to monitor for suspicious individuals, incidents, accidents, and the like. The surveillance camera 12 is an example of an "imaging device" according to the technology of the present disclosure.

[0027] The surveillance camera 12 is housed in a case 13. The case 13 is, for example, a tightly sealed box-shaped container made of stainless steel, and protects the surveillance camera 12 from direct sunlight, wind, rain, etc. A transparent window 16 is fitted into the front opening of the case 13, and the surveillance camera 12 captures images of the surveillance location through this transparent window 16.

[0028] The swivel / oscillation mechanism 14 is attached to the center of the underside of the case 13. The swivel / oscillation mechanism 14 rotates the case 13, and therefore the surveillance camera 12, endlessly through 360° in the horizontal direction in response to operation by the user U of the surveillance camera system 2 via the management device 11. The swivel / oscillation mechanism 14 also oscillates the case 13, and therefore the surveillance camera 12, through ±90° in the vertical direction in response to operation by the user U via the management device 11. In other words, the shooting area of ​​the surveillance camera 12 can be changed in response to operation by the user U. The swivel / oscillation mechanism 14 is an example of a "swivel mechanism" according to the technology of the present disclosure.

[0029] The support post 15 is attached to the lower part of the swivel / oscillation mechanism 14. The support post 15 is attached to an appropriate position in the monitoring location, such as a telephone pole, gatepost, ceiling, or wall.

[0030] The surveillance camera units 10 and the management device 11 are connected to each other via a network 17 so as to be able to communicate with each other. The network 17 is, for example, a wide area network (WAN) such as the Internet or a public communication network. The management device 11 is installed in a control room of a security company that manages the surveillance camera system 2. The management device 11 is operated by a user U, such as an employee of the security company. The management device 11 is, for example, a desktop personal computer, and has a display 18 and an input device 19 such as a keyboard, a mouse, a touch panel, and / or a microphone for voice input. The display 18 mainly displays images (distributed images 85 (see FIG. 6 )) obtained by capturing images of the monitoring location by the surveillance camera 12 and distributed from the surveillance camera 12 via the network 17. The display 18 is an example of a "display unit" according to the technology of the present disclosure. The input device 19 is operated by the user U to input various operational instructions. The operation instructions include an instruction to start or stop shooting for the surveillance camera 12, an instruction to operate the swivel / oscillation mechanism 14, an instruction to change the position of the zoom lens 39 (see FIG. 3) (hereinafter referred to as a zoom instruction), etc.

[0031] 2, the computer constituting the management device 11 includes, in addition to the above-mentioned display 18 and input device 19, a storage 25, a memory 26, a CPU (Central Processing Unit) 27, and a communication unit 28. The display 18, the input device 19, the storage 25, the memory 26, the CPU 27, and the communication unit 28 are interconnected via a bus line 29.

[0032] The storage 25 is a hard disk drive built into the computer constituting the management device 11 or connected via a cable or network. Alternatively, the storage 25 is a disk array with multiple hard disk drives connected in series. The storage 25 stores control programs such as an operating system, various application programs, and various data associated with these programs. The storage 25 also stores images 85 to be distributed. The storage 25 is an example of a "storage unit" according to the technology of the present disclosure. Note that a solid state drive may be used instead of a hard disk drive.

[0033] The memory 26 is a work memory for the CPU 27 to execute processing. The CPU 27 loads programs stored in the storage 25 into the memory 26 and executes processing in accordance with the programs. In this way, the CPU 27 comprehensively controls each part of the computer. The memory 26 may be built into the CPU 27. The communication unit 28 is a network interface that controls the transmission of various information to and from the surveillance camera unit 10 via the network 17.

[0034] As an example, as shown in FIG. 3, the surveillance camera 12 has an imaging optical system 35 and an imaging element 36 built in. The imaging optical system 35 has multiple types of lenses for forming an image on the imaging element 36. Specifically, the imaging optical system 35 has an objective lens 37, a focus lens 38, and a zoom lens 39. These lenses 37 to 39 are arranged in this order from the object side toward the image side. Although simplified in FIG. 1, each of the lenses 37 to 39 is actually a lens group made up of a combination of multiple lenses.

[0035] The imaging optical system 35 also has a diaphragm 40. The diaphragm 40 is disposed on the imaging side of the zoom lens 39. The diaphragm 40 is a so-called iris diaphragm that is configured by combining a plurality of diaphragm blades.

[0036] A focus lens drive mechanism 41 is connected to focus lens 38, a zoom lens drive mechanism 42 is connected to zoom lens 39, and an aperture drive mechanism 43 is connected to aperture 40. Focus lens drive mechanism 41 holds focus lens 38 and includes a focus cam ring with cam grooves formed on its outer periphery, a focus motor that moves the focus cam ring along the optical axis OA by rotating the focus cam ring about the optical axis OA, and a focus motor driver, etc. Similarly, zoom lens drive mechanism 42 holds zoom lens 39 and includes a zoom cam ring with cam grooves formed on its outer periphery, a zoom motor that moves the zoom cam ring along the optical axis OA by rotating the zoom cam ring about the optical axis OA, and a zoom motor driver, etc.

[0037] The diaphragm 40 uses a cam mechanism to simultaneously move multiple diaphragm blades to open and close a central opening formed by the inner edges of the multiple diaphragm blades, i.e., to change the opening degree of the opening, thereby adjusting the amount of light passing through. The diaphragm drive mechanism 43 includes a diaphragm motor that opens and closes the diaphragm blades, a driver for the diaphragm motor, etc.

[0038] The focus motor, zoom motor, and diaphragm motor are, for example, stepping motors. In this case, the positions of focus lens 38 and zoom lens 39 on the optical axis OA and the opening of diaphragm 40 can be derived from the drive amounts of the focus motor, zoom motor, and diaphragm motor. Note that instead of the drive amounts of the focus motor and zoom motor, position sensors may be provided to detect the positions of focus lens 38 and zoom lens 39.

[0039] The motors (focus motor, zoom motor, and aperture motor) or electrical components such as drivers of each of the drive mechanisms 41 to 43 are connected to the control unit 44. The electrical components of each of the drive mechanisms 41 to 43 are driven under the control of the control unit 44. For example, the control unit 44 issues drive signals in response to various operation instructions input via the input device 19 of the management device 11 to drive the electrical components of each of the drive mechanisms 41 to 43. As an example, when a zoom instruction to change the angle of view to the telephoto side is input, the control unit 44 issues a drive signal to the driver of the zoom motor of the zoom lens drive mechanism 42 to move the zoom lens 39 to the telephoto side.

[0040] The focus motor, zoom motor, and diaphragm motor output their drive amounts to the control unit 44. The control unit 44 derives the positions of the focus lens 38 and zoom lens 39 on the optical axis OA and the opening of the diaphragm 40 from the drive amounts.

[0041] The imaging element 36 is, for example, a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor. As shown in FIG. 4 as an example, the imaging element 36 has a rectangular imaging surface 50 that captures an image. The imaging surface 50 is formed by a plurality of pixels 51 arranged two-dimensionally. The pixels 51 accumulate signal charges corresponding to incident light and output image signals (voltage signals) corresponding to the signal charges. The imaging element 36 is arranged so that the center of the imaging surface 50 coincides with the optical axis OA and the imaging surface 50 is perpendicular to the optical axis OA. Note that the terms "coincidence" and "orthogonality" used here refer to perfect coincidence and orthogonality, as well as coincidence and orthogonality that include errors generally accepted in the technical field to which the technology of the present disclosure pertains.

[0042] Reference numeral 52 denotes a display / storage area. The display / storage area 52 is centered on the entire area of ​​the imaging surface 50 (hereinafter referred to as the entire area) 53, and is an area obtained by reducing the entire area 53 by a preset magnification. Therefore, naturally, the display / storage area 52 is smaller than the entire area 53. The entire area 53 is also different in size and position from the display / storage area 52. The display / storage area 52 is an area of ​​a delivered image 85 that is delivered from the surveillance camera 12 to the management device 11, displayed on the display 18, and stored in the storage 25. The display / storage area 52 is an example of a "specific area" according to the technology of the present disclosure. The entire area 53 is also an example of an "effective area" according to the technology of the present disclosure. While the entire area 53 is given as an example of an effective area, this is not limiting. The effective area may be an area slightly smaller (for example, several pixels) than the entire area 53, specifically, an area that is preferably 80% or more, and more preferably 90% or more of the entire area 53.

[0043] 3, an imaging element driver 45 is connected to the imaging element 36. The imaging element driver 45 is connected to the control unit 44. Under the control of the control unit 44, the imaging element driver 45 controls the timing of capturing an image by the imaging element 36 by, for example, supplying a vertical scanning signal and a horizontal scanning signal to the imaging element 36. The imaging element driver 45 also sets a gain to be applied to the image signal output from the pixel 51, which gain corresponds to ISO (International Organization for Standardization) sensitivity.

[0044] A connector 46 is further connected to the control unit 44. A cable for connecting to the network 17 is attached to the connector 46.

[0045] As an example, as shown in FIG. 5 , the control unit 44 is realized by a computer including a storage 55, a memory 56, and a CPU 57. The storage 55, the memory 56, and the CPU 57 are interconnected via a bus line 58. The control unit 44 is an example of a "computer" and a "control device" according to the technology of the present disclosure. The storage 55, which is a non-transitory storage medium, is a non-volatile storage device such as an EEPROM (Electrically Erasable Programmable Read-Only Memory). The storage 55 stores various parameters and various programs. Note that instead of an EEPROM, a ferroelectric random access memory (FeRAM) or a magnetoresistive random access memory (MRAM) may be used as the storage 55.

[0046] The memory 56 is, for example, a RAM (Random Access Memory) or the like, and temporarily stores various types of information. The CPU 57 loads a program stored in the storage 55 into the memory 56 and executes processing in accordance with the program, thereby comprehensively controlling the operation of each part of the surveillance camera 12. The CPU 57 is an example of a "processor" according to the technology of the present disclosure. The memory 56 may be built into the CPU 57.

[0047] As an example, as shown in Fig. 6, an operating program 65 is stored in the storage 55. The operating program 65 is an application program for causing the computer constituting the control unit 44 to function as a control device. In other words, the operating program 65 is an example of an "operating program for a control device" according to the technology of the present disclosure. In addition to the operating program 65, the storage 55 also stores a threshold time group THG, display / storage area information 66, an object detection model 67, and the like.

[0048] When the operating program 65 is started, the CPU 57, in cooperation with the memory 56, etc., functions as an image processing unit 70, a subject detection area setting unit 71, a subject detection unit 72, a timing unit 73, a display / storage area image generation unit 74, a delivery image generation unit 75, and a delivery control unit 76.

[0049] The image processing unit 70 acquires a captured image 80 output from the imaging element 36. The captured image 80 is composed of image signals from all pixels 51 of the imaging element 36, i.e., image signals from the entire area 53. The image processing unit 70 performs various image processing on the captured image 80. The various image processing includes, for example, offset correction processing, sensitivity correction processing, pixel interpolation processing, white balance correction processing, gamma correction processing, demosaic processing, luminance signal and color difference signal generation processing, edge enhancement processing, color correction processing, etc. The image processing unit 70 outputs the captured image 80 after the various image processing to the subject detection area setting unit 71 and the display / storage area image generation unit 74.

[0050] The subject detection area setting unit 71 sets an area in which a subject is detected (hereinafter referred to as the subject detection area) in the subject detection unit 72. A threshold time group THG is input to the subject detection area setting unit 71. As an example, as shown in FIG. 7, the threshold time group THG includes a first threshold time TH1, a second threshold time TH2, and a third threshold time TH3.

[0051] Subject detection area setting unit 71 generates subject detection area image 90 (see FIG. 8) by extracting the subject detection area portion from captured image 80. Subject detection area setting unit 71 outputs subject detection area setting information 81 including subject detection area image 90 to subject detection unit 72 and timing unit 73. As an example, as shown in FIG. 8, subject detection area setting information 81 includes the subject detection area that has been set in addition to subject detection area image 90.

[0052] The subject detection unit 72 is responsible for subject detection processing. That is, the subject detection unit 72 uses the subject detection model 67 to detect a subject present in the subject detection area set by the subject detection area setting unit 71. More specifically, as shown in FIG. 9 as an example, the subject detection unit 72 inputs a subject detection area image 90 to the subject detection model 67 and causes the subject detection model 67 to output subject detection information 82. The subject detection model 67 is, for example, a machine learning model such as a convolutional neural network. The subject detection information 82 is an example of a "subject detection result" according to the technology of the present disclosure.

[0053] As shown in the figure, if the subject of interest, in this case person P, is captured in the subject detection area image 90, the position coordinates of a rectangular detection frame (called a bounding box) 92 surrounding person P are registered in the subject detection information 82. On the other hand, if person P is not captured in the subject detection area image 90, nothing is registered in the subject detection information 82. The detection frame 92 is an example of a "detected subject area" according to the technology of the present disclosure. The subject detection unit 72 outputs the subject detection information 82 to the subject detection area setting unit 71 and the delivery image generation unit 75. Note that in FIG. 9, person P is depicted by a dashed line in the subject detection information 82 for ease of explanation. The same applies to the subsequent FIG. 10 and the like.

[0054] The timer 73 measures various times to be compared with the threshold time group THG. The timer 73 outputs time measurement information 83 to the subject detection area setting unit 71.

[0055] The display and storage area information 66 is information defining the display and storage area 52, such as the position coordinates of two pixels 51 diagonally located at the upper left and lower right corners of the display and storage area 52. The display and storage area image generation unit 74 performs display and storage processing based on this display and storage area information 66. That is, the display and storage area image generation unit 74 generates a display and storage area image 84 by extracting the display and storage area 52 portion from the captured image 80 based on the display and storage area information 66. The process of generating the display and storage area image 84 in the display and storage area image generation unit 74 is an example of the "specific processing," "display-related processing," and "storage-related processing" according to the technology of the present disclosure. The display and storage area image 84 is also an example of the "image of a specific area" according to the technology of the present disclosure. The display and storage area image generation unit 74 outputs the display and storage area image 84 to the distribution image generation unit 75.

[0056] The delivery image generation unit 75 generates a delivery image 85 based on the subject detection information 82 and the display / storage area image 84. More specifically, as shown in FIG. 10 , the delivery image generation unit 75 generates the delivery image 85 by combining a detection frame 92 registered in the subject detection information 82 with the display / storage area image 84. The delivery image generation unit 75 outputs the delivery image 85 to the delivery control unit 76. The delivery control unit 76 controls the delivery of the delivery image 85 to the management device 11.

[0057] Each of the processing units 70 to 76 performs the above-described process each time a captured image 80 is input from the imaging element 36. The frame rate of the captured image 80 is, for example, 15 to 60 fps (frames per second).

[0058] 11, when user U issues an instruction to start shooting to surveillance camera 12 via input device 19 and surveillance camera 12 starts shooting the monitored location, subject detection area setting unit 71 sets the subject detection area to full area 53. In this case, full area 53 is registered as the subject detection area of ​​subject detection area setting information 81, as exemplified in FIG. 8. Furthermore, captured image 80 is registered as subject detection area image 90 of subject detection area setting information 81. Full area 53 is an example of a "second area" according to the technology of the present disclosure.

[0059] The timing unit 73 measures the time during which the subject detection area setting unit 71 has set the subject detection area to the entire area 53 (hereinafter referred to as the set time of the entire area 53) and outputs the timekeeping information 83 to the subject detection area setting unit 71 at predetermined intervals. The subject detection area setting unit 71 compares the set time of the entire area 53 in the timekeeping information 83 with a second threshold time TH2. If the set time of the entire area 53 continues for the second threshold time TH2, the subject detection area is switched from the entire area 53 to the display and storage area 52. The second threshold time TH2 is, for example, one minute. In this case, the display and storage area 52 is registered as the subject detection area in the subject detection area setting information 81. Furthermore, a display and storage area image 84 is registered as the subject detection area image 90 in the subject detection area setting information 81. The display and storage area 52 is an example of a "first area" according to the technology of the present disclosure.

[0060] The timing unit 73 counts the time during which the subject detection area is set in the display and storage area 52 by the subject detection area setting unit 71 (hereinafter referred to as the set time of the display and storage area 52) and outputs the time count information 83 to the subject detection area setting unit 71 at predetermined intervals. The subject detection area setting unit 71 compares the set time of the display and storage area 52 in the time count information 83 with a first threshold time TH1. If the set time of the display and storage area 52 continues for the first threshold time TH1, the subject detection area is switched again from the display and storage area 52 to the full area 53. The first threshold time TH1 is shorter than the second threshold time TH2, and is, for example, 45 seconds. The subject detection area setting unit 71 repeatedly switches the subject detection area alternately between the display and storage area 52 and the full area 53 as long as the subject detection unit 72 continues to detect no subject (hereinafter referred to as the subject non-detection state).

[0061] As an example, as shown in FIG. 12, when the subject detection area is set to the display / storage area 52 or the entire area 53 and a subject is detected by the subject detection unit 72, the subject detection area setting unit 71 switches the subject detection area from the display / storage area 52 or the entire area 53 to the detection frame enlargement area 95 (see FIG. 13). The maximum size of the detection frame enlargement area 95 is the entire area 53. FIG. 12 illustrates a case where the display / storage area 52 is switched to the detection frame enlargement area 95. The detection frame enlargement area 95 is an example of a "third area" according to the technology of the present disclosure. In the following description, the case where a subject is detected by the subject detection unit 72 will be referred to as a subject detection state.

[0062] As an example, as shown in FIG. 13 , the detection frame expansion region 95, as its name suggests, is an area obtained by expanding the detection frame 92 based on a set expansion magnification, using the center CF of the detection frame 92 of the subject detected in the previous frame as a reference. Here, the detection frame expansion region 95 is an area obtained by expanding the vertical side of the detection frame 92 by 1.25 times and the horizontal side by 2.5 times. In this case, 1.25 times and 2.5 times are examples of the "set expansion magnification" according to the technology of the present disclosure. Furthermore, the center CF of the detection frame 92 is an example of the "center of gravity of the subject area" according to the technology of the present disclosure. Note that, as can be seen from the example of FIG. 13 , the detection frame expansion region 95 may extend beyond the display / storage area 52 depending on the position of the original detection frame 92.

[0063] When person P is moving, detection frame 92 changes in position and / or size in accordance with the movement of person P. Therefore, detection frame expansion region 95, which is set based on detection frame 92, also changes in position and / or size in accordance with the movement of person P.

[0064] As an example, as shown in FIG. 14 , when the subject detection area is set to the detection frame enlarged area 95 and the subject detection unit 72 enters a state in which no subject is detected (hereinafter referred to as the subject non-detection state), the timing unit 73 counts the duration of the subject non-detection state and outputs timing information 83 to the subject detection area setting unit 71 at predetermined intervals. The subject detection area setting unit 71 compares the duration of the subject non-detection state in the timing information 83 with a third threshold time TH3. If the subject non-detection state continues for the third threshold time TH3, the subject detection area is switched from the detection frame enlarged area 95 to the entire surface area 53. The subject non-detection state occurs when the entire body of person P is no longer captured in the captured image 80 (when the entire body of person P moves out of the entire surface area 53). Note that, from the time the subject non-detection state is entered until the third threshold time TH3 has elapsed, the subject detection area setting unit 71 continues to set the subject detection area to the detection frame enlarged area 95 that was set immediately before the subject non-detection state was entered. Third threshold time TH3 is shorter than first threshold time TH1 and second threshold time TH2, and is, for example, 5 seconds. After switching the subject detection area from detection frame enlargement area 95 to full area 53, subject detection area setting unit 71 repeatedly switches the subject detection area between display / storage area 52 and full area 53.

[0065] Here, the subject undetected state can be rephrased as the state from when surveillance camera 12 starts capturing images of the monitored location until a subject is first detected by subject detection unit 72. Alternatively, the subject undetected state can be rephrased as the state from when the subject detection area is set to detection frame enlarged area 95, when the subject undetected state continues for third threshold time TH3 or longer, when the subject detection area is switched from detection frame enlarged area 95 to full area 53, until a subject is detected again by subject detection unit 72.

[0066] 9 illustrates an example in which only one person P appears in the subject detection area image 90, but this is not limiting. As an example, as shown in FIG. 15, if multiple people P, three people P1, P2, and P3 in this case, appear in the subject detection area image 90, the subject detection model 67 detects the people P1 to P3 and outputs subject detection information 82 in which the position coordinates of the detection frames 92 for the people P1 to P3 are registered. In this case, the subject detection area setting unit 71 sets a detection frame expansion region 95 for each of the people P1 to P3. For this reason, the detection frame expansion region 95 may partially overlap with other detection frame expansion regions 95.

[0067] 10 illustrates an example in which person P and detection frame 92 fit within display / storage area 52, but this is not limiting. As an example, as shown in FIG. 16, person P and a part of detection frame 92 may be outside display / storage area 52. Note that, for example, if only the head of person P is captured in display / storage area 52, detection frame 92 may be changed to a smaller one that surrounds only the head of person P rather than the entire body.

[0068] Next, the operation of the above configuration will be described with reference to the flowcharts shown in Figures 17, 18, 19, 20, and 21. When the operating program 65 is started in the surveillance camera 12, the CPU 57 of the control unit 44 of the surveillance camera 12 functions as an image processing unit 70, a subject detection area setting unit 71, a subject detection unit 72, a timing unit 73, a display / storage area image generation unit 74, a distribution image generation unit 75, and a distribution control unit 76, as shown in Figure 6.

[0069] In the management device 11, when a user U issues an instruction to start shooting with the surveillance camera 12 via the input device 19 and the surveillance camera 12 starts shooting the monitored location, a captured image 80 of the monitored location is output from the image sensor 36. The captured image 80 is input to the image processing unit 70, where various image processes are performed (step ST100 in FIG. 17 ). After the various image processes, the captured image 80 is output from the image processing unit 70 to the subject detection area setting unit 71 and the display / storage area image generation unit 74.

[0070] If the state of surveillance camera 12 is a subject undetected state (YES in step ST110), subject detection area setting unit 71 performs subject detection area setting processing for the subject undetected state (step ST120). Details of the subject detection area setting processing for the subject undetected state will be described in Figures 18 and 19. On the other hand, if the state of surveillance camera 12 is not a subject undetected state (NO in step ST110), the processing proceeds to step ST130.

[0071] 12 and 13, if the subject detection unit 72 detected a subject in the previous frame, that is, if the previous frame was in a subject detection state (YES in step ST130), the subject detection region setting unit 71 sets the subject detection region to detection frame enlarged region 95 obtained by enlarging detection frame 92 based on the set enlargement magnification, using the center CF of detection frame 92 of the subject detected in the previous frame as a reference (step ST140). On the other hand, if the subject detection unit 72 did not detect a subject in the previous frame, that is, if the previous frame was in a subject non-detection state (NO in step ST130), the subject detection region setting unit 71 sets the subject detection region to detection frame enlarged region 95 that was set immediately before the subject non-detection state was entered (step ST150).

[0072] When the subject detection area is set in subject detection area setting unit 71, subject detection area setting information 81 shown in Fig. 8 is generated. Subject detection area setting information 81 is output from subject detection area setting unit 71 to subject detection unit 72 and timing unit 73.

[0073] In subject detection unit 72, a subject existing in the subject detection area is detected (step ST160). More specifically, as shown in FIG. 9 , subject detection area image 90 is input to subject detection model 67, and subject detection information 82 is output from subject detection model 67. Subject detection information 82 is output from subject detection unit 72 to subject detection area setting unit 71 and distribution image generation unit 75.

[0074] In the display / storage area image generation unit 74, the portion of the display / storage area 52 is extracted from the captured image 80 based on the display / storage area information 66, and a display / storage area image 84 is generated. The display / storage area image 84 is output from the display / storage area image generation unit 74 to the distribution image generation unit 75.

[0075] 10, in the delivery image generation unit 75, the detection frame 92 registered in the subject detection information 82 is combined with the display / storage area image 84 to generate a delivery image 85. The delivery image 85 is output from the delivery image generation unit 75 to the delivery control unit 76, and is delivered to the management device 11 under the control of the delivery control unit 76 (step ST170).

[0076] Subsequently, subject detection state determination processing is performed in subject detection area setting section 71 (step ST180). Details of the subject detection state determination processing will be described with reference to FIGS.

[0077] The processes of steps ST100 to ST180 are repeated until the user U issues an instruction to end filming by the surveillance camera 12 via the input device 19 in the management device 11 and the surveillance camera 12 has not finished filming the monitored location (NO in step ST190).

[0078] 18 and 19 show the procedure for the subject detection area setting process in the subject undetected state of step ST120. First, if it is the first frame immediately after monitoring camera 12 starts capturing the monitoring location (YES in step ST12001), subject detection area setting unit 71 sets the subject detection area to full area 53 (step ST12002), as shown in Fig. 11. Then, timing unit 73 starts timing the set time for full area 53 (step ST12003).

[0079] If it is two frames or later (NO in step ST12001), the subject detection area setting unit 71 compares the set time of the full area 53 in the timing information 83 with the second threshold time TH2, and determines whether the set time of the full area 53 has continued for the second threshold time TH2 (step ST12004). If the set time of the full area 53 has not continued for the second threshold time TH2 (NO in step ST12004), the subject detection area setting unit 71 sets the subject detection area to the full area 53 (step ST12005). On the other hand, if the set time of the full area 53 has continued for the second threshold time TH2 (YES in step ST12004), the process proceeds to step ST12006.

[0080] In step ST12006, the subject detection area setting unit 71 compares the set time for the display and storage area 52 in the timing information 83 with the first threshold time TH1 to determine whether the set time for the display and storage area 52 has continued for the first threshold time TH1. If the set time for the display and storage area 52 has continued for the first threshold time TH1 (YES in step ST12006), the subject detection area setting unit 71 sets the subject detection area to the full surface area 53 (step ST12007). In other words, the subject detection area is switched from the display and storage area 52 to the full surface area 53. Then, the timing unit 73 resets the set time for the full surface area 53 in the timing information 83 to 0 and starts timing the set time for the full surface area 53 (step ST12008). Furthermore, timing of the set time for the display and storage area 52 is stopped, and the set time for the display and storage area 52 in the timing information 83 is reset to 0 (step ST12009).

[0081] If the set time of display / storage area 52 has not continued for first threshold time TH1 (NO in step ST12006), subject detection area setting unit 71 sets the subject detection area to display / storage area 52 (step ST12010). In other words, the subject detection area is switched from full-screen area 53 to display / storage area 52. Then, timing unit 73 starts timing the set time of display / storage area 52 (step ST12011).

[0082] 20 and 21 show the procedure of the subject detection state determination process in step ST180. First, if a subject is detected in step ST160 (YES in step ST18001), the duration of the subject non-detection state in timing information 83 is reset to 0 in timing section 73 (step ST18002).

[0083] If the subject detected in step ST160 is a subject detected from a subject undetected state (YES in step ST18003), subject detection area setting unit 71 determines that the state of surveillance camera 12 has transitioned from a subject undetected state to a subject detected state (step ST18004). On the other hand, if the subject detected in step ST160 is not a subject detected from a subject undetected state (NO in step ST18003), subject detection area setting unit 71 determines that the state of surveillance camera 12 is a subject detected state (step ST18005). The case where the subject detected in step ST160 is not a subject detected from a subject undetected state occurs when the subject detection area is set in detection frame enlargement area 95 and the subject is captured in detection frame enlargement area 95.

[0084] If a subject is not detected in step ST160 (NO in step ST18001), the process proceeds to step ST18006. In step ST18006, subject detection area setting unit 71 determines whether or not surveillance camera 12 is in a subject undetected state. If surveillance camera 12 is not in a subject undetected state (NO in step ST18006), timing unit 73 times the duration of the subject undetected state (step ST18007). If a subject is not detected in step ST160 and surveillance camera 12 is not in a subject undetected state, this refers to the case where the subject detection area is set to detection frame enlargement area 95 and a subject is not captured in detection frame enlargement area 95.

[0085] In the following step ST18008, the subject detection area setting unit 71 compares the duration of the subject non-detection state in the timing information 83 with a third threshold time TH3 to determine whether the subject non-detection state has continued for the third threshold time TH3. If the subject non-detection state has continued for the third threshold time TH3 (YES in step ST18008), the subject detection area setting unit 71 determines that the state of the surveillance camera 12 has transitioned from the subject detection state to the subject non-detection state (step ST18009). On the other hand, if the subject non-detection state has not continued for the third threshold time TH3 (NO in step ST18008), the subject detection area setting unit 71 determines that the state of the surveillance camera 12 is the subject detection state (step ST18010). The state of the surveillance camera 12 determined in the subject detection state determination process in step ST180 (subject detection state or subject non-detection state) is used in the determination in step ST110 of the next frame.

[0086] As described above, the subject detection area setting unit 71 and display / storage area image generation unit 74 of the CPU 57 of the control unit 44 of the surveillance camera 12 set, for the captured image 80, a subject detection area for performing subject detection processing and a display / storage area 52 for performing display / storage processing. The subject detection area setting unit 71 switches the subject detection area between the display / storage area 52, which is a first area, and the entire area 53, which is a second area different in size and position from the display / storage area 52. This makes it possible to detect subjects outside the display / storage area 52.

[0087] As shown in FIG. 4, the entire area 53 is larger than the display and storage area 52. Therefore, it is possible to detect a subject that exists in the entire area 53, which is larger than the display and storage area 52. Furthermore, even if the subject is the same, the subject will appear relatively larger in the display and storage area 52 than in the entire area 53. Therefore, when the subject detection area is set to the display and storage area 52, the subject can be detected with higher accuracy than when the subject detection area is set to the entire area 53.

[0088] The first area is the display / storage area 52, and the second area is the entire area 53. This makes it possible to meet both the need to detect an object present in the display / storage area 52 with high accuracy and the need to detect an object present in the entire area 53 outside the display / storage area 52 in advance.

[0089] 11, subject detection area setting unit 71 repeatedly switches the subject detection area between display / storage area 52 and full area 53. This increases the opportunities to meet both the need to detect subjects present in display / storage area 52 with high accuracy and the need to detect subjects present in full area 53 outside display / storage area 52 in advance.

[0090] 11, if a state in which a subject is not detected continues for a first threshold time TH1 after setting the subject detection area in display / storage area 52, subject detection area setting unit 71 switches the subject detection area to full area 53. Also, if a state in which a subject is not detected continues for a second threshold time TH2 after setting the subject detection area in full area 53, subject detection area setting unit 71 switches the subject detection area to display / storage area 52. This increases the opportunities to meet both the need to detect a subject present in display / storage area 52 with high accuracy and the need to detect a subject present in full area 53 outside display / storage area 52 in advance.

[0091] As shown in FIG. 11, the second threshold time TH2 is different from the first threshold time TH1. Specifically, the second threshold time TH2 is longer than the first threshold time TH1. This increases the opportunities to respond to a request to detect an object present in the entire area 53 outside the display / storage area 52 in advance. Note that if priority is given to highly accurate detection of an object present in the display / storage area 52 over a request to detect an object present in the entire area 53 outside the display / storage area 52 in advance, the first threshold time TH1 may be set longer than the second threshold time TH2.

[0092] 12 and 13, when a subject is detected by subject detection section 72, subject detection area setting section 71 switches the subject detection area to detection frame expansion area 95 defined based on subject detection information 82 of the previous frame. According to detection frame expansion area 95 defined based on subject detection information 82 of the previous frame, there is a high probability that a subject will also be present in detection frame expansion area 95 in the next frame. Furthermore, even if the subject is the same, the subject will appear relatively larger in detection frame expansion area 95 than in full area 53. This makes it possible to detect the subject with high accuracy.

[0093] 13, detection frame expansion region 95 is a region obtained by expanding detection frame 92 based on a set expansion magnification, using the center CF of detection frame 92 that surrounds the subject detected in the previous frame as a reference. This makes it easy to set detection frame expansion region 95.

[0094] 14, when the subject detection area is set to detection frame enlargement area 95 and a state in which a subject is not detected continues for third threshold time TH3, subject detection area setting unit 71 switches the subject detection area from detection frame enlargement area 95 to full area 53. Therefore, when a subject is no longer detected, it is possible to automatically return to a state in which a subject existing in full area 53 outside display / storage area 52 is detected in advance.

[0095] As the specific processing, the display / storage area image generation unit 74 performs processing to generate a display / storage area image 84. Therefore, a delivery image 85 based on the display / storage area image 84 can be displayed on the display 18 or stored in the storage 25.

[0096] Since the surveillance camera 12 is tasked with monitoring the location, it is highly necessary to detect subjects such as suspicious individuals. Therefore, it is possible to greatly demonstrate the effect of being able to detect subjects outside the display / storage area 52.

[0097] [Second embodiment] 22 as an example, in the second embodiment, if the subject detection area is the entire surface area 53 and the user U operates the swivel / oscillation mechanism 14, the subject detection area setting unit 71 switches the subject detection area from the entire surface area 53 to the display / storage area 52. If the set time for the display / storage area 52 continues for a fourth threshold time TH4, the subject detection area setting unit 71 resets the subject detection area to the entire surface area 53. The fourth threshold time TH4 is stored in the storage 55 and is, for example, one minute.

[0098] The case where the user U operates the swivel / swing mechanism 14 mainly occurs when a subject that the user U is paying attention to is located at the destination where the swivel / swing mechanism 14 is operated. For this reason, when the user U operates the swivel / swing mechanism 14, the subject that the user U is paying attention to, which is located at the destination where the swivel / swing mechanism 14 is operated, can be detected with high accuracy by switching the subject detection area to the display / storage area 52, which has higher subject detection accuracy than the entire area 53.

[0099] When the user U issues an instruction to zoom the surveillance camera 12, the subject detection area may be switched from the entire area 53 to the display / storage area 52.

[0100] [Third embodiment] As an example, as shown in FIG. 23 , in the third embodiment, the CPU 57 functions as a brightness detection unit 100 in addition to the processing units 70 to 76. The brightness detection unit 100 detects the brightness of each of the display / storage area 52 and the full-area area 53 at predetermined intervals. The brightness is, for example, an average value of luminance values ​​calculated from the pixel values ​​of the pixels 51 that make up the display / storage area 52 and the full-area area 53. The brightness detection unit 100 outputs brightness information 101 indicating the detected brightness of the display / storage area 52 and the full-area area 53 to the subject detection area setting unit 71.

[0101] As an example, as shown in FIG. 24, subject detection area setting unit 71 switches the subject detection area to the brighter of display / storage area 52 and full area 53. FIG. 24 illustrates a case where the subject detection area is set to display / storage area 52, and when brightness detection unit 100 detects brightness, full area 53 is brighter than display / storage area 52, so the subject detection area is switched to full area 53. In this way, it is possible to always detect a subject in the brighter area, which has higher subject detection accuracy. For example, if display / storage area 52 is not illuminated by a street light, but full area 53 outside display / storage area 52 is illuminated by a street light, it is possible to detect a subject present in full area 53 illuminated by the street light by switching the subject detection area to full area 53, which is brighter.

[0102] As can be seen from the second and third embodiments, the technique of the present disclosure does not require the repeated switching of the subject detection area between the display and storage area 52 and the full area 53, as in the first embodiment. The subject detection area may be set to the full area 53 by default, and the subject detection area may be switched to the display and storage area 52 only when the rotation and swing mechanism 14 is operated by the user U. Alternatively, the subject detection area may be set to the full area 53 by default, and the subject detection area may be switched to the display and storage area 52 only when the display and storage area 52 is brighter than the full area 53. The second and third embodiments may be combined and implemented. Of course, the first, second, and / or third embodiments may be combined and implemented.

[0103] [Fourth embodiment] As an example, as shown in FIG. 25 , in the fourth embodiment, the CPU 57 functions as a subject speed detection unit 105 in addition to the processing units 70 to 76. The subject speed detection unit 105 detects the moving speed of the subject detected by the subject detection unit 72 within the captured image 80. The subject speed detection unit 105 detects the moving speed of the subject based on, for example, the amount of change in the position of the center CF of the detection frame 92 between two consecutive frames. The subject speed detection unit 105 outputs subject speed information 106 indicating the detected moving speed of the subject to the subject detection area setting unit 71.

[0104] A speed threshold STH is input to the subject detection region setting unit 71. The speed threshold STH is stored in storage 55. The subject detection region setting unit 71 compares the moving speed of the subject indicated by subject speed information 106 with the speed threshold STH. Then, as shown in FIG. 26(A) as an example, if the moving speed of the subject is less than the speed threshold STH, the subject detection region setting unit 71 enlarges the detection frame 92 based on the set enlargement magnifications (1.25x vertical, 2.5x horizontal) shown in FIG. 13 of the first embodiment, and sets a detection frame enlarged region 95. On the other hand, if the moving speed of the subject is equal to or greater than the speed threshold STH, the subject detection region setting unit 71 enlarges the detection frame 92 based on set enlargement magnifications (2x vertical, 4x horizontal) that are larger than the set enlargement magnifications (1.25x vertical, 2.5x horizontal) shown in FIG. 13 of the first embodiment, and sets a detection frame enlarged region 95. That is, the subject detection region setting unit 71 changes the set enlargement magnification of the detection frame enlargement region 95 according to the movement speed of the detected subject within the captured image 80.

[0105] If the movement speed is equal to or greater than the speed threshold STH, there is a risk that the subject in the current frame will fall outside of the detection frame expansion area 95 that was set based on the detection frame 92 of the previous frame. For this reason, if the movement speed is equal to or greater than the speed threshold STH, the subject detection area setting unit 71 sets the detection frame expansion area 95 at a set expansion magnification that is larger than the normal set expansion magnification, thereby reducing the possibility of the subject falling outside of the detection frame expansion area 95 and missing from detection.

[0106] The movement speed of the subject in the captured image 80 includes not only the movement speed when the subject itself is walking or running, but also the movement speed when the subject appears to be moving due to a change in the position of the subject in the captured image 80 when the swivel / tilt mechanism 14 is operated.

[0107] If the state in which the movement speed is equal to or greater than the speed threshold STH continues for a set number of frames, the set enlargement factor may be changed to a larger value.Alternatively, if the state in which the movement speed is less than the speed threshold STH continues for a set number of frames after the set enlargement factor is changed to a larger value, the set enlargement factor may be returned to the original small value.

[0108] The set magnification factor may be changed in three or more stages, for example, by setting the set magnification factor to a value smaller than normal when the subject's moving speed is less than the first speed threshold, setting the set magnification factor to a normal value when the subject's moving speed is equal to or greater than the first speed threshold and less than the second speed threshold, and setting the set magnification factor to a value larger than normal when the subject's moving speed is equal to or greater than the second speed threshold.

[0109] The "display-related processing" may be processing for controlling the display of the display / storage area image 84 on a display unit such as the display 18. Similarly, the "storage-related processing" may be processing for controlling the storage of the display / storage area image 84 in a storage unit such as the storage 25.

[0110] The "specific processing" includes not only the exemplified "processing related to display" and "processing related to storage" but also general processing other than subject detection processing that is performed on the captured image 80. Furthermore, the "specific processing" does not have to be the same processing for each frame.

[0111] Note that when display / storage area 52 is set as the subject detection area and subject detection area image 90 generated by subject detection area setting unit 71 is display / storage area image 84, display / storage area image 84 may be output from subject detection area setting unit 71 to delivery image generation unit 75 and used to generate delivery image 85. In this case, display / storage area image generation unit 74 is not operated.

[0112] The center of gravity of the subject area is not limited to the center CF of the exemplary detection frame 92. It may be the center of gravity of a polygon that surrounds the subject.

[0113] Although the display / storage area 52 has been exemplified as the first area, this is not limiting. Either the display area on the display 18 or the storage area on the storage 25 may be the first area. The display area also includes an area cropped by electronic zoom. The first area may also be an area that is slightly smaller or larger than the display / storage area 52 (for example, by a few pixels). Furthermore, although the detection frame enlargement area 95 has been exemplified as the third area, this is not limiting. The detection frame 92 itself may be the third area.

[0114] Although the example has been given in which the display / storage area 52 as the first area and the entire area 53 as the second area are both rectangular and have the same shape, this is not limitative. The shapes of the first area and the second area may be different, such as the first area being circular or elliptical and the second area being rectangular.

[0115] The display / storage area 52 does not have to be an area of ​​the same size, shape, and position for every frame, and may be, for example, different in size, shape, and position for every set frame.

[0116] The subject is not limited to the illustrated person P. It may be an animal such as a deer, boar, monkey, bear, or bird, or a vehicle such as a car, train, or airplane. If the subject is a car, it is preferable to change the set magnification ratio of the detection frame enlargement area 95 depending on the type of subject, for example, by making the horizontal magnification ratio larger than the vertical magnification ratio.

[0117] Although the subject is detected using the subject detection model 67, the present invention is not limited to this and well-known pattern recognition techniques may also be used to detect the subject.

[0118] In the above embodiments, the surveillance camera 12 is shown as an example of an "imaging device" according to the technology of the present disclosure, but this is not limiting. Instead of the surveillance camera 12, a digital camera, a smartphone, a tablet terminal, or the like used by a general user may be used. Furthermore, an electronic endoscope may also be used.

[0119] In the above-described embodiments, the control unit 44 corresponding to the "control device" according to the technology of the present disclosure is provided in the surveillance camera 12, but this is not limiting. As an example, as shown in FIG. 27 , the "control device" according to the technology of the present disclosure may be provided in the management device 11. Furthermore, the "control device" according to the technology of the present disclosure may be provided in such a manner that its functions are distributed between the management device 11 and the surveillance camera 12.

[0120] In each of the above embodiments, the following various processors may be used as the hardware structure of the processing units that perform various processes, such as the image processing unit 70, subject detection area setting unit 71, subject detection unit 72, timing unit 73, display / storage area image generation unit 74, distribution image generation unit 75, distribution control unit 76, brightness detection unit 100, and subject speed detection unit 105. As described above, the various processors include the CPU 57, which is a general-purpose processor that executes software (operation program 65) to function as various processing units, as well as dedicated electrical circuits that are processors having a circuit configuration specifically designed to perform specific processes, such as a programmable logic device (PLD) that is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit).

[0121] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs and / or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor.

[0122] Examples of configuring multiple processing units with a single processor include, first, a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units, as typified by client and server computers. Second, a form in which a processor is used to realize the functions of an entire system including multiple processing units with a single IC (Integrated Circuit) chip, as typified by System on Chip (SoC). In this way, various processing units are configured using one or more of the above-mentioned various processors as a hardware structure.

[0123] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements.

[0124] From the above description, the technology described in the following supplementary paragraphs can be understood.

[0125] [Additional note 1] A control device that performs subject detection processing and identification processing on an image captured by an image sensor, a processor; The processor: a subject detection area for performing the subject detection processing and a specific area for performing the specific processing are set for the captured image; switching the subject detection area between a first area defined based on the specific area and a second area that is different from the first area in at least one of size, shape, and position; Control device. [Additional note 2] The control device according to supplementary item 1, wherein the second region is larger in size than the first region. [Additional note 3] 3. The control device according to claim 2, wherein the first area is the specific area, and the second area is an effective area of ​​the imaging surface of the imaging element. [Additional note 4] The processor: The control device according to any one of supplementary items 1 to 3, wherein the subject detection area is repeatedly switched between the first area and the second area. [Additional note 5] The processor: when a state in which a subject is not detected continues for a first threshold time after the subject detection area is set to the first area, the subject detection area is switched to the second area; The control device according to claim 4, wherein if a state in which the subject is not detected continues for a second threshold time after the subject detection area is set to the second area, the control device switches the subject detection area to the first area. [Additional note 6] The control device according to supplementary item 5, wherein the second threshold time is different from the first threshold time. [Additional note 7] The control device according to supplementary item 6, wherein the second threshold time is longer than the first threshold time. [Additional note 8] The camera is provided in a surveillance camera system having a rotation mechanism, The processor: A control device according to any one of appendix 1 to appendix 7, wherein when the subject detection area is the second area and the rotation mechanism is operated, the subject detection area is switched from the second area to the first area. [Additional note 9] The processor: acquiring brightness information indicating brightness of the first region and the second region; The control device according to any one of supplementary items 1 to 8, wherein the subject detection area is switched to the area that is brighter out of the first area and the second area. [Additional Note 10] The processor: A control device according to any one of claims 1 to 9, wherein when a subject is detected in the subject detection area, the control device switches the subject detection area to a third area defined based on the detection result of the subject in the previous frame. [Additional Note 11] The control device described in Appendix 10, wherein the third area is an area obtained by enlarging the area of ​​the subject detected in the previous frame based on a set magnification ratio, using the center of gravity of the area of ​​the subject detected in the previous frame as a reference. [Additional Note 12] The processor: The control device according to supplementary item 11, wherein the set magnification ratio is changed in accordance with a moving speed of the detected subject within the captured image. [Additional Note 13] The processor: A control device according to any one of appendix 10 to appendix 12, which switches the subject detection area from the third area to the second area when the subject detection area is set to the third area and the state in which the subject is not detected continues for a third threshold time. [Additional Note 14] The processor: A control device according to any one of claims 1 to 13, wherein the specific processing is at least one of processing related to displaying an image of the specific area on a display unit and processing related to storing an image of the specific area in a memory unit. [Additional Note 15] An imaging device comprising the control device according to any one of supplementary items 1 to 14. [Additional Note 16] Item 15. The imaging device according to item 14, which is a surveillance camera. [Additional Note 17] A display device comprising the control device according to any one of supplementary items 1 to 14. [Additional Note 18] Item 18. The display device according to item 17, which is a management device for a surveillance camera system.

[0126] The technology of the present disclosure can be appropriately combined with the various embodiments and / or various modified examples described above. Furthermore, it is not limited to the above-described embodiments, and various configurations can be adopted without departing from the spirit of the present disclosure. Furthermore, the technology of the present disclosure extends not only to programs but also to storage media that non-temporarily store programs.

[0127] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0128] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0129] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0130] 2. Surveillance camera system 10 surveillance camera units 11 Management device 12. Surveillance Cameras 13 cases 14 Swing and oscillating mechanism 15 Posts 16 Transparent window 17 Network 18 Display 19 Input Devices 25, 55 storage 26, 56 memory 27, 57 CPU 28 Communications Department Bus lines 29 and 58 35 Imaging optical system 36 image sensor 37 Objective Lens 38 Focus Lens 39 Zoom Lens 40 apertures 41 Focus lens drive mechanism 42 Zoom lens drive mechanism 43 Aperture drive mechanism 44 Control Unit 45 Image sensor driver 46 Connectors 50 imaging surface 51 pixels 52 Display / storage area 53 Full area 65 Operating Program 66 Display / storage area information 67 Object Detection Model 70 Image processing section 71 Subject detection area setting section 72 Subject detection section 73 Timing section 74 Display / storage area image generation section 75 Distribution image generation unit 76 Distribution control section 80 captured images 81 Object detection area setting information 82 Subject detection information 83 Timing information 84 Display and storage area images 85 Delivery images 90 Object detection area image 92 detection frame 95 Detection frame expansion area 100 Brightness detection unit 101 Brightness Information 105 Subject speed detection unit 106 Subject speed information CF detection frame center OA optical axis P, P1~P3 people ST100, ST110, ST120, ST130, ST140, ST150, ST160, ST170, ST180, ST190, ST12001, ST12002, ST12003, ST12004, ST12005, ST12006, ST12007, ST12008, ST12009, ST12010, ST12011, ST18001, ST18002, ST18003, ST18004, ST18005, ST18006, ST18007, ST18008, ST18009, ST18010 Step STH Speed ​​Threshold TH1 First threshold time TH2 Second threshold time TH3 Third threshold time TH4 Fourth Threshold Time THG threshold time group U User

Claims

1. A control device that performs subject detection processing and identification processing on an image captured by an image sensor, a processor; The processor: a subject detection area for performing the subject detection processing and a specific area for performing the specific processing are set for the captured image; switching the subject detection area between a first area defined based on the specific area and a second area that is different from the first area in at least one of size, shape, and position; Control device.

2. The control device according to claim 1 , wherein the second region is larger in size than the first region.

3. The control device according to claim 2 , wherein the first area is the specific area, and the second area is an effective area of ​​the imaging surface of the imaging element.

4. The processor: The control device according to claim 1 , wherein the object detection area is repeatedly switched between the first area and the second area.

5. The processor: when a state in which a subject is not detected continues for a first threshold time after the subject detection area is set to the first area, the subject detection area is switched to the second area; The control device according to claim 4 , wherein when a state in which the subject is not detected continues for a second threshold time after the subject detection area is set to the second area, the subject detection area is switched to the first area.

6. The control device of claim 5 , wherein the second threshold time is different from the first threshold time.

7. The control device according to claim 6 , wherein the second threshold time is longer than the first threshold time.

8. The camera is provided in a surveillance camera system having a rotation mechanism, The processor: The control device according to claim 1 , wherein when the subject detection area is the second area and the rotation mechanism is operated, the subject detection area is switched from the second area to the first area.

9. The processor: acquiring brightness information indicating brightness of the first region and the second region; The control device according to claim 1 , wherein the object detection area is switched to the area having the higher brightness out of the first area and the second area.

10. The processor: The control device according to claim 1 , wherein when a subject is detected in the subject detection area, the subject detection area is switched to a third area defined based on a result of detection of the subject in a previous frame.

11. The control device according to claim 10 , wherein the third area is an area obtained by enlarging the area of ​​the subject detected in the previous frame based on a set enlargement magnification, using the center of gravity of the area of ​​the subject detected in the previous frame as a reference.

12. The processor: The control device according to claim 11 , wherein the set magnification is changed in accordance with a moving speed of the detected subject within the captured image.

13. The processor:

11. The control device according to claim 10, wherein when the subject detection area is set to the third area and a state in which the subject is not detected continues for a third threshold time, the control device switches the subject detection area from the third area to the second area.

14. The processor: The control device according to claim 1 , wherein the specific processing is at least one of processing related to displaying the image of the specific area on a display unit and processing related to storing the image of the specific area in a storage unit.

15. A method for operating a control device that performs subject detection processing and identification processing on an image captured by an image sensor, comprising: setting a subject detection area in which the subject detection processing is performed and a specific area in which the specific processing is performed for the captured image; and switching the subject detection area between a first area defined based on the specific area and a second area that is different from the first area in at least one of size, shape, and position; A method of operating a control device comprising:

16. An operating program for a control device that performs subject detection processing and identification processing on an image captured by an image sensor, setting a subject detection area in which the subject detection processing is performed and a specific area in which the specific processing is performed for the captured image; and switching the subject detection area between a first area defined based on the specific area and a second area that is different from the first area in at least one of size, shape, and position; An operating program for a control device that causes a computer to perform processing including the above.

17. An imaging device comprising the control device according to claim 1.

18. 18. The imaging device according to claim 17, which is a surveillance camera.

19. A display device comprising the control device according to claim 1.

20. 20. The display device according to claim 19, which is a management device for a surveillance camera system.

Images