Imaging apparatus and control method of the same

Abstract

To determine whether or not it is necessary to change a zoom angle of view of an object included in an image input to an image recognition apparatus to improve the recognition accuracy.SOLUTION: An imaging apparatus includes: an imaging part capable of photographing an image by switching between a normal angle of view and a zoom angle of view by optical zooming; a recognition result acquisition part configured to acquire a recognition result obtained by performing recognition processing of an object by an image recognition device on an input image; a determination part configured to determine whether or not to change the object to the zoom angle of view based on the recognition result of the normal angle of view image photographed at the normal angle of view; a control instruction part configured to control switching to the zoom angle of view zoomed on a zoom target when the zoom target is determined to need to be changed to the zoom angle of view; and an output processing part configured to output, to the image recognition device, the zoom angle of view image photographed by the imaging part that has been switched to the zoom angle of view.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an imaging device and a control method thereof. [Background technology]

[0002] Patent Document 1 discloses a technology for sequentially performing facial recognition of multiple targets by using two cameras: a first camera that captures an overall image including multiple targets, and a second camera that zooms in on one target selected as the target from the overall image captured by the first camera and performs facial recognition. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-249298 Summary of the Invention [Problem to be solved by the invention]

[0004] Some image recognition devices have the ability to capture an unspecified number of objects and recognize multiple objects within the entire image. In this case, it is necessary to select and zoom in on objects that are not highly recognizable, thereby improving authentication accuracy.

[0005] In Patent Document 1, two cameras are used, and multiple authentication targets included in an overall image captured by one camera are sequentially zoomed in with the other camera, and facial authentication is performed for each of the authentication targets, but no consideration is given to which of the multiple targets is selected as the zoom target.

[0006] In view of the above-mentioned problems, an object of the present disclosure is to provide an imaging device and a control method that can improve recognition accuracy by determining whether or not it is necessary to change the zoom angle of view of multiple objects included in an image input to an image recognition device. [Means for solving the problem]

[0007] The imaging device according to the present disclosure includes an imaging unit capable of capturing images by switching between a normal angle of view and a zoom angle of view using optical zoom; a recognition result acquisition unit that acquires a recognition result obtained by performing object recognition processing on an input image using an image recognition device; a determination unit that determines whether or not the object needs to be changed to a zoom angle of view based on the recognition result for the normal angle of view image captured at the normal angle of view; a control instruction unit that, if there is a zoom object that has been determined to require a change to a zoom angle of view, controls switching to a zoom angle of view zoomed on the zoom object; and an output processing unit that outputs the zoom angle of view image captured by the imaging unit that has switched to the zoom angle of view to the image recognition device.

[0008] The control method according to the present disclosure includes a process in which a computer captures an image by switching between a normal angle of view and a zoom angle of view using optical zoom, a process in which an image recognition device performs an object recognition process on an input image and acquires a recognition result, a process in which, based on the recognition result for the normal angle of view image captured at the normal angle of view, it determines whether or not it is necessary to change to a zoom angle of view for the object, and, if it is determined that a change to a zoom angle of view is necessary for an object, it controls switching to a zoom angle of view zoomed in on the object, and a process in which the computer outputs the zoom angle of view image captured after switching to the zoom angle of view to the image recognition device. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to improve recognition accuracy by determining whether or not it is necessary to change the zoom angle of view of multiple objects included in an image input to an image recognition device. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram illustrating an example of the overall configuration of an image recognition system including an imaging device according to an embodiment. [Figure 2] 1 is a block diagram showing the configuration of an imaging device according to a first embodiment. [Figure 3]FIG. 3 is a block diagram showing the configuration of a processing unit in FIG. 2. [Figure 4] FIG. 3 is a flowchart illustrating the processing flow of the imaging apparatus according to the first embodiment. [Figure 5] FIG. 2 is a diagram showing a signal flow in the imaging device of the first embodiment. [Figure 6] FIG. 2 is a diagram showing a signal flow in the imaging device of the first embodiment. [Figure 7] FIG. 2 is a diagram showing a signal flow in the imaging device of the first embodiment. [Figure 8] 4 is a diagram illustrating the timing of reading data in the imaging device of the first embodiment. FIG. [Figure 9] 1A and 1B are diagrams illustrating an example of a normal angle of view image and a zoom angle of view image. [Figure 10] FIG. 10 is a block diagram showing the configuration of an imaging device according to a second embodiment. [Figure 11] FIG. 10 is a diagram showing the flow of signals in the imaging device of the second embodiment. [Figure 12] FIG. 10 is a diagram showing the flow of signals in the imaging device of the second embodiment. [Figure 13] FIG. 10 is a diagram illustrating the timing of reading data in the imaging device of the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, specific embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same elements are denoted by the same reference numerals, and for clarity of explanation, duplicated explanations will be omitted as necessary.

[0012] The embodiments relate to, as an example, an imaging device that captures recognition images to be input to an image recognition device that performs object recognition processing in an image recognition system. In an image recognition system, the image quality of an image input to the image recognition device that performs recognition processing has a significant impact on identification judgment. One example of image quality is image resolution. When an object in an image is small and has low resolution, enlarging the object using electronic zoom increases the size, but the resolution is simply stretched or interpolated, resulting in degradation of image quality. Objects in images with degraded image quality have a lower recognition rate in the image recognition device compared to objects of the same size in images without degradation.

[0013] In order to improve the recognition rate of objects with low recognition rates, the imaging device of the embodiment determines whether or not it is necessary to change multiple object zoom angles of view contained in images input to the image recognition device to zoom angles of view using optical zoom.

[0014] Furthermore, the imaging device according to the embodiment employs a method for improving the recognition rate of an image recognition device by adjusting the image quality of an input image to be input to the image recognition device so as to improve the recognition rate. This method analyzes the recognition results of the image recognition device, sets adjustment parameters for adjusting the image quality to be advantageous for the recognition process, and performs image quality adjustment using the set adjustment parameters for the input image.

[0015] An image with a high recognition rate in an image recognition device is not necessarily the same as an image that a user wants to visually confirm. When an image with a high recognition rate in an image recognition device is displayed on a display device, the user may not be able to visually confirm the optimal image. An imaging device according to an embodiment supplies an image with a high recognition rate in the image recognition device to the image recognition device and provides the user with a visually optimal image.

[0016] Embodiment 1. Fig. 1 is a block diagram showing the overall configuration of an image recognition system 100 including an imaging device 10 according to an embodiment. As shown in Fig. 1, the image recognition system 100 includes the imaging device 10, an image recognition device 20, and a display device 30. The image recognition system 100 has a function of displaying an image captured by the imaging device 10 for visual confirmation by a user. The functions provided by the image recognition system 100 also include a function of recognizing objects included in the captured image and displaying the objects so that they can be visually recognized by the user.

[0017] Note that if the image recognition system 100 does not need a function to display images captured by the imaging device 10 for user viewing, it does not need to include the display device 30. The image recognition system 100 may transmit images captured by the imaging device 10 to an external device other than the display device 30. The external device may be, for example, a computer, a server device, or a smartphone.

[0018] <Imaging device 10> The imaging device 10 generates display image data and recognition image data from an imaging signal obtained by capturing an image of a landscape including multiple objects at a predetermined angle of view. Fig. 2 is a block diagram showing the configuration of the imaging device 10 according to the first embodiment. As shown in Fig. 2, the imaging device 10 includes a camera unit 11, a signal processing unit 12, an output unit 13, a processing unit 14, a control unit 15, a first memory 16, a second memory 17, and a storage unit 18.

[0019] The camera unit 11 includes an imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The camera unit 11 also includes a lens group (imaging optical system) including a zoom lens and a focus lens, an iris diaphragm, a mechanical shutter, and the like. The camera unit 11 adjusts the level of a signal based on an image formed by the imaging optical system according to a predetermined amplification gain, performs A / D conversion, and sequentially outputs the signal to the signal processing unit 12 as an imaging signal. In other words, the imaging signal can be RAW data. RAW data is data obtained from an imaging element having a pixel arrangement pattern such as a Bayer array, and contains information on only one color component per pixel. The camera unit 11 is sometimes referred to as an imaging unit.

[0020] The camera unit 11 adjusts the brightness of the captured image by adjusting the amount of light incident on the imaging element using a mechanical shutter or iris diaphragm of the imaging device 10. Note that the imaging device 10 can also adjust the brightness of the captured image by adjusting the opening size of diaphragm blades (not shown).

[0021] The camera unit 11 has a PTZ (Pan-Tilt-Zoom) function. The PTZ function is a function that performs horizontal panning, vertical tilting, and zooming in (telephoto) and out (wide-angle) operations. The camera unit 11 has a drive unit (not shown) that performs PTZ operations based on control signals from the control unit 15.

[0022] The camera unit 11 is capable of capturing images by switching between a normal angle of view and a zoom angle of view using an optical zoom that controls a zoom lens and a focus lens (not shown). Here, an image obtained by the camera unit 11 not performing PTZ operation and capturing images using default settings is referred to as a "normal angle of view image." Also, an image captured by the camera unit 11 performing PTZ operation and zooming in on a zoom target is referred to as a "zoom angle of view image." In other words, a normal angle of view image is a wider-angle image than a zoom angle of view image.

[0023] Here, the normal angle of view image is an image that allows multiple objects to be viewed from a bird's eye view and is an image that the user wants to visually confirm, whereas the zoom angle of view image is an image in which the zoom object is enlarged by optical zoom and is an image that increases the recognition rate of the object in the image recognition device 20.

[0024] The signal processing unit 12 acquires the image signals captured continuously by the camera unit 11 and performs various image processing on the image data in units of frames (hereinafter referred to as image frames). The processing performed by the signal processing unit 12 includes, for example, de-Bayering, linear matrix processing, white balance adjustment, gamma correction, and aperture correction. De-Bayering is a color interpolation process that converts an image signal, in which each pixel represents one color component, into full-color image data, in which each pixel represents each color component. Linear matrix processing is a process that generates an image signal with a target color tone to improve color reproducibility. White balance processing is a process that corrects imbalances between colors by adjusting RGB gain. Note that the image processing in the signal processing unit 12 is not limited to these examples and may include other processes such as noise removal, defective pixel correction, and edge enhancement.

[0025] The signal processing unit 12 can generate image frames conforming to a predetermined display signal format. For example, when generating image data in full high-definition format, the signal processing unit 12 may perform the above-described image processing on each image frame and then perform enlargement / reduction processing to an image size of 1920 x 1080 pixels. The signal processing unit 12 passes the image data that has undergone each processing to the output unit 13.

[0026] The output unit 13 can acquire the recognition results from the image recognition device 20 (described later) via the processing unit 14 through a data flow (not shown). The output unit 13 can process the image data using the recognition results to generate display image data. The output unit 13 outputs the generated display image data to the display device 30. The display device 30 is a liquid crystal display device or the like with an image display function. The display device 30 can display a captured image with high visibility and a display image including the recognition result based on the display image data. Note that if the image recognition system 100 does not include the display device 30, the imaging device 10 does not need to include the output unit 13.

[0027] The output unit 13 may use the recognition result to generate information to be displayed superimposed on the captured image. The display device 30 can, for example, superimpose on the captured image a recognition frame surrounding the area of the recognized object, character information corresponding to the type of the recognized object, recognition determination information such as a recognition rate, and the like. The output unit 13 may also create auxiliary display information such as a menu setting display that is used when the user performs various inputs using an input device (not shown). The output unit 13 can superimpose a menu setting display on the captured image.

[0028] The image data processed by the signal processing unit 12 is used for display on the display device 30 and is also supplied to the processing unit 14. The image data processed by the signal processing unit 12 is input to the processing unit 14. The processing unit 14 generates recognition image data suitable for object recognition processing from the input image data. The recognition image data is supplied to the image recognition device 20 and used to recognize the object. Note that the photographic signal generated by the camera unit 11 may be supplied directly to the processing unit 14 without going through the signal processing unit 12. Also, a signal in the middle of being processed by the signal processing unit 12 may be supplied to the processing unit 14.

[0029] The processing unit 14 can acquire the recognition result from the image recognition device 20, which will be described later. Based on the recognition result of the normal angle of view image, the processing unit 14 can determine whether or not a change to a zoom angle of view is necessary for each object included in the normal angle of view image, and determine a zoom object that requires a change to a zoom angle of view. The processing unit 14 will be described in detail later. The control unit 15 controls the PTZ operation of the camera unit 11 to capture a zoom angle of view image of the zoom object determined by the processing unit 14.

[0030] Specifically, the imaging device 10 can use optical zoom to switch to a zoom angle of view that is narrower than the normal angle of view, thereby enlarging an object with a low authentication rate to a predetermined magnification. Note that the camera unit 11 may perform panning and tilting operations so that the center position of the zoom object coincides with the center position of the zoom angle of view image. However, the camera unit 11 only needs to perform at least a zooming operation to acquire a zoom angle of view image of the zoom object, and does not need to perform panning and tilting operations.

[0031] Furthermore, in addition to the process of switching to the zoom angle of view of the zoom object described above, the processing unit 14 may also perform image quality adjustment processing on the recognition image data based on image quality adjustment parameters. For example, an image recognition support device (not shown) can generate image quality adjustment parameters that affect the recognition rate of the object and are used when the processing unit 14 performs the image quality adjustment processing. Such image quality adjustment parameters include a luminance gain for controlling the luminance (brightness) of the captured image, tone mapping characteristics, an aperture gain for edge enhancement, and the like.

[0032] The image quality adjustment support device may generate image quality adjustment parameters to be used when adjusting the image quality of the recognition image to improve the recognition accuracy of the image for recognition from the next time onward, depending on the recognition result. Note that the generation of these image quality adjustment parameters may be performed multiple times. For example, image quality adjustment may be repeatedly performed until the "recognition frequency," which is the number of successful image recognitions out of the total number of image recognitions within a certain period of time, reaches or exceeds a predetermined value.

[0033] The first memory 16 stores the photographic signal (RAW data). The photographic signal output from the camera unit 11 is saved in the first memory 16 in accordance with instructions from the processing unit 14, which will be described later. The second memory 17 stores image data that has undergone signal processing in the signal processing unit 12. The image data output from the signal processing unit 12 is saved in the second memory 17 in accordance with instructions from the processing unit 14, which will be described later. The first memory 16 and the second memory 17 may be volatile storage devices such as RAM (Random Access Memory) that can hold data for several frames, for example.

[0034] When the object to be zoomed is present in the normal angle of view image, the processing unit 14 stores the normal angle of view image in the first memory 16. Then, while the camera unit 11 is switching from the normal angle of view to the zoom angle of view, the processing unit 14 can output the normal angle of view image stored in the memory to the outside.

[0035] <Image recognition device 20> The image recognition device 20 performs image recognition processing on the recognition images supplied from the processing unit 14, and feeds back the recognition results to the imaging device 10. The recognition images are continuously input to the image recognition device 20, and the recognition processing is continuously executed at any time.

[0036] The recognition result includes the presence or absence of an object, the type of object, the area or position of the object, and the recognition rate. The presence or absence of an object is information indicating whether or not an object has been recognized, i.e., identified, by image recognition processing on the recognition image. The type of object is information indicating the type of object to be recognized. The area of the object is a group of coordinates that defines the range of the area including the recognized object in the recognition image. The area of the object is, for example, a range specified by pixel values in an XY coordinate system. The position of the object is, for example, a representative point such as the center coordinates of the object recognized in the recognition image.

[0037] The recognition rate is an example of the degree of accuracy of recognition by image recognition. In other words, the recognition rate is numerical information indicating the presence or absence, type, and recognition accuracy of the area of an object recognized by image recognition processing in an image for recognition. The recognition rate may be expressed, for example, as a range from 0 to 100%. Furthermore, the recognition rate may be calculated using, for example, a threshold indicating the similarity with the object, the number of stages passed by a classifier, etc. When multiple recognition objects are recognized, the recognition result may generate a set of type, area, and recognition rate for each object.

[0038] The image recognition device 20 is hardware or software, or a combination thereof, capable of executing known image recognition processing. For example, the image recognition device 20 is realized by executing a known image recognition processing program on a computer. The image recognition device 20 may be redundantly configured on multiple computers, and each functional block may be realized by multiple computers. The image recognition device 20 may also be realized as a client-server system, a cloud computing system, or the like, in which each system is connected via a communication network. The functions of the image recognition device 20 may also be provided in a Software as a Service (SaaS) format. Alternatively, the image recognition device 20 may be incorporated into part of the imaging device 10 and realized by the same computer.

[0039] The image recognition process by the image recognition device 20 may involve storing multiple images for each object and recognizing the object using pattern matching. In this case, deep learning may be performed using multiple images captured from various angles to create a model for object recognition. It is generally known that the recognition rate of such image recognition processes varies depending on image characteristics, such as the brightness of the recognition image and the ease of distinguishing the object from the background in the recognition image.

[0040] The image recognition device 20 may also have a function of tracking each recognized object using known techniques such as motion compensation between image frames. The image recognition device 20 can link identification information, such as an identification number for identifying the same object, with location information and provide the linked information to the imaging device 10. The above-mentioned recognition result may also include the identification information.

[0041] The processing unit 14 of the imaging device 10 will now be described with reference to Fig. 3. Fig. 3 is a block diagram showing detailed configurations of the processing unit 14 and the storage unit 18 of Fig. 2. As shown in Fig. 3, the processing unit 14 includes a recognition result acquisition unit 141, a determination unit 142, a storage processing unit 143, a control instruction unit 144, an output processing unit 145, and a parameter setting unit 146.

[0042] The processing unit 14 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), or a quantum processor (quantum computer control chip). The processing unit 14 loads an imaging control program stored in a storage unit 18 provided in the imaging device 10 into a memory and executes the program. As a result, the processing unit 14 realizes the functions of a recognition result acquisition unit 141, a determination unit 142, a control instruction unit 144, and an output processing unit 145, and performs zooming on the zoom target object described above. The processing unit 14 also realizes the function of a parameter setting unit 146, thereby adjusting the image quality of the recognition image input to the image recognition device 20. Some or all of the components of the processing unit 14 may be realized, for example, by a general-purpose or dedicated circuit realized in a semiconductor device.

[0043] The storage unit 18 may include a non-volatile storage device such as a hard disk or flash memory, and a memory such as RAM, i.e., a volatile storage device. The storage unit 18 stores, for example, an imaging control program and recognition results. The imaging control program is a computer program that implements the processing of the control method for the imaging device 10 according to the embodiment. The recognition results include the recognition rate and position information for each object, identification information for identifying the same object, etc.

[0044] The recognition result acquisition unit 141 acquires the recognition result of object recognition processing performed on the captured image by the image recognition device 20. The determination unit 142 determines whether or not the object needs to be changed to a zoom angle of view based on the recognition result for the normal angle of view image. The determination unit 142 may determine whether or not the object needs to be changed to a zoom angle of view based on the recognition result for the normal angle of view image.

[0045] Specifically, the determination unit 142 can determine, for example, an object whose recognition rate is lower than a predetermined threshold as a zoom object, i.e., as requiring a change to the zoom angle of view, and can determine, for an object whose recognition rate is equal to or higher than the predetermined threshold, as not requiring a change to the zoom angle of view. Here, the recognition result may include information on whether the object is a person. For example, if the recognition rate, which ranges from 0 to 100%, is equal to or higher than an arbitrary human determination threshold, the object is determined to be a person, and if the recognition rate is lower than the human determination threshold, the predetermined threshold for determining whether the object is a zoom object can be set to the same value as or a value close to the human determination threshold.

[0046] Alternatively, the determination unit 142 can determine that an object whose size is smaller than a predetermined value in the normal angle of view image is a zoom object. Note that the size of the object may be the pixel size of the area where the object exists, or the size of a recognition frame that surrounds the area of the object and is displayed superimposed on the captured image using the recognition result.

[0047] The determination unit 142 may determine that a change to a zoom angle of view is necessary when at least one of the following cases is true: the size of the object in the normal angle of view image is smaller than a predetermined value; and the recognition rate of the object in the normal angle of view image is smaller than a threshold value.

[0048] Furthermore, the determination unit 142 may determine that a change to a zoom angle of view is unnecessary if the recognition rate of the object in the normal angle of view image is significantly smaller than a predetermined threshold. That is, the determination unit 142 determines that a change to a zoom angle of view is necessary if the recognition rate of the object in the normal angle of view image is smaller than the predetermined threshold and larger than a second threshold that is smaller than the predetermined threshold. For example, the predetermined threshold may be the same value as or a value close to the aforementioned human determination threshold, and the second threshold may be a value obtained by multiplying the predetermined threshold by 0.8, but is not limited to this. This is because if the recognition rate is significantly low, it is considered unlikely that the object is an object that should be recognized (e.g., a person) in the first place, and therefore it is considered that optical zoom is also unlikely to be necessary.

[0049] The determination unit 142 may also monitor changes over time in the recognition rate of the same object in normal angle-of-view images taken at multiple points in time, and determine whether or not a change to a zoom angle of view is necessary based on the change over time in the recognition rate. Here, the determination unit 142 determines that the same object can be associated with the same identifier in recognition results taken at multiple points in time, based on information about the object's area included in the recognition results. For example, the determination unit 142 determines that a change to a zoom angle of view is necessary if the recognition rate of a given object has been equal to or greater than the human determination threshold for a certain period of time, i.e., the object has been continuously determined to be a person, but at some point the recognition rate falls below the human determination threshold. This is because it is considered that the recognition rate of an object that has already been continuously determined to be a person has temporarily dropped due to some factor, and optical zoom is likely to be necessary.

[0050] When the determination unit 142 determines that a zoom object exists in the normal angle of view image, the save processing unit 143 saves image data obtained by image processing the normal angle of view image in the second memory 17. When a zoom object exists that has been determined to require a change to a zoom angle of view, the control instruction unit 144 outputs an instruction signal to the control unit 15 to instruct switching to a zoom angle of view zoomed into the zoom object.

[0051] Based on an instruction signal from the control instruction unit 144, the control unit 15 controls the PTZ operation of the camera unit 11 to capture an image of the zoom object at the zoom angle of view. Specifically, the camera unit 11 calculates the amount of movement based on the center point of the normal angle of view image and a representative point such as the center coordinates of the zoom object, and can perform panning and tilting. Furthermore, the camera unit 11 can perform zooming so that the size of the zoom object becomes a predetermined size, for example.

[0052] In the image recognition system 100, the image quality of the recognition image input to the image recognition device 20 that performs the recognition process has a significant effect on the identification and judgment of the object. As described above, by using optical zoom to capture a zoomed angle of view image of the zoom object, it is possible to increase the recognition rate of the zoom object.

[0053] The output processing unit 145 outputs the zoom angle of view image to the image recognition device 20. Furthermore, the output processing unit 145 outputs the normal angle of view image to the external display device 30 during the period from when the control unit 15 starts switching the camera unit 11 from the normal angle of view to the zoom angle of view until the camera unit 11 returns to the normal angle of view. Specifically, the output processing unit 145 outputs the image data stored in the second memory 17 to the external display device 30 during the period from when the control unit 15 starts switching the camera unit 11 from the normal angle of view to the zoom angle of view until the camera unit 11 returns to the normal angle of view. In this way, while the camera unit 11 is capturing an image of a zoom target at the zoom angle of view, the normal angle of view image can be displayed on the display device 30, allowing the user to visually confirm the target. Furthermore, it is also possible to add a display such as a recognition frame for the zoom target to the normal angle of view image by utilizing the recognition result of the zoom target recognized in the zoom angle of view image.

[0054] Here, a control method for the imaging device 10 will be described with reference to Fig. 4. Fig. 4 is a flow diagram illustrating the flow of processing in the imaging device 10. The control method for the imaging device 10 mainly includes (1) recognition processing of a normal angle of view image, (2) processing of switching to a zoom angle of view based on the recognition result of the normal angle of view image, (3) processing of outputting image data for display and image data for recognition, and (4) image quality adjustment processing based on the recognition result of the zoom angle of view image.

[0055] (1) Recognition process for normal angle of view images 4, the imaging device 10 captures a normal angle of view image at a normal angle of view (step S10) and outputs the image to the image recognition device 20. When capturing a normal angle of view image, the imaging device 10 does not perform a PTZ operation and is in an initial setting state. At this time, the image signal output from the camera unit 11 is not stored in the first memory 16, but is processed by the signal processing unit 12 and delivered to the output unit 13. In addition, the image data processed by the signal processing unit 12 is input to the image recognition device 20 via the processing unit 14.

[0056] The processing unit 14 may perform image quality adjustment using image quality adjustment parameters so as to increase the recognition rate of the object in the recognition image. The image quality adjustment parameters may be determined by performing feedback control of the image quality adjustment parameters in accordance with the recognition results for the recognition image. The processing unit 14 may repeat the feedback control until the recognition rate of the object stabilizes at a high level. The flow of image data for recognition during feedback control is indicated by the dashed line in FIG. 5. In the example of FIG. 5, the image quality adjustment parameters determined by feedback control are reflected only in the processing unit 14, but may also be reflected in the signal processing unit 12.

[0057] At this time, based on the recognition results of image recognition performed on the recognition image after image quality adjustment, the image quality type to be adjusted is selected, and the adjustment value for each image quality type is fine-tuned, thereby optimizing the image quality adjustment parameters. Image quality types include, but are not limited to, brightness, S / N ratio, and resolution. In other words, optimizing the image quality adjustment parameters means adjusting the recognition image to an image quality that is advantageous for image recognition processing.

[0058] Then, the image recognition device 20 performs a recognition process for each object included in the normal angle of view image. The image capture device 10 acquires the recognition result transmitted from the image recognition device 20 (step S11). Thereafter, based on the recognition result of the object, it determines whether or not a change to a zoom angle of view is necessary (step S12). If the determination of whether or not a change to a zoom angle of view of the object is necessary is "no" (step S12, NO), the process returns to step S11. If there are multiple objects in the normal angle of view image, the process of step S12 may be performed for all of the objects. If the determination of whether or not a change to a zoom angle of view of the object is "necessary" (step S12, YES), the image capture device 10 can determine the object as an object to be zoomed. Here, if there are multiple objects in the normal angle of view image and the determination of whether or not a change to a zoom angle of view of the multiple objects is "necessary" may determine any one of the objects as an object to be zoomed. For example, the object with the recognition rate closest to a predetermined threshold value is determined as an object to be zoomed. In yet another example where the necessity of changing the zoom angle of view of multiple objects is "necessary," multiple objects may be determined as objects to be zoomed in. In this case, for example, the processes from step S13 to step S21 described below are repeated the number of times equal to the number of objects to be zoomed in.

[0059] (2) Switching to zoom angle The imaging device 10 calculates the amount of PTZ movement to perform PZT operation on the zoom object (step S13). The imaging device 10 can calculate the amount of pan and tilt movement based on, for example, the positions of the center point of the normal angle of view image and the representative point of the zoom object. The imaging device 10 can also calculate the amount of optical zoom movement so that the size of the zoom object becomes a predetermined size. The imaging device 10 may perform zoom operation with a preset optical zoom amount, for example, 1.5x or 2x.

[0060] Furthermore, if the object to be zoomed is present in the normal angle of view image, the imaging device 10 saves the normal angle of view image (image data) in the second memory 17 (step S14). Then, it is determined whether saving of the normal angle of view image is complete (step S15). If saving of the normal angle of view image is not complete (step S15, NO), S14 and S15 are repeated until saving of the normal angle of view image is complete.

[0061] When the storage of the normal angle of view image has finished (step S15, YES), the imaging device 10 reads out the normal angle of view image stored in the second memory 17 based on the PTZ movement amount while switching to the zoom angle of view (step S16). That is, the processing of step S16 is continuously performed from step S17 to step S19, which will be described later. Therefore, the imaging device 10 can output the normal angle of view image to the display device 30 while switching to the zoom angle of view.

[0062] Then, the imaging device 10 captures a zoom angle of view image at the zoom angle of view, and stores data (capture signal) of the zoom angle of view image in the first memory 16 (step S17). The flow of image data in step S16 is indicated by dashed line 42 in Fig. 6, and the flow of image data in step S17 is indicated by dashed line 41 in Fig. 6. Then, it is determined whether or not the saving of the zoom angle of view image has finished (step S18). If the saving of the zoom angle of view image has not finished (step S18, NO), S17 and S18 are repeated until the saving of the zoom angle of view image has finished.

[0063] When the storage of the zoom angle of view image is completed (step S18, YES), the imaging device 10 returns to the normal angle of view and cancels the reading of the normal angle of view image from the second memory 17 (step S19).

[0064] (3) Output processing of image data for display and image data for recognition Then, the zoom angle of view image is read from the first memory 16 and output as image data for recognition to the image recognition device 20. At this time, the normal angle of view image captured at the normal angle of view and stored in the second memory 17 is output as image data for display to the display device 30 (step S20). Here, the normal angle of view image stored in the second memory 17 has already been processed by the signal processing unit 12, and is therefore output to the output unit 13 and the display device 30 without passing through the signal processing unit 12. The flow of the image data for recognition in step S20 is indicated by the dashed line 43 in FIG. 7, and the flow of the image data for display is indicated by the dashed line 44.

[0065] Then, it is determined whether the image recognition process has ended (step S21). If feedback control of image quality adjustment parameters is not performed, processing unit 14 determines that the image quality recognition process has ended when the recognition result of the image data for recognition output from image recognition device 20 in step S20 is transmitted to processing unit 14 (step S21, YES), and ends the process. If feedback control of image quality adjustment parameters is performed, processing unit 14 determines that the image quality recognition process has ended when the recognition rate of the object has stabilized at a high level, based on the recognition result transmitted from image recognition device 20 (step S21, YES), and ends the process. If feedback control of image quality adjustment parameters is performed, and processing unit 14 determines that the recognition rate of the object has not stabilized at a high level, based on the recognition result transmitted from image recognition device 20 (step S21, NO), the process returns to step S20.

[0066] FIG. 8 is a diagram illustrating the timing of reading data in the imaging device 10. In FIG. The timing of reading data in the imaging device 10 will be described with reference to FIG. 8. Each box shown in FIG. 8 corresponds to one frame of data. FIG. 8 shows an example in which the second frame has a zoom angle of view and the remaining frames have a normal angle of view. Note that here, the imaging device 10 is capable of performing a PTZ operation from the normal angle of view to the zoom angle of view within one frame (step S16), capturing a zoom angle of view image (step S17), and performing a PTZ operation from the zoom angle of view to the normal angle of view (step S19).

[0067] For the sake of explanation, the captured signals for each frame output from the camera unit 11 are labeled A to G. The same symbols are assigned to the data that has undergone signal processing for display or recognition, corresponding to these captured signals A to G. Furthermore, the data stored in the first memory 16 is labeled M1, and the data stored in the second memory 17 is labeled M2.

[0068] As shown in Fig. 8, data A, which is a normal angle of view image captured by camera unit 11 at a normal angle of view and subjected to image processing by signal processing unit 12, is output to display device 30 as image data for display, and is also output to image recognition device 20 as image data for recognition. Image recognition device 20 performs recognition processing on data A. Note that in the image recognition processing on data A, feedback control of image quality adjustment parameters is not performed. If it is determined based on the result of the recognition processing that an object to be zoomed is present, data A is saved in second memory 17. Then, PTZ operation is performed, and the image changes to the zoom angle of view.

[0069] Data B, which is a zoom angle of view image captured by camera unit 11 at a zoom angle of view, is stored in first memory 16, and data B subjected to image processing by signal processing unit 12 is output as image data for recognition to image recognition device 20. At this time, data M2A stored in second memory 17 is output to display device 30 as image data for display.

[0070] Thereafter, the camera unit 11 returns from the zoom angle of view to the normal angle of view. Then, the zoom angle of view image (data M1B) stored in the first memory 16 is processed by the signal processing unit 12 and input to the image recognition device 20 via the processing unit 14 as an image for recognition.

[0071] (4) Image quality adjustment processing based on the recognition results of the zoom angle image 8, feedback control of image quality adjustment parameters is performed for three frames on the zoom angle of view image (data M1B) stored in first memory 16. Data M1B that has undergone a first image quality adjustment using the image quality adjustment parameters is designated M1B-1, data M1B-2 that has undergone a second image quality adjustment using the image quality adjustment parameters, and data M1B-3 that has undergone a third image quality adjustment.

[0072] The second image quality adjustment parameters are adjusted in processing unit 14 according to the recognition result for data M1B-1. Furthermore, the third image quality adjustment parameters are adjusted in processing unit 14 according to the recognition result for data M1B-2. In this way, by performing feedback control of the image quality adjustment parameters according to the recognition result for the recognition image, it is possible to improve the recognition rate of the object. Furthermore, while feedback control is being performed, data M2A stored in second memory 17 is output to display device 30 as display image data. In this way, the zoom angle of view image is not output to display device 30, and the user can continue to view the normal angle of view image.

[0073] Then, after the recognition rate of the object in the zoom angle of view image (data M1B) stabilizes at a high level, the operation of reading data M1B from the first memory 16 is stopped, and the operation of reading data MB2 from the second memory 17 is stopped. Thereafter, data F and G captured at the normal angle of view are output to the display device 30 as image data for display, and to the image recognition device 20 as image data for recognition.

[0074] Note that the signal processing unit 12 may retain the setting values of the image processing executed on the normal angle of view image when saving the normal angle of view image in the second memory 17. When reading out the zoom angle of view image from the first memory 16 and outputting it to the signal processing unit 12, the image processing may be performed using the retained setting values of the image processing. Furthermore, the signal processing unit 12 may determine the setting values of the image processing on the zoom angle of view image based on information related to feedback control of the image quality adjustment parameters in the processing unit 14.

[0075] 9 is a diagram showing an example of a normal angle of view image and a zoom angle of view image. In FIG. 9, it is assumed that an object T1 in the normal angle of view image is determined as a zoom object. In this case, the object T1 moves to the center position of the zoom angle of view image, and a PTZ operation is performed so that the object T1 becomes larger than a predetermined pixel size, and the zoom angle of view image is captured.

[0076] As described above, according to the first embodiment, it is possible to determine whether or not to change the angle of view of the object to a zoom angle of view and optically enlarge the image based on the recognition result of the normal angle of view image, thereby improving the recognition rate of the zoom object. Furthermore, while the imaging device 10 is capturing an image of the zoom object at the zoom angle of view, it is possible to output a normal angle of view image to the display device 30. In other words, the imaging device 10 can output an image with a wide angle of view to the display device 30 and output an optically enlarged image of the object to the image recognition device 20. This allows the imaging device 10 to effectively switch between and output an image that the user wants to visually confirm and an image that improves the recognition accuracy of the object in the image recognition device 20.

[0077] Embodiment 2. Fig. 10 is a block diagram showing the configuration of an imaging device according to the second embodiment. In Fig. 10, the same components as those in Fig. 2 are assigned the same reference numerals, and description thereof will be omitted. The second embodiment differs from the first embodiment in that the zoom angle of view image stored in the first memory 16 is stored after being processed by the signal processing unit 12. When data stored in the first memory 16 is read out, the processed zoom angle of view image is input directly to the signal processing unit 12 without passing through the signal processing unit 12. In the second embodiment, the processing unit 14 performs feedback control of the image quality adjustment parameters.

[0078] The operational flow of the imaging device 10 according to the second embodiment is substantially the same as the example shown in FIG. 4. Differences from the first embodiment will be described below for some of the steps in FIG. 4. After the imaging device 10 shown in FIG. 10 has finished saving the normal angle of view image in the second memory 17 (step S15 in FIG. 4, YES), the imaging device 10 reads out the normal angle of view image saved in the second memory 17 while switching to a zoom angle of view. At this time, the imaging device 10 processes the zoom angle of view image captured at the zoom angle of view in the signal processing unit 12 and saves it in the first memory 16 (step S17). The flow of image data in step S16 is indicated by dashed line 46 in FIG. 11, and the flow of image data in step S17 is indicated by dashed line 45 in FIG. 11.

[0079] After the processed zoom angle of view image has been saved (step S18, YES), the imaging device 10 returns to the normal angle of view and cancels reading of the normal angle of view image from the second memory 17 (step S19). Then, while the normal angle of view image captured at the normal angle of view is being output to the display device 30, the processed zoom angle of view image is output directly from the first memory 16 to the image recognition device 20 as image data for recognition (S20). The processing unit 14 can adjust the image quality of the recognition image using image quality adjustment parameters that are feedback-controlled. The flow of the recognition image data in step S20 in the second embodiment is indicated by dashed lines 48 and 49 in FIG. 12, and the flow of the display image data is indicated by dashed line 47.

[0080] Fig. 13 is a diagram illustrating the timing of reading data in the imaging device of the second embodiment. Note that the reference symbols assigned to each piece of data in Fig. 13 are the same as those in Fig. 8. As shown in Fig. 13, data A captured at a normal angle of view is output to the display device 30 as image data for display, and is also output to the image recognition device 20 as image data for recognition. The image recognition device 20 performs a recognition process on the data A. If it is determined based on the results of the recognition process that there is an object to zoom, the data A is stored in the second memory 17. Then, a PTZ operation is performed, and the camera switches to a zoom angle of view.

[0081] Data B, which is a zoom angle of view image captured at a zoom angle of view, is image processed by signal processing unit 12 and stored in first memory 16, and is also output as image data for recognition to image recognition device 20. At this time, data M2A stored in second memory 17 is output to display device 30 as image data for display.

[0082] Thereafter, the camera unit 11 returns from the zoom angle of view to the normal angle of view. Then, the processed zoom angle of view image (data M1B) is input to the image recognition device 20 as an image for recognition.

[0083] 13, feedback control of image quality adjustment parameters is performed for three frames on the zoom angle of view image (data M1B) that has been image processed by the signal processing unit 12 and stored in the first memory 16. Furthermore, the imaging device 10 outputs data C captured at a normal angle of view to the display device 30 as an image to be displayed. Thereafter, the imaging device 10 outputs data D to G to the display device 30 as images to be displayed.

[0084] The image quality adjustment parameters are adjusted a second time in processing unit 14 according to the recognition result for data M1B-1. Furthermore, the image quality adjustment parameters are adjusted a third time in processing unit 14 according to the recognition result for data M1B-2. In this way, by performing feedback control of the image quality adjustment parameters according to the recognition result for the recognition image, it is possible to improve the recognition rate of the object. In this way, the zoom angle of view image is not output to display device 30, and the user can continue to view the normal angle of view image.

[0085] Then, after the recognition rate of the object in the zoomed angle of view image (data M1B) has stabilized at a high level, the read operation of data M1B from the first memory 16 is terminated, and data F and G captured at a normal angle of view are output to the image recognition device 20 as image data for recognition.

[0086] As described above, in the second embodiment, the zoom angle of view image processed by the signal processing unit 12 is stored in the first memory 16 and directly input to the processing unit 14. Therefore, while the normal angle of view image is processed by the signal processing unit 12 and output to the display device 30, the processed data stored in the first memory 16 can be directly input to the processing unit 14. This prevents the signal processing unit 12 from simultaneously processing the zoom angle of view image and the normal angle of view image, reducing the risk of discomfort in viewing the normal angle of view image due to a delay in feedback control that cannot be addressed by anything other than image quality adjustment parameters. Furthermore, according to the second embodiment, the imaging device 10 can effectively switch between and output an image that the user wants to visually confirm and an image that improves the object recognition accuracy of the image recognition device 20.

[0087] Each functional block shown in the drawings that performs various processes can be configured in hardware with a processor, memory, and other circuits. The processes described above can also be implemented by having a processor execute a program. Therefore, these functional blocks can be implemented in various forms, such as hardware only, software only, or a combination of both, and are not limited to any one of these.

[0088] The above-described program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include semiconductor memory (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, and RAM). The program may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0089] The contents of the present disclosure can be used in various fields that utilize image recognition. Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0090] (Appendix A1) an imaging unit capable of capturing an image by switching between a normal angle of view and a zoom angle of view using optical zoom; a recognition result acquisition unit that acquires a recognition result by performing object recognition processing on an input image using an image recognition device; a determination unit that determines whether or not the angle of view of the object needs to be changed to a zoom angle of view based on a recognition result for the normal angle of view image captured at the normal angle of view; a control instruction unit that controls switching to a zoom angle of view zoomed onto a zoom object when it is determined that a change to the zoom angle of view is required for the zoom object; an output processing unit that outputs a zoom angle of view image captured by the imaging unit that has switched to the zoom angle of view to the image recognition device; Equipped with Imaging device. (Appendix A2) the determination unit determines that a change to a zoom angle of view is necessary when a recognition rate of the object in the normal angle of view image is smaller than a predetermined threshold value; 10. The imaging device according to claim 1. (Appendix A3) the determination unit determines that a change to a zoom angle of view is necessary when the recognition rate of the object in the normal angle of view image is smaller than the threshold value and larger than a second threshold value that is smaller than the threshold value. 10. The imaging device according to claim 9, wherein the imaging device is a (Appendix A4) a storage processing unit that stores the normal angle of view image in a memory when the object to be zoomed is present; the output processing unit outputs the normal angle of view image stored in the memory to an external device while switching from the normal angle of view to the zoom angle of view. An imaging device according to any one of appendices A1 to A3. (Appendix B1) The computer A process of capturing an image by switching between a normal angle of view and a zoomed angle of view using optical zoom; A process of performing object recognition processing on an input image using an image recognition device and acquiring a recognition result; a process of determining whether or not the angle of view of the object needs to be changed to a zoom angle of view based on a recognition result for the normal angle of view image captured at the normal angle of view; If it is determined that a change to the zoom angle of view is required for an object, a process of controlling switching to a zoom angle of view that zooms in on the object is performed; a process of switching to the zoom angle of view and outputting the zoom angle of view image captured by the imaging unit to an image recognition device; To execute A method for controlling an imaging device. (Appendix C1) A process of capturing an image by switching between a normal angle of view and a zoomed angle of view using optical zoom; A process of performing object recognition processing on an input image using an image recognition device and acquiring a recognition result; a process of determining whether or not the angle of view of the object needs to be changed to a zoom angle of view based on a recognition result for the normal angle of view image captured at the normal angle of view; If it is determined that a change to the zoom angle of view is required for an object, a process of controlling switching to a zoom angle of view that zooms in on the object is performed; a process of switching to the zoom angle of view and outputting a zoom angle of view image captured by the zoom angle of view to an image recognition device; to the computer, program.

[0091] Some or all of the elements described in Appendix A2 to Appendix A4 that are subordinate to Appendix A1 (imaging device) may also be subordinate to Appendix B1 (control method for imaging device) and Appendix C1 (program) in a similar subordinate relationship. [Explanation of symbols]

[0092] 100 Image Recognition System 10. Imaging device 11 Camera unit 12 Signal Processing Section 13 Output section 14 Processing section 15 Control Unit 16 First Memory 17 Second Memory 18 Memory section 141 Recognition result acquisition unit 142 Judgment section 143 Preservation Processing Department 144 Control instruction section 145 Output Processing Unit 146 Parameter setting section 20 Image Recognition Device 30 Display device

Claims

1. an imaging unit capable of capturing an image by switching between a normal angle of view and a zoom angle of view using optical zoom; a recognition result acquisition unit that acquires a recognition result by performing object recognition processing on an input image using an image recognition device; a determination unit that determines whether or not the angle of view of the object needs to be changed to a zoom angle of view based on a recognition result for the normal angle of view image captured at the normal angle of view; a control instruction unit that controls switching to a zoom angle of view zoomed onto a zoom object when it is determined that a change to the zoom angle of view is required for the zoom object; an output processing unit that outputs a zoom angle of view image captured by the imaging unit that has switched to the zoom angle of view to the image recognition device; Equipped with Imaging device.

2. the determination unit determines that a change to a zoom angle of view is necessary when a recognition rate of the object in the normal angle of view image is smaller than a predetermined threshold value; The imaging device according to claim 1 .

3. the determination unit determines that a change to a zoom angle of view is necessary when the recognition rate of the object in the normal angle of view image is smaller than the threshold value and larger than a second threshold value that is smaller than the threshold value; The imaging device according to claim 2 .

4. a storage processing unit that stores the normal angle of view image in a memory when the object to be zoomed is present; the output processing unit outputs the normal angle of view image stored in the memory to an external device while switching from the normal angle of view to the zoom angle of view. The imaging device according to claim 1 .

5. The computer A process of capturing an image by switching between a normal angle of view and a zoomed angle of view using optical zoom; A process of performing object recognition processing on an input image using an image recognition device and acquiring a recognition result; a process of determining whether or not the angle of view of the object needs to be changed to a zoom angle of view based on a recognition result for the normal angle of view image captured at the normal angle of view; If it is determined that a change to the zoom angle of view is required for an object, a process of controlling switching to a zoom angle of view that zooms in on the object is performed; a process of switching to the zoom angle of view and outputting a zoom angle of view image captured by the zoom angle of view to an image recognition device; To execute A method for controlling an imaging device.

Images