Image processing device, image processing method, and program
By calculating the optimal imaging ratio and adjusting the ratio of the imaging area to the cropping area of the PTZ camera, the problem of image quality degradation and loss when the main subject moves at a high speed was solved, and more reliable video generation of main subject tracking was achieved.
Patent Information
- Application Number
- CN202380029302.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-31
- Filing Date
- 2023-02-24
- Publication Date
- 2024-11-19
AI Technical Summary
Existing technologies using PTZ cameras to generate video tracking of prominent subjects pose risks of reduced image quality and loss of the prominent subject, especially when the prominent subject moves at high speeds, where excessively long mechanical drive time makes it impossible to adjust the camera orientation in time.
By calculating the optimal imaging ratio and adjusting the ratio of the imaging area and the cropping area of the PTZ camera, combined with camera control parameters, more reliable image cropping and video tracking generation are achieved, suppressing image quality degradation.
It improves the success rate of image capture in situations with highly mobile and prominent subjects, reduces image quality degradation, and ensures reliable tracking of prominent subjects.
Smart Images

Figure CN118985134A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an image processing device, an image processing method, and a program. More specifically, the present disclosure relates to an image processing device, an image processing method, and a program capable of more reliably performing a video generation process of a prominent subject through image interception while suppressing a decrease in image quality. Background Art
[0002] For example, there is a case where a process of generating video data obtained by extracting a specific prominent subject, for example, an image region following a target person, is required from a video shot using a surveillance camera.
[0003] Furthermore, for example, in a sports video such as soccer, there is a case where it is necessary to generate follow-up video data obtained by cutting out an image region of a specific prominent object, for example, a specific player.
[0004] As a process of generating such follow-up video data of a prominent subject, for example, a process of capturing a wide-angle image including the prominent subject using a fixed camera and cutting out an image area of the prominent subject from the captured video to generate the follow-up video data may be performed.
[0005] As described above, by capturing a wide-angle image including a prominent subject using a fixed camera, the prominent subject is less likely to move out of the capturing range even if the prominent subject moves slightly, so that video data following the prominent subject can be generated.
[0006] For example, in the case of generating a video following a prominent player in a football match, a video including a wide area such as an entire or half football field is shot with a fixed camera, and an image area of the prominent player is captured from each frame of the video, thereby generating a video similar to that of only shooting the prominent player.
[0007] Note that a process of cutting out an image region of a prominent subject is described in, for example, Patent Document 1 (WO 2016 / 167016A).
[0008] However, if the image area of the prominent player is cut out from a wide area of the shot image taken by the fixed camera in this way, there arises a problem that the image quality of the cut-out image is degraded in the case where the image area of the prominent player is small.
[0009] In order not to significantly reduce the image quality of the cut-out image of the prominent subject, the cut-out image including the prominent subject needs to have an image size equal to or larger than a predetermined number of pixels.
[0010] In order to realize the generation of a cut-out image with suppressed image quality degradation, a method using a PTZ camera is effective.
[0011] PTZ cameras can pan, tilt and zoom, and can point the camera's shooting direction to a prominent subject. Even if the prominent subject moves, it can be shot while maintaining the prominent subject within a shooting area of a predetermined image size for a longer period of time.
[0012] However, for example, pan processing of rotating the shooting direction of a PTZ camera in the horizontal direction and tilt processing of rotating the shooting direction of a camera in the vertical direction require operation of a mechanical drive mechanism for rotating the camera itself, and there is a problem that a predetermined drive time is required.
[0013] Therefore, when the moving speed of the prominent subject as the tracking target is high, if the time for driving the camera shooting direction to a direction in which the prominent subject can be shot becomes long, there is a possibility that the prominent subject will be lost.
[0014] Citation List Patent Literature Patent Document 1: WO 2016 / 167016A Summary of the invention Problems to be solved by the present invention The present disclosure is made in view of the above-mentioned problems, and an object of the present disclosure is to provide an image processing device, an image processing method, and a program that can more reliably perform the generation process of subject tracking video data accompanying image capture while suppressing the reduction of image quality.
[0015] An object of the embodiments of the present disclosure is to provide an image processing device, an image processing method, and a program capable of more reliably performing processing of intercepting and recording or distributing a partial image area including a prominent subject from a captured image of a PTZ camera while suppressing a reduction in image quality.
[0016] Solution to the problem A first aspect of the present disclosure is an image processing device, comprising: a clipping region calculation unit, wherein the clipping region calculation unit calculates an image region including a prominent subject as an image clipping region based on an image captured by a camera; a clipping execution unit, the clipping execution unit clipping the image of the image clipping area calculated by the clipping area calculation unit, and generating a clipped image; an imaging ratio calculation unit that calculates an imaging ratio for clipped image generation as an index value for executing a clipped image generation process including the prominent subject, the imaging ratio for clipped image generation being a size ratio between a size of the clipped image and a size of an imaging area equivalent to a whole image area of a captured image of the camera; and a camera control parameter calculation unit for calculating a camera control parameter for causing the camera to capture an image having an imaging area size determined based on the calculated imaging ratio for generating the clipped image, The camera is caused to capture an image to which the camera control parameters calculated by the camera control parameter calculation unit are applied.
[0017] In addition, a second aspect of the present disclosure is an image processing method performed in an image processing device, the method comprising: a clipping area calculation step, in which a clipping area calculation unit calculates an image area including a prominent subject as an image clipping area based on an image captured by a camera; A capture execution step, in which a capture execution unit captures the image of the image capture area calculated in the capture area calculation step and generates a capture image; an imaging ratio calculation step of calculating, by an imaging ratio calculation unit, an imaging ratio for cropped image generation as an index value for executing cropped image generation processing including the prominent subject, the imaging ratio for cropped image generation being a size ratio between the size of the cropped image and the size of an imaging area equivalent to the entire image area of the captured image of the camera; and a camera control parameter calculation step, in which a camera control parameter calculation unit calculates a camera control parameter camera, wherein the camera control parameter is used to cause the camera to capture an image having an imaging area size determined based on the calculated imaging ratio for generating the clipped image, The camera is caused to capture an image to which the camera control parameters calculated by the camera control parameter calculation unit are applied.
[0018] Furthermore, a third aspect of the present disclosure is a program for causing an image processing device to perform image processing, the program causing the image processing device to perform: A clipping area calculation step, wherein the clipping area calculation unit calculates an image area including a prominent subject as an image clipping area based on an image captured by the camera; A capture execution step, causing the capture execution unit to capture the image of the image capture area calculated in the capture area calculation step and generate a capture image; a cropped image generation imaging ratio calculation step of causing an imaging ratio calculation unit to calculate an imaging ratio for cropped image generation as an index value for executing cropped image generation processing including the prominent subject, the cropped image generation imaging ratio being a size ratio between a size of the cropped image and a size of an imaging area equivalent to a whole image area of an image captured by the camera; and a camera control parameter calculation step, causing a camera control parameter calculation unit to calculate a camera control parameter camera, wherein the camera control parameter is used to cause the camera to capture an image having an imaging area size determined based on the calculated imaging ratio for generating the clipped image, The camera is caused to capture an image to which the camera control parameters calculated by the camera control parameter calculation unit are applied.
[0019] Note that the program of the present disclosure is, for example, a program that can be provided by providing a storage medium or communication medium in a computer-readable format to an image processing device or a computer system that can execute various program codes. By providing such a program in a computer-readable format, processing corresponding to the program is implemented on the image processing device or the computer system.
[0020] Other purposes, features and advantages of the present disclosure will become apparent from a more detailed description based on the embodiments of the present disclosure described later and the accompanying drawings. Note that the system herein is a logical collection of multiple devices and is not limited to a system in which the various devices are in the same housing.
[0021] According to the configuration of an embodiment of the present disclosure, a device and method for calculating an optimal imaging ratio (imaging ratio for cropped image generation) is implemented, wherein the optimal imaging ratio is an index value for more reliably performing cropped image generation processing including a prominent subject based on a captured image of a PTZ camera.
[0022] Specifically, for example, in a configuration in which an image cut-out area including a prominent subject is calculated based on a captured image of a PTZ camera, and an image of the calculated area is cut out to generate a cut-out image, an optimal imaging ratio (imaging ratio for cut-out image generation) which is a ratio of the size of the imaging area of the camera to the cut-out image is calculated as an index value for more reliably performing a cut-out image generation process including a prominent subject. Furthermore, a zoom parameter for capturing an image corresponding to the size of the imaging area of the camera calculated based on the optimal imaging ratio is calculated, and the PTZ camera is caused to perform image capturing to which the calculated parameter is applied.
[0023] With this configuration, an apparatus and method for calculating an optimal imaging ratio (imaging ratio for cropped image generation) which is an index value for more reliably performing a process of generating a cropped image including a prominent subject based on an image captured by a PTZ camera are realized.
[0024] Note that the effects described herein are merely examples and are not limiting, and other effects may be provided. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 11 is a diagram for explaining an overview of a process of generating a follow-up image of a prominent subject accompanying image capture using a PTZ camera.
[0026] Figure 2 is a diagram illustrating a configuration example of an image processing system including an external device that generates a cut-out image of an image region including a prominent subject.
[0027] Figure 3 1 and 2 are diagrams for explaining a clipped region setting example including a full-body region of a prominent subject and a clipped region setting example including only an upper-body region of a prominent subject.
[0028] Figure 4 It is a diagram for explaining a problem in image interception processing.
[0029] Figure 5 It is a diagram for explaining a problem in image interception processing.
[0030] Figure 6 1 is a diagram for explaining a specific example of processing performed by the image processing apparatus of the present disclosure.
[0031] Figure 7 is a diagram for explaining the imaging ratio.
[0032] Figure 8 It is a diagram for explaining a specific example of the imaging ratio changing process performed by the image processing device of the present disclosure.
[0033] Fig. 9 It is a diagram for explaining a specific example of the imaging ratio changing process performed by the image processing device of the present disclosure.
[0034] Fig.10 1 is a diagram for explaining an embodiment of changing the imaging ratio according to the speed of a prominent subject.
[0035] Fig.11 is a diagram for explaining rMIN (allowable minimum imaging ratio).
[0036] Fig.12 is a diagram for explaining an example of rMAX (allowable maximum imaging ratio).
[0037] Fig.13 1 is a diagram for explaining an embodiment of changing an imaging ratio according to acceleration of a prominent subject.
[0038] Fig.14 1 is a diagram for explaining an embodiment of changing an imaging ratio according to the speed and acceleration of a significant subject.
[0039] Fig.15is a diagram illustrating a flowchart for explaining a processing sequence of an embodiment of changing an imaging ratio according to a speed of a prominent subject.
[0040] Fig.16 is a diagram illustrating a flowchart for explaining a processing sequence of an embodiment of changing an imaging ratio according to acceleration of a prominent subject.
[0041] Fig.17 is a diagram illustrating a flowchart for explaining a processing sequence of an embodiment of changing an imaging ratio according to a speed and an acceleration of a significant subject.
[0042] Fig.18 is a diagram illustrating a flowchart for explaining a processing sequence of an embodiment of changing an imaging ratio according to a speed and an acceleration of a significant subject.
[0043] Fig.19 1 is a diagram for explaining a specific processing example of an embodiment of changing an imaging ratio according to a communication delay time between a camera and an external device.
[0044] Fig. 20 is a diagram illustrating a flowchart for explaining a processing sequence of an embodiment of changing an imaging ratio according to a communication delay time between a camera and an external device.
[0045] Fig.21 is a diagram illustrating a flowchart for explaining a processing sequence of an embodiment of changing an imaging ratio according to a speed and acceleration of a prominent subject, and a communication delay time between a camera and an external device.
[0046] Fig. 22 is a diagram illustrating a flowchart for explaining a processing sequence of an embodiment of changing an imaging ratio according to a speed and acceleration of a prominent subject, and a communication delay time between a camera and an external device.
[0047] Fig.23 is a diagram for explaining an example of processing performed by a camera as an example of the image processing device of the present disclosure.
[0048] Fig.24 : are diagrams for explaining configuration examples of a camera and an external device as an example of the image processing device of the present disclosure, and processes to be performed.
[0049] Fig.25 : are diagrams for explaining configuration examples of a camera and an external device as an example of the image processing device of the present disclosure, and processes to be performed.
[0050] Fig.26 is a diagram for explaining a configuration example of a camera as an example of the image processing device of the present disclosure.
[0051] Fig. 27 : is a diagram for explaining a configuration example of a camera and an external device as an example of the image processing device of the present disclosure.
[0052] Fig.28 : is a diagram for explaining a configuration example of a camera and an external device as an example of the image processing device of the present disclosure.
[0053] Fig.29 It is a diagram for explaining a hardware configuration example of the image processing device of the present disclosure. DETAILED DESCRIPTION
[0054] Hereinafter, details of the image processing device, image processing method, and program of the present disclosure will be described with reference to the drawings. Note that the description will be made according to the following items.
[0055] 1. Overview of the tracking image generation process of a prominent subject accompanying image capture using a PTZ camera 2. Processing performed by the image processing device of the present disclosure 3. (Example 1) (Example 1) Example of changing the imaging ratio according to at least one of the speed or acceleration of the prominent subject 3-1. (Example 1-1) Example of changing the imaging ratio according to the speed of the prominent subject 3-2. (Example 1-2) Example of changing the imaging ratio according to the acceleration of the prominent subject 3-3. (Example 1-3) Example of changing the imaging ratio according to the speed and acceleration of the prominent subject 4. Processing Sequence Executed by the Image Processing Apparatus of Embodiment 1 5. (Example 2) Example of changing the imaging ratio according to the communication delay time between the camera and the external device 6. (Example 3) Example of changing the imaging ratio according to the speed and acceleration of the prominent subject and the communication delay time between the camera and the external device 7. Configuration Example of the Image Processing Device Disclosed in the Present Disclosure 8. Detailed Structure of the Image Processing Device Disclosed 9. Hardware Configuration Example of Image Processing Device 10. Summary of the present disclosure [1. Overview of the tracking image generation process of a prominent subject accompanying image capture using a PTZ camera] First, refer to Figure 1 and subsequent figures, describe an outline of a tracking image generation process of a prominent subject accompanying image capture using a PTZ camera.
[0056] As described above, in the case of generating tracking video data obtained by extracting an image area of a specific tracking target person from a video shot using a surveillance camera, or in the case of generating tracking video data obtained by cutting out an image area of a specific player from a video shot of a sport such as football, a process of cutting out and generating an image area including a prominent subject from an image shot by a fixed camera that shoots a wide area can be performed.
[0057] However, if an image region of a prominent player is cut out from a wide region of a shot image shot by a fixed camera, the image quality of the cut-out image is degraded if the image region of the prominent player is small.
[0058] In order not to significantly reduce the image quality of the cut-off image of the prominent subject, it is necessary to set the cut-off image including the prominent subject to an image size equal to or larger than a predetermined number of pixels, and to achieve such a shooting process, using a PTZ camera is effective.
[0059] A PTZ camera is a camera that can perform pan, tilt, and zoom operations. That is, a PTZ camera is a camera that can perform the following operations: Panning process that rotates the camera's shooting direction in the horizontal direction; Pitch processing to rotate the camera's shooting direction in the vertical direction; and Zoom processing that changes the camera's shooting angle of view (= enlargement or reduction of the shooting area).
[0060] By using such a PTZ camera, the shooting direction of the camera can be directed toward a prominent subject, and even if the prominent subject moves, shooting can be performed while maintaining the prominent subject within a shooting area having a predetermined image size for a longer period of time.
[0061] Note that in recent years, by performing AI analysis using at least one of a machine learning model such as a deep neural network (DNN) which is a multi-layer neural network or a rule-based model, it is possible to perform processing for detecting and tracking a specific person from a captured image with high accuracy.
[0062] By using such AI analysis, it is possible to automatically control the PTZ camera (control of pan, tilt, and zoom) and shoot follow-up videos of prominent subjects.
[0063] However, for example, in the panning process of rotating the shooting direction of the PTZ camera in the horizontal direction and the pitching process of rotating the shooting direction of the camera in the vertical direction, since the camera itself is rotated, it is necessary to operate a mechanical driving mechanism, and there is a problem that a predetermined driving time is required.
[0064] When the moving speed of a prominent subject as a tracking target is high, if the time required to drive the camera shooting direction to a direction in which the prominent subject can be shot becomes long, there is a possibility that the prominent subject will be lost.
[0065] Will refer to Figure 1 and subsequent drawings, describe an overview of a process of photographing an image (video (=moving image)) including a significant subject and a process of generating a cut-out image of the significant subject using a PTZ camera and problems thereof.
[0066] Figure 1 This diagram illustrates a state in which an image of a prominent subject 21 as a tracking target person is captured using the PTZ camera 10 .
[0067] Note that in the following description, it is assumed that “image” includes video as a moving image and image frames constituting the video.
[0068] like Figure 1 As shown in the diagram, the PTZ camera 10 can perform pan processing to rotate the camera shooting direction in the horizontal direction, tilt processing to rotate the camera shooting direction in the vertical direction, and zoom processing to change the camera shooting field angle (= enlargement and reduction of the shooting area).
[0069] Figure 2 : is a diagram illustrating a configuration example of an image processing system including an external device 30 that receives a captured image of a PTZ camera 10, performs image analysis processing for extracting an image region of a significant subject, and generates a cut-out image of the image region including the significant subject.
[0070] Figure 2 The diagram illustrates processing steps S01 to S04 performed by the PTZ camera 10 and the external device 30 .
[0071] The individual processing steps will be described below.
[0072] (Step S01) First, in step S01, the PTZ camera 10 captures an image. The image is a moving image (video).
[0073] The captured image 20 illustrated in the figure is an example of one image frame of a moving image (video) captured by the PTZ camera 10 .
[0074] In the captured image 20 , for example, a prominent subject 21 that is a tracking target is captured.
[0075] (Step S02) Next, in step S02 , the PTZ camera 10 transmits the captured image to the external device 30 which is a device for executing image analysis processing and image interception processing.
[0076] (Step S03) The external device 30 detects a significant subject 21 from the captured image 20 received from the PTZ camera 10 , and performs an image clipping process of clipping an image region of the detected significant subject 21 .
[0077] Note that for the prominent subject 21, for example, a facial image, body features, etc. are analyzed in advance in the external device 30, and the acquired feature information is registered in the storage unit. The external device 30 performs image matching processing using these registered feature information to perform detection processing of the prominent subject 21.
[0078] Specifically, for example, by performing the AI analysis using at least one of the machine learning model such as a deep neural network and the rule-based model described above, the prominent subject 21 is detected from the captured image 20 .
[0079] The external device 30 also performs an image clipping process of clipping an image region of the detected prominent subject 21 .
[0080] As illustrated in the figure, a process of cutting out a partial image area as a cut-out area 23 from an imaging area 22 which is an area of the entire captured image 20 is performed.
[0081] For example, an image region including a whole body region of the prominent subject 21 is set as the cutout region 23 , and image cutout is performed.
[0082] Note that the setting of the clipped area of the image of the prominent subject 21 is predefined. That is, a clipped area setting algorithm is predefined, and the external device 30 sets the clipped area 23 including the prominent subject 21 from the image according to the algorithm and performs image clipping processing.
[0083] There are various types of clipped area setting algorithms. Specifically, for example, a clipped area setting algorithm including the whole body area of the prominent subject 21, a clipped area setting algorithm including only the upper body area of the prominent subject 21, a clipped area setting algorithm including only the face area of the prominent subject 21, etc. are used. Note that the types of clipped area setting algorithms used in the external device 30 are not limited to these examples.
[0084] Figure 3 The illustrations illustrate a clipped region setting example including a full-body region of the prominent subject 21 and a clipped region setting example including only an upper-body region of the prominent subject 21 .
[0085] Figure 3The “(1) Full body shot” diagram shown in the figure illustrates an example of a cut-out region setting algorithm including a full body region of the prominent subject 21. The position of the full body region of the prominent subject 21 is set at a substantially central position of the cut-out region 23, and a rectangular region is set as the cut-out region 23, in which predetermined lengths are set on the upper side, the lower side, the left side, and the right side of the prominent subject 21, for example, a margin region of approximately 10% of the height of the prominent subject is set on the upper side and the lower side of the prominent subject 21, and a margin region of approximately 50% of the lateral width of the prominent subject is set on the left side and the right side of the prominent subject 21.
[0086] also, Figure 3 The “(2) Upper Body Shot” diagram shown in the figure illustrates an example of a cut-out region setting algorithm including the upper body region of the prominent subject 21. The position of the upper body region of the prominent subject 21 is set at a substantially central position of the cut-out region 23, and a rectangular region is set as the cut-out region 23, in which a margin region of approximately 10% of the height of the prominent subject is set on the prominent subject 21, and a margin region of approximately 30% of the lateral width of the prominent subject is set on the left and right of the prominent subject 21.
[0087] For example, as described above, an image clipping region setting algorithm for the prominent subject 21 is defined in advance, and the external device 30 sets the clipping region 23 including the prominent subject 21 from the image according to the defined algorithm and performs image clipping processing.
[0088] return Figure 2 , the processing performed by the PTZ camera 10 and the external device 30 will be described continuously.
[0089] In step S03, the external device 30 sets the clipped area 23 in each image frame of the captured image 20 captured by the PTZ camera 10, and generates a clipped image corresponding to each image frame. By reproducing these clipped images corresponding to the corresponding image frames, a video centered on the significant subject 21 on which the significant subject is mainly projected can be generated.
[0090] Note that, although not shown, a video including a captured image generated by the external device 30 may be distributed from the external device 30 to a user terminal such as a television or a smartphone, and may be stored in a storage unit in the external device 30 .
[0091] (Step S04) When performing the image interception processing in step S03, the external device 30 also performs image analysis. The external device 30 estimates the moving direction and moving speed of the prominent subject 21 detected from the captured image 20, and in step S04, sends a camera driving instruction to the PTZ camera 10 based on these estimation results.
[0092] That is, the orientation (pan and tilt settings) and zoom setting information of the PTZ camera 10 are transmitted so that the PTZ camera 10 can follow and photograph the significant subject 21 .
[0093] (Step S05) When receiving the camera driving instruction data from the external device 30 , in step S05 , the PTZ camera 10 changes and updates the camera orientation (pan and tilt) and zoom setting based on the received camera driving instruction information.
[0094] By continuously repeating these processes, the PTZ camera 10 can continuously capture images of the significant subject 21 , and the external device 30 can detect the significant subject 21 from the captured images received from the PTZ camera and cut out an image region of the significant subject 21 .
[0095] Note that a video composed of cut-out images is a video in which the prominent subject 21 is captured larger than the prominent subject 21 in the captured image before the cut-out.
[0096] However, as described above, in the panning process of the shooting direction of the PTZ camera 10 by rotating in the horizontal direction and the pitching process of the shooting direction of the camera by rotating in the vertical direction, since the camera itself is rotated, a mechanical driving mechanism must be operated, which requires a predetermined driving time, and when the moving speed of the prominent subject 21 is fast, there is a possibility that the prominent subject is lost. As a result, there is a problem that the interception process including the prominent subject 21 cannot be performed.
[0097] A specific example of this problem will refer to Figure 4 to describe.
[0098] Figure 4 Graphical instructions reference Figure 2 The PTZ camera 10 and the external device 30 are described.
[0099] The PTZ camera 10 captures captured images 20 a to 20 c including a significant subject 21 at times ( t0 ) to ( t2 ), and transmits the captured images to the external device 30 .
[0100] The external device 30 detects a prominent subject 21 from the captured image 20 received from the PTZ camera 10 , and performs a process of setting and clipping a clipped area 23 including a whole body area of the prominent subject 21 from an imaging area 22 corresponding to the entire area of the captured image 20 .
[0101] The prominent subject 21 in the captured image 20a at time (t0) is stationary. The external device 30 may perform a process of setting and clipping a cutout region 23a including a whole body region of the prominent subject 21 from an imaging region 22a corresponding to the entire region of the captured image 20a at time (t0).
[0102] Through the image analysis processing of the captured image 20 a at time ( t0 ), the external device 30 confirms that the significant subject 21 is stationary, and does not send a driving instruction to the PTZ camera 10 .
[0103] The prominent subject 21 in the captured image 20b at the next time (t1) is running to the right in the figure. The external device 30 sets and cuts out a cutout area 23b including the whole body area of the prominent subject 21 from the imaging area 22b corresponding to the entire area of the captured image 20b at the time (t1).
[0104] Through image analysis processing of the captured image 20b at time (t1), the external device 30 confirms that the prominent subject 21 is running toward the right side in the figure, and sends a camera drive (PTZ drive) instruction to the PTZ camera 10 to move the current shooting direction toward the right side.
[0105] That is, execute Figure 4 The process (process b) shown in the diagram.
[0106] When a camera driving (PTZ driving) instruction is input from the external device 30, the PTZ camera 10 performs a panning operation to move the current shooting direction to the right. The shot image of the operation result is a shot image 20c at time (t2).
[0107] A portion of the prominent subject 21 in the captured image 20 c is near the end of the capturing range of the PTZ camera 10 or outside the imaging range.
[0108] This is because the panning operation for moving the photographing direction to the right is performed by mechanical driving of a gear mechanism or the like of the PTZ camera 10 , which takes time.
[0109] As a result, at time ( t2 ), the PTZ camera 10 transmits a captured image 20 c in which a part of the prominent subject 21 is outside the capturing range to the external device 30 .
[0110] As in Figure 4 As shown in the diagram of (processing c), even if the external device 30 attempts to set and cut out the cut-out area 23c of the whole body area including the prominent subject 21 from the imaging area 22c corresponding to the entire area of the captured image 20c at time (t2), the external device 30 cannot set the cut-out area 23c of the whole body area including the prominent subject 21.
[0111] As above reference Figure 3 As described above, for example, a clipped region setting algorithm including a whole body region of the prominent subject 21 is defined in advance. That is, an algorithm for clipping a region in which prescribed margins are respectively set in the upper, lower, left, and right directions of the prominent subject 21 is defined.
[0112] exist Figure 4 In (processing c) shown in the diagram, even if image clipping according to the clipping algorithm is performed, a predetermined margin cannot be set on the right side of the significant subject 21 (=moving direction of the significant subject), and as a result, the clipping area 23c cannot be set.
[0113] Furthermore, in a case where the PTZ camera 10 and the external device 30 are located at positions separated from each other so that there is a possibility of a communication delay occurring, the possibility of a tracking error of a prominent subject or an image interception error occurring due to the communication delay further increases.
[0114] refer to Figure 5 , a specific example of a problem that occurs in an environment where the PTZ camera 10 and the external device 30 are located at locations separated from each other and a communication delay may occur will be described.
[0115] Figure 5 The diagram is similar to Figure 4 The PTZ camera 10 and the external device 30 are provided. Figure 5 The PTZ camera 10 and the external device 30 illustrated in FIG. 1 are located at locations separated from each other and are connected via a communication network such as the Internet, for example, so that there is a possibility that a communication delay occurs.
[0116] The PTZ camera 10 captures a captured image 20 p including the prominent subject 21 at time ( t1 ), and transmits the captured image 20 p to the external device 30 .
[0117] However, due to the communication delay of the captured image 20p, the external device 30 receives the captured image 20p at time (t1+dt1). (dt1) is the time required for the communication process of sending the captured image 20p from the PTZ camera 10 to the external device 30 (=communication delay time).
[0118] The external device 30 detects the prominent subject 21 from the captured image 20p received from the PTZ camera 10 at time (t1+dt1), and performs processing to set and capture the whole body area including the prominent subject 21 from the imaging area 22 corresponding to the entire area of the captured image 20p.
[0119] Through the image analysis processing of the captured image 20 p , the external device 30 confirms that the prominent subject 21 is running toward the right direction in the figure, and sends a camera drive (PTZ drive) instruction to the PTZ camera 10 to move the current capturing direction to the right.
[0120] That is, execute Figure 5 The process (process x) shown in the diagram.
[0121] The timing at which the camera driving (PTZ driving) command is transmitted from the external device 30 to the PTZ camera 10 is time (t1+dt2).
[0122] A time dt2 has elapsed from the photographing timing (t1) at which an image is photographed.
[0123] The PTZ camera 10 receives a camera driving (PTZ driving) instruction from the external device 30 at time (t1+dt3).
[0124] A communication delay also occurs in this data communication. (dt3-dt2) corresponds to the time required for the communication process of sending a camera drive (PTZ drive) command from the external device 30 to the PTZ camera 10 (=communication delay time).
[0125] The PTZ camera 10 performs a pan operation to move the current shooting direction rightward according to the camera driving (PTZ driving) instruction received from the external device 30. The timing of the completion of the pan operation is time (t1+dt4), and the captured image after the completion of the pan operation is the captured image 20q.
[0126] A part of the body of the prominent subject 21 in the captured image 20 q is outside the capturing range of the PTZ camera 10 .
[0127] As described above, when the communication delay with the external device 30 is added to the delay attributable to the mechanical driving of the PTZ camera 10 , the possibility of occurrence of a tracking error or an image interception error of a prominent subject further increases.
[0128] The process of the present disclosure solves such a problem and enables more reliable execution of a process of cutting out and recording or distributing a partial image area including a prominent subject from a captured image of a PTZ camera while suppressing a decrease in image quality.
[0129] Hereinafter, details of the configuration and processing of the image processing apparatus of the present disclosure will be described.
[0130] [2. Processing performed by the image processing device of the present disclosure] Hereinafter, a process performed by the image processing apparatus of the present disclosure will be described.
[0131] Will refer to Figure 6 3 and subsequent figures to describe the processing performed by the image processing device of the present disclosure.
[0132] Figure 6 1 is a diagram for explaining a specific example of processing performed by the image processing apparatus of the present disclosure.
[0133] like Figure 6 As shown in the diagram, the image processing device according to the present disclosure performs control for changing the imaging ratio according to at least one of, for example, the moving speed or acceleration of a prominent subject as a tracking target, a processing delay time including a communication delay, and the like.
[0134] Figure 6 The following two image capturing examples of the PTZ camera 10 are illustrated.
[0135] (a) Example of image capture before the imaging ratio is changed (b) Example of image capture after changing the imaging ratio In (a) and (b), the same prominent subject 21 is photographed, and the prominent subject 21 is running and moving toward the right side in the figure.
[0136] Comparing (a) the image capturing example before the imaging ratio change and (b) the image capturing example after the imaging ratio change, the ratio between the size of the imaging area 22 corresponding to the entire area of the captured image 20 and the size of the clipped area 23 is different.
[0137] (b) The size ratio (imaging ratio) between the size of the imaging area 22b and the size of the cut-out area 23b in the image capturing example after the imaging ratio is changed is greater than the size ratio (imaging ratio) between the size of the imaging area 22a and the size of the cut-out area 23a in the image capturing example before the imaging ratio is changed (a).
[0138] Specifically, the size of the imaging region 22 b in the (b) image capturing example after the imaging ratio change is larger than the size of the imaging region 22 a in the (a) image capturing example before the imaging ratio change.
[0139] That is, (b) the captured images b and 20b (= the captured images with the imaging area 22b) after the imaging ratio is changed are images captured using a setting having a wider viewing angle than (a) the captured images a and 20a (= the captured images with the imaging area 22a) before the imaging ratio is changed.
[0140] As described above, the image processing device of the present disclosure changes the size ratio (imaging ratio) of the size of the imaging area 22 and the size of the clipped area based on whether the prominent subject 21 as the tracking target moves at a certain speed. Then, in the case where it is detected that the prominent subject 21 is moving at a predetermined speed or more, the change processing of the "imaging ratio" is performed so that the size ratio between the size of the imaging area 22 and the size of the clipped area 23 is greater than the size ratio in the case where it is detected that the prominent subject 21 is not moving at a predetermined speed or more. That is, the zoom of the PTZ camera 10 is adjusted to perform the setting of widening the shooting range, that is, the processing of widening the angle of view.
[0141] The “imaging ratio” is the ratio of the image area size to the clipping area size. That is, Imaging ratio = (imaging area size / cut-off area size).
[0142] The "imaging ratio" will refer to Figure 7 to describe.
[0143] Figure 7 Illustrations of (a) captured image examples and (b) imaging ratio diagrams.
[0144] In (a) the captured image example, a captured image 20 including a prominent subject 21 is illustrated. An imaging region 22 corresponds to the entire region of the captured image 20. A clipped region 23 is set in the imaging region 22. The clipped region 23 is set to a region including the entire body of the prominent subject 21, for example.
[0145] In the (b) imaging ratio explanation diagram, only the rectangular frames of the imaging area 22 and each clipped area 23 shown in the (a) captured image example are illustrated.
[0146] For example, Figure 7 As illustrated in (b), the imaging ratio can be calculated as the ratio between the height H of the imaging area 22 (=the entire image area of the captured image 20) and the height h of the clipped area 23.
[0147] Imaging ratio = (imaging area size / intercepted area size) = H / h.
[0148] Note that the aspect ratio of the imaging region 22 and the aspect ratio of the cutout region 23 are the same.
[0149] Figure 8The diagram illustrates the above Figure 6 Calculation example and comparison example of the imaging ratio of each captured image in (a) and (b).
[0150] like Figure 8 As shown in the diagram, (a) The imaging ratio of the image before the imaging ratio change is Imaging ratio = Ha / ha.
[0151] In addition, the imaging ratio in the image shooting example (b) after the imaging ratio is changed is Imaging ratio = Hb / hb.
[0152] Comparing the imaging ratios in these two image capture examples (a) and (b), (Ha / ha)<(Hb / hb). That is, the imaging ratio (Hb / hb) in the image capturing example after the imaging ratio change (b) is changed to a value (large ratio) greater than the imaging ratio (Ha / ha) in the image capturing example after the imaging ratio change (a).
[0153] As described above, the image processing device of the present disclosure performs control to increase the imaging ratio when, for example, a prominent subject moves at a certain speed. Specifically, the image processing device performs processing to widen the viewing angle by zoom adjustment of the PTZ camera.
[0154] This process expands the range in which the PTZ camera 10 can capture images, and can continue to follow and capture without losing the important subject 21 .
[0155] Fig. 9 Graphic instructions and Figure 8 Examples of different changing imaging ratios.
[0156] and Figure 8 similar, Fig. 9 The following two image capturing examples of the PTZ camera 10 are illustrated.
[0157] (a) Example of image capture before the imaging ratio is changed (b) Example of image capture after changing the imaging ratio In (a) and (b), the same prominent subject 21 is photographed, but Fig. 9 In the example illustrated in FIG. 2 , the main subject 21 is stationary.
[0158] like Fig. 9 As shown in the diagram, (a) The imaging ratio of the image before the imaging ratio change is Imaging ratio = Ha / ha.
[0159] In addition, the imaging ratio in the image shooting example (b) after the imaging ratio is changed is Imaging ratio = Hb / hb.
[0160] Comparing the imaging ratios in these two image capture examples (a) and (b), (Ha / ha)>(Hb / hb). That is, the imaging ratio (Hb / hb) in the image capturing example after the imaging ratio change (b) is changed to a value (small ratio) smaller than the imaging ratio (Ha / ha) in the image capturing example after the imaging ratio change (a).
[0161] As described above, the image processing device of the present disclosure controls to reduce the imaging ratio when the prominent subject is stationary or when the prominent subject stays within a predetermined range for a predetermined time or longer. Specifically, the process of reducing the viewing angle is performed by zoom adjustment of the PTZ camera.
[0162] This processing narrows the range in which the PTZ camera 10 can capture an image, and can increase the ratio of the clipped area 23 in the imaging area 22. That is, the ratio of the clipped area 23 including the prominent subject 21 increases, and as a result, the number of pixels in the image area of the clipped image 23 increases, and the image quality of the clipped image 23 can be improved.
[0163] [3. (Example 1) Example of changing the imaging ratio according to at least one of the speed or acceleration of the prominent subject] Next, a specific example of processing performed by the image processing apparatus of the present disclosure will be described.
[0164] The following embodiments will be described sequentially.
[0165] (Embodiment 1) Embodiment in which the imaging ratio is changed according to at least one of the speed or acceleration of the prominent subject (Example 2) Example of changing the imaging ratio according to the communication delay time between the camera and the external device (Example 3) Example of changing the imaging ratio according to the speed and acceleration of the prominent subject and the communication delay time between the camera and the external device First, as (Embodiment 1), an embodiment in which the imaging ratio is changed according to at least one of the speed or the acceleration of the prominent subject will be described.
[0166] Note that Example 1 will be further subdivided into the following examples.
[0167] (Example 1-1) Example of changing the imaging ratio according to the speed of the prominent subject (Example 1-2) Example of changing the imaging ratio according to the acceleration of the prominent subject (Example 1-3) Example of Changing the Imaging Ratio According to the Speed and Acceleration of the Prominent Subject (3-1. (Example 1-1) Example of changing the imaging ratio according to the speed of the prominent subject) First, an embodiment in which the imaging ratio is changed according to the speed of a prominent subject will be described.
[0168] Will refer to Fig.10 2 and subsequent figures describe an embodiment of changing the imaging ratio according to the speed of a prominent subject.
[0169] Fig.10 The graph illustrated in the figure is a graph in which the significant object speed (v) is set on the horizontal axis and the imaging ratio is set on the vertical axis.
[0170] Note that the significant subject speed (v) on the horizontal axis is, for example, the speed in the captured image of the PTZ camera 10, specifically, for example, the number of moving pixels of the significant subject per unit time, for example, the number of moving pixels per second (pixels / s).
[0171] As above reference Figure 7 As described above, the imaging ratio on the vertical axis is the ratio between the height H of the imaging area 22 corresponding to the entire area of the captured image 20 and the height h of the cut-out area 23. Imaging ratio = (imaging area size / intercepted area size) = H / h.
[0172] Fig.10 The graph illustrated in FIG. 1 is a graph illustrating an example of change control of the imaging ratio according to the velocity (v) of the prominent subject performed by the image processing device of the present disclosure.
[0173] As illustrated in the graph, the image processing device of the present disclosure changes the imaging ratio as follows according to the speed (v) of the prominent subject.
[0174] (a) When the speed of the main subject (v) = 0 to v1, the imaging ratio = rMIN (the minimum allowable imaging ratio) (b) When the speed (v) of the prominent subject is between v1 and v2, the imaging ratio is changed from rMIN (the minimum allowable imaging ratio) to rMAX (the maximum allowable imaging ratio) according to the speed (v) of the prominent subject. (c) When the speed of the main subject (v) = v2~, the imaging ratio = rMAX (the maximum allowable imaging ratio) Note that (b) when the speed (v) of the prominent subject = v1~v2, in the example of the curve graph illustrated in the figure, the imaging ratio is set to change linearly from rMIN (minimum allowable imaging ratio) to rMAX (maximum allowable imaging ratio), that is, the imaging ratio is set to change proportionally with the change in the speed (v) of the prominent subject, but this is an example, for example, the imaging ratio can be increased according to an ascending curve depicting a predetermined curve, or the imaging ratio can be changed exponentially as the speed (v) of the prominent subject increases.
[0175] like Fig.10 As shown in the diagram, (a) when the speed (v) of the significant subject = 0 to v1, that is, when the significant subject is stationary or moving at a speed less than a predetermined value, the imaging ratio is set to rMIN (the minimum allowable imaging ratio), and the difference (ratio) between the size of the imaging area 22 and the size of the cut-out area 23 is set to be smaller than the imaging ratio when the speed (v) of the significant subject is greater than v1.
[0176] Through this processing, the number of pixels of the clipped area 23 can be set larger, and the reduction in the image quality of the clipped image can be reduced.
[0177] On the other hand, (c) when the speed (v) of the prominent subject = v2~, that is, when the moving speed of the prominent subject is greater than a predetermined value, the imaging ratio is set to rMAX (the maximum allowable imaging ratio), and the difference (ratio) between the size of the imaging area 22 and the size of the cut-out area 23 is set to be greater than the imaging ratio when the speed (v) of the prominent subject is less than v2.
[0178] As a result, even in the case where the prominent subject moves quickly, the possibility that the prominent subject is out of the photographing range can be reduced, and reliable tracking processing of the prominent subject can be performed.
[0179] Note that rMIN (allowable minimum imaging ratio) and rMAX (allowable maximum imaging ratio) shown in the graph are predefined ratios.
[0180] rMIN (minimum allowable imaging ratio) and rMAX (maximum allowable imaging ratio) can be predefined fixed values, but can be determined according to the state of the scene of the captured image, the setting of the capture area, the minimum allowable number of pixels of the capture area 23 (preset minimum allowable number of pixels), etc.
[0181] Will refer to Fig.11 and Fig.12A specific example of determining rMIN (allowable minimum imaging ratio) and rMAX (allowable maximum imaging ratio) according to the state of the scene of the captured image, the setting of the clipped area, the allowable minimum number of pixels of the clipped area 23, etc. is described.
[0182] First, refer to Fig.11 Describes rMIN (minimum allowed imaging ratio).
[0183] rMIN (allowable minimum imaging ratio) is a value indicating a lower limit ratio of how small the imaging area can be set relative to the clipped area 23 .
[0184] This rMIN (allowable minimum imaging ratio) is predetermined before image capture starts, but there are various methods for determining this value, for example, Fig.11 Various determination processing examples are shown in the figure.
[0185] Fig.11 (a) is a diagram illustrating a determination example of rMIN (allowable minimum imaging ratio) according to the lens size of a prominent subject, that is, a setting mode of a cropped area for the prominent subject.
[0186] Note that the setting of the cropped area for the prominent subject is also determined in advance before image capturing starts.
[0187] For example, Fig.11 As shown in the diagram (a1) of FIG. 1 , when the setting of the clipped area for the prominent subject is a setting of a full-body shot including the whole body of the prominent subject, rMIN (the minimum permissible imaging ratio) is the ratio between the height Ha1 of the imaging area 22 and the height ha1 of the clipped area 23, rMIN (minimum permissible imaging ratio) = Ha1 / ha1, And the value (ratio) of rMIN (allowable minimum imaging ratio) is set relatively small (compared to the case of lens setting including only a part of the prominent subject).
[0188] On the other hand, Fig.11 As shown in the diagram (a2) of FIG. 1 , in the case of an upper body shot including only the upper body of a prominent subject, rMIN (minimum permissible imaging ratio) is the ratio between the height Ha2 of the imaging area 22 and the height ha2 of the clipped area 23, rMIN (minimum permissible imaging ratio) = Ha2 / ha2, And the value (ratio) of rMIN (allowable minimum imaging ratio) is set relatively large (compared to the case of lens setting including an area larger than the upper body of the prominent subject).
[0189] In this way, rMIN (the minimum allowable imaging ratio) changes according to the lens size, that is, the size of the body area of the prominent subject to be included in the cropped area on the captured image (the ratio of the body area to be included in the cropped area to the imaging area).
[0190] In addition, if Fig.11 As shown in the diagram in (b), rMIN (minimum allowable imaging ratio) can be determined according to the scene.
[0191] For example, Fig.11 (b1) is a scene in which the motion of the prominent subject is intense (large motion or large amount of movement).
[0192] In this case, rMIN (the minimum permissible imaging ratio) is the ratio between the height Hb1 of the imaging area 22 and the height hb1 of the cut-off area 23, rMIN (minimum permissible imaging ratio) = Hb1 / hb1, And the value (ratio) of rMIN (allowable minimum imaging ratio) is set relatively large (compared to the case of a scene in which the movement of the prominent subject is small).
[0193] On the other hand, Fig.11 As shown in the diagram (b2) of FIG. 1 , in the case of a scene where the movement of the prominent subject is small, rMIN (the minimum permissible imaging ratio) is the ratio between the height Hb2 of the imaging area 22 and the height hb2 of the clipping area 23, rMIN (minimum permissible imaging ratio) = Hb2 / hb2, And the value (ratio) of rMIN (allowable minimum imaging ratio) is set relatively small (compared to the case of a scene in which the movement of a prominent subject is large).
[0194] In this way, rMIN (the minimum permissible imaging ratio) can be set to vary according to the scene, that is, according to the movement of the prominent subject.
[0195] Next, we will refer to Fig.12 describe Fig.10 An example of rMAX (maximum permissible imaging ratio) illustrated in the graph of FIG. rMAX (maximum permissible imaging ratio) is a value indicating an upper limit ratio of how large the imaging area can be set relative to the clipped area 23.
[0196] For example, rMAX (allowable maximum imaging ratio) is determined based on an allowable value as a resolution of a clipped image clipped based on the clipped area 23 , that is, an allowable minimum number of pixels included in the clipped area 23 .
[0197] Specifically, Fig.12As shown in the diagram, for example, when the number of pixels (resolution) of the imaging area 22 corresponding to the entire captured image is 3840×2160, and the minimum number of pixels to be included in the clipped area 23, that is, the minimum allowable number of pixels is 1280×720, rMAX (maximum allowable imaging ratio) is calculated as follows.
[0198] rMAX (maximum permissible imaging ratio) = (number of pixels in height of imaging area 22) / (number of pixels in height of clipping area 23) = 2160 / 720 = 3 As described above, when the number of pixels (resolution) of the imaging area 22 is 3840×2160 and the minimum permissible number of pixels of the clipping area 23 is 1280×720, rMAX (maximum allowable imaging ratio) is 3.
[0199] In this way, rMAX (allowable maximum imaging ratio) is calculated based on the minimum number of pixels to be included in the clipped area 23, that is, the allowable minimum number of pixels.
[0200] Note, reference Fig.11 and Fig.12 The described determination examples of rMIN (allowable minimum imaging ratio) and rMAX (allowable maximum imaging ratio) are examples, and determination may be performed using other methods. In addition, as described above, predefined fixed values may be used.
[0201] return Fig.10 , an example of change control of the imaging ratio according to the speed (v) of the prominent subject performed by the image processing device of the present disclosure will be described again.
[0202] like Fig.10 As shown in the graph of , the image processing device according to Embodiment 1-1 of the present disclosure changes the imaging ratio as follows according to the speed (v) of the prominent subject.
[0203] (a) When the speed of the main subject (v) = 0 to v1, the imaging ratio = rMIN (the minimum allowable imaging ratio) (b) When the speed of the prominent subject (v) = v1-v2, the imaging ratio changes from rMIN (the minimum permissible imaging ratio) to rMAX (the maximum permissible imaging ratio) (c) When the speed of the main subject (v) = v2~, the imaging ratio = rMAX (the maximum allowable imaging ratio) Based on Fig.10The external device 30 uses the imaging ratio calculated by the graph illustrated in FIG. 1 as the optimal imaging ratio (imaging ratio for cropped image generation), and causes the PTZ camera 10 to capture an image according to the imaging area size determined based on the calculated optimal imaging ratio.
[0204] Note that the "optimal imaging ratio (imaging ratio for cropped image generation)" is an indicator value for reliably performing cropped image generation processing including a prominent subject compared to the prior art, and is an indicator value for setting the image quality of the cropped image containing the prominent subject to a predetermined image quality or higher.
[0205] For example, the velocity (v) of the current prominent subject 21 is Fig.10 The current speed (vt) is shown graphically in the graph.
[0206] use Fig.10 The graph shown in the figure is used to calculate the imaging ratio corresponding to the current speed (vt) of the prominent subject 21 as the optimal imaging ratio. The optimal imaging ratio corresponding to the current speed (vt) of the prominent subject 21 is Fig.10 The optimal imaging ratio (r1) is shown in the figure.
[0207] The external device 30 causes the PTZ camera 10 to Fig.10 The graph shown in the illustration calculates the optimal imaging ratio (r1) and determines the imaging area size to capture the image.
[0208] Specifically, the PTZ camera 10 is caused to perform image capturing processing while setting the height H of the imaging area 22 (=the entire image area of the captured image 20) to H=r1×h.
[0209] Note that h is the height of the cutout region 23 .
[0210] By causing the PTZ camera 10 to perform image capture with such settings, the captured area 23 including the prominent subject 21 can be more reliably captured from the captured image 20 (imaging area 22) of the PTZ camera 10, and degradation in image quality of the captured image can also be mitigated.
[0211] In this embodiment 1-1, Fig.10 As shown in the diagram, (a) when the speed (v) of the significant subject = 0 to v1, that is, when the significant subject is stationary or the moving speed is less than a predetermined value, the imaging ratio is set to rMIN (the minimum allowable imaging ratio), and the difference (ratio) between the size of the imaging area 22 and the size of the cut-out area 23 is set to be smaller than the difference (ratio) when the moving speed of the significant subject is greater than a predetermined value.
[0212] Through this processing, the number of pixels of the clipped area 23 can be set larger, and the reduction in the image quality of the clipped image can be reduced.
[0213] On the other hand, (c) when the speed (v) of the prominent subject = v2~, that is, when the moving speed of the prominent subject is greater than a predetermined value, the imaging ratio is set to rMAX (the maximum allowable imaging ratio), and the difference (ratio) between the size of the imaging area 22 and the size of the cut-out area 23 is set to be greater than the difference (ratio) when the moving speed of the prominent subject is less than a predetermined value.
[0214] As a result, even in the case where the prominent subject moves quickly, the possibility that the prominent subject is out of the photographing range can be reduced, and reliable tracking processing of the prominent subject can be performed.
[0215] (3-2. (Example 1-2) Example of changing the imaging ratio according to the acceleration of the prominent subject) Next, an embodiment in which the imaging ratio is changed according to the acceleration of the prominent subject will be described.
[0216] Will refer to Fig.13 An embodiment in which the imaging ratio is changed according to the acceleration of a prominent subject is described.
[0217] Fig.13 The graph illustrated in the figure is a graph in which the significant subject acceleration (α) is set on the horizontal axis and the imaging ratio is set on the vertical axis.
[0218] Note that the dominant subject acceleration (α) on the horizontal axis is, for example, the acceleration in an image captured by the PTZ camera 10 .
[0219] As above reference Figure 7 As described above, the imaging ratio on the vertical axis is the ratio between the height H of the imaging area 22 corresponding to the entire area of the captured image 20 and the height h of the cut-out area 23. Imaging ratio = (imaging area size / intercepted area size) = H / h.
[0220] Fig.13 The graph illustrated in FIG. 2 is a graph illustrating an example of change control of the imaging ratio according to the acceleration (α) of the prominent subject performed by the image processing device of the present disclosure.
[0221] As illustrated in the graph, the image processing device of the present disclosure changes the imaging ratio as follows according to the acceleration (α) of the prominent subject.
[0222] (a) When the acceleration (α) of the main subject is 0 to α1, the imaging ratio is rMIN (the minimum allowable imaging ratio). (b) When the acceleration (α) of the prominent subject is between α1 and α2, the imaging ratio is changed from rMIN (the minimum allowable imaging ratio) to rMAX (the maximum allowable imaging ratio) according to the acceleration (a) of the prominent subject. (c) When the acceleration (α) of the main subject is α2~, the imaging ratio is rMAX (the maximum allowable imaging ratio). Note that (b) when the acceleration (α) of the prominent subject = α1~α2, in the example of the graph illustrated in the figure, the imaging ratio is set to change linearly from rMIN (the minimum allowable imaging ratio) to rMAX (the maximum allowable imaging ratio), that is, the imaging ratio is set to change in proportion to the change in the acceleration (α) of the prominent subject, but this is an example, and for example, the imaging ratio may be increased according to an ascending curve depicting a predetermined curve, or the imaging ratio may be set to change exponentially as the acceleration (α) of the prominent subject increases.
[0223] In this embodiment 1-2, according to Fig.13 The external device 30 uses the imaging ratio calculated by the graph illustrated in FIG. 1 as the optimal imaging ratio (imaging ratio for cropped image generation), and causes the PTZ camera 10 to capture an image according to the imaging area size determined based on the calculated optimal imaging ratio.
[0224] For example, assuming that the acceleration (α) of the current prominent subject 21 is Fig.13 The current acceleration (αt) is shown in the graph shown in the figure.
[0225] use Fig.13 The graph shown in the figure is used to calculate the imaging ratio corresponding to the current acceleration (αt) of the prominent subject 21 as the optimal imaging ratio. The optimal imaging ratio corresponding to the current acceleration (αt) of the prominent subject 21 is Fig.13 The optimal imaging ratio (r2) is shown in the figure.
[0226] The external device 30 causes the PTZ camera 10 to Fig.13 The graph shown in the illustration calculates the optimal imaging ratio (r2) and determines the imaging area size to capture the image.
[0227] Specifically, the PTZ camera 10 is caused to perform image capturing processing while setting the height H of the imaging area 22 (=the entire image area of the captured image 20) to H=r2×h.
[0228] Note that h is the height of the cutout region 23 .
[0229] By causing the PTZ camera 10 to perform image capture with such settings, the captured area 23 including the prominent subject 21 can be more reliably captured from the captured image 20 (imaging area 22) of the PTZ camera 10, and degradation in image quality of the captured image can also be mitigated.
[0230] like Fig.13 As shown in the diagram, in the image processing device of the present embodiment 1-2, (a) when the acceleration (α) of the prominent subject is 0 to α1, that is, when the prominent subject is stationary or the acceleration is less than a predetermined value, the imaging ratio is set to rMIN (the minimum allowable imaging ratio), and the difference (ratio) between the size of the imaging area 22 and the size of the cut-out area 23 is set to be smaller than the difference (ratio) when the moving acceleration of the prominent subject is greater than a predetermined value.
[0231] Through this processing, the number of pixels of the clipped area 23 can be set larger, and the reduction in the image quality of the clipped image can be reduced.
[0232] On the other hand, (c) when the acceleration (α) of the prominent subject = α2~, that is, when the moving acceleration of the prominent subject is greater than a predetermined value, the imaging ratio is set to rMAX (the maximum allowable imaging ratio), and the difference (ratio) between the size of the imaging area 22 and the size of the cut-out area 23 is set to be greater than the difference (ratio) when the moving acceleration of the prominent subject is less than a predetermined value.
[0233] As a result, even in a case where the moving speed of the prominent subject increases rapidly, the possibility that the prominent subject goes beyond the photographing range can be reduced, and reliable tracking processing of the prominent subject can be performed.
[0234] Note that, as described above, rMIN (minimum allowable imaging ratio) and rMAX (maximum allowable imaging ratio) illustrated in the graph are predefined ratios.
[0235] For example, rMIN (the minimum permissible imaging ratio) and rMAX (the maximum permissible imaging ratio) may be predefined fixed values. Fig.11 and Fig.12 As described above, it can be determined based on the state of the scene of the captured image, the setting of the clipping area, the minimum allowable number of pixels of the clipping area 23, etc.
[0236] (3-3. (Example 1-3) Example of changing the imaging ratio according to the speed and acceleration of the prominent subject) Next, an embodiment in which the imaging ratio is changed according to the speed and acceleration of the prominent subject will be described.
[0237] Will refer to Fig.14An embodiment in which the imaging ratio is changed according to the speed and acceleration of a prominent subject is described.
[0238] This embodiment 1-3 is achieved by combining the above reference Fig.10 Embodiment 1-1 described above, that is, an embodiment in which the imaging ratio is changed according to the speed (v) of the prominent subject, and Fig.13 Embodiments 1 and 2 described above are obtained by combining the two embodiments of the embodiment in which the imaging ratio is changed according to the acceleration (α) of the prominent subject.
[0239] Fig.14 (Process A) Illustrated in the above reference Fig.10 The graph described in the embodiment 1-1, that is, the embodiment in which the imaging ratio is changed according to the speed (v) of the significant subject, is a graph in which the speed (v) of the significant subject is set on the horizontal axis and the imaging ratio is set on the vertical axis.
[0240] Fig.14 (Process B) Illustrated in the above reference Fig.13 The graph described in the embodiment 1-2, that is, the embodiment in which the imaging ratio is changed according to the acceleration (α) of the significant subject, is a graph in which the acceleration (α) of the significant subject is set on the horizontal axis and the imaging ratio is set on the vertical axis.
[0241] In the present embodiment 1-3, first, the optimum imaging ratio (imaging ratio for clipped image generation) corresponding to the current velocity (v) and acceleration (α) of the significant subject 21 is calculated respectively using the two graphs.
[0242] First, as (Processing A), the optimum imaging ratio (r1) corresponding to the current speed (vt) of the current prominent subject 21 is acquired.
[0243] The velocity (v) of the current prominent subject 21 is defined as Fig.14 The current velocity (vt) is graphically shown in the graph of (Process A).
[0244] When using Fig.14 When the graph illustrated in (Processing A) calculates the optimal imaging ratio corresponding to the current speed (vt) of the prominent subject 21, the optimal imaging ratio corresponding to the current speed (vt) of the prominent subject 21 is Fig.14 Optimal imaging ratio (r1) shown graphically in (Treatment A).
[0245] Next, as (processing B), the optimum imaging ratio (r2) corresponding to the current acceleration (αt) of the current prominent subject 21 is acquired.
[0246] The acceleration (α) of the current prominent subject 21 is defined as Fig.14 The current acceleration (αt) is illustrated in the graph of (Process B).
[0247] When using Fig.14 When the graph illustrated in (Processing B) calculates the optimal imaging ratio corresponding to the current acceleration (αt) of the prominent subject 21, the optimal imaging ratio corresponding to the current acceleration (αt) of the prominent subject 21 is Fig.14 The optimal imaging ratio (r2) is shown graphically in (Treatment B).
[0248] Next, (process C) is executed.
[0249] In (Processing C), a process of comparing the speed-corresponding optimal imaging ratio r1 and the acceleration-corresponding optimal imaging ratio r2 and selecting the larger value as the final optimal imaging ratio is performed.
[0250] First, the optimal imaging ratio r1 corresponding to the speed and the optimal imaging ratio r2 corresponding to the acceleration are compared.
[0251] In the example shown in the diagram, r1>r2, Therefore, a larger value, ie, the speed corresponding optimal imaging ratio r1 is selected as the final optimal imaging ratio.
[0252] That is, the final optimal imaging ratio = r1.
[0253] The external device 30 causes the PTZ camera 10 to capture an image according to the imaging area size calculated based on the final optimal imaging ratio=r1.
[0254] Specifically, the image capturing process is performed in a case where the height H of the imaging area 22 (=the entire image area of the captured image 20) is H=r1×h.
[0255] Note that h is the height of the cutout region 23 .
[0256] By causing the PTZ camera 10 to perform image capture with such settings, the captured area 23 including the prominent subject 21 can be more reliably captured from the captured image 20 (imaging area 22) of the PTZ camera 10, and degradation in image quality of the captured image can also be mitigated.
[0257] [4. Processing sequence executed by the image processing device of Example 1] Next, a processing sequence executed by the above-described image processing apparatus according to Embodiment 1 of the present disclosure will be described.
[0258] The processing sequence of the following embodiment will be sequentially described.
[0259] (Example 1-1) Example of changing the imaging ratio according to the speed of the prominent subject (Example 1-2) Example of changing the imaging ratio according to the acceleration of the prominent subject (Example 1-3) Example of Changing the Imaging Ratio According to the Speed and Acceleration of the Prominent Subject Note that the image processing device of the present disclosure is, for example, configured Figure 2 The image processing system illustrated in the figure is an external device 30 or a PTZ camera 10.
[0260] As described later, the process of the present disclosure may be executed by the PTZ camera 10 alone, or may be configured to be executed using a combination of the PTZ camera 10 and the external device 30 .
[0261] In the following, as a representative processing example, as shown in FIG. Figure 2 Hereinafter, a processing example will be described in which the external device 30 receives a captured image of the PTZ camera 10 , the external device 30 performs image analysis and image interception processing, and controls (pan, tilt, and zoom driving) the PTZ camera 10 .
[0262] The following description will be given assuming that the processing is performed by the external device 30. Fig.15 However, as described above, the processing may also be performed in the PTZ camera 10.
[0263] First, refer to Fig.15 The flowchart shown in the figure, (Example 1-1) Example of changing the imaging ratio according to the speed of the prominent subject A processing sequence performed by the image processing device (external device 30) according to the present embodiment 1-1 will be described.
[0264] Note that the processing according to the flow described below can be executed according to a program stored in a storage unit of the image processing apparatus, for example, and can be executed under the control of a control unit having a program execution function such as a CPU. Fig.15 The details of the processing of each step of the process shown in the diagram.
[0265] (Step S101) First, in step S101 , the image processing apparatus of the present disclosure calculates the velocity (v) of the prominent subject 21 .
[0266] Image processing device, that is, Figure 2The external device 30 illustrated in FIG. 1 receives a captured image (moving image) from the PTZ camera 10 and calculates a velocity (v) of the prominent subject by analyzing changes in the position of the prominent subject 21 on the captured image in each image frame constituting the moving image.
[0267] Specifically, for example, the number of moving pixels of the prominent object per unit time, for example, the number of moving pixels per second (pixel / s) is calculated.
[0268] (Step S102) Next, in step S102 , the image processing apparatus according to the present disclosure calculates the optimum imaging ratio (imaging ratio for clipped image generation) corresponding to the speed (v) of the current prominent subject calculated in step S101 based on the speed-imaging ratio correspondence data.
[0269] This process is referenced above Fig.10 The speed-to-image ratio data are Fig.10 The curve diagram shown in the figure. Fig.10 As described above, the speed-imaging ratio correspondence data is a graph that defines the optimal imaging ratio according to the speed (v) of the significant subject as follows.
[0270] (a) When the speed of the main subject (v) = 0 ~ v1, the optimal imaging ratio = rMIN (the minimum allowable imaging ratio) (b) When the speed (v) of the prominent subject = v1 to v2, the optimum imaging ratio is changed from rMIN (the minimum allowable imaging ratio) to rMAX (the maximum allowable imaging ratio) according to the speed (v) of the prominent subject.
[0271] (c) When the speed of the main subject (v) = v2~, the optimal imaging ratio = rMAX (the maximum allowable imaging ratio) use Fig.10 The graph shown in the diagram in FIG. 2 calculates the imaging ratio corresponding to the current speed (v) of the prominent subject 21 as the optimal imaging ratio. Fig.10 In the example shown in the diagram, the optimal imaging ratio corresponding to the current speed (vt) of the prominent subject 21 is Fig.10 The optimal imaging ratio (r1) is shown in the figure.
[0272] Note that the optimal imaging ratio calculated in step S102 is Fig.10 The graph shown in the figure shows values ranging from rMIN (minimum allowable imaging ratio) to rMAX (maximum allowable imaging ratio).
[0273] (Step S103) Next, in step S103 , the image processing apparatus determines whether the difference (ratio) between the optimum imaging ratio calculated in step S102 and the current imaging ratio is smaller than a prescribed threshold value.
[0274] In the case where the difference (ratio) between the optimal imaging ratio calculated in step S102 and the current imaging ratio is smaller than the specified threshold value, the determination in step S103 is yes, and then it is determined in step S107 whether the shooting is finished. In the case where the shooting is not finished, the process proceeds to the processing of the next processing frame without performing the zoom control of the camera (PTZ camera 10).
[0275] This is a process for preventing fine control from being frequently performed for each image frame and the captured image from becoming unstable if zoom control is performed before the difference (ratio) between the optimum imaging ratio calculated in step S102 and the current imaging ratio becomes smaller than a specified threshold.
[0276] As described above, in a case where the difference (ratio) between the optimum imaging ratio calculated in step S102 and the current imaging ratio is smaller than a prescribed threshold, the determination processing in step S103 is performed to continue stable image capturing without performing control.
[0277] On the other hand, in a case where the difference (ratio) between the optimum imaging ratio calculated in step S102 and the current imaging ratio is equal to or greater than the prescribed threshold, the determination in step S103 is NO, and the process proceeds to step S104.
[0278] Note that as the designated threshold, a value predefined in the image processing device is used. For example, by setting the designated threshold to about 5% in advance, it is possible to determine whether to perform zoom control based on whether the difference (ratio) between the optimal imaging ratio calculated in step S102 and the current imaging ratio is less than the designated threshold (e.g., 5%). For example, when the designated threshold is 5%, the optimal imaging ratio calculated in step S102 is 2.2 times (220%), and the current imaging ratio is 2.0 times (200%), the difference between the calculated optimal imaging ratio and the current imaging ratio is 0.1 (10%), which is greater than the designated threshold of 5%, and thus zoom control is performed.
[0279] In other words, when the following (Formula 1) is satisfied, zoom control is not performed.
[0280] Current imaging ratio - specified threshold < calculated optimal imaging ratio < current imaging ratio + specified threshold ... (Formula 1) On the other hand, when the following (Formula 2) or (Formula 3) is satisfied, zoom control is performed.
[0281] The calculated optimal imaging ratio ≥ the current imaging ratio + the specified threshold... (Formula 2) The calculated optimal imaging ratio ≤ the current imaging ratio - the specified threshold value (Formula 3) (Step S104) In step S103, in a case where the difference (ratio) between the optimum imaging ratio calculated in step S102 and the current imaging ratio is equal to or greater than a prescribed threshold, the process proceeds to step S104.
[0282] In this case, in step S104, the image processing apparatus determines whether the optimum imaging ratio calculated in step S102 is larger or smaller than the current imaging ratio.
[0283] In a case where the optimum imaging ratio is greater than the current imaging ratio, the process proceeds to step S105.
[0284] On the other hand, in a case where the optimum imaging ratio is smaller than the current imaging ratio, the process proceeds to step S106.
[0285] (Step S105) In the case where it is determined in step S104 that the optimum imaging ratio calculated in step S102 is greater than the current imaging ratio, the process proceeds to step S105 and the following process is performed.
[0286] In this case, in step S105, the image processing apparatus performs a process of widening the angle of view by adjusting the zoom (angle of view) of the camera so as to increase the current imaging ratio to the optimal imaging ratio.
[0287] Specifically, for example, in Figure 2 In the system configuration illustrated in FIG. 1 , the external device 30 outputs a camera driving instruction to the PTZ camera 10 to adjust the zoom (angle of view) of the camera to widen the angle of view so as to increase the current imaging ratio to the optimal imaging ratio.
[0288] This process corresponds to, for example, the above reference Figure 8 Description of the process.
[0289] That is, the processing corresponds to Figure 8 The zoom (angle of view) state before the imaging ratio change shown in the diagram in (a) is changed to Figure 8 Processing of the zoom (angle of view) state after the imaging ratio is changed as illustrated in (b).
[0290] Figure 8 The captured image b and the captured area 20b after the imaging ratio change in (b) are in the form of images with a ratio of Figure 8 The captured image a before the imaging ratio is changed in (a) and the image captured when the imaging area 20a is set with a wide viewing angle.
[0291] As described above, when it is determined that the optimal imaging ratio calculated in step S102 is greater than the current imaging ratio, in step S105 , the zoom of the PTZ camera 10 is adjusted to make settings for widening the shooting range, that is, processing for widening the viewing angle is performed.
[0292] (Step S106) On the other hand, in the event that it is determined in step S104 that the optimum imaging ratio calculated in step S102 is smaller than the current imaging ratio, the process proceeds to step S106 , and the following processing is performed.
[0293] In this case, in step S106, the image processing apparatus performs a process of narrowing the angle of view by adjusting the zoom (angle of view) of the camera so as to reduce the current imaging ratio to the optimal imaging ratio.
[0294] Specifically, for example, in Figure 2 In the system configuration illustrated in FIG. 1 , the external device 30 outputs a camera driving instruction to the PTZ camera 10 to reduce the viewing angle by adjusting the zoom (viewing angle) of the camera so as to reduce the current imaging ratio to the optimal imaging ratio.
[0295] This process corresponds to, for example, the above reference Fig. 9 Description of the process.
[0296] That is, the processing corresponds to Fig. 9 The zoom (angle of view) state before the imaging ratio change shown in the diagram in (a) is changed to Fig. 9 Processing of the zoom (angle of view) state after the imaging ratio is changed as illustrated in (b).
[0297] Fig. 9 The captured image b and the captured area 20b after the imaging ratio change in (b) are in the form of images with a ratio of Fig. 9 The captured image a before the imaging ratio is changed in (a) and the image captured when the imaging area 20a is set with a narrow viewing angle.
[0298] As described above, when it is determined that the optimal imaging ratio calculated in step S102 is smaller than the current imaging ratio, in step S106 , the zoom of the PTZ camera 10 is adjusted to perform settings for narrowing the shooting range, that is, processing for narrowing the viewing angle.
[0299] (Step S107) After the processing of step S105 and step S106 , the processing proceeds to step S107 , and in step S107 , it is determined whether the photographing processing is ended.
[0300] In a case where the photographing process is not completed, the process returns to step S101 , and the process for the next photographed image frame is performed.
[0301] When the photographing process is completed, the process ends.
[0302] By executing these processes, the success probability of the process of cutting out a cutout region including a prominent subject from a captured image of the PTZ camera 10 can be improved, and degradation in the image quality of the cutout image can be mitigated.
[0303] Next, refer to Fig.16 The flowchart shown in the figure, (Example 1-2) Example of changing the imaging ratio according to the acceleration of the prominent subject A processing sequence executed by the image processing device (external device 30) according to the present embodiment 1-2 will be described.
[0304] (Step S121) First, in step S121 , the image processing apparatus of the present disclosure calculates the acceleration (α) of the prominent subject 21 .
[0305] Image processing device, i.e. Figure 2 The external device 30 illustrated in FIG. 1 receives a captured image (moving image) from the PTZ camera 10 and calculates acceleration (α) of the prominent subject by analyzing changes in the position of the prominent subject 21 on the captured image in each image frame constituting the moving image.
[0306] (Step S122) Next, in step S122 , the image processing apparatus of the present disclosure calculates the optimum imaging ratio (imaging ratio for clipped image generation) corresponding to the acceleration (α) of the current prominent subject calculated in step S121 based on the acceleration-imaging ratio correspondence data.
[0307] This process is referenced above Fig.13 The acceleration-imaging ratio corresponds to the data Fig.13 The curve diagram shown in the figure. Fig.13 As described above, the acceleration-imaging ratio correspondence data is a graph that defines the optimal imaging ratio as follows according to the acceleration (α) of the significant subject.
[0308] (a) When the acceleration (α) of the main subject is 0 to α1, the optimal imaging ratio is rMIN (the minimum allowable imaging ratio). (b) When the acceleration (α) of the prominent subject is between α1 and α2, the optimum imaging ratio is changed from rMIN (the minimum allowable imaging ratio) to rMAX (the maximum allowable imaging ratio) according to the acceleration (α) of the prominent subject.
[0309] (c) When the acceleration (α) of the main subject is α2~, the optimal imaging ratio is rMAX (the maximum allowable imaging ratio). use Fig.13 The graph shown in the figure is used to calculate the imaging ratio corresponding to the current acceleration (α) of the prominent subject 21 as the optimal imaging ratio. Fig.13 In the example shown in the diagram, the optimal imaging ratio corresponding to the current acceleration (αt) of the prominent subject 21 is Fig.13 The optimal imaging ratio (r2) is shown in the figure.
[0310] Note that the optimal imaging ratio calculated in step S122 is Fig.10 The graph shown in the figure shows values ranging from rMIN (minimum allowable imaging ratio) to rMAX (maximum allowable imaging ratio).
[0311] (Step S123) Next, in step S123, the image processing apparatus determines whether the difference (ratio) between the optimum imaging ratio calculated in step S122 and the current imaging ratio is smaller than a prescribed threshold value.
[0312] In the case where the difference (ratio) between the optimal imaging ratio calculated in step S122 and the current imaging ratio is smaller than the specified threshold value, the determination in step S123 is yes, and then it is determined in step S127 whether the shooting is finished. In the case where the shooting is not finished, the process proceeds to the processing of the next processing frame without performing the zoom control of the camera (PTZ camera 10).
[0313] This is a process for preventing fine control from being frequently performed for each image frame and the captured image from becoming unstable if zoom control is performed before the difference (ratio) between the optimum imaging ratio calculated in step S122 and the current imaging ratio becomes smaller than a specified threshold.
[0314] As described above, when the difference (ratio) between the optimum imaging ratio calculated in step S122 and the current imaging ratio is smaller than the prescribed threshold, the determination process in step S123 is performed to continue stable image capturing without performing control.
[0315] On the other hand, in a case where the difference (ratio) between the optimum imaging ratio calculated in step S122 and the current imaging ratio is equal to or larger than the prescribed threshold, the determination in step S123 is NO, and the processing proceeds to step S124.
[0316] Note that as the specified threshold, a value predefined in the image processing apparatus is used. For example, the process of determining the difference (ratio) between the optimal imaging ratio calculated in step S122 and the current imaging ratio can be performed by using a value of approximately 5% of the current imaging ratio.
[0317] That is, the process of determining whether to perform zoom control is performed based on whether the difference (ratio) between the optimal imaging ratio calculated in step S122 and the current imaging ratio is smaller than a specified threshold value (eg, 5%).
[0318] (Step S124) In step S123, in a case where the difference (ratio) between the optimum imaging ratio calculated in step S122 and the current imaging ratio is equal to or greater than a prescribed threshold, the process proceeds to step S124.
[0319] In this case, in step S124, the image processing apparatus determines whether the optimum imaging ratio calculated in step S122 is larger or smaller than the current imaging ratio.
[0320] In a case where the optimum imaging ratio is greater than the current imaging ratio, the process proceeds to step S125.
[0321] On the other hand, in a case where the optimum imaging ratio is smaller than the current imaging ratio, the process proceeds to step S126.
[0322] (Step S125) In a case where it is determined in step S124 that the optimum imaging ratio calculated in step S122 is greater than the current imaging ratio, the process proceeds to step S125 , and the following processing is performed.
[0323] In this case, in step S125, the image processing apparatus performs a process of widening the angle of view by adjusting the zoom (angle of view) of the camera so as to increase the current imaging ratio to the optimal imaging ratio.
[0324] Specifically, for example, in Figure 2 In the system configuration illustrated in FIG. 1 , the external device 30 outputs a camera driving instruction to the PTZ camera 10 to adjust the zoom (angle of view) of the camera to widen the angle of view so as to increase the current imaging ratio to the optimal imaging ratio.
[0325] This process corresponds to, for example, the above reference Figure 8 Description of the process.
[0326] That is, the processing corresponds to Figure 8 The zoom (angle of view) state before the imaging ratio change shown in the diagram in (a) is changed to Figure 8 Processing of the zoom (angle of view) state after the imaging ratio is changed as illustrated in (b).
[0327] Figure 8 The captured image b and the captured area 20b after the imaging ratio change in (b) are in the form of images with a ratio of Figure 8 The captured image a before the imaging ratio is changed in (a) and the image captured when the imaging area 20a is set with a wide viewing angle.
[0328] As described above, when it is determined that the optimal imaging ratio calculated in step S122 is greater than the current imaging ratio, in step S125, the zoom of the PTZ camera 10 is adjusted to make settings for widening the shooting range, that is, processing for widening the viewing angle is performed.
[0329] (Step S126) On the other hand, in a case where it is determined in step S124 that the optimum imaging ratio calculated in step S122 is smaller than the current imaging ratio, the process proceeds to step S126 , and the following processing is performed.
[0330] In this case, in step S126, the image processing apparatus performs a process of narrowing the angle of view by adjusting the zoom (angle of view) of the camera so as to reduce the current imaging ratio to the optimal imaging ratio.
[0331] Specifically, for example, in Figure 2 In the system configuration illustrated in FIG. 1 , the external device 30 outputs a camera driving instruction to the PTZ camera 10 to reduce the viewing angle by adjusting the zoom (viewing angle) of the camera so as to reduce the current imaging ratio to the optimal imaging ratio.
[0332] This process corresponds to, for example, the above reference Fig. 9 Description of the process.
[0333] That is, the processing corresponds to Fig. 9 The zoom (angle of view) state before the imaging ratio change shown in the diagram in (a) is changed to Fig. 9 Processing of the zoom (angle of view) state after the imaging ratio is changed as illustrated in (b).
[0334] Fig. 9 The captured image b and the captured area 20b after the imaging ratio change in (b) are in the form of images with a ratio of Fig. 9 The captured image a before the imaging ratio is changed in (a) and the image captured when the imaging area 20a is set with a narrow viewing angle.
[0335] As described above, when it is determined that the optimal imaging ratio calculated in step S122 is smaller than the current imaging ratio, in step S126, the zoom of the PTZ camera 10 is adjusted to perform settings for narrowing the shooting range, that is, processing for narrowing the viewing angle.
[0336] (Step S127) After the processing of step S125 and step S126, the processing proceeds to step S127, and in step S127, it is determined whether the photographing processing is ended.
[0337] In a case where the photographing process is not completed, the process returns to step S121 , and the process for the next photographed image frame is performed.
[0338] When the photographing process is completed, the process ends.
[0339] By executing these processes, the success probability of the process of cutting out a cutout region including a prominent subject from a captured image of the PTZ camera 10 can be improved, and degradation in the image quality of the cutout image can be mitigated.
[0340] Next, refer to Fig.17 and Fig.18 The flowchart shown in the figure, (Example 1-3) Example of Changing the Imaging Ratio According to the Speed and Acceleration of the Prominent Subject A processing sequence executed by the image processing device (external device 30) according to the present embodiment 1-3 will be described.
[0341] (Step S141) First, in step S141 , the image processing apparatus of the present disclosure calculates the velocity (v) and the acceleration (α) of the prominent subject 21 .
[0342] Image processing device, i.e. Figure 2 The external device 30 illustrated in FIG. 1 receives a captured image (moving image) from the PTZ camera 10 and calculates a velocity (v) and an acceleration (α) of the prominent subject by analyzing a position change of the prominent subject 21 in each image frame constituting the moving image.
[0343] (Step S142) Next, in step S142, the image processing device of the present disclosure calculates the speed corresponding optimal imaging ratio based on the speed-imaging ratio corresponding data, and the speed corresponding optimal imaging ratio is the optimal imaging ratio corresponding to the current speed (v) of the prominent subject calculated in step S141.
[0344] This process is referenced above Fig.14 (Process A) describes the process. The speed-to-image ratio data is Fig.14 Graph shown graphically in (Treatment A).
[0345] use Fig.14The graph shown in the diagram of (Processing A) calculates the imaging ratio corresponding to the current speed of the prominent subject 21 as the speed-corresponding optimal imaging ratio. Fig.14 In the example illustrated in (Processing A), the optimal imaging ratio corresponding to the current speed (vt) of the prominent subject 21 is Fig.14 The speeds shown in the diagram in (Treatment A) correspond to the optimal imaging ratio (r1).
[0346] (Step S143) Next, in step S143, the image processing device according to the present disclosure calculates the acceleration corresponding optimal imaging ratio based on the acceleration-imaging ratio corresponding data, and the acceleration corresponding optimal imaging ratio is the optimal imaging ratio corresponding to the acceleration (α) of the current prominent subject calculated in step S141.
[0347] This process is referenced above Fig.14 (Process B) describes the process. The acceleration-imaging ratio corresponding data is Fig.14 (Treatment B) is a graph shown in the figure.
[0348] use Fig.14 The graph shown in the diagram of (Processing B) calculates the imaging ratio corresponding to the current acceleration of the significant subject 21 as the acceleration corresponding optimal imaging ratio. Fig.14 In the example illustrated in (Processing B), the optimal imaging ratio corresponding to the current acceleration (αt) of the prominent subject 21 is Fig.14 The acceleration illustrated in the diagram in (Process B) corresponds to the optimal imaging ratio (r2).
[0349] (Steps S144 to S145) Next, in step S144, the image processing device compares the optimal imaging ratio corresponding to the velocity (v) of the prominent subject (velocity corresponding optimal imaging ratio vx) and the optimal imaging ratio corresponding to the acceleration (α) of the prominent subject (acceleration corresponding optimal imaging ratio αx).
[0350] This process corresponds to the one described above. Fig.14 (Processing C) Processing.
[0351] In a case where it is determined in step S145 that the velocity-corresponding optimal imaging ratio vx of the prominent subject is equal to or greater than the acceleration-corresponding optimal imaging ratio αx, the process proceeds to step S146 .
[0352] On the other hand, in the case where it is determined that the velocity-corresponding optimal imaging ratio vx of the prominent subject is smaller than the acceleration-corresponding optimal imaging ratio αx, the process proceeds to step S147 .
[0353] (Step S146) In a case where it is determined in step S145 that the velocity-corresponding optimal imaging ratio vx of the prominent subject is equal to or greater than the acceleration-corresponding optimal imaging ratio αx, the process proceeds to step S146 .
[0354] In this case, the image processing device selects the speed-corresponding optimal imaging ratio vx of the prominent subject which is a larger value as the final optimal imaging ratio.
[0355] (Step S147) On the other hand, in a case where it is determined in step S145 that the velocity-corresponding optimal imaging ratio vx of the prominent subject is smaller than the acceleration-corresponding optimal imaging ratio αx, the process proceeds to step S147 .
[0356] In this case, the image processing device selects the acceleration-corresponding optimal imaging ratio αx of the prominent subject which is a larger value as the final optimal imaging ratio.
[0357] (Step S148) Next, in step S148, the image processing apparatus determines whether the difference (ratio) between the final optimal imaging ratio selected in step S146 or step S147 and the current imaging ratio is smaller than a prescribed threshold value.
[0358] In the case where the difference (ratio) between the final optimal imaging ratio selected in step S146 or step S147 and the current imaging ratio is smaller than the specified threshold value, the determination in step S148 is yes, and then it is determined in step S153 whether the shooting is finished. In the case where the shooting is not finished, the process proceeds to the processing of the next processing frame without performing the zoom control of the camera (PTZ camera 10).
[0359] This is a process for preventing fine control from being frequently performed for each image frame and the captured image from becoming unstable if zoom control is performed before the difference (ratio) between the optimum imaging ratio and the current imaging ratio is smaller than a specified threshold.
[0360] As described above, in a case where the difference (ratio) between the optimum imaging ratio and the current imaging ratio is smaller than the prescribed threshold value, the determination process in step S148 is performed so as to continue stable image capturing without performing control.
[0361] On the other hand, in a case where the difference (ratio) between the final optimal imaging ratio selected in step S146 or step S147 and the current imaging ratio is equal to or greater than the specified threshold, the determination in step S148 is NO, and the processing proceeds to step S149.
[0362] Note that as the specified threshold, a value predefined in the image processing device is used. For example, the process of determining the difference (ratio) between the final optimal imaging ratio selected in step S146 or step S147 and the current imaging ratio can be performed by using a value of approximately 5% of the current imaging ratio.
[0363] That is, the process of determining whether to perform zoom control is performed based on whether the difference (ratio) between the final optimal imaging ratio selected in step S146 or step S147 and the current imaging ratio is smaller than a specified threshold (e.g., 5%).
[0364] (Step S149) In step S148, in a case where the difference (ratio) between the final optimum imaging ratio selected in step S146 or step S147 and the current imaging ratio is equal to or greater than a prescribed threshold, the process proceeds to step S149.
[0365] In this case, in step S149, the image processing apparatus determines whether the final optimum imaging ratio selected in step S146 or step S147 is larger than or smaller than the current imaging ratio.
[0366] In a case where the optimum imaging ratio is greater than the current imaging ratio, the process proceeds to step S151.
[0367] On the other hand, in a case where the optimum imaging ratio is smaller than the current imaging ratio, the process proceeds to step S152.
[0368] (Step S151) In a case where it is determined in step S146 or step S147 that the selected final optimal imaging ratio is greater than the current imaging ratio, the process proceeds to step S151 and the following processing is performed.
[0369] In this case, in step S151, the image processing apparatus performs a process of widening the angle of view by adjusting the zoom (angle of view) of the camera so as to increase the current imaging ratio to the optimal imaging ratio.
[0370] Specifically, for example, in Figure 2 In the system configuration illustrated in FIG. 1 , the external device 30 outputs a camera driving instruction to the PTZ camera 10 to adjust the zoom (angle of view) of the camera to widen the angle of view so as to increase the current imaging ratio to the optimal imaging ratio.
[0371] This process corresponds to, for example, the above reference Figure 8 Description of the process.
[0372] That is, the processing corresponds to Figure 8 The zoom (angle of view) state before the imaging ratio change shown in the diagram in (a) is changed to Figure 8Processing of the zoom (angle of view) state after the imaging ratio is changed as illustrated in (b).
[0373] Figure 8 The captured image b and the captured area 20b after the imaging ratio change in (b) are in the form of images with a ratio of Figure 8 The captured image a before the imaging ratio is changed in (a) and the image captured when the imaging area 20a is set with a wide viewing angle.
[0374] As described above, when it is determined in step S146 or step S147 that the selected final optimal imaging ratio is greater than the current imaging ratio, the zoom of the PTZ camera 10 is adjusted in step S125 to make settings for widening the shooting range, that is, a process for widening the viewing angle is performed.
[0375] (Step S152) On the other hand, in a case where it is determined in step S146 or step S147 that the selected final optimal imaging ratio is smaller than the current imaging ratio, the process proceeds to step S152, and the following processing is performed.
[0376] In this case, in step S152, the image processing apparatus performs a process of narrowing the angle of view by adjusting the zoom (angle of view) of the camera so as to reduce the current imaging ratio to the optimal imaging ratio.
[0377] Specifically, for example, in Figure 2 In the system configuration illustrated in FIG. 1 , the external device 30 outputs a camera driving instruction to the PTZ camera 10 to reduce the viewing angle by adjusting the zoom (viewing angle) of the camera so as to reduce the current imaging ratio to the optimal imaging ratio.
[0378] This process corresponds to, for example, the above reference Fig. 9 Description of the process.
[0379] That is, the processing corresponds to Fig. 9 The zoom (angle of view) state before the imaging ratio change shown in the diagram in (a) is changed to Fig. 9 Processing of the zoom (angle of view) state after the imaging ratio is changed as illustrated in (b).
[0380] Fig. 9 The captured image b and the captured area 20b after the imaging ratio change in (b) are in the form of images with a ratio of Fig. 9 The captured image a before the imaging ratio is changed in (a) and the image captured when the imaging area 20a is set with a narrow viewing angle.
[0381] As described above, when it is determined in step S146 or step S147 that the selected final optimal imaging ratio is smaller than the current imaging ratio, in step S152, the zoom control parameters of the PTZ camera 10 are adjusted to reduce the shooting range, that is, to reduce the viewing angle.
[0382] (Step S153) After the processing of step S151 and step S152, the processing proceeds to step S153, and in step S153, it is determined whether the shooting processing is ended.
[0383] In a case where the photographing process is not completed, the process returns to step S141 , and the process for the next photographed image frame is performed.
[0384] When the photographing process is completed, the process ends.
[0385] By executing these processes, the success probability of the process of cutting out a cutout region including a prominent subject from a captured image of the PTZ camera 10 can be improved, and degradation in the image quality of the cutout image can be mitigated.
[0386] [5. (Example 2) Example of changing the imaging ratio according to the communication delay time between the camera and the external device] Next, as (Embodiment 2), an embodiment in which an imaging ratio is changed according to a communication delay time between a camera and an external device will be described.
[0387] As above reference Figure 5 As described above, when a communication delay occurs between the PTZ camera 10 and the external device 30 that generates a cutout image by inputting a captured image from the PTZ camera, the probability of a tracking error of a prominent subject or a cutout image generation error increases.
[0388] Embodiment 2 described below is an embodiment having a configuration for preventing this from happening.
[0389] Will refer to Fig.19 A specific processing example of an embodiment in which an imaging ratio is changed according to a communication delay time between a camera and an external device is described.
[0390] Fig.19 The graph illustrated in FIG. 1 is a graph in which the communication delay time (dt) between the camera and the external device is set on the horizontal axis, and the imaging ratio correction coefficient is set on the vertical axis.
[0391] The communication delay time (dt) between the camera and the external device on the horizontal axis is, for example, the number of seconds (sec) required for communication between the camera and the external device. Note that the communication delay time can be acquired using the number of delayed frames and the frame rate (FPS).
[0392] For example, the imaging ratio correction factor on the vertical axis is used to multiply the above reference Fig.11 The correction coefficient of the multiplication value of the optimal imaging ratio is calculated by using the "minimum permissible imaging ratio rMIN" described above.
[0393] In the case where the communication delay time (dt) between the camera and the external device is 0, the imaging ratio correction coefficient is 1.0, and the imaging ratio correction coefficient also increases as the communication delay time (dt) between the camera and the external device increases.
[0394] However, the maximum value is the correction coefficient allowable maximum value kMAX.
[0395] The maximum allowable value kMAX of the correction coefficient is calculated by the following calculation formula.
[0396] The maximum allowable value of the correction coefficient kMAX = (the maximum allowable imaging ratio rMAX) / (the minimum allowable imaging ratio rMIN) The image processing device according to the second embodiment of the present disclosure is based on Fig.19 The graph shown in the figure changes the imaging ratio as follows.
[0397] (a) When the communication delay time (dt) between the camera and the external device is 0 to t1, the imaging ratio correction coefficient is changed from 1.0 to the correction coefficient allowable maximum value kMAX according to the communication delay time.
[0398] (b) When the communication delay time (dt) between the camera and the external device = t1~, the imaging ratio correction coefficient is set to the correction coefficient allowable maximum value kMAX.
[0399] Note that when (a) the communication delay time (dt) between the camera and the external device = 0 to t1, the imaging ratio correction coefficient is set to change linearly from 1.0 to the maximum allowable correction coefficient value kMAX, but this is an example, and for example, the imaging ratio correction coefficient may be increased according to an ascending curve depicting a predetermined curve, or the imaging ratio may be set to change exponentially as the communication delay time (dt) increases.
[0400] In this embodiment 2, the external device 30 applies Fig.19 The imaging ratio correction coefficient is calculated from the curve graph shown in the diagram, and the optimal imaging ratio (imaging ratio for generating the intercepted image) is calculated by the following calculation formula, that is, Optimum imaging ratio = (minimum allowable imaging ratio rMIN) × (imaging ratio correction coefficient).
[0401] The value calculated according to the above formula is set as the optimal imaging ratio, and the PTZ camera 10 is caused to perform image capturing according to the imaging area size determined based on the calculated optimal imaging ratio.
[0402] For example, the current communication delay time (dt) between the camera and the external device is set to Fig.19 The current delay time (ta) is shown graphically in the graph of FIG.
[0403] use Fig.19 The graph shown in the diagram calculates the imaging ratio correction coefficient = 1.2 corresponding to the current communication delay time (dt) = ta between the camera and the external device.
[0404] Furthermore, the calculated imaging ratio correction coefficient = 1.2 is applied, and the optimum imaging ratio is calculated according to the following formula.
[0405] Optimal imaging ratio = (minimum allowable imaging ratio rMIN) × (imaging ratio correction factor) =(minimum allowable imaging ratio rMIN)×(1.2) That is, in the case where the communication delay time (dt)=ta between the current camera and the external device, the optimum imaging ratio is an imaging ratio that is 1.2 times the (minimum allowable imaging ratio rMIN).
[0406] The external device 30 enables the PTZ camera 10 to Fig.19 The graph shown in the illustration calculates the imaging ratio correction coefficient to calculate the optimal imaging ratio determined by the imaging area size to capture the image.
[0407] Specifically, the PTZ camera 10 is caused to perform image capturing processing while setting the height H of the imaging area 22 (=the entire image area of the captured image 20) to H=(1.2 (rMIN))×h.
[0408] Note that h is the height of the cutout region 23 .
[0409] By causing the PTZ camera 10 to perform image capture with such settings, the captured area 23 including the prominent subject 21 can be more reliably captured from the captured image 20 (imaging area 22) of the PTZ camera 10, and degradation in image quality of the captured image can also be mitigated.
[0410] like Fig.19 As shown in the diagram, when the communication delay time (dt) between the camera and the external device is 0 to t1, the image processing device of the second embodiment changes the imaging ratio correction coefficient from 1.0 to the correction coefficient allowable maximum value kMAX.
[0411] As a result, the imaging ratio is set to increase in sequence from rMIN (allowable minimum imaging ratio) according to the delay time. As the delay time is shorter, the difference (ratio) between the size of the imaging area 22 and the size of the clipping area 23 is set to be smaller.
[0412] Through this processing, the number of pixels of the clipped area 23 can be set larger, and the reduction in the image quality of the clipped image can be reduced.
[0413] On the other hand, when (b) the communication delay time (dt) between the camera and the external device = t1~, the imaging ratio correction coefficient is set to the correction coefficient allowable maximum value kMAX. As a result, the imaging ratio is set to rMAX (allowable maximum imaging ratio), and the difference (ratio) between the size of the imaging area 22 and the size of the clipping area 23 is set to be greater than the difference (ratio) when the communication delay time is less than the predetermined value (t1).
[0414] As a result, even in a case where the communication delay time is longer than a predetermined value, the possibility that the prominent subject is out of the photographing range can be reduced, and reliable tracking processing of the prominent subject can be performed.
[0415] Note that, as described above, similarly to Embodiments 1-1 to 1-3, rMIN (allowable minimum imaging ratio) and rMAX (allowable maximum imaging ratio) are predefined ratios.
[0416] For example, rMIN (the minimum permissible imaging ratio) and rMAX (the maximum permissible imaging ratio) may be predefined fixed values. Fig.11 and Fig.12 As described above, it can be determined based on the state of the scene of the captured image, the setting of the clipping area, the minimum allowable number of pixels of the clipping area 23, etc.
[0417] Next, refer to Fig. 20 The flowchart shown in the figure, (Example 2) Example of changing the imaging ratio according to the communication delay time between the camera and the external device A processing sequence executed by the image processing device (external device 30) according to the present embodiment 2 will be described.
[0418] (Step S201) First, in step S201 , the image processing apparatus of the present disclosure calculates a communication delay time (dt) between the PTZ camera 10 and the external device 30 .
[0419] Image processing device, that is, Figure 2The external device 30 illustrated in FIG. 2 calculates a communication delay time (dt) between the PTZ camera 10 and the external device 30 at each timing of receiving a captured image (moving image) from the PTZ camera 10 .
[0420] The frame rate of the captured image (moving image) of the PTZ camera 10 is defined in advance, and the communication delay time (dt) between the PTZ camera 10 and the external device 30 can be calculated by analyzing the reception timing of each image frame.
[0421] (Step S202) Next, in step S202 , the image processing apparatus of the present disclosure calculates an imaging ratio correction coefficient corresponding to the communication delay time (dt) between the PTZ camera 10 and the external device 30 calculated in step S201 based on the communication delay time-imaging ratio correction coefficient correspondence data.
[0422] This process is applied to the above reference Fig.19 Describes the processing of the graph.
[0423] As reference Fig.19 As described above, the communication delay time-imaging ratio correction coefficient correspondence data is a graph defining the imaging ratio correction coefficient according to the communication delay time (dt) between the PTZ camera 10 and the external device 30 as follows.
[0424] (a) When the communication delay time (dt) between the camera and the external device is 0 to t1, the imaging ratio correction coefficient is changed from 1.0 to the correction coefficient allowable maximum value kMAX according to the communication delay time.
[0425] (b) When the communication delay time (dt) between the camera and the external device = t1~, the imaging ratio correction coefficient is set to the correction coefficient allowable maximum value kMAX.
[0426] use Fig.19 The graph illustrated in FIG. 1 calculates an imaging ratio correction coefficient corresponding to a communication delay time (dt) between the PTZ camera 10 and the external device 30 .
[0427] For example, in Fig.19 In the example illustrated in FIG. 2 , in the case where the communication delay time (dt)=ta between the PTZ camera 10 and the external device 30 , the imaging ratio correction coefficient=1.2.
[0428] (Step S203) Next, in step S203 , the image processing apparatus calculates an optimum imaging ratio (imaging ratio for clipped image generation) according to the communication delay time (dt) between the PTZ camera 10 and the external device 30 using the imaging ratio correction coefficient calculated in step S202 .
[0429] Specifically, the optimum imaging ratio (imaging ratio for generating a cutout image) is calculated according to the following formula.
[0430] Optimal imaging ratio = (minimum allowable imaging ratio rMIN) × (imaging ratio correction factor) (Step S204) Next, in step S204, the image processing apparatus determines whether the difference (ratio) between the optimum imaging ratio calculated in step S203 and the current imaging ratio is smaller than a prescribed threshold value.
[0431] When the difference (ratio) between the optimal imaging ratio calculated in step S203 and the current imaging ratio is less than the specified threshold value, the determination in step S204 is yes, and then it is determined in step S208 whether the shooting is finished. In the case where the shooting is not finished, the process proceeds to the processing of the next processing frame without performing the zoom control of the camera (PTZ camera 10).
[0432] This is a process for preventing fine control from being frequently performed for each image frame and thereby causing the captured image to become unstable if zoom control is performed before the difference (ratio) between the optimum imaging ratio calculated in step S203 and the current imaging ratio is small.
[0433] As described above, when the difference (ratio) between the optimum imaging ratio calculated in step S203 and the current imaging ratio is smaller than the prescribed threshold, the determination process in step S204 is performed to continue stable image capturing without performing control.
[0434] On the other hand, when the difference (ratio) between the optimum imaging ratio calculated in step S203 and the current imaging ratio is equal to or greater than the prescribed threshold, the determination in step S204 is NO, and the process proceeds to step S205.
[0435] Note that as the specified threshold, a value predefined in the image processing apparatus is used. For example, the process of determining the difference (ratio) between the optimal imaging ratio calculated in step S203 and the current imaging ratio can be performed by using a value of approximately 5% of the current imaging ratio.
[0436] That is, the process of determining whether to perform zoom control is performed based on whether the difference (ratio) between the optimal imaging ratio calculated in step S203 and the current imaging ratio is smaller than a specified threshold value (eg, 5%).
[0437] (Step S205) In step S204, in a case where the difference (ratio) between the optimum imaging ratio calculated in step S203 and the current imaging ratio is equal to or greater than a prescribed threshold, the process proceeds to step S205.
[0438] In this case, in step S205, the image processing apparatus determines whether the optimum imaging ratio calculated in step S203 is larger than or smaller than the current imaging ratio.
[0439] In a case where the optimum imaging ratio is greater than the current imaging ratio, the process proceeds to step S206.
[0440] On the other hand, in a case where the optimum imaging ratio is smaller than the current imaging ratio, the process proceeds to step S207.
[0441] (Step S206) In a case where it is determined in step S205 that the optimum imaging ratio calculated in step S203 is greater than the current imaging ratio, the process proceeds to step S206 , and the following processing is performed.
[0442] In this case, in step S206, the image processing apparatus performs a process of widening the angle of view by adjusting the zoom (angle of view) of the camera so as to increase the current imaging ratio to the optimal imaging ratio.
[0443] Specifically, for example, in Figure 2 In the system configuration illustrated in FIG. 1 , the external device 30 outputs a camera driving instruction to the PTZ camera 10 to adjust the zoom (angle of view) of the camera to widen the angle of view so as to increase the current imaging ratio to the optimal imaging ratio.
[0444] This process corresponds to, for example, the above reference Figure 8 Description of the process.
[0445] That is, the processing corresponds to Figure 8 The zoom (angle of view) state before the imaging ratio change shown in the diagram in (a) is changed to Figure 8 Processing of the zoom (angle of view) state after the imaging ratio is changed as illustrated in (b).
[0446] Figure 8 The captured image b and the captured area 20b after the imaging ratio change in (b) are in the form of images with a ratio of Figure 8 The captured image a before the imaging ratio is changed in (a) and the image captured when the imaging area 20a is set with a wide viewing angle.
[0447] As described above, when it is determined that the optimal imaging ratio calculated in step S203 is greater than the current imaging ratio, in step S206 , the zoom control parameter of the PTZ camera 10 is adjusted to perform settings for widening the shooting range, that is, processing for widening the viewing angle.
[0448] (Step S207) On the other hand, in a case where it is determined in step S205 that the optimum imaging ratio calculated in step S203 is smaller than the current imaging ratio, the process proceeds to step S207 , and the following processing is performed.
[0449] In this case, in step S207, the image processing apparatus performs a process of narrowing the angle of view by adjusting the zoom (angle of view) of the camera so as to reduce the current imaging ratio to the optimal imaging ratio.
[0450] Specifically, for example, in Figure 2 In the system configuration illustrated in FIG. 1 , the external device 30 outputs a camera driving instruction to the PTZ camera 10 to reduce the viewing angle by adjusting the zoom (viewing angle) of the camera so as to reduce the current imaging ratio to the optimal imaging ratio.
[0451] This process corresponds to, for example, the above reference Fig. 9 Description of the process.
[0452] That is, the processing corresponds to Fig. 9 The zoom (angle of view) state before the imaging ratio change shown in the diagram in (a) is changed to Fig. 9 Processing of the zoom (angle of view) state after the imaging ratio is changed as illustrated in (b).
[0453] Fig. 9 The captured image b and the captured area 20b after the imaging ratio change in (b) are in the form of images with a ratio of Fig. 9 The captured image a before the imaging ratio is changed in (a) and the image captured when the imaging area 20a is set with a narrow viewing angle.
[0454] As described above, when it is determined that the optimal imaging ratio calculated in step S203 is smaller than the current imaging ratio, in step S207 , the zoom control parameters of the PTZ camera 10 are adjusted to perform settings for narrowing the shooting range, that is, processing for narrowing the viewing angle.
[0455] (Step S208) After the processing of step S206 and step S207, the process proceeds to step S208, and in step S208, it is determined whether the photographing process ends.
[0456] In a case where the photographing process is not completed, the process returns to step S201 , and the process for the next photographed image frame is performed.
[0457] When the photographing process is completed, the process ends.
[0458] By executing these processes, the success probability of the process of cutting out a cutout region including a prominent subject from a captured image of the PTZ camera 10 can be improved, and degradation in the image quality of the cutout image can be mitigated.
[0459] [6. (Example 3) Example of changing the imaging ratio according to the speed and acceleration of the prominent subject and the communication delay time between the camera and the external device] Next, as (Embodiment 3), an embodiment in which the imaging ratio is changed according to the speed and acceleration of the prominent subject and the communication delay time between the camera and the external device is described.
[0460] The embodiment described below corresponds to the case where the following two above-mentioned embodiments are performed together, namely, (Example 1-3) Example of Changing the Imaging Ratio According to the Speed and Acceleration of the Prominent Subject (Embodiment 2) An embodiment in which the imaging ratio is changed according to the communication delay time between the camera and the external device.
[0461] refer to Fig.21 and Fig. 22 The flowchart shown in the figure, (Example 3) Example of changing the imaging ratio according to the speed and acceleration of the prominent subject and the communication delay time between the camera and the external device A processing sequence executed by the image processing device (external device 30) according to the present Embodiment 3 will be described.
[0462] (Step S301) First, in step S301 , the image processing apparatus of the present disclosure calculates the velocity (v) and acceleration (α) of the prominent subject 21 .
[0463] Image processing device, that is, Figure 2 The external device 30 illustrated in FIG. 1 receives a captured image (moving image) from the PTZ camera 10 and calculates a velocity (v) and an acceleration (α) of the prominent subject by analyzing a position change of the prominent subject 21 in each image frame constituting the moving image.
[0464] (Step S302) Next, in step S302, the image processing device of the present disclosure calculates the speed corresponding optimal imaging ratio based on the speed-imaging ratio corresponding data, and the speed corresponding optimal imaging ratio is the optimal imaging ratio corresponding to the current speed (v) of the prominent subject calculated in step S301.
[0465] This process is referenced above Fig.14 (Process A) describes the process. The speed-to-image ratio data is Fig.14 Graph shown graphically in (Treatment A).
[0466] use Fig.14 The graph shown in the diagram of (Processing A) calculates the imaging ratio corresponding to the current speed of the prominent subject 21 as the speed-corresponding optimal imaging ratio. Fig.14 In the example illustrated in (Processing A), the optimal imaging ratio corresponding to the current speed (vt) of the prominent subject 21 is Fig.14 The speeds shown in the diagram in (Treatment A) correspond to the optimal imaging ratio (r1).
[0467] (Step S303) Next, in step S303, the image processing device according to the present disclosure calculates the acceleration corresponding optimal imaging ratio based on the acceleration-imaging ratio corresponding data, and the acceleration corresponding optimal imaging ratio is the optimal imaging ratio corresponding to the acceleration (α) of the current prominent subject calculated in step S301.
[0468] This process is referenced above Fig.14 (Process B) describes the process. The acceleration-imaging ratio corresponding data is Fig.14 (Treatment B) is a graph shown in the figure.
[0469] use Fig.14 The curve diagram shown in the figure (processing B) calculates the imaging ratio corresponding to the current acceleration of the significant subject 21 as the acceleration corresponding optimal imaging ratio. Fig.14 In the example illustrated in (Processing B), the optimal imaging ratio corresponding to the current acceleration (αt) of the prominent subject 21 is Fig.14 The acceleration illustrated in the diagram in (Process B) corresponds to the optimal imaging ratio (r2).
[0470] (Steps S304-S305) Next, in step S304, the image processing device compares the optimal imaging ratio corresponding to the velocity (v) of the prominent subject (velocity corresponding optimal imaging ratio vx) and the optimal imaging ratio corresponding to the acceleration (α) of the prominent subject (acceleration corresponding optimal imaging ratio αx).
[0471] This process corresponds to the one described above. Fig.14 (Processing C) Processing.
[0472] In a case where it is determined in step S305 that the velocity-corresponding optimal imaging ratio vx of the prominent subject is equal to or greater than the acceleration-corresponding optimal imaging ratio αx, the process proceeds to step S306 .
[0473] On the other hand, in the case where it is determined that the velocity-corresponding optimal imaging ratio vx of the prominent subject is smaller than the acceleration-corresponding optimal imaging ratio αx, the process proceeds to step S307 .
[0474] (Step S306) In a case where it is determined in step S305 that the velocity-corresponding optimal imaging ratio vx of the prominent subject is equal to or greater than the acceleration-corresponding optimal imaging ratio αx, the process proceeds to step S306 .
[0475] In this case, the image processing device selects the speed-corresponding optimal imaging ratio vx of the prominent subject, which is a larger value, as the subject motion-corresponding optimal imaging ratio.
[0476] (Step S307) On the other hand, in a case where it is determined in step S305 that the velocity-corresponding optimal imaging ratio vx of the prominent subject is smaller than the acceleration-corresponding optimal imaging ratio αx, the process proceeds to step S307 .
[0477] In this case, the image processing device selects the acceleration corresponding optimal imaging ratio αx of the prominent subject which is a larger value as the subject motion corresponding optimal imaging ratio.
[0478] (Step S311) Next, in step S311 , the image processing apparatus of the present disclosure calculates a communication delay time (dt) between the PTZ camera 10 and the external device 30 .
[0479] Image processing device, that is, Figure 2 The external device 30 illustrated in FIG. 2 calculates a communication delay time (dt) between the PTZ camera 10 and the external device 30 at each timing of receiving a captured image (moving image) from the PTZ camera 10 .
[0480] The frame rate of the captured image (moving image) of the PTZ camera 10 is defined in advance, and the communication delay time (dt) between the PTZ camera 10 and the external device 30 can be calculated by analyzing the reception timing of each image frame.
[0481] (Step S312) Next, in step S312 , the image processing apparatus of the present disclosure calculates an imaging ratio correction coefficient corresponding to the communication delay time (dt) between the PTZ camera 10 and the external device 30 calculated in step S311 based on the communication delay time-imaging ratio correction coefficient correspondence data.
[0482] This process is applied to the above reference Fig.19 Describes the processing of the graph.
[0483] As reference Fig.19 As described above, the communication delay time-imaging ratio correction coefficient correspondence data is a graph defining the imaging ratio correction coefficient according to the communication delay time (dt) between the PTZ camera 10 and the external device 30 as follows.
[0484] (a) When the communication delay time (dt) between the camera and the external device is 0 to t1, the imaging ratio correction coefficient is changed from 1.0 to the correction coefficient allowable maximum value kMAX according to the communication delay time.
[0485] (b) When the communication delay time (dt) between the camera and the external device = t1~, the imaging ratio correction coefficient is set to the correction coefficient allowable maximum value kMAX.
[0486] use Fig.19 The graph illustrated in FIG. 1 calculates an imaging ratio correction coefficient corresponding to a communication delay time (dt) between the PTZ camera 10 and the external device 30 .
[0487] For example, in Fig.19 In the example illustrated in FIG. 2 , in the case where the communication delay time (dt)=ta between the PTZ camera 10 and the external device 30 , the imaging ratio correction coefficient=1.2.
[0488] (Step S313) Next, in step S313, the image processing device calculates the optimal imaging ratio according to the subject motion and the communication delay time (dt) between the PTZ camera 10 and the external device 30 by using the subject motion corresponding optimal imaging ratio determined in step S306 or step S307 and the imaging ratio correction coefficient calculated in step S312.
[0489] Specifically, the optimal imaging ratio is calculated according to the following formula.
[0490] Optimal imaging ratio = (optimal imaging ratio corresponding to the subject's motion) × (imaging ratio correction coefficient) (Step S314) Next, in step S314, the image processing apparatus determines whether the difference (ratio) between the optimum imaging ratio calculated in step S313 and the current imaging ratio is smaller than a prescribed threshold value.
[0491] In the case where the difference (ratio) between the optimal imaging ratio calculated in step S313 and the current imaging ratio is smaller than the specified threshold value, the determination in step S314 is yes, and then it is determined in step S318 whether the shooting is finished. In the case where the shooting is not finished, the process proceeds to the processing of the next processing frame without performing the zoom control of the camera (PTZ camera 10).
[0492] This is a process for preventing fine control from being frequently performed for each image frame and thereby causing the captured image to become unstable if zoom control is performed before the difference (ratio) between the optimum imaging ratio calculated in step S313 and the current imaging ratio is small.
[0493] As described above, when the difference (ratio) between the optimum imaging ratio calculated in step S313 and the current imaging ratio is smaller than the prescribed threshold, the determination process in step S314 is performed to continue stable image capturing without performing control.
[0494] On the other hand, when the difference (ratio) between the optimum imaging ratio calculated in step S313 and the current imaging ratio is equal to or greater than the prescribed threshold, the determination in step S314 is NO and the process proceeds to step S315.
[0495] Note that as the specified threshold, a value predefined in the image processing apparatus is used. For example, the process of determining the difference (ratio) between the optimal imaging ratio calculated in step S313 and the current imaging ratio can be performed by using a value of approximately 5% of the current imaging ratio.
[0496] That is, the process of determining whether to perform zoom control is performed based on whether the difference (ratio) between the optimal imaging ratio calculated in step S313 and the current imaging ratio is smaller than a specified threshold value (eg, 5%).
[0497] (Step S315) In step S314, in a case where the difference (ratio) between the optimum imaging ratio calculated in step S313 and the current imaging ratio is equal to or greater than a prescribed threshold, the process proceeds to step S315.
[0498] In this case, in step S315, the image processing apparatus determines whether the optimum imaging ratio calculated in step S313 is larger than or smaller than the current imaging ratio.
[0499] In a case where the optimum imaging ratio is greater than the current imaging ratio, the process proceeds to step S316.
[0500] On the other hand, in a case where the optimum imaging ratio is smaller than the current imaging ratio, the process proceeds to step S317.
[0501] (Step S316) In a case where it is determined in step S315 that the optimum imaging ratio calculated in step S313 is greater than the current imaging ratio, the process proceeds to step S316 , and the following processing is performed.
[0502] In this case, in step S316, the image processing apparatus performs a process of widening the angle of view by adjusting the zoom (angle of view) of the camera so as to increase the current imaging ratio to the optimal imaging ratio.
[0503] Specifically, for example, in Figure 2 In the system configuration illustrated in FIG. 1 , the external device 30 outputs a camera driving instruction to the PTZ camera 10 to adjust the zoom (angle of view) of the camera to widen the angle of view so as to increase the current imaging ratio to the optimal imaging ratio.
[0504] This process corresponds to, for example, the above reference Figure 8 Description of the process.
[0505] That is, the processing corresponds to Figure 8 The zoom (angle of view) state before the imaging ratio change shown in the diagram in (a) is changed to Figure 8 Processing of the zoom (angle of view) state after the imaging ratio is changed as illustrated in (b).
[0506] Figure 8 The captured image b and the captured area 20b after the imaging ratio change in (b) are in the form of images with a ratio of Figure 8 The captured image a before the imaging ratio is changed in (a) and the image captured when the imaging area 20a is set with a wide viewing angle.
[0507] As described above, when it is determined that the optimal imaging ratio calculated in step S3133 is greater than the current imaging ratio, in step S316, the zoom control parameters of the PTZ camera 10 are adjusted to perform settings for widening the shooting range, that is, processing for widening the viewing angle.
[0508] (Step S317) On the other hand, in a case where it is determined in step S315 that the optimum imaging ratio calculated in step S313 is smaller than the current imaging ratio, the process proceeds to step S317 , and the following processing is performed.
[0509] In this case, in step S317, the image processing apparatus performs a process of narrowing the angle of view by adjusting the zoom (angle of view) of the camera so as to reduce the current imaging ratio to the optimal imaging ratio.
[0510] Specifically, for example, in Figure 2 In the system configuration illustrated in FIG. 1 , the external device 30 outputs a camera driving instruction to the PTZ camera 10 to reduce the viewing angle by adjusting the zoom (viewing angle) of the camera so as to reduce the current imaging ratio to the optimal imaging ratio.
[0511] This process corresponds to, for example, the above reference Fig. 9 Description of the process.
[0512] That is, the processing corresponds to Fig. 9 The zoom (angle of view) state before the imaging ratio change shown in the diagram in (a) is changed to Fig. 9 Processing of the zoom (angle of view) state after the imaging ratio is changed as illustrated in (b).
[0513] Fig. 9 The captured image b and the captured area 20b after the imaging ratio change in (b) are in the form of images with a ratio of Fig. 9 The captured image a before the imaging ratio is changed in (a) and the image captured when the imaging area 20a is set with a narrow viewing angle.
[0514] As described above, when it is determined that the optimal imaging ratio calculated in step S313 is smaller than the current imaging ratio, in step S317 , the zoom of the PTZ camera 10 is adjusted to perform settings for narrowing the shooting range, that is, processing for narrowing the viewing angle.
[0515] (Step S318) After the processing of step S316 and step S317, the process proceeds to step S318, and in step S318, it is determined whether the photographing process ends.
[0516] In a case where the photographing process is not completed, the process returns to step S311 , and the process for the next photographed image frame is performed.
[0517] When the photographing process is completed, the process ends.
[0518] By executing these processes, the success probability of the process of cutting out a cutout region including a prominent subject from a captured image of the PTZ camera 10 can be improved, and degradation in the image quality of the cutout image can be mitigated.
[0519] [7. Configuration Example of the Image Processing Device Disclosed in the Present Invention] Next, a configuration example of the image processing device of the present disclosure will be described.
[0520] The above-mentioned embodiments 1 to 3, namely, (Embodiment 1) Embodiment in which the imaging ratio is changed according to at least one of the velocity and acceleration of a prominent subject (Example 2) Example of changing the imaging ratio according to the communication delay time between the camera and the external device (Example 3) Example of changing the imaging ratio according to the speed and acceleration of the prominent subject and the communication delay time between the camera and the external device The processing according to these embodiments can be mainly performed in the form of Figure 2The image processing system illustrated in FIG. 1 may be executed in the external device 30 , or may be executed by the PTZ camera 10 alone without using the external device 30 .
[0521] Will refer to Fig.23 A processing sequence in the case where the processing according to the above-described embodiment is executed by the PTZ camera 10 alone is described.
[0522] Fig.23 1 is a diagram for explaining a processing sequence in a case where the PTZ camera 100 performs all of image capturing, image interception, distribution of intercepted images, and camera control processing.
[0523] Fig.23 The PTZ camera 100 shown in the diagram corresponds to Figure 2 and the PTZ camera 10 illustrated in the other figures.
[0524] like Fig.23 As shown in the diagram, the PTZ camera 100 repeatedly executes the following five processes in sequence.
[0525] Step S501 = image capture processing; Step S502 = image analysis processing; Step S503 = image capture processing; Step S504 = Camera control processing Step S505 = Capture image output processing The PTZ camera 100 is a camera that captures moving images (video), and the processes in steps S501 to S505 are repeatedly performed for each frame or each batch of a plurality of frames captured by the PTZ camera.
[0526] The image capturing process in step S501 is a process of capturing an image (moving image) by the PTZ camera 100 .
[0527] The image analysis process in step S502 is an image analysis process of the image captured by the PTZ camera 100. For example, detection of a person to be captured, detection of a facial area, etc. are performed.
[0528] The process of detecting an image region of a prominent subject as a candidate for clipping from a captured image can be performed by applying existing processes such as pattern matching, face detection processing, skeleton detection processing, and segmentation processing.
[0529] Note that the modes of person detection processing include processing for detecting a head or face area, processing for detecting an upper body, processing for detecting a whole body, etc. Which mode of person detection processing is performed is determined based on, for example, a predetermined subject tracking algorithm.
[0530] The image interception processing in step S503 is a processing of intercepting a partial image area of the captured image captured by the PTZ camera 100 based on the result of the image analysis processing in step S502 .
[0531] Note that in the image capture processing in step S503, for example, while using the AI analysis described above using at least one of the machine learning models such as a deep neural network or a rule-based model to detect and track a specific person, processing is performed to capture an image with a predetermined perspective according to a prescribed algorithm.
[0532] The camera control process in step S504 is a process of calculating camera control parameters, specifically, zoom (angle of view) setting values, etc., for setting a cut-out region including a prominent subject in the image cut-out process in step S503 and acquiring a high-quality cut-out image. That is, the camera control parameters that are optimal for image capture in the cut-out image region are calculated, the calculated camera control parameters are set to the PTZ camera 100, and image capture is performed.
[0533] Reference above Fig.15 The processing described in the flowchart in, that is, the following embodiments 1 to 3, (Embodiment 1) Embodiment in which the imaging ratio is changed according to at least one of the velocity and acceleration of a prominent subject (Example 2) Example of changing the imaging ratio according to the communication delay time between the camera and the external device (Example 3) Example of changing the imaging ratio according to the speed and acceleration of the prominent subject and the communication delay time between the camera and the external device The processing according to these embodiments is as follows Fig.23 The following steps are performed as shown in the diagram.
[0534] Step S502 = image analysis processing; Step S503 = image capture processing; Step S504 = Camera control processing The cut-out image output processing in step S505 is processing of distributing the cut-out image 25 including the prominent subject cut-out in the image cut-out processing in step S503 to the outside, or storing the cut-out image in a storage unit.
[0535] For example, as illustrated in the figure, the captured image 25 is distributed (displayed) to a user terminal 80 such as a smartphone.
[0536] The processes in steps S501 to S505 are repeatedly performed in units of processed image frames captured by the PTZ camera 100 .
[0537] Fig.23The example illustrated in FIG. 1 is a configuration example in which the image processing of the present disclosure is performed in the PTZ camera 100 .
[0538] In addition to such a configuration example, a configuration is also possible in which a part of the image processing of the present disclosure is performed in an external device other than the PTZ camera 100 .
[0539] Will refer to Fig.24 An example of such a configuration example is described.
[0540] Fig.24 The diagram illustrates a PTZ camera 100 and an external device 120. The external device 120 corresponds to the Figure 2 The external device 30 described in the above description.
[0541] The PTZ camera 100 and the external device 120 are configured to be communicable.
[0542] The external device 120 includes a device connected to the PTZ camera 100 in a wireless or wired manner, for example, at least one of a PC, a server (cloud), a switch, other image processing devices, and the like.
[0543] The PTZ camera 100 captures an image (moving image) and transmits the captured image data to the external device 120 .
[0544] The external device 120 performs image analysis processing, image interception processing, camera control parameter calculation processing, intercepted image output processing, and the like on the captured image received from the PTZ camera 100 .
[0545] The external device 120 calculates the camera control parameters generated by the above process, ie, control parameters such as zoom control parameters optimal for generation of a cut-out image, and transmits the control parameters to the PTZ camera 100. The PTZ camera 100 sets the camera control parameters received from the external device 120 and performs image capture.
[0546] exist Fig.24 In the configuration illustrated in FIG. 2 , the recording process, the display process, and the distribution process of the captured image are executed by the external device 120 .
[0547] The external device 120 stores and records the captured image generated by the external device 120 in the recording medium 121 .
[0548] Furthermore, the external device 120 performs a process of distributing the generated captured image to the user terminal 130 such as a smartphone or a television owned by the user.
[0549] Fig.25 Graphic instructions for use Fig.24An example of a processing sequence of the PTZ camera 100 and the external device 120 is illustrated in FIG.
[0550] like Fig.25 As shown in the diagram, the PTZ camera 100 performs the following processing.
[0551] Step S521 = Image capture processing Step S522 = captured image output processing Step S528=Camera control processing On the other hand, for example, the external device 120 such as a PC, a server (cloud), a switch, or other image processing devices performs the following processing.
[0552] Step S523 = Photographed image input processing Step S524 = Image analysis processing Step S525 = Image capture processing Step S526 = Camera control parameter generation and transmission process Step S527 = Capture image output processing The PTZ camera 100 is a camera that captures a moving image (video), and the PTZ camera 100 and the external device 120 repeatedly perform the processes in steps S521 to S527 for each frame captured by the PTZ camera or for each of a plurality of frames.
[0553] The image capturing process in step S521 is a process of capturing an image (moving image) by the PTZ camera 100 .
[0554] In step S522 , the PTZ camera 100 transmits the captured image to the external device 120 .
[0555] In step S523 , the external device 120 inputs the captured image from the PTZ camera 100 .
[0556] Next, the image analysis process in step S524 executed by the external device 120 is an image analysis process of the captured image captured by the PTZ camera 100. For example, detection of a person to be captured, detection processing of a face area, and the like are performed.
[0557] The process of detecting an image region of a prominent subject as a candidate for clipping from a captured image can be performed by applying existing processes such as pattern matching, face detection processing, skeleton detection processing, and segmentation processing.
[0558] Note that the modes of person detection processing include processing for detecting a head or face area, processing for detecting an upper body, processing for detecting a whole body, etc. Which mode of person detection processing is performed is determined based on, for example, a predetermined subject tracking algorithm.
[0559] The image interception processing in step S525 is a processing of intercepting a partial image area of the captured image captured by the PTZ camera 100 based on the result of the image analysis processing in step S524 .
[0560] Note that in the image capture processing in step S525, for example, while using the AI analysis described above using at least one of the machine learning models such as a deep neural network or a rule-based model to detect and track a specific person, processing is performed to capture an image with a predetermined perspective according to a prescribed algorithm.
[0561] The camera control parameter calculation and transmission processing in step S526 is a processing for calculating and transmitting camera control parameters, specifically, zoom (angle of view) setting values, etc., for setting a cut-out area including a prominent subject and acquiring a high-quality cut-out image in the image cut-out processing in step S524. That is, the camera control parameters that are optimal for image shooting in the area of the cut-out image are calculated, and the calculated camera control parameters are transmitted to the PTZ camera 100.
[0562] In step S528 , the PTZ camera 100 captures an image by applying the camera control parameters calculated by the external device 120 .
[0563] Reference above Fig.15 The processes described in the flowcharts in the above, i.e., the following embodiments 1 to 3, (Embodiment 1) Embodiment in which the imaging ratio is changed according to at least one of the velocity and acceleration of a prominent subject (Example 2) Example of changing the imaging ratio according to the communication delay time between the camera and the external device (Example 3) Example of changing the imaging ratio according to the speed and acceleration of the prominent subject and the communication delay time between the camera and the external device The processing according to these embodiments is as follows Fig.25 The following steps are performed as shown in the diagram.
[0564] Step S524 = image analysis processing; Step S525 = image capture processing; Step S526 = Camera control parameter calculation and transmission processing The cut-out image output processing in step S527 is processing of distributing the cut-out image 25 including the prominent subject cut-out in the image cut-out processing in step S525 to the outside, or storing the cut-out image in a storage unit.
[0565] For example, as illustrated in the figure, the captured image 25 is distributed (displayed) to a user terminal 80 such as a smartphone.
[0566] The processes in steps S521 to S528 are repeatedly performed in units of processed image frames captured by the PTZ camera 100 .
[0567] As above reference Figure 23 to Figure 25 As described above, the image processing of the present disclosure may be performed by the camera alone, or may be performed as a cooperative process between the camera and other external devices.
[0568] [8. Detailed Configuration of the Image Processing Device of the Present Disclosure] Next, the detailed configuration of the image processing device of the present disclosure will be described.
[0569] As described above, the image processing of the present disclosure may be performed by the camera alone, or may be performed as a cooperative process between the camera and other external devices.
[0570] First, refer to Fig.26 A configuration example of an image processing apparatus in a case where image processing of the present disclosure is performed by a camera alone, that is, a PTZ camera 100 is described.
[0571] like Fig.26 As shown in the diagram, the PTZ camera 100 as an example of the image processing device of the present disclosure includes an imaging unit 201, an image analysis unit 202, a clipping area calculation unit 203, a significant subject motion (speed or acceleration) analysis unit 204, an optimal imaging ratio calculation unit 205, a clipping execution unit 206, an output unit 207, a recording processing unit 208, a recording medium 209, an optimal imaging area determination unit 211, a camera control parameter calculation unit 212 and a camera control unit 213.
[0572] The imaging unit 201 performs image capturing processing.
[0573] Note that a notable subject such as a tracking target is determined in advance, and a moving image for capturing a tracking image of the notable subject is captured.
[0574] As described above, by performing AI analysis using at least one of a machine learning model such as a deep neural network or a rule-based model, the process of detecting and tracking a specific person from a captured image can be performed with high accuracy.
[0575] By using such AI analysis, the PTZ camera (control of pan, tilt and zoom) is automatically controlled to shoot a follow-up video of a prominent subject.
[0576] The image analysis unit 202 performs image analysis processing on the captured image captured by the imaging unit 201. For example, detection of a person to be captured, detection processing of a face area, tracking processing, and the like are performed.
[0577] The image analysis unit 202 performs human detection processing by applying processing such as pattern matching, face detection processing, skeleton detection processing, and segmentation processing.
[0578] The clipped area calculation unit 203 performs processing for calculating an image clipped area including a predetermined prominent subject, for example, a position and a size of a clipped rectangular captured image.
[0579] The clipped region calculation unit 203 calculates a clipped region including a prominent subject from an image according to a predetermined algorithm.
[0580] As above reference Figure 3 As described above, there are various types of clipping area setting algorithms. Specifically, for example, a clipping area setting algorithm including the whole body area of the prominent subject, a clipping area setting algorithm including only the upper body area of the prominent subject, a clipping area setting algorithm including only the face area of the prominent subject, etc. are used.
[0581] The significant subject motion (speed or acceleration) analysis unit 204 calculates the motion of the significant subject, specifically, at least one of the speed or the acceleration.
[0582] The significant subject movement (speed or acceleration) analysis unit 204 calculates at least one of the speed (v) or acceleration (α) of the significant subject by analyzing the position change of the significant subject in each image frame constituting the moving image captured by the imaging unit 201 .
[0583] These calculated values are input to the optimum imaging ratio calculation unit 205 .
[0584] The optimal imaging ratio calculation unit 205 inputs the clipping area information calculated by the clipping area calculation unit 203 and at least one of the speed (v) or acceleration (α) of the significant subject analyzed by the significant subject motion (speed or acceleration) analysis unit 204, and calculates the optimal imaging ratio corresponding to at least one of the speed (v) or acceleration (α) of the significant subject, that is, the "imaging ratio for clipping image generation".
[0585] That is, the processing according to any one of the following embodiments described above is performed, (Example 1-1) Example of changing the imaging ratio according to the speed of the prominent subject (Example 1-2) Example of changing the imaging ratio according to the acceleration of the prominent subject (Example 1-3) Example of Changing the Imaging Ratio According to the Speed and Acceleration of the Prominent Subject An optimal imaging ratio corresponding to at least one of the velocity (v) or the acceleration (α) of the prominent subject is calculated.
[0586] The optimal imaging ratio corresponding to at least one of the velocity (v) or acceleration (α) of the prominent subject calculated by the optimal imaging ratio calculation unit 205 is output to the optimal imaging area determination unit 211 together with the clipped area information calculated by the clipped area calculation unit 203.
[0587] The optimal imaging region determination unit 211 determines the size of the imaging region of the image captured by the imaging unit 201 using the clipped region information calculated by the clipped region calculation unit 203 and the optimal imaging ratio calculated by the optimal imaging ratio calculation unit 205 .
[0588] Specifically, the height H of the imaging area 22 (=the entire image area of the captured image 20) is calculated according to the following calculation formula, that is, H=(optimal imaging ratio)×h.
[0589] Note that h is the height of the cutout region 23 .
[0590] Note that the aspect ratio of the imaging area is predefined, and if the height H of the imaging area is calculated, the width L of the imaging area can also be calculated.
[0591] The size of the imaging area calculated by the optimal imaging area determination unit 211 , for example, the height×width (H×L) of the imaging area is input to the camera control parameter calculation unit 212 .
[0592] The camera control parameter calculation unit 212 calculates zoom control parameters required to capture an image having an angle of view of the size of the imaging area (height×width (H×L) of the imaging area) calculated by the optimal imaging area determination unit 211 .
[0593] The zoom control parameter calculated by the camera control parameter calculation unit 212 is output to the camera control unit 213 .
[0594] The camera control unit 213 sets the zoom control parameter calculated by the camera control parameter calculation unit 212 in the PTZ camera, and causes the PTZ camera 100 to perform image capturing to which the zoom control parameter is applied.
[0595] Through these processes, it is possible to more reliably set a cut-out region of a prominent subject from an image captured by the PTZ camera 100 , and also to reduce degradation in image quality of the cut-out image.
[0596] The clipping execution unit 206 executes processing for clipping an image from the captured image based on the image clipping area calculated by the clipping area calculation unit 203 .
[0597] As described above, the clipping area calculation unit 203 calculates the Figure 3 The predefined algorithm calculates the clipping area including the prominent subject. The clipping execution unit 206 performs image clipping processing according to the clipping area, and the clipping area is determined according to the predefined algorithm.
[0598] The output unit 207 outputs the intercepted image intercepted by the interception execution unit 206 to at least one of various user terminals such as an external device, a smartphone, and a television.
[0599] The recording processing unit 208 records the cutout image cutout by the cutout execution unit 206 on the recording medium 209 .
[0600] Next, we will refer to Fig. 27 The configurations and processes of the PTZ camera 100 and the external device 120 are described in the case where the PTZ camera 100 and the external device 120 jointly perform the image processing of the present disclosure.
[0601] Fig. 27 is a diagram illustrating a configuration example of the PTZ camera 100 and the external device 120 .
[0602] Note that the external device 120 includes, for example, at least one of a PC, a server (cloud), a switch, a broadcasting device, other image processing devices, and the like.
[0603] In addition, the PTZ camera 100 and the external device 120 are connected to each other in at least one of a wired manner and a wireless manner, and can communicate with each other.
[0604] Fig. 27 The PTZ camera 100 illustrated in FIG. 2 includes an imaging unit 221 , an output unit 222 , a recording processing unit 223 , a recording medium 224 , and a camera control unit 225 .
[0605] In addition, the external device 120 includes an input unit 301, an image analysis unit 302, a clipping area calculation unit 303, a prominent subject motion (speed or acceleration) analysis unit 304, an optimal imaging ratio calculation unit 305, a clipping execution unit 306, an output unit 307, a recording processing unit 308, a recording medium 309, an optimal imaging area determination unit 311 and a camera control parameter calculation unit 312.
[0606] The imaging unit 221 of the PTZ camera 100 performs image capturing processing.
[0607] Note that a notable subject such as a tracking target is determined in advance, and a moving image for capturing a tracking image of the notable subject is captured.
[0608] As described above, by performing AI analysis using at least one of a machine learning model such as a deep neural network or a rule-based model, the process of detecting and tracking a specific person from a captured image can be performed with high accuracy.
[0609] By using such AI analysis, the PTZ camera (control of pan, tilt and zoom) is automatically controlled to shoot a follow-up video of a prominent subject.
[0610] The image captured by the imaging unit 221 is output to the external device 120 via the output unit 222 , and is recorded on the recording medium 224 via the recording processing unit 223 .
[0611] The camera control unit 225 applies the camera control parameters input from the camera control parameter calculation unit 312 of the external device 120 to cause the imaging unit 221 to capture an image.
[0612] Through this process, the PTZ camera 100 can perform image capturing to which camera control parameters, specifically, zoom setting parameters and the like, which are optimal for capturing an image determined by the external device 120, are applied.
[0613] The input unit 301 of the external device 120 inputs the image captured by the imaging unit 221 of the PTZ camera from the output unit 222 of the PTZ camera 100 , and outputs the image to the image analysis unit 302 .
[0614] The processing from the image analysis unit 302 to the camera control parameter calculation unit 312 of the external device 120 is similar to that described in the above reference. Fig.26 The processes from the image analysis unit 202 to the camera control parameter calculation unit 212 of the PTZ camera 100 are described.
[0615] exist Fig. 27 In the configuration illustrated in FIG. 1 , the external device 120 performs image analysis processing, that is, detection of a person or the like to be captured, and also performs image capture processing.
[0616] Furthermore, the optimum imaging ratio calculation unit 305 of the external device 120 performs optimum imaging ratio calculation processing.
[0617] That is, the processing according to any one of the following embodiments described above is performed, (Example 1-1) Example of changing the imaging ratio according to the speed of the prominent subject (Example 1-2) Example of changing the imaging ratio according to the acceleration of the prominent subject (Example 1-3) Example of Changing the Imaging Ratio According to the Speed and Acceleration of the Prominent Subject An optimal imaging ratio corresponding to at least one of the velocity (v) or the acceleration (α) of the prominent subject is calculated.
[0618] The optimal imaging ratio corresponding to at least one of the velocity (v) or acceleration (α) of the prominent subject calculated by the optimal imaging ratio calculation unit 305 is output to the optimal imaging area determination unit 311 together with the clipping area information calculated by the clipping area calculation unit 303 of the external device 120.
[0619] The optimal imaging area determination unit 311 applies the clipped area information calculated by the clipped area calculation unit 303 and the optimal imaging ratio calculated by the optimal imaging ratio calculation unit 305 to decide the size of the imaging area of the image captured by the PTZ camera 100 .
[0620] Specifically, the height H of the imaging area 22 (=the entire image area of the captured image 20) is calculated according to the following calculation formula, that is, H=(optimal imaging ratio)×h.
[0621] Note that h is the height of the cutout region 23 .
[0622] The size of the imaging area calculated by the optimal imaging area determination unit 311 , for example, the height×width (H×L) of the imaging area is input to the camera control parameter calculation unit 312 .
[0623] The camera control parameter calculation unit 312 calculates zoom control parameters required to capture an image having an angle of view of the size of the imaging area (height×width (H×L) of the imaging area) calculated by the optimal imaging area determination unit 311 .
[0624] The zoom control parameter calculated by the camera control parameter calculation unit 312 is output to the camera control unit 225 of the PTZ camera 100 .
[0625] The camera control unit 225 of the PTZ camera 100 sets the zoom control parameter calculated by the camera control parameter calculation unit 312 of the external device 120 in the PTZ camera, and causes the PTZ camera 100 to perform image capturing to which the zoom control parameter is applied.
[0626] Through these processes, it is possible to more reliably set a cut-out region of a prominent subject from an image captured by the PTZ camera 100 , and also to reduce degradation in image quality of the cut-out image.
[0627] Fig.28 is a diagram illustrating a configuration example of a PTZ camera 100 and an external device 120 capable of executing the above-described Embodiments 2 and 3, that is, (Example 2) Example of changing the imaging ratio according to the communication delay time between the camera and the external device (Embodiment 3) An embodiment in which the imaging ratio is changed according to the speed and acceleration of a significant subject and the communication delay time between a camera and an external device.
[0628] Fig.28 The PTZ camera 100 shown in the diagram has Fig. 27 The PTZ camera 100 shown in the figure has the same structure. Fig.28 The external device 120 shown in the diagram has a Fig. 27 The external device 120 shown in the diagram has a configuration in which a communication delay time calculation unit 313 is added.
[0629] Other components and Fig. 27 The structure shown in the figure is similar.
[0630] As above reference Figure 5 As described above, when a communication delay occurs between the PTZ camera 100 and the external device 120 that generates a captured image by inputting a captured image from the PTZ camera 100 , the possibility of a tracking error of a prominent subject or a capture image generation error increases.
[0631] Fig.28 The external device 120 illustrated in FIG. 1 has a configuration that prevents such a situation from occurring, and can change the imaging ratio according to the communication delay time between the PTZ camera 100 and the external device 120 .
[0632] The communication delay time calculation unit 313 of the external device 120 calculates the communication delay time between the PTZ camera 100 and the external device 120 .
[0633] The frame rate of the captured image (moving image) of the PTZ camera 100 is predefined, and the communication delay time calculation unit 313 of the external device 120, for example, analyzes the reception timing of the image frame received by the input unit 301 from the PTZ camera 100 to calculate the communication delay time between the PTZ camera 100 and the external device 120.
[0634] Information indicating the communication delay time between the PTZ camera 100 and the external device 120 calculated by the communication delay time calculation unit 313 is output to the optimum imaging ratio calculation unit 305 .
[0635] The optimal imaging ratio calculation unit 305 calculates the above reference Fig.19 The imaging ratio correction coefficient described above is added, and further processing according to any one of the following embodiments is performed to calculate the optimal imaging ratio.
[0636] (Example 2) Example of changing the imaging ratio according to the communication delay time between the camera and the external device (Example 3) Example of changing the imaging ratio according to the speed and acceleration of the prominent subject and the communication delay time between the camera and the external device For example, in the case of executing the processing according to the above (Example 2), executing according to Fig. 20 The process of the flowchart shown in the diagram is used to calculate the optimal imaging ratio.
[0637] Furthermore, when the processing according to the above (Embodiment 3) is performed, the processing according to Fig.21 and Fig. 22 The process of illustrating the flowchart shown in the figure is to calculate the optimal imaging ratio.
[0638] The optimal imaging ratio calculated by the optimal imaging ratio calculation unit 305 is output to the optimal imaging region determination unit 311 together with the clipped region information calculated by the clipped region calculation unit 303 .
[0639] Subsequent processing is similar to reference Fig. 27 Description of the process.
[0640] By Application Fig.28 The structure shown in the diagram can be processed according to the above-mentioned (Example 2) or (Example 3).
[0641] That is, as described in (Embodiment 2), it is possible to calculate an optimal imaging ratio corresponding to the communication delay time between the PTZ camera 100 and the external device 120, and capture an image at the calculated optimal imaging ratio.
[0642] Furthermore, as described in (Embodiment 3), it is possible to calculate an optimum imaging ratio corresponding to the speed and acceleration of a significant subject and the communication delay time between a camera and an external device, and capture an image at the calculated optimum imaging ratio.
[0643] [9. Hardware Configuration Example of Image Processing Device] Next, we will refer to Fig.29 A hardware configuration example of an image processing apparatus that executes the processing according to the above-described embodiment is described.
[0644] Fig.29 The hardware shown in the diagram is for example the hardware shown in the diagram above. Figure 20 to Figure 23 Describes an example of the hardware configuration of a camera and external devices.
[0645] Will describe Fig.29 The hardware configuration is shown in the diagram.
[0646] A central processing unit (CPU) 701 functions as a data processing unit that performs various processes according to a program stored in a read-only memory (ROM) 702 or a storage unit 708. For example, a process according to the sequence described in the above embodiment is performed. A random access memory (RAM) 703 stores a program executed by the CPU 701, data, etc. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704.
[0647] The CPU 701 is connected to an input / output interface 705 via a bus 704 , and an input unit 706 including various sensors, cameras, switches, keyboards, mice, microphones, etc. and an output unit 707 including a display, speakers, etc. are connected to the input / output interface 705 .
[0648] The storage unit 708 connected to the input / output interface 705 includes, for example, a hard disk and the like, and stores programs executed by the CPU 701 and various data. The communication unit 709 functions as a transmission / reception unit of data communication via a network such as the Internet or a local area network, and communicates with an external device.
[0649] The drive 710 connected to the input / output interface 705 drives a removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory such as a memory card, and records or reads data.
[0650] [10. Summary of the Configuration of the Present Disclosure] As described above, the embodiments of the present disclosure are described in detail with reference to specific embodiments. However, it is apparent that those skilled in the art may modify or replace the embodiments without departing from the gist of the present disclosure. That is, the present invention is disclosed in the form of examples and should not be interpreted in a restricted manner. In order to determine the gist of the present disclosure, the claims should be considered.
[0651] Note that the technology disclosed herein can have the following configurations.
[0652] (1) An image processing device, comprising: a clipping region calculation unit, wherein the clipping region calculation unit calculates an image region including a prominent subject as an image clipping region based on an image captured by a camera; a clipping execution unit, the clipping execution unit clipping the image of the image clipping area calculated by the clipping area calculation unit, and generating a clipped image; an imaging ratio calculation unit that calculates an imaging ratio for clipped image generation as an index value for executing a clipped image generation process including the prominent subject, the imaging ratio for clipped image generation being a size ratio between a size of the clipped image and a size of an imaging area equivalent to a whole image area of a captured image of the camera; and a camera control parameter calculation unit for calculating a camera control parameter for causing the camera to capture an image having an imaging area size determined based on the calculated imaging ratio for generating the clipped image, The camera is caused to capture an image to which the camera control parameters calculated by the camera control parameter calculation unit are applied.
[0653] (2) The image processing device according to (1), wherein the camera is PTZ cameras capable of pan, tilt and zoom, and The camera control parameter calculation unit is configured to A zoom control parameter for photographing an image according to a size of an imaging area of the camera calculated based on an imaging ratio for cropped image generation is calculated.
[0654] (3) The image processing device according to (1) or (2), wherein the image processing device includes a significant subject motion analysis unit, the significant subject motion analysis unit analyzing the motion of the significant subject, and The imaging ratio calculation unit is configured to: inputting the analysis result of the prominent subject motion analysis unit; and An imaging ratio for generating a cut-off image corresponding to at least one of a velocity or an acceleration of the significant subject is calculated.
[0655] (4) The image processing device according to (3), wherein the imaging ratio calculation unit is configured as follows: As the speed of the prominent subject increases, the imaging ratio for clipped image generation is calculated to be a larger value, and The camera control parameter calculation unit is configured to A zoom control parameter is calculated, the zoom control parameter increasing an imaging area of the camera as the speed of the prominent subject increases.
[0656] (5) The image processing device according to (3) or (4), wherein the imaging ratio calculation unit is configured as follows: As the acceleration of the prominent subject increases, the imaging ratio for clipped image generation is calculated to be a larger value, and The camera control parameter calculation unit is configured to A zoom control parameter is calculated, the zoom control parameter increasing an imaging area of the camera as the acceleration of the prominent subject increases.
[0657] (6) The image processing device according to any one of (3) to (5), wherein the imaging ratio calculation unit is configured as follows: An imaging ratio for clipped image generation corresponding to at least one of the velocity or the acceleration of the prominent subject is calculated as a value between a predefined allowable minimum imaging ratio and an allowable maximum imaging ratio.
[0658] (7) The image processing device according to (6), wherein the allowable maximum imaging ratio is an imaging ratio determined according to an allowable minimum number of pixels included in the clipped image.
[0659] (8) The image processing device according to any one of (3) to (7), wherein the imaging ratio calculation unit is configured as follows: The imaging ratio for clipped image generation is calculated using correspondence data between the significant subject speed and the imaging ratio for clipped image generation.
[0660] (9) The image processing device according to any one of (3) to (8), wherein the imaging ratio calculation unit is configured as follows: The imaging ratio for clipped image generation is calculated using correspondence data between the significant subject acceleration and the imaging ratio for clipped image generation.
[0661] (10) The image processing device according to any one of (3) to (9), wherein the imaging ratio calculation unit is configured as follows: comparing a speed-corresponding imaging ratio calculated according to the speed of the significant subject and an acceleration-corresponding imaging ratio calculated according to the acceleration of the significant subject; and The imaging ratio having the larger value is determined as the imaging ratio for final clipped image generation.
[0662] (11) The image processing device according to any one of (1) to (10), wherein the image processing device is an external device capable of communicating with the camera, The image processing device comprises a communication delay time calculation unit that calculates a communication delay time between the camera and the image processing device, and The imaging ratio calculation unit is configured to An imaging ratio for clipped image generation according to the communication delay time between the camera and the image processing device calculated by the communication delay time calculation unit is calculated.
[0663] (12) The image processing device according to (11), wherein the imaging ratio calculation unit is configured as follows: As the communication delay time between the camera and the image processing device increases, the imaging ratio for clipped image generation is calculated to be a larger value, and The camera control parameter calculation unit is configured to A zoom control parameter is calculated that increases an imaging area of the camera as a communication delay time between the camera and the image processing device increases.
[0664] (13) The image processing device according to (11) or (12), wherein the imaging ratio calculation unit is configured as follows: An imaging ratio for clipped image generation corresponding to a communication delay time between the camera and the image processing device is calculated as a value between a predefined allowable minimum imaging ratio and an allowable maximum imaging ratio.
[0665] (14) The image processing device according to any one of (11) to (13), wherein the imaging ratio calculation unit is configured as follows: The imaging ratio for clipped image generation is calculated using correspondence data between a communication delay time between the camera and the image processing device and an imaging ratio correction coefficient.
[0666] (15) The image processing device according to (14), wherein the imaging ratio correction coefficient is a multiplication coefficient for calculating an imaging ratio for clipped image generation by multiplying a predefined permissible minimum imaging ratio.
[0667] (16) An image processing device according to (14) or (15), wherein the imaging ratio correction coefficient is a multiplication coefficient for calculating the imaging ratio for generating a final cropped image by multiplying the imaging ratio corresponding to the subject movement calculated based on the movement of the prominent subject by the imaging ratio correction coefficient.
[0668] (17) The image processing device according to any one of (1) to (16), wherein the image processing device is an external device capable of communicating with the camera, The image processing device comprises: a significant subject motion analysis unit, the significant subject motion analysis unit analyzing the motion of the significant subject; and a communication delay time calculation unit that calculates a communication delay time between the camera and the image processing device, and The imaging ratio calculation unit is configured to: inputting the analysis result of the significant subject motion analysis unit and the communication delay time between the camera and the image processing device calculated by the communication delay time calculation unit; and The imaging ratio for generating the cut-out image is calculated according to the speed and acceleration of the prominent subject and the communication delay time between the camera and the image processing device.
[0669] (18) An image processing method executed in an image processing device, the method comprising: a clipping area calculation step, in which a clipping area calculation unit calculates an image area including a prominent subject as an image clipping area based on an image captured by a camera; A capture execution step, in which a capture execution unit captures the image of the image capture area calculated in the capture area calculation step and generates a capture image; an imaging ratio calculation step of calculating, by an imaging ratio calculation unit, an imaging ratio for cropped image generation as an index value for executing cropped image generation processing including the prominent subject, the imaging ratio for cropped image generation being a size ratio between the size of the cropped image and the size of an imaging area equivalent to the entire image area of the captured image of the camera; and a camera control parameter calculation step, in which a camera control parameter calculation unit calculates a camera control parameter for causing the camera to capture an image having an imaging area size determined based on the calculated imaging ratio for generating the clipped image; The camera is caused to capture an image to which the camera control parameters calculated by the camera control parameter calculation unit are applied.
[0670] (19) A program for causing an image processing device to execute image processing, the program causing the image processing device to execute: A clipping area calculation step, wherein the clipping area calculation unit calculates an image area including a prominent subject as an image clipping area based on an image captured by the camera; A capture execution step, causing the capture execution unit to capture the image of the image capture area calculated in the capture area calculation step and generate a capture image; a cropped image generation imaging ratio calculation step of causing an imaging ratio calculation unit to calculate an imaging ratio for cropped image generation as an index value for executing cropped image generation processing including the prominent subject, the cropped image generation imaging ratio being a size ratio between a size of the cropped image and a size of an imaging area equivalent to a whole image area of an image captured by the camera; and a camera control parameter calculation step of causing a camera control parameter calculation unit to calculate camera control parameters for causing the camera to capture an image having an imaging area size determined based on the calculated imaging ratio for generating the clipped image, The camera is caused to capture an image to which the camera control parameters calculated by the camera control parameter calculation unit are applied.
[0671] In addition, a series of processes described in this article can be performed by hardware, software or a composition obtained by combining hardware and software. In the case of performing the process with software, the program in which the process sequence is recorded can be installed in a memory in a computer incorporated into special hardware and the program can be executed, or the program can be installed in a general-purpose computer capable of performing various processes and the program can be executed. For example, the program can be recorded in a recording medium in advance. In addition to being installed in a computer from a recording medium, the program can also be received via a network such as a local area network (LAN) or the Internet and installed in a recording medium such as a built-in hard disk.
[0672] Note that the various processes described herein may be performed not only sequentially according to the description, but also in parallel or individually according to the processing capability of the device performing the process or according to necessity. In addition, the system described herein is a logical collection of multiple devices, and is not limited to a system in which the various devices are in the same housing.
[0673] Industrial Applicability As described above, according to the configuration of an embodiment of the present disclosure, a device and method for calculating an optimal imaging ratio (imaging ratio for cropped image generation) is implemented, wherein the optimal imaging ratio is an indicator value for more reliably performing cropped image generation processing including a prominent subject based on an image captured by a PTZ camera.
[0674] Specifically, for example, in a configuration in which an image cut-out area including a prominent subject is calculated based on a captured image of a PTZ camera, and an image of the calculated area is cut out to generate a cut-out image, an optimal imaging ratio (imaging ratio for cut-out image generation) which is a ratio of the size of the imaging area of the camera to the cut-out image is calculated as an index value for more reliably performing a cut-out image generation process including the prominent subject. Furthermore, a zoom parameter for capturing an image corresponding to the size of the imaging area of the camera calculated based on the optimal imaging ratio is calculated, and the PTZ camera is caused to perform image capturing to which the calculated parameter is applied.
[0675] With this configuration, an apparatus and method for calculating an optimal imaging ratio (imaging ratio for cropped image generation) which is an index value for more reliably performing a process of generating a cropped image including a prominent subject based on an image captured by a PTZ camera are realized.
[0676] Reference numerals list 10PTZ Camera 20 Capture images 21 Important Subjects 22 Imaging Area 23 Intercept area 30 External devices 100PTZ Camera 201 Imaging Unit 202 Image Analysis Unit 203 Intercepted area calculation unit 204: Subject motion (speed or acceleration) analysis unit 205 Optimal imaging ratio calculation unit 206 Intercept Execution Unit 207 Output Unit 208 Record Processing Unit 209 Recording Media 211 Optimal imaging area determination unit 212 Camera control parameter calculation unit 213 Camera control unit 221 Imaging Unit 222 Output Unit 223 Record Processing Unit 224 Recording Media 225 Camera control unit 301 Input Unit 302 Image Analysis Unit 303 Intercepted area calculation unit 304: Subject motion (speed or acceleration) analysis unit 305 Optimal imaging ratio calculation unit 306 Intercept Execution Unit 307 Output Unit 308 Record Processing Unit 309 Recording Media 311 Optimal Imaging Area Determination Form 312 Camera control parameter calculation unit 313 Communication delay time calculation unit 701CPU 702ROM 703RAM 704 bus 705 Input / Output Interface 706 Input Unit 707 Output Unit 708 Storage Unit 709 Communication Unit 710 Drive 711 Removable Media
Claims
1. An image processing device, comprising: a clipping region calculation unit, wherein the clipping region calculation unit calculates an image region including a prominent subject as an image clipping region based on an image captured by a camera; a clipping execution unit, the clipping execution unit clipping the image of the image clipping area calculated by the clipping area calculation unit, and generating a clipped image; an imaging ratio calculation unit that calculates an imaging ratio for clipped image generation as an index value for executing a clipped image generation process including the prominent subject, the imaging ratio for clipped image generation being a size ratio between a size of the clipped image and a size of an imaging area equivalent to a whole image area of a captured image of the camera; and a camera control parameter calculation unit for calculating a camera control parameter for causing the camera to capture an image having an imaging area size determined based on the calculated imaging ratio for generating the clipped image, The camera is caused to capture an image to which the camera control parameters calculated by the camera control parameter calculation unit are applied.
2. The image processing device according to claim 1, wherein the camera is PTZ cameras capable of pan, tilt and zoom, and The camera control parameter calculation unit is configured to A zoom control parameter for photographing an image according to a size of an imaging area of the camera calculated based on an imaging ratio for cropped image generation is calculated.
3. The image processing device according to claim 1, wherein the image processing device comprises a significant subject motion analysis unit, the significant subject motion analysis unit analyzing the motion of the significant subject, and The imaging ratio calculation unit is configured to: inputting the analysis result of the prominent subject motion analysis unit; and An imaging ratio for generating a cut-off image corresponding to at least one of a velocity or an acceleration of the significant subject is calculated.
4. The image processing apparatus according to claim 3, wherein the imaging ratio calculation unit is configured to: As the speed of the prominent subject increases, the imaging ratio for clipped image generation is calculated to be a larger value, and The camera control parameter calculation unit is configured to A zoom control parameter is calculated, the zoom control parameter increasing an imaging area of the camera as the speed of the prominent subject increases.
5. The image processing apparatus according to claim 3, wherein the imaging ratio calculation unit is configured to: As the acceleration of the prominent subject increases, the imaging ratio for clipped image generation is calculated to be a larger value, and The camera control parameter calculation unit is configured to A zoom control parameter is calculated, the zoom control parameter increasing an imaging area of the camera as the acceleration of the prominent subject increases.
6. The image processing apparatus according to claim 3, wherein the imaging ratio calculation unit is configured to: An imaging ratio for clipped image generation corresponding to at least one of the velocity or the acceleration of the prominent subject is calculated as a value between a predefined allowable minimum imaging ratio and an allowable maximum imaging ratio. 7 . The image processing apparatus according to claim 6 , wherein the permissible maximum imaging ratio is an imaging ratio determined based on a permissible minimum number of pixels included in the clipped image.
8. The image processing apparatus according to claim 3, wherein the imaging ratio calculation unit is configured to: The imaging ratio for clipped image generation is calculated using correspondence data between the significant subject speed and the imaging ratio for clipped image generation.
9. The image processing apparatus according to claim 3, wherein the imaging ratio calculation unit is configured to: The imaging ratio for clipped image generation is calculated using correspondence data between the significant subject acceleration and the imaging ratio for clipped image generation.
10. The image processing apparatus according to claim 3, wherein the imaging ratio calculation unit is configured to: comparing a speed-corresponding imaging ratio calculated according to the speed of the significant subject and an acceleration-corresponding imaging ratio calculated according to the acceleration of the significant subject; and The imaging ratio having the larger value is determined as the imaging ratio for generating the final clipped image.
11. The image processing device according to claim 1, wherein the image processing device is an external device capable of communicating with the camera, The image processing device comprises a communication delay time calculation unit that calculates a communication delay time between the camera and the image processing device, and The imaging ratio calculation unit is configured to An imaging ratio for clipped image generation according to the communication delay time between the camera and the image processing device calculated by the communication delay time calculation unit is calculated.
12. The image processing apparatus according to claim 11, wherein the imaging ratio calculation unit is configured to: As the communication delay time between the camera and the image processing device increases, the imaging ratio for clipped image generation is calculated to be a larger value, and The camera control parameter calculation unit is configured to A zoom control parameter is calculated that increases an imaging area of the camera as a communication delay time between the camera and the image processing device increases.
13. The image processing apparatus according to claim 11, wherein the imaging ratio calculation unit is configured to: An imaging ratio for clipped image generation corresponding to a communication delay time between the camera and the image processing device is calculated as a value between a predefined allowable minimum imaging ratio and an allowable maximum imaging ratio.
14. The image processing apparatus according to claim 11, wherein the imaging ratio calculation unit is configured to: The imaging ratio for clipped image generation is calculated using correspondence data between a communication delay time between the camera and the image processing device and an imaging ratio correction coefficient. 15 . The image processing apparatus according to claim 14 , wherein the imaging ratio correction coefficient is a multiplication coefficient for calculating an imaging ratio for clipped image generation by multiplying a predefined permissible minimum imaging ratio.
16. An image processing device according to claim 14, wherein the imaging ratio correction coefficient is a multiplication coefficient for calculating an imaging ratio for generating a final clipped image by multiplying the imaging ratio corresponding to the subject movement calculated according to the movement of the prominent subject by the imaging ratio correction coefficient.
17. The image processing device according to claim 1, wherein the image processing device is an external device capable of communicating with the camera, The image processing device comprises: a significant subject motion analysis unit, the significant subject motion analysis unit analyzing the motion of the significant subject; and a communication delay time calculation unit that calculates a communication delay time between the camera and the image processing device, and The imaging ratio calculation unit is configured to: inputting the analysis result of the significant subject motion analysis unit and the communication delay time between the camera and the image processing device calculated by the communication delay time calculation unit; as well as The imaging ratio for generating the cut-out image is calculated according to the speed and acceleration of the prominent subject and the communication delay time between the camera and the image processing device.
18. An image processing method performed in an image processing device, the method comprising: a clipping area calculation step, in which a clipping area calculation unit calculates an image area including a prominent subject as an image clipping area based on an image captured by a camera; A capture execution step, in which a capture execution unit captures the image of the image capture area calculated in the capture area calculation step and generates a capture image; an imaging ratio calculation step, in which an imaging ratio calculation unit calculates an imaging ratio for clipped image generation as an index value for executing a clipped image generation process including the prominent subject, the imaging ratio for clipped image generation being a size ratio between a size of the clipped image and a size of an imaging area corresponding to a whole image area of an image captured by the camera; as well as a camera control parameter calculation step, in which a camera control parameter calculation unit calculates a camera control parameter for causing the camera to capture an image having an imaging area size determined based on the calculated imaging ratio for generating the clipped image; The camera is caused to capture an image to which the camera control parameters calculated by the camera control parameter calculation unit are applied.
19. A program for causing an image processing device to perform image processing, the program causing the image processing device to perform: A clipping area calculation step, wherein the clipping area calculation unit calculates an image area including a prominent subject as an image clipping area based on an image captured by the camera; A capture execution step, causing the capture execution unit to capture the image of the image capture area calculated in the capture area calculation step and generate a capture image; a cropped image generation imaging ratio calculation step of causing an imaging ratio calculation unit to calculate an imaging ratio for cropped image generation as an index value for executing cropped image generation processing including the prominent subject, the cropped image generation imaging ratio being a size ratio between a size of the cropped image and a size of an imaging area equivalent to a whole image area of an image captured by the camera; and a camera control parameter calculation step of causing a camera control parameter calculation unit to calculate camera control parameters for causing the camera to capture an image having an imaging area size determined based on the calculated imaging ratio for generating the clipped image, The camera is caused to capture an image to which the camera control parameters calculated by the camera control parameter calculation unit are applied.
