Information processing device, information processing method, and computer program

Abstract

To provide an information processing device that can control a tracking target to be included in a cropped image generated for tracking a subject.SOLUTION: An information processing device includes: crop region determining means that determines crop regions for images acquired in a time series; cropping means that generates cropped images from the images according to the crop regions; and tracking region detection means that detects tracking regions for a tracking target in the cropped images. The crop region determining means determines the crop region for a current frame such that the tracking target is included in the crop region for the current frame on the basis of the tracking region calculated by the tracking region detection means for the previous frame.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, a computer program, and the like. [Background technology]

[0002] A variety of methods have been proposed for computers and other machines to learn from images and recognize object regions. These recognition methods are called recognition tasks here.

[0003] Recognition tasks include, for example, a detection task to detect human body parts (head, face, upper body, whole body, etc.) from an image, and a tracking task to search for and track a specific subject from an image. If the area of ​​an object in an image can be identified through a detection task or tracking task, for example, in a camera, the lens can be focused on that area.

[0004] Furthermore, the exposure of that area can be adjusted appropriately. This dramatically improves user operability. The present invention can be used for a variety of purposes, not just cameras.

[0005] Neural networks (hereafter referred to as "NN") are known as a technology for learning and executing the above recognition tasks. NN is an abbreviation for Neural Networks. A deep (large number of layers) multi-layer NN is called a deep NN (DNN).

[0006] DNN is an abbreviation for Deep Neural Networks. In particular, deep convolutional NNs are called DCNNs. DCNN is an abbreviation for Deep Convolutional Neural Networks.

[0007] DCNN is known for its high performance (detection accuracy, detection performance). In recent years, a technology called Vision Transformer, which incorporates an attention mechanism into image recognition, has also been attracting attention.

[0008] When using recognition tasks such as camera autofocus (AF), high-speed response is required, and there are limitations on the circuit size that can be installed on the device, so there are also limitations on computational resources. As a result, the input resolution of the NN that can be installed on the device cannot be very high.

[0009] On the other hand, it is desirable for camera AF to focus on a local area of ​​the subject, such as the pupil of a human or animal, the tip of an airplane, etc. Since a local area is generally small compared to the entire subject, it is desirable to process the local area while it is still captured at high resolution.

[0010] Furthermore, in the tracking task, the amount of information increases when the size of the object in the image is large, so it is expected that the tracking accuracy will improve. Therefore, it is desirable to process the object after capturing it at high resolution.

[0011] To achieve both of these, for example, in a tracking task, which is a type of recognition task, a crop region containing the main subject is calculated for input data based on the subject detection or tracking results of the previous frame.The crop region is then resized, and an image (hereinafter referred to as a cropped image) cut out from the input data in the enlarged or reduced (hereinafter referred to as resized) region is generated and tracking processing is performed.Similarly, local region detection tasks also perform processing on the cropped image.

[0012] One method for generating a cropped image from input data is to use the size (for example, area) of the area where the main subject exists as a reference and calculate the crop range by multiplying it by a fixed number.

[0013] The advantage of the area-based crop image generation method is that the area of ​​the main subject in the cropped image can be kept constant even if there are differences in the area of ​​the main subject in the time-series data. If the area of ​​the main subject is a rectangle (bounding box), cropping can be performed based on the height and width of the rectangle.

[0014] Furthermore, in Patent Document 1, a subject tracking task is performed using DCNN, and the tracking difficulty is quantified based on whether there are any objects in the background that are the same color as the periphery of the output result of the tracking task, and whether the size of the tracking target area is small, and a crop range is calculated based on the tracking difficulty.

[0015] Furthermore, in Patent Document 2, a multitasking DCNN that performs multiple tasks, such as a tracking task and a task of detecting detailed parts of the main subject, learns the crop range using time-series data, and estimates the optimal crop range from the output results of the tracking task and the detection task. [Prior art documents] [Patent documents]

[0016] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-11441 [Patent Document 2] Japanese Patent Publication No. 2023-110521 Summary of the Invention [Problem to be solved by the invention]

[0017] In a method of generating a cropped image based on the size of the area where the main subject exists, if a local area of ​​the main subject exists on the edge of the main subject, it may not fit within the range of the cropped image. In particular, if the main subject is long horizontally or vertically, setting the size of the crop range based on the area may cause the local area to not fit within the cropped area.

[0018] If such a local area exists outside the cropped image, it will be excluded from processing, making it impossible to detect the local area. Furthermore, the local area will remain undetected in the cropped image used to detect the area where the main subject exists, even in the next frame and thereafter.

[0019] The present invention aims to solve the above-mentioned problem and to provide an information processing device that can be controlled so that a tracking target is included in a cropped image generated for tracking a subject. [Means for solving the problem]

[0020] An information processing device according to one aspect of the present invention includes: a crop region determining means for determining a crop region for the images acquired in time series; a cropping means for generating a cropped image from the image in accordance with the cropping area; a tracking area detection means for detecting a tracking area of ​​a tracking target in the cropped image, The crop area determination means determines the crop area of ​​the current frame based on the tracking area calculated by the tracking area detection means in the previous frame so that the tracking target fits within the crop area of ​​the current frame. [Effects of the Invention]

[0021] According to the present invention, it is possible to realize an information processing device that can perform control so that a tracking target fits into a cropped image generated for tracking a subject. [Brief explanation of the drawings]

[0022] [Figure 1] 1A is a diagram showing an example of the hardware configuration of an information processing device 110 according to the first embodiment, and FIG. 1B is a functional block diagram showing an example of the configuration of functional blocks of the information processing device according to the first embodiment. [Figure 2] FIG. 2 is a diagram for explaining the flow of data in the information processing apparatus 110 of the first embodiment. [Figure 3] 10 is a flowchart showing an example of processing in the information processing apparatus 110 of the first embodiment. [Figure 4] 10A and 10B are diagrams showing the flow of processing for determining a crop reference area according to the first embodiment. [Figure 5]10A is a functional block diagram showing an example of the functional arrangement of an information processing device 502 according to a second embodiment, and FIG. 10B is a diagram for explaining the flow of data in the information processing device 502 according to the second embodiment. [Figure 6] 10 is a flowchart showing an example of processing in an information processing apparatus 502 according to the second embodiment. [Figure 7] 10A and 10B are diagrams illustrating an example of a process for determining a crop reference area according to the second embodiment. [Figure 8] 11A and 11B are diagrams illustrating the flow of processing for determining a crop reference area according to the third embodiment. [Figure 9] 13A and 13B are diagrams illustrating the flow of processing for determining a crop reference area according to the fourth embodiment. [Figure 10] 10 is a flowchart showing an example of processing in an information processing device according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0023] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same members or elements are designated by the same reference numerals, and duplicate descriptions will be omitted or simplified.

[0024] In the following explanation, the time when a video frame (hereinafter abbreviated as frame) is acquired will be represented by t, the first frame image acquired will be t=1, the time of the current frame image will be t=T, the time of the previous frame image will be t=T-1, the time of the next frame image will be t=T+1, etc.

[0025] In addition, the pre-trained model in the following description refers to a model that has been trained using DCNN (Deep Convolutional Neural Networks) to detect the target object.

[0026] In the following, an example will be described in which a vehicle photographed by a camera is tracked as a subject, and a specific part of the subject (for example, the front part of an airplane, hereinafter referred to as a local area) is detected in parallel. However, the subject is not limited to a vehicle, and the present invention can also be applied to, for example, the head or ankles of a person, or the head or tail of an animal.

[0027] (Embodiment 1) In the first embodiment, when the detection accuracy of the position and size of the tracking region is insufficient, a crop region is calculated using the detection results of the local region of the subject and the tracking region.

[0028] Fig. 1(A) is a diagram showing an example of the hardware configuration of an information processing device 110 according to embodiment 1. In Fig. 1(A), a CPU 101 as a computer executes a control computer program stored in a ROM 103 to control the entire information processing device.

[0029] The RAM 102 is used as a temporary storage area such as the main memory and work area of ​​the CPU 101, and develops control computer programs to make them executable by the CPU 101. The input unit 105 is composed of a keyboard, touch panel, etc., and is used to receive input from the user, and can also receive image input, etc.

[0030] The display unit 106 is configured with a liquid crystal display or the like, and can display various data and processing results to the user. The information processing device 110 can also communicate with other devices via the communication unit 104, and receives image inputs from other devices, acquires pre-trained models, and receives instructions from users via the communication unit 104. The information processing device 110 also outputs processing results to other devices.

[0031] The storage unit 107 stores data used in the processing of this embodiment, for example, a trained model. The storage unit 107 may be configured as a medium such as a hard disk drive (HDD), a flash memory, or various optical media.

[0032] In this embodiment, a frame that serves as a reference when determining a cropping area (hereinafter referred to as a cropping reference area) is calculated using the presence area of ​​the subject to be tracked (hereinafter referred to as a tracking area) and local areas calculated by a pre-trained DCNN.

[0033] Fig. 1(B) is a functional block diagram showing an example of the configuration of functional blocks of an information processing device according to embodiment 1. Note that some of the functional blocks shown in Fig. 1(B) are realized by causing a CPU or the like serving as a computer included in the information processing device to execute a computer program stored in a memory serving as a storage medium.

[0034] However, some or all of these functions may be implemented by hardware, which may be a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP).

[0035] Furthermore, the functional blocks shown in Fig. 1(B) do not have to be built into the same housing, but may be configured as separate devices connected to each other via signal paths. The above explanation regarding Fig. 1(B) also applies to Fig. 5(A).

[0036] 1(B), an image acquisition unit 111 receives input of time-series data from a user. A cropping area determination unit 112 determines a cropping area for the next frame using information output from a tracking area detection unit 115 and a local area detection unit 116, which will be described later. Here, the cropping area determination unit 112 functions as a cropping area determination means that determines a cropping area for images acquired in time series.

[0037] The cropping unit 113 generates a cropped image from the frame based on the cropping area determined by the cropping area determination unit 112. Here, the cropping unit 113 functions as a cropping means that generates a cropped image from an image according to the cropping area.

[0038] The tracking area detection unit 115 uses a pre-trained DCNN to calculate the area in the cropped image where the subject exists as a rectangle that holds four parameters: the x-coordinate, y-coordinate, width, and height of the center coordinates of the tracking area.

[0039] Here, the tracking area detection unit 115 functions as a tracking area detection means that detects the tracking area of ​​the tracking target in the cropped image. Note that the information held as parameters by the rectangle of the tracking area is not limited to the above four parameters as long as it can represent a rectangle.

[0040] The local area detection unit 116 uses a pre-trained DCNN to calculate the area where the local area of ​​the subject of the cropped image (for example, the front part of an airplane) exists as a rectangle that holds four parameters: the x-coordinate, y-coordinate, width, and height of the center coordinate of the local area.

[0041] As with the output from the tracking area detection unit 115, the information held as parameters by the rectangle of the local area is not limited to the above four parameters as long as it can represent a rectangle. Note that the local area detection unit 116 functions as a local area detection means that determines at least one local area of ​​the tracking target.

[0042] The pre-trained model used in the tracking area detection unit 115 and the local area detection unit 116 may be a single pre-trained model for multi-task learning that performs multiple recognition tasks, or multiple pre-trained models may be used to specialize for each recognition task.

[0043] Next, the flow of processing in the information processing device 110 in this embodiment will be described with reference to FIGS.

[0044] FIG. 2 is a diagram for explaining the flow of data in the information processing apparatus 110 of the first embodiment, and FIG. 3 is a flowchart showing an example of processing in the information processing apparatus 110 of the first embodiment.

[0045] The CPU or the like serving as a computer within the information processing device 110 executes a computer program stored in memory, thereby sequentially performing the operations of the steps in the flowchart of FIG.

[0046] 3, the image acquisition unit 111 acquires an image at time t = 1. In step S301, the cropping region determination unit 112 determines a cropping region for the frame acquired at time t = 1. Note that step S301, together with step S307 described below, functions as a cropping region determination step for determining a cropping region for images acquired in time series.

[0047] To determine the crop area, the subject presence area of ​​the previous frame (hereafter referred to as the crop reference area) is required, but since there is no previous frame at time t = 1, it cannot be used. Therefore, the user must somehow initialize the subject presence area.

[0048] For example, the subject presence region may be determined as a region of a certain size based on the center position of the input image, or may be determined by detecting an object near coordinates in the input image specified by the user using a DCNN pre-trained on an object detection task.

[0049] In this embodiment, the crop area is a rectangle that holds four parameters: the x-coordinate of the center in the input image coordinate system, the y-coordinate, the width, and the height. Once the subject existence area is determined, a constant multiple (for example, 2 times) of the area of ​​the subject existence area is determined as the crop area.

[0050] In step S302, a loop is started for all input time-series data. Specifically, the processes of steps S303 to S308 are repeated. In step S303, the image acquisition unit 111 acquires an image at time t=T.

[0051] In step S304, the cropping unit 113 generates a cropped image from the cropping region. Note that step S304 functions as a cropping step for generating a cropped image from the image in accordance with the cropping region.

[0052] At time t=1, a cropped image is generated from the cropping area calculated in step S301. At time t=T (t≠1), a cropped image 211 is generated from the cropping area calculated in step S306 for the previous frame, time t=T−1, as shown in FIG.

[0053] In this embodiment, an example will be described in which a cropping area calculated for one frame before is used, but a cropping area calculated for two or more frames before may also be used, which can reduce the amount of processing when the frame rate is high or the subject movement is small.

[0054] The cropped image is input to a tracking region detection unit 115 and a local region detection unit 116, which are pre-trained DCNNs. When DCNN is implemented using hardware such as a circuit, it is desirable to set the image size of the input data to a fixed value, so the image size of the cropped image 211 is resized to be a constant size throughout all frames, regardless of the area of ​​the cropped region.

[0055] For example, consider the case where the original image size before cropping is full HD (1920x1200 pixels) and the image size of the input data for DCNN is VGA (640x480 pixels). In this case, the cropped image is resized to, for example, VGA size for all frames.

[0056] In step S305, the cropped image 211 at time t=T generated in step S303 is input, and the tracking area of ​​the subject is calculated by the tracking area detection unit 115. Here, step S305 functions as a tracking area detection step that detects the tracking area of ​​the tracking target in the cropped image.

[0057] 4A and 4B are diagrams showing the flow of processing for determining the crop reference area in embodiment 1. In the following description, as shown in FIG. 4A, the upper left edge 405 of the tracking area 402 is set to (lxb, lyb) and the lower right edge 406 of the tracking area is set to (rxb, ryb).

[0058] In step S306, the cropped image 211 at time t=T generated in step S303 is input, and the local area detection unit 116 calculates local areas of the subject.

[0059] In the following description, as shown in FIG. 4A, the upper left end 403 of a local region 401 is defined as (lxa, lya), and the lower right end 404 of the local region is defined as (rxa, rya).

[0060] In step S307, a crop region at time t=T+1 is calculated using the tracking region 402 calculated in step S305 and the local region 401 calculated in step S306. Note that step S307, together with step S301, functions as a crop region determination step for determining a crop region for images acquired in time series.

[0061] In this case, if the detection accuracy of the tracking area is not sufficiently high, and the result of the detection of the tracking area is used as the cropping reference area, the cropping area may become inappropriate.

[0062] Therefore, in step S307 of this embodiment, the cropping standard is determined using not only the detection result of the tracking region but also the detection result of the local region. The processing in step S307 will be described with reference to FIG.

[0063] (i) First, a corrected tracking region 407 in Fig. 4(A) is calculated from four points in Fig. 4(A), namely, an upper left edge 403 of the local region, a lower right edge 404 of the local region, an upper left edge 405 of the tracking region, and a lower right edge 406 of the tracking region, so as to encompass the local region 401 and the tracking region 402. That is, an upper left edge 408 (lx', ly') of the corrected tracking region 407 and a lower right edge 409 (rx', ry') of the corrected tracking region 407 are calculated using the following equations 1 and 2.

[0064]

number

number

[0065] (ii) Next, the crop reference area 410 in FIG. 4B is calculated from the four points calculated in (i): the upper left edge 408 of the corrected tracking area, the lower right edge 409 of the corrected tracking area, the upper left edge 405 of the tracking area, and the lower right edge 406 of the tracking area.

[0066] That is, the reference area upper left corner 411 and reference area lower right corner 412 of the crop reference area 410 are calculated as a weighted sum using the following equations 3 and 4. Note that α in equations 3 and 4 indicates the weight (importance) of the tracking area 402 relative to the crop reference area 410, and is defined in the range of 0 to 1.

[0067]

number

number

[0068] Note that, although this embodiment shows an example in which a weighted sum of a local region and a tracking region is used, correction does not necessarily have to be performed using a weighted sum. For example, the size and position of a corrected tracking region 407 that encompasses the local region and the tracking region may be used as a cropping reference. In this case, the corrected tracking region 407 does not have to be rectangular.

[0069] (iii) Then, an area that is a constant multiple (for example, 1.2 times) of the area of ​​the crop reference area 410 is determined as the crop area at time t=T+1. The above-mentioned magnification value used when determining the crop area can be set in advance by the user.

[0070] When determining the crop area, if the crop area is larger than the VGA size, it is resized to reduce it, or an area up to the size of the cropped image 211 is adopted as the crop area. Also, if the crop area is smaller than the VGA size, it is resized to enlarge it.

[0071] In this embodiment, the crop area is determined as a constant multiple of the area of ​​the crop reference area 410, but this is not limitative. For example, the crop area may be determined based on the length of the long side of the crop reference area 410.

[0072] In this way, in step S307, the crop area determination unit 112 determines the crop area of ​​the current frame based on the tracking area calculated in the tracking area detection step in the previous frame so that the tracking target fits within the crop area of ​​the current frame.

[0073] In the first embodiment, step S307 determines the crop region based on the tracking region and at least one local region.

[0074] Returning to FIG. 3, in step S308, the crop region calculated at time t=T is held, and the process moves to processing for the frame at time t=T+1.

[0075] Next, proceed to step S309, and if the processing of steps S303 to S308 has not been completed for all frames, return to step S302. Steps S302 to S309 are then repeated until the processing of steps S303 to S308 has been completed for all frames. If the loop processing for all frames has been completed in step S309, the flow of FIG. 3 ends.

[0076] In this embodiment, the two detection results of the tracking area and the local area are used in calculating the crop reference area, and the crop area is determined so that the local area fits within the cropped image. This makes it possible to handle situations where the detection accuracy of the position and size of the tracking area is insufficient when generating a cropped image intended to detect a local area of ​​a subject in each frame.

[0077] (Embodiment 2) In the first embodiment, the detection results of the tracking region and local region are used when calculating the crop reference region, but there is a possibility that the detection accuracy of the position and size of the tracking region and local region may not be sufficient. Therefore, in the second embodiment, detection of the whole body region of the tracking target is performed together with detection of the tracking region and local region, and the crop region is calculated using the detection results of the whole body region, the tracking region, and the local region.

[0078] Fig. 5(A) is a functional block diagram showing an example of the functional configuration of an information processing device 502 in embodiment 2. In Fig. 5(A), in addition to the tracking area detection unit 115 and the local area detection unit 116, a whole body area detection unit 501 is included as whole body area detection means that detects the whole body area of ​​the subject as a tracking target.

[0079] FIG. 5B is a diagram for explaining the flow of data in the information processing device 502 of the second embodiment, in which the tracking area detection unit 115, the local area detection unit 116, and the whole body area detection unit 501 each perform detection processing on the cropped image 211.

[0080] Fig. 6 is a flowchart showing an example of processing in the information processing device 502 of embodiment 2. Note that the operation of each step in the flowchart of Fig. 6 is performed sequentially by a CPU or the like serving as a computer in the information processing device 502 executing a computer program stored in a memory.

[0081] Steps S600 to S606, step S609, and step S610 in Fig. 6 are similar to steps S300 to S306, step S308, and step S309 in Fig. 3, and therefore will not be described further. Steps S607 and S608 will be described below.

[0082] In step S607, the cropped image 211 at time t=T generated in step S604 is input, and the whole body area detection unit 501 calculates the whole body area of ​​the subject.

[0083] 7A and 7B are diagrams for explaining an example of the process of determining the crop reference area in embodiment 2. As shown in Fig. 7A, in a whole body area 701 calculated by the whole body area detection unit 501, an upper left edge 702 of the whole body area is set to (lxc, lyc), and a lower right edge 703 of the whole body area is set to (rxc, ryc).

[0084] On the other hand, as with the first embodiment, the upper left edge 405 of the tracking area 402 is set to (lxb, lyb) and the lower right edge 406 of the tracking area is set to (rxb, ryb).

[0085] In step S608, a crop region at time t=T+1 is calculated using the whole body region 701 of the subject calculated in step S607 and, for example, the tracking region 402. That is, in step S608, the crop region determination unit 112, which serves as crop region determination means, determines a crop region based on the tracking region and the whole body region. The processing in this step will be described with reference to FIG. 7.

[0086] (i) First, a corrected tracking area 704 that surrounds the whole body area 701 and the tracking area 402 is calculated from four points in Fig. 7(A), namely, the upper left edge 702 of the whole body area, the lower right edge 703 of the whole body area, the upper left edge 405 of the tracking area, and the lower right edge 406 of the tracking area. That is, the upper left edge 705 (lx", ly") of the corrected tracking area 704 and the lower right edge 706 (rx", ry") of the corrected tracking area 704 are calculated using the following equations 5 and 6.

[0087]

number

number

[0088] (ii) Next, a crop reference area 707 in FIG. 7B is calculated from four points: the upper left edge 705 of the corrected tracking area, the lower right edge 706 of the corrected tracking area, the upper left edge 405 of the tracking area, and the lower right edge 406 of the tracking area.

[0089] That is, the reference area upper left corner 708 and reference area lower right corner 709 of the crop reference area 707 are calculated as weighted sums using the following equations 7 and 8. β in equations 7 and 8 indicates the weight (importance) of the tracking area 402 relative to the crop reference area 707, and is defined in the range of 0 to 1.

[0090]

number

number

[0091] (iii) Next, similar to the process described in step S305 in the first embodiment, an area that is a constant multiple of the area of ​​the crop reference area 707 is determined as the crop area at time t=T+1.

[0092] As described above, in the second embodiment, for example, two detection results of the tracking region and the whole-body region are used to calculate a crop region so that the local region fits within the cropped image. However, in step S608, the crop region at time t=T+1 may be calculated using the whole-body region 701 of the subject calculated in step S607 and at least one of the tracking region 402 and the local region 401.

[0093] By performing processing in this manner in the second embodiment, when generating a cropped image for the purpose of detecting local areas of a subject in each frame, it is possible to cope with situations in which the detection accuracy of the positions and sizes of the tracking area and local areas is insufficient.

[0094] (Embodiment 3) For example, in the first embodiment, a method was shown in which the coordinates of the upper left and lower right corners of the local area and the tracking area are used to calculate the upper left and lower right corners of the crop reference area, thereby determining a crop area such that the entire subject, including the local area, is present in the cropped image.

[0095] However, since the main purpose of correcting the crop reference area is to fit a local area into the cropped image, the entire subject does not necessarily need to be present in the cropped image. A specific example would be a case where the front part of an airplane, which corresponds to a local area, is present in the cropped image, but the tail is not included in the cropped image.

[0096] Therefore, in the third embodiment, the center coordinates of the crop reference area are calculated using the center coordinates of the local area and the tracking area, and a crop area is calculated so that at least the local area fits within the cropped image. That is, the crop area determination unit 112 as a crop area determination means determines the crop area based on the center positions of the local area and the tracking area.

[0097] The processing flowchart in the third embodiment is the same as that in FIG. 3, but the processing in steps S304 and S305 differs from that in the first embodiment, and therefore steps S304 and S305 will be described below.

[0098] 8A and 8B are diagrams illustrating the flow of processing for determining a crop reference area according to embodiment 3. In step S304 of embodiment 3, central coordinates of the local area 401 and the tracking area, etc., shown in FIG.

[0099] That is, in step S304, the x and y coordinates (cxa, cya) of the center coordinates 801 of the local region 401, and the x and y coordinates, width and height (cxb, cyb, Wb, Hb) of the center coordinates 802 of the tracking region are acquired.

[0100] In step S305 in the third embodiment, as shown in FIG. 8B, the coordinates obtained by correcting the center coordinate 802 of the tracking region 402 so as to approach the center coordinate 801 of the local region 401 are calculated as the center coordinate 803 of the crop reference region 804.

[0101] Specifically, when the distance between the centers of a center coordinate 801 of the local region 401 and a center coordinate 802 of the tracking region 402 is 1, the shift amount of the center coordinate of the local region 401 is set to β, and the center coordinate 803 (cx, cy) of the crop reference region 804 is calculated using the following equation 9. β is set between 0 and 1.

[0102]

number

[0103] The width and height of the crop reference area 804 are Wb and Hb, which are the width and height of the tracking area 402, and the x-coordinate, y-coordinate, width and height of the center coordinates of the crop reference area 804 calculated in step S305 are (cx, cy, Wb, Hb).

[0104] Then, an area that is a constant multiple of the calculated area of ​​the crop reference area 804 is determined as the crop area at time t=T+1.

[0105] As described above, in the third embodiment, the cropping reference area is calculated from the central coordinates of the local area and tracking area, and the width and height of the tracking area. As a result, when generating a cropped image for the purpose of detecting a local area of ​​a subject in each frame, the cropping area can be determined so that at least the local area fits within the cropped image.

[0106] (Embodiment 4) In the fourth embodiment, when determining the cropping region for the next frame based on the area of ​​the cropping reference region calculated for the current frame, the size and aspect ratio of the cropping reference region are corrected so that they are equal to or smaller than predetermined thresholds.

[0107] That is, in the fourth embodiment, the crop region determination unit 112 serving as crop region determination means corrects the crop region when the aspect ratio of the tracking region is greater than a predetermined threshold value.

[0108] 9A and 9B are diagrams illustrating the flow of processing for determining a cropping reference area according to embodiment 4. As shown in FIG. 9A, even if a cropping reference area 901 including a local area is calculated according to embodiments 1 to 3, the local area may fall outside the cropping area 902 depending on the aspect ratio of the cropping reference area.

[0109] Therefore, in the fourth embodiment, in order to always include a local region within a crop region, for example, a threshold is set for the aspect ratio of the crop reference region, and the crop reference region is corrected if the aspect ratio is greater than the threshold.

[0110] Fig. 10 is a flowchart showing an example of processing in the information processing device of embodiment 4. Note that the operation of each step in the flowchart of Fig. 10 is performed sequentially by a CPU or the like serving as a computer in the information processing device 110 executing a computer program stored in a memory.

[0111] The processing of the fourth embodiment is similar to the flow of Fig. 3, but in steps S301 and S305 of Fig. 3, processing as shown in the flowchart of Fig. 10 is executed. Note that the processing in step S305 at t=T will be described with reference to Fig. 10.

[0112] In step S1001, the parameters of the crop reference area at t=T are acquired and the aspect ratio is calculated. Note that the parameters acquired here may be the aspect ratio of the crop reference area calculated using any one of the first to third embodiments, or may be the aspect ratio of the tracking area in the cropped image generated in step S304.

[0113] However, the aspect ratio is a value that allows the width and height of the cropping reference area to be calculated, such as the x-coordinate and y-coordinate of the upper left corner and the lower right corner of the frame. In the example of the fourth embodiment, it is assumed that the cropping reference area acquired in step S305 is a rectangle with an aspect ratio such as that shown in cropping reference area 901 in Figure 9(A).

[0114] In step S1002, it is determined whether the aspect ratio of the crop reference area 901 acquired in step S1001 is greater than a predetermined threshold value. If the determination in step S1002 is Yes, the process proceeds to step S1003, and if the determination is No, the process proceeds to step S1004.

[0115] In step S1003, the length of the short side of the crop reference area 901 is extended as shown in Fig. 9B so that the aspect ratio of the crop reference area 901 matches the threshold aspect ratio. For example, if the threshold aspect ratio is 4:1 and the crop reference area 901 has a width of 700 and a height of 100, the height of the crop reference area is extended to 175, as shown in corrected crop reference area 1103 shown in Fig. 9B, and the aspect ratio is corrected to match the threshold aspect ratio.

[0116] In the fourth embodiment, the length of the short side of the crop reference area is extended, but this is not limitative, and it is sufficient that the aspect ratio of the corrected crop reference area 1103 is equal to or less than a predetermined threshold value.

[0117] In step S1004, a crop area 1104 in Fig. 9B is calculated from a corrected crop reference area 1103 having an aspect ratio equal to or less than the threshold value, as in the first to third embodiments. In this way, in the fourth embodiment, the length of the short side of the tracking area is corrected based on the aspect ratio of the tracking area to calculate the crop reference area.

[0118] Therefore, regardless of the detection accuracy of the positions and sizes of the local region and the whole body region, the crop region can be calculated so that the local region always fits within the cropped image. Note that, although the aspect ratio of the crop reference region is corrected to be equal to or less than a predetermined threshold in the fourth embodiment, the aspect ratio of the crop region may also be corrected.

[0119] The present invention has been described above in detail based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible based on the spirit of the present invention, and these are not excluded from the scope of the present invention.

[0120] The present invention also includes those that realize the functions of the above embodiments using, for example, at least one processor such as a CPU, memory, or circuit (for example, ASIC). Also, multiple processors may be used to perform distributed processing.

[0121] In order to realize part or all of the control in the above-described embodiments, a computer program that realizes the functions of the above-described embodiments may be supplied to an information processing device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the information processing device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. The present invention also includes the following combinations.

[0122] (Configuration 1) An information processing apparatus comprising: a crop area determination means for determining a crop area for images acquired in time series; a cropping means for generating a cropped image from the image according to the crop area; and a tracking area detection means for detecting a tracking area of ​​a tracking target in the cropped image, wherein the crop area determination means determines the crop area of ​​the current frame based on the tracking area calculated by the tracking area detection means in a previous frame so that the tracking target fits within the crop area of ​​the current frame.

[0123] (Configuration 2) The information processing device according to Configuration 1, further comprising a local area detection means for determining at least one local area of ​​the tracking target, wherein the crop area determination means determines the crop area based on the tracking area and the at least one local area.

[0124] (Configuration 3) The information processing device according to Configuration 2, wherein the crop region determination means determines the crop region so that the tracking region and the at least one local region fit within the crop region of the current frame.

[0125] (Configuration 4) The information processing device according to configuration 2 or 3, characterized in that the crop region determination means corrects the tracking region calculated by the tracking region detection means in the previous frame so as to include the tracking region and the at least one local region, and determines the crop region based on the corrected tracking region and the calculated tracking region.

[0126] (Configuration 5) The information processing device according to configuration 2, wherein the cropping region determining means determines the cropping region based on the center position of the local region and the center position of the tracking region.

[0127] (Configuration 6) Further comprising a whole body region detection means for detecting a whole body region of the tracking target, 6. The information processing device according to any one of configurations 1 to 5, wherein the crop region determining means determines the crop region based on the tracking region and the whole body region.

[0128] (Configuration 7) The information processing device according to any one of configurations 1 to 6, wherein the cropping region determination means corrects the cropping region when the aspect ratio of the tracking region is greater than a predetermined threshold value.

[0129] (Method) An information processing method comprising: a crop area determination step of determining a crop area for images acquired in time series; a crop step of generating a cropped image from the image in accordance with the crop area; and a tracking area detection step of detecting a tracking area of ​​a tracking target in the cropped image, wherein the crop area determination step determines the crop area of ​​the current frame based on the tracking area calculated by the tracking area detection step in a previous frame so that the tracking target fits within the crop area of ​​the current frame.

[0130] (Program) A computer program for controlling each means of the information processing device according to any one of configurations 1 to 6 by a computer. [Explanation of symbols]

[0131] 101:CPU 102:RAM 103:ROM 104: Communications Department 105: Input section 106: Display section 107: Storage section 110: Information processing device 111: Image acquisition unit 112: Crop area determination unit 113: Crop section 115: Tracking area detection unit 116: Local area detection unit

Claims

1. a crop region determining means for determining a crop region for the images acquired in time series; a cropping means for generating a cropped image from the image in accordance with the cropping area; a tracking area detection means for detecting a tracking area of ​​a tracking target in the cropped image, the crop region determination means determines the crop region of the current frame based on the tracking region calculated by the tracking region detection means in the previous frame so that the tracking target fits within the crop region of the current frame.

2. further comprising a local area detection means for determining at least one local area of ​​the tracking target; 2. The information processing apparatus according to claim 1, wherein the crop region determining means determines the crop region based on the tracking region and the at least one local region.

3. 3. The information processing apparatus according to claim 2, wherein the crop region determining means determines the crop region so that the tracking region and the at least one local region fit within the crop region of the current frame.

4. the crop region determination means corrects the tracking region calculated by the tracking region detection means in the previous frame so as to include the tracking region and the at least one local region; The information processing apparatus according to claim 2 , wherein the cropping area is determined based on the corrected tracking area and the calculated tracking area.

5. The crop region determining means The information processing apparatus according to claim 2 , wherein the cropping area is determined based on a center position of the local area and a center position of the tracking area.

6. further comprising a whole body region detection means for detecting a whole body region of the tracking target; 2. The information processing apparatus according to claim 1, wherein the crop region determining means determines the crop region based on the tracking region and the whole body region.

7. 2. The information processing apparatus according to claim 1, wherein the cropping region determining means corrects the cropping region when the aspect ratio of the tracking region is greater than a predetermined threshold value.

8. a cropping region determination step of determining a cropping region for the images acquired in time series; a cropping step of generating a cropped image from the image in accordance with the cropping region; a tracking area detection step of detecting a tracking area of ​​a tracking target in the cropped image, the cropping region determination step determines the cropping region of the current frame based on the tracking region calculated by the tracking region detection step in the previous frame so that the tracking target fits within the cropping region of the current frame.

9. A computer program for controlling each means of the information processing apparatus according to any one of claims 1 to 7 by a computer.

Images