Information processing apparatus, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing method, and a program that can create a stable cropped image.SOLUTION: An information processing apparatus comprises: an image input unit that receives one or more images to be input as input data; a tracking target specification unit that specifies a target to be tracked included in the image as a tracking target; a local area specification unit that specifies, in the image, a local area including at least part of a detection target included in the tracking target; a cropping area determination unit that determines size information of a cropping area for cutting out the tracking target from the image on the basis of size information of the local area; and a cropping unit that cuts out an image on the basis of the cropping area to create a cropped image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, a technique has become known in which a partial region including a target to be tracked (hereinafter referred to as the tracking target) in an image is cut out (hereinafter referred to as a crop) and the target is tracked using an image of the partial region (hereinafter referred to as a crop image).

[0003] Patent Document 1 discloses a technology that uses information about the size of the entire body of a main subject and information about other parts of the main subject to determine a single cropped image for performing multiple recognition tasks. The technology in Patent Document 1 crops an image to generate a cropped image by cutting out a partial region that includes the subject from the entire image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-141421 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology of Patent Document 1, when determining the cropped image, the cropping is based on the overall size of the tracking target, which is easily affected by changes in the tracking target's posture, so the size of the cropped area is likely to change due to changes in the tracking target's posture, and the cropped image is not stable.

[0006] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an information processing device, an information processing method, and a program that can generate a stable cropped image. [Means for solving the problem]

[0007] In order to solve this problem, for example, an information processing device of the present invention has the following arrangement: a tracking target identification means for identifying a target to be tracked included in an image as the tracking target; a local area specifying means for specifying, within the image, a local area including at least a part of a detection target included in the tracking target; a crop region determination means for determining size information of a crop region for cutting out the tracking target from the image based on size information of the local region; a cropping unit that cuts out the image based on the crop area to generate a cropped image; Equipped with. [Effects of the Invention]

[0008] According to the present invention, a stable cropped image can be generated. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a hardware configuration diagram of an information processing apparatus according to an embodiment. [Figure 2] FIG. 2 is a functional block diagram illustrating functions of the information processing apparatus according to the first to fourth embodiments. [Figure 3] 1A to 1C are diagrams showing time-series images for explaining the first and second embodiments. [Figure 4] FIG. 4 is a flowchart showing a recognition process according to the first to fourth embodiments. [Figure 5] FIG. 4 is a flowchart showing a crop region determination process according to the first and second embodiments. [Figure 6] FIG. 10 is a diagram of pre-registration information according to the first to fifth embodiments. [Figure 7] 10A to 10C are diagrams showing time-series images for explaining the third and fourth embodiments. [Figure 8] FIG. 11 is a flowchart showing a crop region determination process according to the third embodiment. [Figure 9] FIG. 13 is a flowchart showing a crop region determination process according to the fourth embodiment. [Figure 10]FIG. 10 is a diagram illustrating the configuration of an information processing apparatus according to a fifth embodiment. [Figure 11] 13A to 13C are diagrams showing time-series images for explaining the fifth embodiment. [Figure 12] FIG. 10 is a flowchart of the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] <First embodiment: Cropping method using local areas with low size fluctuation rate against posture changes> 1 shows an example of a hardware configuration diagram of an information processing device 200. The information processing device 200 may be a so-called computer. The information processing device 200 has a CPU 100, a ROM 110, a RAM 120, a HDD 130, an input unit 140, a display unit 150, a communication unit 160, and a bus 170. The CPU 100, the ROM 110, the RAM 120, the HDD 130, the input unit 140, the display unit 150, and the communication unit 160 are connected via the bus 170 so as to be able to send and receive information.

[0012] CPU 100 is an abbreviation for Central Processing Unit, and is a central processing unit. Instead of or in addition to CPU 100, information processing device 200 may have other processors such as an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), or a QPU (Quantum Processing Unit). CPU 100 performs calculations and logical decisions for various processes. For example, CPU 100 reads out programs stored in ROM 110 or HDD 130 and loads them into RAM 120 to realize various functions and execute various processes. Furthermore, some or all of the functions of information processing device 200 may be realized by one or more circuits, such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array).

[0013] The ROM 110 is an abbreviation for Read-Only-Memory and is a non-volatile memory that stores control programs such as an OS (Operating System).

[0014] The RAM 120 is an abbreviation for Random Access Memory, and is used as the main memory of the CPU 100 and as a temporary storage area such as a work area.

[0015] HDD 130 is an abbreviation for Hard Disk Drive and is a large-capacity nonvolatile storage device. HDD 130 stores electronic data, programs, and data required for executing programs according to this embodiment. Instead of or in addition to HDD 130, information processing device 200 may have an external storage device that performs a similar function. Here, the external storage device can be realized, for example, by media (recording media) and an external storage drive for realizing access to the media. Known examples of such media include flexible disks (FDs), CD-ROMs, DVDs, USB memories, MOs, and flash memories. The external storage device may also be a server device connected via a network.

[0016] The input unit 140 receives input from the user and passes it to the CPU 100. The input unit 140 includes a mouse, a keyboard, a touch panel, and the like.

[0017] Display unit 150 displays images such as various data and processing results to the user based on image data acquired from CPU 100, etc. Display unit 150 is configured with a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. The term "image" may be used to include a still image, a video, an image of one frame of a video, and data thereof.

[0018] The communication unit 160 relays communication with other devices. As a result, the information processing device 200 communicates data with other devices via the communication unit 160. The information processing device 200 may receive instructions from a user from other devices via the communication unit 160, or may output processing results and the like to other devices.

[0019] FIG. 2 is a functional block diagram illustrating functions of an information processing device 200 according to this embodiment. The configuration of this embodiment will be described using FIG. 2. Note that only an overview will be described here, and details will be described later. The information processing device 200 has the functions of an image input unit 220, a tracking target setting unit 230, a crop region determination unit 240, a cropping unit 250, a multitasking unit 260, a tracking target identification unit 270, a local region identification unit 280, and an output unit 290. For example, the CPU 100 realizes the functions of the image input unit 220, the tracking target setting unit 230, the crop region determination unit 240, the cropping unit 250, the multitasking unit 260, the tracking target identification unit 270, the local region identification unit 280, and the output unit 290 by reading and executing a program stored in the ROM 110 or the HDD 130.

[0020] The input data 210 represents data of a captured image or a group of images including a plurality of images. The input data 210 is, for example, a plurality of images in time series obtained from an imaging device such as a digital camera or a surveillance camera.

[0021] The image input unit 220 receives one or more images input as input data 210. The input data 210 is, for example, a moving image in which a plurality of frame images are successively generated in time series.

[0022] The tracking target setting unit 230 sets at least one of the type of the tracking target to be tracked in the initial frame of the input data 210, its position in the image, and its size indicated by the number of pixels, etc. The type of the tracking target may be, for example, any of a person, an animal including a cat or a dog, a car, etc.

[0023] The crop region determination unit 240 determines a crop region for cutting out the tracking target from the image passed from the image input unit 220. Specifically, the crop region determination unit 240 determines the size information of the crop region based on size information of a local region that includes at least a portion of a detection target included in the tracking target. The size information is information about the size of an area within the image (here, the local region) and may be at least one of the number of pixels within the area and the length and width of the area. The detection target is a portion of the tracking target, and in the case of an animal, for example, includes the head, face, etc. There may be multiple detection targets. For example, the detection target may include the head, face, feet, hands, etc. When multiple types of detection targets are detected, the crop region determination unit 240 may select a detection target for determining the size information of the crop region from the multiple types of detection targets in accordance with a priority order described below. In this case, the crop region determination unit 240 may determine the size information of the crop region based on the size information of the local region of the selected detection target. The cropping region determination unit 240 may determine the position of the cropping region along with the size information.

[0024] The cropping unit 250 cuts out (crops) the image passed from the image input unit 220 based on the cropping region determined by the cropping region determination unit 240. In this way, the cropping unit 250 generates a cropped image including the tracking target to be used by the multitasking unit 260.

[0025] The multitasking unit 260 executes multiple recognition tasks on the cropped image generated by the cropping unit 250. In this embodiment, a cat whole body detector, a head detector, and a face detector are used as examples of the multiple recognition tasks. Here, various recognition models can be used for the executed recognition tasks, including, for example, convolutional neural networks, a Vision Transformer (ViT), and a support vector machine (SVM) combined with a feature extractor. While this embodiment is not limited to the above format, the multitasking unit 260 will be described as a CNN in this embodiment.

[0026] Tracking target identification unit 270 identifies a target to be tracked that is included in the image as the tracking target from the detection result obtained from multitasking unit 260. Tracking target identification unit 270 identifies the tracking target from the similarity between feature information of the tracking target obtained from tracking target setting unit 230 and feature information of the detection result obtained from multitasking unit 260.

[0027] The local region identification unit 280 identifies local regions for determining size information of crop regions. Specifically, the local region identification unit 280 identifies local regions within the image that include at least a portion of a detection target included in the tracking target identified by the tracking target identification unit 270. The image in which the local region is identified may be a cropped image. That is, the local region identification unit 280 may identify local regions within the image using either the original image or the cropped image. For example, if the tracking target is a cat and the detection targets are the cat's entire body, head, and face, the local region identification unit 280 identifies local regions within the image that include at least a portion of each detection target for each detection target. The local region identification unit 280 passes information about the identified one or more local regions to the crop region determination unit 240. The information about the local regions is used when determining a crop region for the next frame.

[0028] The output unit 290 outputs the results obtained from the multitasking unit 260, the tracking target identification unit 270, and the local area identification unit 280. In this way, the information processing device 200 performs processing on the input scene in time series, thereby enabling the tracking target to be tracked with high accuracy.

[0029] FIG. 3 shows a time series of images in the case of tracking a cat as an example in this embodiment. FIG. 4 is a flowchart of the recognition process in this embodiment. Hereinafter, it is assumed that the flowchart is realized by the CPU 100 executing a control program. FIG. 6(a) shows pre-registered information in this embodiment. The pre-registered information is stored in the HDD 130, and in this embodiment, the information processing device 200 can refer to the information as needed. The pre-registered information may be read from the HDD 130 and stored in the RAM 120.

[0030] The processing of this embodiment will be described in detail below with reference to FIG.

[0031] In S401, the image input unit 220 acquires one frame of image input as the input data 210. Here, the image input unit 220 acquires the image 301 shown in FIG. 3(a). The image 301 in FIG. 3(a) is an image of the initial frame at time t=0. In the image 301, a cat is walking to the left.

[0032] In S402, the image input unit 220 determines whether the image of the acquired input data 210 is the initial frame. If the image 301 is the initial frame at time t=0, the image input unit 220 determines that the image is the initial frame, and proceeds to S403.

[0033] In S403, the tracking target setting unit 230 sets a tracking target based on the image of the initial frame. Here, in S403, the tracking target setting unit 230 sets the tracking target to a cat, and also sets the position and size of the tracking target. Any method may be used to set the tracking target. For example, the tracking target setting unit 230 may set the tracking target by receiving a touch on the camera screen or a voice operation by the user, or may set the tracking target using a recognition result by a recognition processing unit of the camera, and various other methods may be used. In this embodiment, it is assumed that the tracking target setting unit 230 sets the entire body of the cat as the tracking target region 302, as shown in FIG. 3(a). After the tracking target setting unit 230 has finished setting the tracking target, the image input unit 220 again performs the process of S401.

[0034] In S401, the image input unit 220 acquires the image 311 shown in Fig. 3(b). Here, the image 311 is an image of a cat at time t = 1. In the image 311, the cat has stopped walking to the left.

[0035] In S402, the image input unit 220 determines that the image 311 is not the initial frame because it is a frame subsequent to the initial frame, and proceeds to the crop region determination process in S404.

[0036] 5(a) shows a detailed flow of the crop region determination process of S404 executed by the crop region determination unit 240 of the first embodiment. The crop region determination unit 240 executes the process of S404 and the processes of S501 to S507, which are detailed versions of S404.

[0037] In S501, the cropping region determination unit 240 determines whether or not there is a local region detected in the previous frame (i.e., the frame at the previous time) that can be used to calculate the cropping region. At time t=1, the local region identification process performed in S408, which will be described later, has not yet been performed, so the cropping region determination unit 240 determines that there is no local region, the determination in S501 is No, and the process proceeds to S503.

[0038] In S503, the crop region determination unit 240 sets the number of pixels of the entire body of the tracking target set by the tracking target setting unit 230 as the cropping standard.

[0039] Here, the cropping criterion refers to size information of the tracking target region on the image, and refers to information for calculating the size (here, the number of pixels) of the cropping region. The size and the number of pixels are examples of size information. The cropping region determination unit 240 can determine the cropping region by using a cropping magnification and cropping criterion, which will be described later. In this embodiment, the cropping criterion is described as the number of pixels, but this is not necessarily required. Any information related to the size of the tracking target may be used. For example, the cropping region determination unit 240 may use size information such as the length of the long side, the length of the short side, and the length of the diagonal of a rectangular frame representing the target region as the cropping criterion. At time t=1, the cropping region determination unit 240 determines the cropping region by calculating the product of the number of pixels of the entire body of the tracking target, which serves as the cropping criterion, and the cropping magnification.

[0040] In S504, the cropping region determination unit 240 acquires a cropping factor corresponding to the cropping reference. The cropping factor is a factor by which a preset cropping reference is multiplied, and can be set for each type of cropping reference (e.g., detection target). FIG. 6A shows pre-registered information 601 that is preset in this embodiment. The pre-registered information 601 determines a cropping factor for each detection target, which is a type of recognition task. The pre-registered information 601 indicates that the cropping factor is 3.0 when the whole body is selected as the cropping reference, 15.0 when the head is selected, and 30.0 when the face is selected. Therefore, since the cropping region determination unit 240 sets the number of pixels of the whole body of the tracking target as the cropping reference in S503, in S504, the cropping region determination unit 240 selects and acquires 3.0, which is associated with the whole body, from the pre-registered information 601 shown in FIG. 6A as the cropping factor. In this embodiment, the number of pixels in the whole body of the tracking target is the number of pixels in the tracking target area 302 set by the tracking target setting unit 230, but this is not limited to this, and the results of the whole body detector in the multitasking unit 260 may also be used.

[0041] In S505, the crop region determination unit 240 calculates the number of pixels in the crop region from the cropping reference and the cropping magnification. Specifically, the crop region determination unit 240 calculates the number of pixels in the crop region as the product of the number of pixels in the entire body of the tracking target set as the cropping reference and the cropping magnification determined in S504.

[0042] In S506, the cropping region determination unit 240 determines the aspect ratio of the cropping region. In this embodiment, the aspect ratio is described as 4:3, but this is not limiting. The cropping region determination unit 240 may, for example, determine the aspect ratio according to the cropping region, or may use a predetermined aspect ratio.

[0043] In S507, the cropping region determination unit 240 determines the position of the cropping region. In this embodiment, the cropping region determination unit 240 determines the center of the tracking target as the position of the cropping region. Note that the position of the cropping region does not have to be the center of the tracking target, and any position determination method may be used. In this way, the cropping region determination unit 240 determines the cropping region 312 shown in FIG. 3(b). The detailed flow of S404 is omitted and the process proceeds to S405.

[0044] In S405, the cropping unit 250 generates a cropped image 313. For example, the cropping unit 250 crops the image 311 using the cropping area 312 determined in S404, and resizes the image. In this embodiment, the image size after resizing is described as QVGA (320 pixels x 240 pixels), but the image size is not limited to this. Through the above processing, the cropping unit 250 generates a cropped image 313 shown in FIG. 3(c).

[0045] In S406, the multitasking unit 260 performs multitasking processing on the cropped image 313 in FIG. 3(c). In this embodiment, the multitasking performed by the multitasking unit 260 is described using a cat's whole body detector, head detector, and face detector as an example, but is not limited to this. For example, the multitasking unit 260 may execute a pupil detector and a function for tracking a tracking target as multitasking processing instead of or in addition to the above detectors. The detection results of the multitasking unit 260 at time t=1 are a whole body detection result 314, a head detection result 315, and a face detection result 316 shown in FIG. 3(c).

[0046] In S407, the tracking target identification unit 270 identifies the tracking target from the detection results of the multitasking unit 260. In this embodiment, the tracking target identification unit 270 compares the tracking target region 302 with the feature amounts of each detection result, and determines that the detection result with the closest feature amount is the tracking target. The identification of the tracking target does not necessarily have to be by feature amount comparison, and any method can be used as long as it can determine that the detection result is the tracking target. It is assumed that the tracking target identification unit 270 has identified the whole body detection result 314 as the tracking target at time t=1.

[0047] In S408, the local region identification unit 280 identifies a local region of the tracking target from the detection results of the multitasking unit 260. In this embodiment, the local region identification unit 280 identifies the local region using the distance from the center position of the whole body detection result 314 in FIG. 3(c) that was identified as the tracking target by the tracking target identification unit 270. Identification of the local region does not necessarily require the use of distance information. Assume that at time t=1, the local region identification unit 280 identifies the head detection result 315 and the face detection result 316 as local regions. Here, since the whole body detection result 314 has been identified as the tracking target, it is not used as a local region for determining the crop region. The regions of the head detection result 315 and the face detection result 316 are examples of multiple local regions of multiple different types of detection targets of the tracking target.

[0048] In S409, it is determined whether the processing of all frames has been completed. Since the processing of all frames has not been completed at time t=1, the process returns to S401 and the processing at time t=2 is performed.

[0049] In S401, the image input unit 220 acquires the image 321 in Fig. 3(d). Image 321 is an image of a cat taken at time t=2. In image 321, the cat, which had stopped in the previous frame, has resumed moving to the left. After that, the image input unit 220 again performs the process of S402, and then the cropping region determination unit 240 performs the process of S404.

[0050] 5A, which shows detailed flow of S404, the cropping region determination unit 240 determines whether there is a local region detected in the previous frame that can be used for cropping region calculation. At time t=1, the cropping region determination unit 240 has detected the head detection result 315 and the face detection result 316 in the previous frame. In this embodiment, the cropping region determination unit 240 proceeds to processing in S502 to determine that all information registered in the pre-registered information 601 can be used for cropping region calculation.

[0051] In S502, the cropping region determination unit 240 sets the number of pixels in a local region as the cropping standard in accordance with the priority order of the pre-registered information 601. Here, if there are multiple types of local regions, a detection target is selected in accordance with the priority order, and the local region of the detection target is set as the cropping standard. Specifically, there are two local regions, in this case, a head detection result 315 and a face detection result 316, but with reference to the pre-registered information 601, the head has the highest priority. Therefore, in S502, the cropping region determination unit 240 sets the number of pixels of the head detection result 315 as the cropping standard.

[0052] Here, the priority of each cropping reference in the pre-registered information 601 is determined in advance taking into consideration the "rate of size change with respect to posture changes." In the example of a cat in this embodiment, it can be seen that the size of the whole body changes greatly between walking and standing still, but the size change of the head is small. Therefore, by having the cropping region determination unit 240 set the cropping reference to the head, it is possible to reduce the change in the cropping region with respect to posture changes. The size change of the face with respect to posture changes is larger than that of the head, but smaller than that of the whole body. Therefore, the priority of the head is set to be first, the priority of the face to be second, and the priority of the whole body to be third.

[0053] In S504, the cropping region determination unit 240 obtains the cropping magnification of 15.0 from the pre-registered information 601, and calculates the number of pixels in the cropping region in S505. Thereafter, the cropping region determination unit 240 performs the processes of S506 and S507 to determine the cropping region 322 in FIG. 3(d).

[0054] In S405, the cropping unit 250 generates a cropped image using the cropping area 322. Thereafter, the processes from S406 to S409 are performed, and the series of processes ends.

[0055] As described above, this embodiment can determine the cropping reference and cropping magnification factor by taking into account the "size change rate relative to posture changes." If an area that is easily affected by posture changes were set as the cropping reference, the cropping area would be extremely narrow for an image such as a curled-up cat. Because the cropping area is determined using information from the previous frame, if the cropping area is narrow, there is a possibility that part or all of the cat may extend outside the cropping area if the cat makes a sudden movement in the current frame. If this happens, the cat will extend outside the cropped image, making it impossible to properly perform multitasking.

[0056] In this embodiment, the knowledge that the size of a cat's entire body varies greatly depending on its posture, but the size of its head varies little, is set as a priority of pre-registered information and processing is performed.

[0057] That is, in this embodiment, size information for the cropping area is determined based on size information for the head or other part of the object to be tracked, which is less susceptible to changes in the object's posture. This makes it possible to generate a stable cropped image, since the size of the cropping area is less susceptible to changes in the object's posture. As a result, this embodiment can reduce the amount of the object to be tracked that falls outside the cropping area, stabilizing the performance of multitasking, which processes multiple recognition tasks, and stabilizing the accuracy of the detection task for the object to be detected.

[0058] In this embodiment, a detection target used to determine size information of a crop region is selected based on a predetermined priority order for multiple detection targets, namely the whole body, the head, and the face. Then, in this embodiment, size information of a crop region is determined based on size information of a local region of the detection target. This makes it possible for this embodiment to more reliably stabilize the crop region. Furthermore, even if the detection target with the highest priority order is not detected, this embodiment can determine size information of a crop region based on size information of a local region of the detection target, which is the next least affected by changes in the posture of the tracking target.

[0059] <Second embodiment: Calculation of crop area using results of multiple recognition tasks> In the second embodiment, a method for determining a crop region using the results of multiple recognition tasks will be described, using the example of a cat walking, as in the first embodiment. In this embodiment, the local region identification unit 280 identifies multiple local regions. For example, the local region identification unit 280 identifies a local region from each of multiple detection targets, including a head, a face, and the like, thereby identifying multiple local regions. The crop region determination unit 240 determines size information for the crop region based on size information for the multiple local regions. For example, the crop region determination unit 240 may determine the crop region based on the result of performing an averaging process on the size information for the multiple local regions.

[0060] An example of the hardware configuration of this embodiment is the same as that of the first embodiment shown in FIG. 1, and the configuration diagram is also the same as that shown in FIG.

[0061] As an example of this embodiment, Fig. 3 shows a time series of images when tracking a cat. Image 301 in Fig. 3(a) is an image of a cat walking to the left in the initial frame at time t=0. In addition, the second embodiment uses the flowcharts in Fig. 4 and Fig. 5(b). Fig. 6(b) shows pre-registration information in this embodiment.

[0062] In processing of image 301 at time t=0, processing is performed in the order of S401, S402, and S403, as in the first embodiment. In S403, tracking target setting unit 230 sets the entire body of the cat as tracking target region 302, as shown in FIG. 3(a).

[0063] Next, in S401, the image input unit 220 acquires image 311 in FIG. 3(b). Here, image 311 is an image of a cat taken at time t=1. In image 311, the cat, which had been walking to the left, has stopped. Because the image input unit 220 determines in S402 that image 311 is a frame subsequent to the initial frame, the cropping region determination unit 240 executes the cropping region determination process in S404.

[0064] Fig. 5(b) shows a detailed flow of the crop region determination process of S404 executed by the crop region determination unit 240 of the second embodiment. Note that, among the processes in Fig. 5(b), the explanation of the same processes as Fig. 5(a) will be simplified. Fig. 6(b) shows pre-registered information 602 of the second embodiment.

[0065] In S511, the cropping region determination unit 240 determines whether there is a local region detected in the previous frame that can be used to calculate the cropping region. At time t=1, the local region identification process performed in S408, which will be described later, has not yet been performed, so the determination in S511 is No, and the process proceeds to S514. In S514, the cropping region determination unit 240 sets the number of pixels of the entire body of the tracking target set by the tracking target setting unit 230 as the cropping standard.

[0066] In S515, the cropping region determination unit 240 acquires, as the cropping magnification, 3.0 times, which corresponds to the whole body as the cropping standard, from the pre-registered information 602 in FIG. 6(b).

[0067] In S516, the crop region determination unit 240 calculates the number of pixels in the crop region by multiplying the number of pixels in the entire body of the tracking target set as the cropping reference by the cropping magnification determined in S515.

[0068] The cropping region determination unit 240 determines the aspect ratio of the cropping region in S517, and determines the position of the cropping region in S518. As a result, the cropping region determination unit 240 determines the cropping region 312 shown in FIG. 3(b).

[0069] In S405, the cropping unit 250 performs cropping using the cropping area 312 determined in S404 and resizes the image to QVGA to generate the cropped image 313 shown in Fig. 3(c). In S406, the multitasking unit 260 performs multitasking to obtain a whole body detection result 314, a head detection result 315, and a face detection result 316.

[0070] In S407, tracking target identification unit 270 identifies a tracking target from among the detection results of multitasking unit 260. At time t=1, it is assumed that tracking target identification unit 270 identifies whole body detection result 314 as the tracking target.

[0071] In S408, local region identification unit 280 identifies a local region of the tracking target from the detection results of multitasking unit 260. At time t=1, it is assumed that tracking target identification unit 270 identifies head detection result 315 and face detection result 316 as local regions. Here, whole body detection result 314 has been identified as the tracking target, and therefore is not used as a local region for determining a crop region.

[0072] In S409, it is determined whether the processing of all frames has been completed. Since the processing of all frames has not been completed at time t=1, the process returns to S401 and the processing at time t=2 is performed.

[0073] In S401, the image input unit 220 acquires the image 331 in Fig. 3(e). The image 331 is an image of a cat taken at time t=2. In the image 331, the cat, which had stopped in the previous frame, has resumed moving to the left. After that, the processes from S402 to S404 are performed again.

[0074] In S511, the cropping region determination unit 240 determines whether there is a local region detected in the previous frame that can be used to calculate a cropping region. At time t=1, a head detection result 315 and a face detection result 316 have been detected. In this embodiment, all information registered in the pre-registered information 602 can be set as cropping reference, so the process proceeds to S512.

[0075] In S512, the crop region determination unit 240 acquires the cropping magnifications and weighting factors of all local regions identified from multiple detection targets from the pre-registered information 602. In S513, the crop region determination unit 240 calculates the number of pixels in the crop region from the number of pixels of the cropping reference, the cropping magnification, and the weighting factor.

[0076] Here, the weighting coefficient indicates the importance of each recognition task, and by using the weighting coefficient, it is possible to calculate the number of pixels in the crop region more stably. In this embodiment, the pre-registered information 602 is set in consideration of the "size variation rate with respect to posture change" so that the weighting coefficient of a detection result with a small size variation rate is increased.

[0077] The method for calculating the number of pixels C in the crop region in this embodiment is shown below.

[0078]

number

[0079] In equation (1.1), W is a weighting coefficient, P is the number of pixels in the local region, and R is the cropping ratio. N is the number of identified local regions, and in this embodiment, there are two in total: a head detection result 315 and a face detection result 316. In this embodiment, the head detection result 315 will have 60 pixels, and the face detection result 316 will have 20 pixels.

[0080] The above formula (1.1) calculates the number of pixels in the crop region for each recognition task and calculates the weighted average. When the head detection result 315 is used as the cropping reference, the number of pixels in the crop region is 900 pixels, calculated by multiplying the head detection result's 60 pixels by the head cropping factor of 15.0. When the face detection result 316 is used as the cropping reference, the number of pixels in the crop region is 600 pixels, calculated by multiplying the face detection result's 20 pixels by the head cropping factor of 30.0. By multiplying each by the head weighting factor of 5.0 and the face weighting factor of 3.0 and calculating the weighted average, the number of pixels C in the crop region is 787.5 pixels. By using multiple detection results to calculate the number of pixels in the crop region, the crop region determination unit 240 can ensure a stable number of pixels in the crop region even if the head detection result 315 is detected incorrectly.

[0081] The cropping region determination unit 240 determines the aspect ratio of the cropping region in S517, and determines the position of the cropping region in S518. As a result, the cropping region determination unit 240 determines the cropping region 332 shown in FIG. 3(e).

[0082] Then, in S405, the cropping unit 250 generates a cropped image using the cropping area 332. Then, the processes from S406 to S409 are performed, and the process of this embodiment ends.

[0083] As described above, in this embodiment, by calculating the size information of the crop area based on the size information of multiple detection targets detected by multiple recognition tasks, it is possible to calculate a stable crop area even if the size of some of the detection results is incorrect. For example, in this embodiment, the size information of the crop region is calculated based on the result of averaging the size information of multiple local regions. This makes it possible to stabilize the crop region even if the size information of multiple local regions includes outliers. Note that, although the weighted average has been used as an example in this embodiment, this is not necessarily required, and the crop region may also be determined using averaging processes such as simple averaging or moving averaging.

[0084] The crop region determination unit 240 may also determine at least one of the upper and lower limit values ​​for the size information of the crop region using the detection results of multiple recognition tasks. For example, if priorities are set in the pre-registered information, the crop region determination unit 240 may calculate a crop region from the detection result of the recognition task with the highest priority and set the upper limit value to 2.0 times the number of pixels in the crop region and the lower limit value to 0.5 times the number of pixels in the crop region. If the crop region determination unit 240 subsequently calculates the crop region using a weighted average as in this embodiment and the number of pixels in the crop region exceeds the upper limit value or falls below the lower limit value, the crop region determination unit 240 sets the number of pixels in the crop region so that it falls within the set upper and lower limit values. By setting the upper and lower limits as described above, the crop region can be set within a range of 0.5 to 2.0 times the number of pixels in the crop region calculated from the recognition task with the highest priority.

[0085] As a result, the performance of the multitasking means can be stabilized by using the detection results of a plurality of recognition tasks.

[0086] <Third embodiment: Determining cropping criteria used to calculate cropping areas using time-series information> In this embodiment, a method for determining a cropping reference using time-series information will be described using an example of a cat walking. For example, in this embodiment, the local region identification unit 280 identifies multiple local regions from multiple images captured at different times. The crop region determination unit 240 determines size information for the crop region using size information for at least one of the multiple local regions. Here, the local region identification unit 280 may identify multiple types of detection targets, such as multiple local regions of a head and a face. In other words, the local region identification unit 280 may identify local regions of multiple different types of detection targets from each of multiple images captured at different times. The crop region determination unit 240 may determine size information for the crop information based on size information for a local region of a detection target selected from multiple types of detection targets. For example, the crop region determination unit 240 may determine size information for the crop region based on size information for the selected local region of the detection target based on a change in the size information for the local region. The change in size information for the local region may be, for example, a rate of change in size of the local region identified from images captured at different times. The size here may be either the number of pixels in the local region or the product of the length and width of the local region.

[0087] An example of the hardware configuration of this embodiment is the same as that of the first embodiment shown in FIG. 1, and the configuration diagram is also the same as that shown in FIG.

[0088] Fig. 7 shows a time series of images when tracking a cat as an example in this embodiment. Image 701 in Fig. 7(a) is an image of a cat walking leftward in the initial frame at time t=0. In addition, the third embodiment uses the flowcharts in Fig. 4 and Fig. 8. Fig. 6(c) shows pre-registration information in this embodiment.

[0089] In processing of image 701 at time t=0, processing is performed in the order of S401, S402, and S403, as in the first embodiment. In S403, tracking target setting unit 230 sets the entire body of the cat as tracking target region 702, as shown in FIG. 7(a).

[0090] Next, in S401, the image input unit 220 acquires the image 711 in FIG. 7(b). Here, the image 711 is an image of the cat taken at time t=1. In the image 711, the cat continues to move to the left. In S402, the image input unit 220 determines that the image 711 is a frame after the initial frame, and therefore proceeds to the crop region determination process in S404.

[0091] 8 shows a detailed flow of the crop region determination process in S404 executed by the crop region determination unit 240 of the third embodiment. Of the processes in FIG. 8, the description of the same processes as those in the above-described embodiments will be simplified.

[0092] In S801, the cropping region determination unit 240 determines whether there is a local region detected in the previous frame that can be used to calculate a cropping region. At time t=1, there is no local region because the local region identification process performed in S408, which will be described later, has not yet been performed. Therefore, the cropping region determination unit 240 determines No in S801 and proceeds to the process of S807.

[0093] In S807, the crop region determination unit 240 sets the number of pixels of the entire body of the tracking target set by the tracking target setting unit 230 as the cropping standard.

[0094] In S808, the cropping region determination unit 240 acquires, as the cropping magnification, 3.0 times, which corresponds to the whole body as the cropping standard, from the pre-registered information 603 in FIG. 6(c).

[0095] In S809, the crop region determination unit 240 calculates the number of pixels in the crop region by multiplying the number of pixels in the entire body of the tracking target set as the crop reference by the cropping magnification determined in S808.

[0096] The cropping region determination unit 240 determines the aspect ratio of the cropping region in S810, and determines the position of the cropping region in S811. In this way, the cropping region determination unit 240 determines the cropping region 712 shown in FIG. 7(b).

[0097] In S405, the cropping unit 250 performs cropping using the cropping area 712 determined in S404 and resizes the image to QVGA, generating the cropped image 713 shown in FIG. 3(c).

[0098] In S406, the multitasking unit 260 performs multitasking processing to obtain a whole body detection result 714, a head detection result 715, and a face detection result 716.

[0099] In S407, the tracking target identification unit 270 identifies the tracking target from among the detection results of the multitasking unit 260. At time t=1, the tracking target identification unit 270 identifies the whole body detection result 714 as the tracking target.

[0100] In S408, the local area identifying unit 280 identifies a local area of ​​the tracking target from the detection results of the multitasking unit 260. At time t=1, the local area identifying unit 280 identifies the head detection result 715 and the face detection result 716 as local areas.

[0101] In S409, it is determined whether the processing of all frames has been completed. Since the processing of all frames has not been completed at time t=1, the process returns to S401 and the processing at time t=2 is performed.

[0102] In S401, the image input unit 220 acquires the image 721 in Fig. 7(d). Here, the image 721 is an image of a cat taken at time t=2. In the image 721, the cat continues to move to the left. Thereafter, the processes from S402 to S404 are performed again.

[0103] In S801, the cropping region determination unit 240 determines whether there is a local region detected in the previous frame that can be used to calculate a cropping region. At time t=1, a head detection result 715 and a face detection result 716 have been detected in the previous frame. In this embodiment, all information registered in the pre-registered information 603 is considered usable for calculating a cropping region, so the cropping region determination unit 240 determines that there is a local region in the previous frame and proceeds to processing in S802.

[0104] In S802, the cropping region determination unit 240 determines whether there are multiple local regions that can be used to calculate the cropping region. At time t=1, the head detection result 715 and face detection result 716 have been detected, so the cropping region determination unit 240 determines that there are multiple local regions that can be used, and proceeds to the process of S803.

[0105] In S803, the cropping region determination unit 240 determines whether the time-series information of the local region detection results is available. At time t=2, only the detection result from time t=1 is available, so information such as detection results or local regions identified from images taken at different times to determine changes in the local regions over time is not available. Therefore, the cropping region determination unit 240 determines that the time-series information is not available, and proceeds to processing in S806.

[0106] In S806, the cropping area determination unit 240 sets the detected local areas as the cropping criterion in accordance with the priority. Since it is clear from the pre-registration information 603 that the face detection result has the highest priority, the cropping area determination unit 240 sets the number of pixels in the face area as the cropping criterion.

[0107] In S808, the cropping region determination unit 240 acquires 30.0 times, which is associated with the face, from the pre-registration information 603 in FIG. 6C and sets it as the cropping magnification.

[0108] In S809, the cropping region determination unit 240 calculates the number of pixels in the cropping region by multiplying the number of pixels in the face set as the cropping reference by the cropping magnification determined in S808. In S810, the cropping region determination unit 240 determines the aspect ratio of the cropping region, and in S811, determines the position of the cropping region. As a result, the cropping region determination unit 240 determines the cropping region 722 shown in FIG. 7(d).

[0109] In S405, the cropping unit 250 performs cropping using the cropping area 722 determined in S404 and resizes the image to QVGA, generating the cropped image 723 shown in FIG. 3(e).

[0110] In S406, the multitasking unit 260 performs multitasking processing to obtain a whole body detection result 724, a head detection result 725, and a face detection result 726.

[0111] In S407, the tracking target identification unit 270 identifies the tracking target from among the detection results of the multitasking unit 260. At time t=2, the tracking target identification unit 270 identifies the whole body detection result 724 as the tracking target.

[0112] In S408, the local area identifying section 280 identifies a local area of ​​the tracking target from the detection results of the multitasking section 260. At time t=2, the local area identifying section 280 identifies the head detection result 725 and the face detection result 726 as local areas.

[0113] In S409, it is determined whether the processing of all frames has been completed. At time t=2, the processing of all frames has not been completed, so the process returns to S401 and the processing at time t=3 is performed.

[0114] In S401, the image input unit 220 acquires the image 731 in Fig. 7(g). The image 731 is an image of the cat taken at time t = 3. In the image 731, the cat continues to move to the left. Thereafter, the processes from S402 to S404 are performed again.

[0115] In S801, the cropping region determination unit 240 determines whether there is a local region detected in the previous frame that can be used to calculate the cropping region. At time t=2, a head detection result 725 and a face detection result 726 have been detected in the previous frame. Therefore, the cropping region determination unit 240 determines that there is a local region, and proceeds to processing in S802.

[0116] In S802, the cropping region determination unit 240 determines whether there are multiple local regions that can be used for cropping. At time t=2, a head detection result 725 and a face detection result 726 have been detected. Therefore, the cropping region determination unit 240 determines that there are multiple local regions, and proceeds to the processing of S803.

[0117] In S803, the cropping region determination unit 240 determines whether the time-series information of the local region detection results is available. At time t=3, the detection results from times t=1 and t=2 are available, so all the information needed to determine changes in the local region over time is available. Therefore, the cropping region determination unit 240 determines that the time-series information is available and proceeds to the process in S804. In this embodiment, if there are local region detection results for the past two frames, it is determined that the time-series information is available, but this is not limited to this.

[0118] In S804, the crop region determination unit 240 calculates the size change rate from the time-series information of each local region.

[0119] The size change rate is a value calculated by calculating how much the size (here, the number of pixels) has changed from the size information of the local region detection results in previous frames. In this embodiment, the crop region determination unit 240 calculates how much the size has changed from the original local region detection size by comparing the size information of the detection results from two frames ago with the size information of the detection results from one frame ago, but this is not necessarily required. For example, the crop region determination unit 240 may calculate the size change rate by calculating the size variance and standard deviation of the time-series information of the local region. Between the whole-body detection results 714 and 724, it can be seen that the whole-body size change rate is small because the orientation of the cat's body has not changed. Between the face detection results 716 and 726, it can be seen that the cat's face has changed from facing forward to facing sideways, resulting in a large size change rate. Between the head detection results 715 and 725, it can be seen that the head size detection result changes only slightly even when the cat turns sideways.

[0120] In S805, the cropping region determination unit 240 sets the number of pixels in the local region with the smallest size change rate as the cropping standard. In this embodiment, the cropping region determination unit 240 determines that the region with the smallest size change rate is the head, and sets the number of pixels in the head as the cropping standard. Here, if the method of the first embodiment were used, the number of pixels in the face would be set as the cropping standard in accordance with the priority order, which could result in a narrower cropping region compared to time t=2.

[0121] In S808, the cropping region determination unit 240 acquires 15.0 times, which is associated with the head, as the cropping magnification from the pre-registered information 603 in FIG. 6(c).

[0122] In S809, the cropping region determination unit 240 calculates the number of pixels in the cropping region by multiplying the number of pixels in the head set as the cropping reference by the cropping magnification determined in S808.

[0123] The cropping region determination unit 240 determines the aspect ratio of the cropping region in S810, and determines the position of the cropping region in S811. As a result, the cropping region determination unit 240 determines the cropping region 732 shown in FIG. 7(g).

[0124] Then, in S405, the cropping unit 250 generates a cropped image using the cropping area 732. Then, the processes from S406 to S409 are performed, and the process of this embodiment ends.

[0125] As described above, the information processing apparatus of the third embodiment can set a stable crop region by calculating a crop region using one of the local regions identified using time-series information of multiple images taken at different times. For example, this embodiment selects a local region for calculating a crop region based on the rate of change in size information of the local region. Specifically, this embodiment determines a crop region using a local region with small changes in size information. This allows this embodiment to calculate an appropriate stable crop region even if a local region that is not suitable for a stable crop region has a high priority or if no priority is set.

[0126] Furthermore, in the present embodiment, a method for determining a cropping reference from among different types of local regions such as the face and head has been described, but the present invention is not limited to this. For example, even if there are multiple candidates for the same type of local region such as the pupils or the feet (for example, two in the case of pupils), one local region suitable for calculating the crop region may be determined using the above method.

[0127] <Fourth embodiment: Determining crop area using time series information when recognition task is not detected> In this embodiment, a method for determining a crop region when a high-priority recognition task has not been detected will be described using the example of a cat walking. Specifically, in this embodiment, the local region identification unit 280 identifies multiple local regions from each of multiple frame images at different times. The crop region determination unit 240 sets a crop region for the current image based on the crop region of the image of the previous frame (i.e., the image at the previous time) based on changes in the multiple local regions at different times. Furthermore, if there is no local region at the previous time, the crop region determination unit 240 determines whether to set the current crop region based on the previous crop region based on changes in the local region.

[0128] Fig. 7 shows a time series of images when tracking a cat as an example in the fourth embodiment. Image 701 in Fig. 7(a) is an image of a cat walking leftward in the initial frame at time t=0. The fourth embodiment uses the flowcharts in Fig. 4 and Fig. 9. Fig. 6(d) shows pre-registration information in the fourth embodiment.

[0129] In processing the image 701 at time t=0, processing is performed in the order of S401, S402, and S403, as in the third embodiment. In S403, the tracking target setting unit 230 sets the entire body of the cat as the tracking target region 702, as shown in FIG. 7(a).

[0130] Next, in S401, image 711 in FIG. 7(b) is acquired. Here, image 711 is an image of a cat taken at time t=1. In image 711, the cat continues to move to the left. Since the image input unit 220 determines in S402 that image 711 is a frame subsequent to the initial frame, the process proceeds to crop region determination processing in S404.

[0131] FIG. 9 shows a detailed flow of the crop region determination process in S404 executed by the crop region determination unit 240 of the fourth embodiment.

[0132] In S901, the cropping region determination unit 240 determines whether there is a local region detected in the previous frame that can be used to calculate the cropping region. At time t=1, the multitasking unit 260 has not yet performed processing, so the cropping region determination unit 240 determines that there is no local region detected and proceeds to processing in S903.

[0133] In S903, the cropping region determination unit 240 determines whether there are detection results for the previous frame. At time t=1, the multitasking unit 260 has not yet performed processing, so the cropping region determination unit 240 determines that there are no detection results for the previous frame and proceeds to processing in S904.

[0134] In S904, the crop region determination unit 240 sets the number of pixels of the entire body of the tracking target set by the tracking target setting unit 230 as the cropping standard.

[0135] In S907, the cropping region determination unit 240 acquires, as the cropping magnification, 3.0 times which corresponds to the whole body as the cropping standard from the pre-registered information 604 in FIG. 6(d).

[0136] In S908, the crop region determination unit 240 calculates the number of pixels in the crop region by multiplying the number of pixels in the entire body of the tracking target set as the cropping reference by the cropping magnification determined in S907.

[0137] The cropping region determination unit 240 determines the aspect ratio of the cropping region in S909, and determines the position of the cropping region in S911. As a result, the cropping region determination unit 240 determines the cropping region 712 shown in FIG. 7(b).

[0138] In S405, the cropping unit 250 performs cropping using the cropping area 712 determined in S404 and resizes the image to QVGA, generating the cropped image 713 shown in FIG. 7(c).

[0139] In S406, the multitasking unit 260 performs multitasking processing to obtain a whole body detection result 714, a head detection result 715, and a face detection result 716.

[0140] In S407, the tracking target identification unit 270 identifies a tracking target from among the detection results of the multitasking unit 260. At time t=1, the tracking target identification unit 270 identifies the whole body detection result 714 as the tracking target.

[0141] In S408, the local area identifying unit 280 identifies a local area of ​​the tracking target from the detection results of the multitasking unit 260. At time t=1, the local area identifying unit 280 identifies the head detection result 715 and the face detection result 716 as local areas.

[0142] In S409, it is determined whether the processing of all frames has been completed. Since the processing of all frames has not been completed at time t=1, the process returns to S401 and the processing at time t=2 is performed.

[0143] In S401, the image input unit 220 acquires the image 721 in Fig. 7(d). Here, the image 721 is an image of a cat taken at time t=2. In the image 721, the cat continues to move to the left. Thereafter, the processes from S402 to S404 are performed again.

[0144] In S901, the cropping region determination unit 240 determines whether there is a local region detected in the previous frame that can be used to calculate the cropping region. At time t=1, a head detection result 715 and a face detection result 716 have been detected in the previous frame. In this embodiment, all information registered in the pre-registered information 604 can be set as cropping reference, so the process proceeds to S902.

[0145] In S902, the cropping region determination unit 240 sets the detected local regions as the cropping criterion in accordance with the priority. Since it is clear from the pre-registered information 604 that the head has the highest priority, the cropping region determination unit 240 sets the number of pixels in the head region as the cropping criterion.

[0146] In S907, the cropping region determination unit 240 acquires the cropping magnification of 15.0, which corresponds to the head and is the cropping standard, from the pre-registered information 604 in FIG. 6(d).

[0147] In S908, the cropping region determination unit 240 calculates the number of pixels in the cropping region by multiplying the number of pixels in the head set as the cropping reference by the cropping magnification determined in S907.

[0148] The cropping region determination unit 240 determines the aspect ratio of the cropping region in S909, and determines the position of the cropping region in S911. As a result, the cropping region determination unit 240 determines the cropping region 722 shown in FIG. 7(d).

[0149] In S405, the cropping unit 250 performs cropping using the cropping area 722 determined in S404 and resizes the image to QVGA, generating the cropped image 727 shown in FIG. 7(f).

[0150] In S406, the multitasking unit 260 performs multitasking processing and obtains only the whole body detection result 728, with the head and face not being detected.

[0151] In S407, the tracking target identification unit 270 identifies the tracking target from among the detection results of the multitasking unit 260. At time t=2, the tracking target identification unit 270 identifies the whole body detection result 728 as the tracking target.

[0152] In S408, the local region identifying section 280 identifies a local region of the tracking target from the detection result of the multitasking section 260. At time t=2, the local region identifying section 280 does not detect any local region.

[0153] In S409, it is determined whether the processing of all frames has been completed. At time t=2, the processing of all frames has not been completed, so the process returns to S401 and the processing at time t=3 is performed.

[0154] In S401, the image input unit 220 acquires the image 733 in Fig. 7(h). The image 733 is an image of the cat taken at time t = 3. In the image 733, the cat continues to move to the left. Thereafter, the processes from S402 to S404 are performed again.

[0155] In S901, the cropping region determination unit 240 determines whether there is a local region detected in the previous frame that can be used to calculate the cropping region. At time t=2, the head and face have not been detected, so it is determined that there is no local region detected, and the process proceeds to S903.

[0156] In S903, the cropping region determination unit 240 determines whether there is a detection result for the previous frame. At time t=2, the whole body detection result 728 has been detected, so the cropping region determination unit 240 determines that there is a detection result for the previous frame and proceeds to the process of S905.

[0157] In S905, the cropping region determination unit 240 calculates a size change rate from the time-series information of the detection results. As described in the third embodiment, the size change rate is a value calculated to indicate how much the size has changed from the size information (here, the number of pixels) of the detection results in past frames. For example, the cropping region determination unit 240 may calculate the change in size between the size of a local region at a certain time and the size of the local region immediately after that time as the size change rate.

[0158] In S906, the cropping region determination unit 240 determines whether the size change rate is equal to or less than a threshold. If the size change rate between whole body detection result 714 and whole body detection result 728 is low and the cropping region determination unit 240 determines that the size change rate is equal to or less than the threshold, the process proceeds to S910. On the other hand, if the cropping region determination unit 240 determines that the size change rate is greater than the threshold, the process proceeds to S904.

[0159] In S910, the crop region determination unit 240 sets the crop region of the previous frame as the crop region of the current frame. In other words, if a local region from the previous time point has not been detected and the size change rate is equal to or less than the threshold, the crop region determination unit 240 determines the crop region from the previous time point as the current crop region.

[0160] Here, for example, in the third embodiment, a local region in the previous frame that can be used to calculate a crop region is not detected in the current frame, so the entire body of the tracking target, which has a large "size change rate relative to posture change," may be set as the cropping reference. In this embodiment, the crop region determination unit 240 determines that there has been no significant change in the posture of the tracking target based on the size change rate of a local region in the current frame that has a large "size change rate relative to posture change." If the crop region determination unit 240 determines that the size change rate is small and that there has been no significant change in the shooting state of the tracking target, it can use the crop region of the previous frame as the crop region for the current frame.

[0161] Then, in S911, the cropping region determination unit 240 determines the position of the cropping region, thereby determining the cropping region 734 shown in FIG.

[0162] Thereafter, in S405, the cropping unit 250 generates a cropped image using the cropping area 734. Thereafter, the processes from S406 to S409 are performed, and the process of this embodiment ends.

[0163] As described above, the information processing device of the fourth embodiment can perform cropping using the crop area of ​​the previous frame as the crop area of ​​the current frame by confirming that there is no significant change in the shooting state of the tracking target from the size change rate of the area with a large "size change rate relative to posture change." However, the position of the crop area does not need to use the coordinates of the previous frame and may be changed according to the movement of the subject. Furthermore, although the size change rate is used to confirm that there is no significant change in the shooting state of the tracking target in this embodiment, it may also be confirmed using the recognition accuracy of the tracking target. Here, recognition accuracy refers to a recognition rate calculated from the number of frames in which the tracking target is recognized, a recognition score indicating the likelihood of the recognized target, or the like.

[0164] In this embodiment, it is determined whether cropping can be performed using the crop region of the previous frame, but it may also be determined whether size information of the local region of the previous frame can be used in the current frame. By making the above determination, it becomes possible to calculate the crop region using the local region of the previous frame even if the local region used to calculate the crop region in the previous frame has not been detected in the current frame.

[0165] As a result, even if a high-priority recognition task has not been detected, a stable crop region can be calculated.

[0166] <Fifth embodiment: Multitasking performed on images of different sizes> Fig. 10 is a diagram illustrating a first information processing device 200 and a second information processing device 1000 according to the fifth embodiment. An image input unit 220, a tracking target setting unit 230, a crop region determination unit 240, a cropping unit 250, a first multitasking unit 260, a tracking target identification unit 270, a local region identification unit 280, and an output unit 290 in Fig. 10 perform the same operations as the corresponding configurations in the first embodiment, and therefore description thereof will be omitted.

[0167] The second information processing apparatus 1000 receives input data 210 from the image input unit 220. The second multitasking unit 1010 executes multiple recognition tasks on the image data of the input data 210. In this embodiment, the second multitasking unit 1010 processes an uncropped image, but this is not limited thereto. The second multitasking unit 1010 may execute recognition tasks on an image larger than a cropped image. The second multitasking unit 1010 may execute recognition tasks on, for example, a cropped image whose size is half the image size, or a cropped image whose aspect ratio is 1:1. Therefore, the local region identification unit 280 identifies local regions from the cropped image and an image larger than the cropped image. The second multitasking unit 1010 may be provided in the first information processing apparatus 200. In this case, the first information processing apparatus 200 and the second information processing apparatus 1000 are integrated into a single information processing apparatus.

[0168] FIG. 11 shows a time series of images in the case of tracking a cat as an example in the fifth embodiment. FIG. 12 shows a flowchart of the processing in this embodiment. FIG. 12(a) shows the entire flowchart. FIG. 12(b) shows a detailed flow of the crop region determination processing in S1204. FIG. 6(a) shows pre-registered information in this embodiment.

[0169] First, in S1201, the image input unit 220 acquires one frame of image input as the input data 210. Here, the image input unit 220 acquires the image 1101 in FIG. 11(a). The image 1101 in FIG. 11(a) is an image of the initial frame at time t=0. In the image 1101, a cat is walking to the left.

[0170] In S1202, the image input unit 220 determines whether the acquired frame is the initial frame of the input data 210. If an image of the initial frame at time t=0 has been acquired, the image input unit 220 determines that it is the initial frame and proceeds to S1203.

[0171] In S1203, the tracking target setting unit 230 sets a tracking target. Here, in S1203, the tracking target setting unit 230 sets the cat as the tracking target and sets the position and size of the tracking target. In this embodiment, the tracking target setting unit 230 sets the entire body of the cat as the tracking target region 1102, as shown in FIG. 11(a).

[0172] In S1207, after the setting of the tracking target is completed, the second multitasking unit 1010 performs the second multitasking process. FIG. 11(b) shows the result of the second multitasking process. An image 1103 in FIG. 11(b) shows an image obtained by the second multitasking process, and in this embodiment, the same image as the image 1101 is used. The second multitasking unit 1010 detects a whole body detection result 1104 and a head detection result 1105 based on the image 1103.

[0173] In S1208, the tracking target identification unit 270 identifies a tracking target from among the detection results of the second multitasking unit 1010. At time t=0, the tracking target identification unit 270 identifies the whole body detection result 1104 as the tracking target.

[0174] In S1209, the local region identifying unit 280 identifies a local region of the tracking target from the detection result of the second multitasking unit 1010. At time t=0, the local region identifying unit 280 identifies the head detection result 1105 as a local region.

[0175] In S1210, it is determined whether the processing of all frames has been completed. Since the processing of all frames has not been completed at time t=0, the process returns to S1201 and the processing for time t=1 is performed.

[0176] In S1201, the image input unit 220 acquires the image 1111 shown in Fig. 11(c). Here, the image 1111 is an image of the cat at time t = 1. In the image 1111, the cat continues to move to the left.

[0177] In S1202, the image input unit 220 determines that the image 1111 is a frame subsequent to the initial frame, and therefore the process proceeds to crop region determination processing in S1204.

[0178] FIG. 12B shows a detailed flow of the crop region determination process in S1204.

[0179] In S1211, the cropping region determination unit 240 determines whether there is a local region detected in the previous frame that can be used to calculate a cropping region. At time t=1, there is a head detection result 1105 obtained from the second multitasking unit 1010, so the cropping region determination unit 240 determines that there is a local region detected in the previous frame and proceeds to processing in S1212. On the other hand, if there is no local region in the previous frame that can be used to calculate a cropping region, such as the head detection result 1105, the cropping region determination unit 240 determines that there is no local region and proceeds to S1214, where it sets the number of pixels in the entire body of the tracking target set by the tracking target setting unit 230 as the cropping standard. Thereafter, the cropping region determination unit 240 proceeds to S1215, which will be described later.

[0180] In S1212, the crop region determination unit 240 integrates the results of the first multitasking process and the second multitasking process. For example, if a head is detected in each multitasking process, the crop region determination unit 240 determines the number of pixels for the head region as a local region. In this embodiment, the crop region determination unit 240 determines the number of pixels for the head region using the average of the pixel counts for the head region detected by the two processes. Note that the determination of the number of pixels is not limited to this, and it may also be determined using a weighted average of the two regions or one of them. Furthermore, if there is only one detection result, the crop region determination unit 240 may use the number of pixels for the only head detection result.

[0181] In S1213, the cropping region determination unit 240 sets the number of pixels of the head detection result 1105 as the cropping standard in accordance with the priority order of the pre-registered information 601 in FIG. 6(a).

[0182] In S1215, the cropping region determination unit 240 acquires 15.0 times, which corresponds to the head, as the cropping magnification from the pre-registered information 601 in FIG. 6(a).

[0183] In S1216, the cropping region determination unit 240 calculates the number of pixels in the cropping region by multiplying the number of pixels in the head set as the cropping reference by the cropping magnification determined in S1215.

[0184] The cropping region determination unit 240 determines the aspect ratio of the cropping region in S1217, and determines the position of the cropping region in S1218. In this way, the cropping region determination unit 240 determines the cropping region 1112 shown in FIG.

[0185] In S1205, the cropping unit 250 generates a cropped image 1116. For example, the cropping unit 250 crops the image 1111 using the cropping area 1112 determined in S1204, and resizes the image. Through the above processing, the cropping unit 250 generates the cropped image 1116 shown in FIG. 11(e). At time t=1, the cat's movements were sudden, so part of the cat's body and head protrude outside the cropping area.

[0186] In S1206, the first multitasking unit 260 performs first multitasking processing on the cropped image 1116 in Fig. 11(e). The result of the first multitasking unit 260 at time t=1 is only the whole body detection result 1117 as shown in Fig. 11(e) because the entire body of the cat is outside the cropped area.

[0187] In S1207, the second multitasking unit 1010 performs second multitasking processing. Fig. 11(d) shows the result of the second multitasking processing. Image 1113 in Fig. 11(d) shows an image that has been subjected to the second multitasking processing, and is the processing result using the same image as image 1111. The second multitasking unit 1010 performs the second multitasking processing on image 1111, and detects a whole body detection result 1114 and a head detection result 1115, as shown in image 1113.

[0188] In S1208, the tracking target identification unit 270 identifies the tracking target from the detection results of the first multitasking unit 260 and the second multitasking unit 1010. At time t=1, the tracking target identification unit 270 identifies the whole body detection result 1114 as the tracking target, as shown in FIG. 11(d).

[0189] In S1209, the local region identifying unit 280 identifies a local region of the tracking target from the detection results of the first multitasking unit 260 and the second multitasking unit 1010. At time t=1, the local region identifying unit 280 identifies the head detection result 1115 as a local region, as shown in FIG. 11(d).

[0190] In S1210, it is determined whether the processing of all frames has been completed. Since the processing of all frames has not been completed at time t=1, the process returns to S1201 and the processing for time t=2 is performed.

[0191] In S1201, the image input unit 220 acquires the image 1121 in Fig. 11(f). The image 1121 is an image of the cat at time t = 2. In the image 1121, the cat continues to move to the left. Thereafter, the processes of S1202 and S1204 are performed again.

[0192] In S1211, the cropping region determination unit 240 determines whether or not there is a local region detected in the previous frame that can be used to calculate the cropping region. At time t=2, the head detection result 1115 has been detected, so the cropping region determination unit 240 determines that there is a local region and proceeds to the processing of S1212.

[0193] In S1212, crop region determination unit 240 integrates the results of the first multitasking process and the results of the second multitasking process. Here, because there is only one detection result for the head, crop region determination unit 240 integrates the processing results by directly using the result of second multitasking unit 1010 for the head. For the whole body, crop region determination unit 240 integrates the processing results by taking the average of the number of pixels of whole body detection result 1114 and whole body detection result 1117 as the number of pixels for the whole body.

[0194] In S1213, the cropping region determination unit 240 sets the number of pixels of the head set in S1212 as the cropping standard in accordance with the priority order of the pre-registered information 601.

[0195] Thereafter, the cropping region determination unit 240 performs the processes from S1215 to S1218 in the same manner as at time t=1, and determines the cropping region 1122 shown in FIG. 11(f).

[0196] In S1205, the cropping unit 250 generates a cropped image using the cropping area 1122.

[0197] Thereafter, the processes from S1206 to S1210 are carried out, and the series of processes ends.

[0198] As described above, the fifth embodiment can more stably calculate the crop area by performing the first multitasking process and the second multitasking process using a cropped image and an image larger than the cropped image. In this embodiment, even if the entire cat protrudes from the cropped image, cropping can be performed using a local area with a small "size change rate relative to pose change" by using the detection results of the second multitasking process performed on an image larger than the cropped image. Furthermore, although this embodiment performs processing on an uncropped image, processing may be performed on any image as long as it is different from the cropped image generated by cropping unit 250.

[0199] <Other embodiments> In the above embodiment, a cat is used as an example of the tracking target. However, other categories, such as people or motorcycles, may also be used as tracking targets. When applying this embodiment to other categories, a new local region serving as the cropping reference may be set using the "size change rate relative to posture change" as an index. For example, for a person, the region with the largest size change rate is the whole body region, and the region with the smallest size change rate is the head region. For a motorcycle, the region with the largest size change rate is the entire body region, and the region with the smallest size change rate is the tire length. Therefore, in the above embodiment, the cropping reference is determined based on the priority and size change rate, but the cropping reference may also be determined based on the category classification results of the tracking target. For example, a combination of category and cropping reference may be determined in advance, such as the cropping reference for tracking a person being the head and the cropping reference for tracking a motorcycle being the tire.

[0200] In the above embodiment, an example has been described in which a crop region is determined based on a local region selected from a plurality of local regions, but the method for determining a crop region is not limited to this. For example, priorities may be set for body parts such as the whole body, head, and face, and local regions may be set only for parts with high priorities among the detected parts, and a crop region may be determined based on the local region.

[0201] In the above embodiment, the explanation is given on the premise of multitasking, but multitasking is not necessary. In this case, the local region identifying unit 280 may detect a predetermined region and identify the local region according to the region.

[0202] In the above embodiment, the input data 210 is a moving image, but the present invention is not limited to this. For example, the input data 210 may be a plurality of still images captured at regular time intervals, a plurality of time-lapse images, or the like.

[0203] The above-described embodiments may be combined as appropriate.

[0204] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0205] The disclosure of this specification includes the following information processing device, information processing method, and program. (Item 1) a tracking target identification means for identifying a target to be tracked included in an image as the tracking target; a local area specifying means for specifying, within the image, a local area including at least a part of a detection target included in the tracking target; a crop region determination means for determining size information of a crop region for cutting out the tracking target from the image based on size information of the local region; a cropping unit that cuts out the image based on the crop area to generate a cropped image; An information processing device comprising: (Item 2) the local area specifying means specifies a plurality of local areas including at least a part of a plurality of types of detection targets included in the tracking target; The crop region determining means determines size information of the crop region based on size information of any one of the plurality of local regions. 2. The information processing device according to item 1, (Item 3) The crop region determination means determines size information of the crop region based on size information of a local region of a detection target selected from the plurality of types of detection targets based on a predetermined priority order of the detection targets. 3. The information processing device according to item 2, (Item 4) the local area specifying means specifies a plurality of local areas including at least a part of the detection target of the tracking target; The crop region determining means determines size information of the crop region based on size information of the plurality of local regions. 4. The information processing device according to any one of items 1 to 3, (Item 5) The crop region determining means determines size information of the crop region based on a result of averaging the size information of the plurality of local regions. 5. The information processing device according to item 4. (Item 6) The cropping area determining means determines the size information of the cropping area based on at least one of an upper limit value and a lower limit value set based on the size information of the cropping area. 6. The information processing device according to item 4 or 5, (Item 7) the local region specifying means specifies a plurality of local regions from a plurality of images taken at different times; The cropping region determining means determines size information of the cropping region based on size information of at least one of the plurality of local regions. 7. The information processing device according to any one of items 1 to 6, (Item 8) the local area specifying means specifies a plurality of local areas including at least a part of a plurality of types of detection targets included in the tracking target; The crop region determination means determines size information of the crop region based on size information of a local region of a detection target selected from the plurality of types of detection targets. 8. The information processing device according to item 7, (Item 9) The crop region determination means determines size information of the crop region based on size information of a local region of a detection target selected from the plurality of types of detection targets, based on a change in size information of the plurality of local regions. 9. The information processing device according to item 8, (Item 10) the local area specifying means specifies a plurality of local areas including at least a part of a plurality of types of detection targets included in the tracking target; The crop region determination means determines size information of the crop region based on size information of a local region of a detection target selected from the plurality of types of detection targets based on recognition accuracy of the detection target. 10. The information processing device according to any one of items 1 to 9, (Item 11) The crop region determination means uses at least one of the recognition rate of the detection object and the recognition score of the detection object as the recognition accuracy. 11. The information processing device according to item 10. (Item 12) the local region specifying means specifies a plurality of local regions from a plurality of images taken at different times; The cropping region determining means determines whether or not to set a cropping region at a previous time as a current cropping region based on a change in size information of the plurality of local regions. 12. The information processing device according to any one of items 1 to 11, (Item 13) The crop region determination means sets the crop region of the previous time as the crop region of the current image when the local region used to determine the crop region of the image of the previous time is not detected in the current image. Item 13. The information processing device according to item 12. (Item 14) The local area specifying means specifies the local area from the cropped image and an image larger than the cropped image. 14. The information processing device according to any one of items 1 to 13, (Item 15) The local area specifying means sets a local area based on a detection target set according to a category of the tracking target. 15. The information processing device according to any one of items 1 to 14, (Item 16) image input means for acquiring the image to be input; a tracking target setting means for setting the tracking target; a multitasking unit that executes a plurality of recognition tasks on the tracking target of the cropped image; Equipped with The tracking target identification means identifies the tracking target included in the image based on results of the plurality of recognition tasks. 16. The information processing device according to any one of items 1 to 15, (Item 17) a tracking target identification step of identifying a target to be tracked included in the image as the tracking target; a local area specifying step of specifying, within the image, a local area including at least a part of a detection target included in the tracking target; a crop region determination step of determining size information of a crop region for cutting out the tracking target from the image based on size information of the local region; a cropping step of cutting out the image based on the cropping area to generate a cropped image; An information processing method comprising: (Item 18) A program for causing a computer to function as each means of the information processing device according to any one of items 1 to 16.

[0206] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0207] 200...information processing device, 220...image input unit, 230...tracking target setting unit, 240...cropping area determination unit, 250...cropping unit, 260...multitasking unit, 270...tracking target identification unit, 280...local area identification unit, 1000...second information processing device, 1010...second multitasking unit.

Claims

1. a tracking target identification means for identifying a target to be tracked included in an image as the tracking target; a local area specifying means for specifying, within the image, a local area including at least a part of a detection target included in the tracking target; a crop region determination means for determining size information of a crop region for cutting out the tracking target from the image based on size information of the local region; a cropping unit that cuts out the image based on the crop area to generate a cropped image; An information processing device comprising:

2. the local area specifying means specifies a plurality of local areas including at least a part of a plurality of types of detection targets included in the tracking target; The crop region determining means determines size information of the crop region based on size information of any one of the plurality of local regions.

2. The information processing apparatus according to claim 1, wherein:

3. The crop region determination means determines size information of the crop region based on size information of a local region of a detection target selected from the plurality of types of detection targets based on a predetermined priority order of the detection targets.

3. The information processing apparatus according to claim 2, wherein:

4. the local area specifying means specifies a plurality of local areas including at least a part of the detection target of the tracking target; The crop region determining means determines size information of the crop region based on size information of the plurality of local regions.

2. The information processing apparatus according to claim 1, wherein:

5. The crop region determining means determines size information of the crop region based on a result of averaging the size information of the plurality of local regions.

5. The information processing apparatus according to claim 4,

6. The cropping area determining means determines the size information of the cropping area based on at least one of an upper limit value and a lower limit value set based on the size information of the cropping area.

5. The information processing apparatus according to claim 4,

7. the local region specifying means specifies a plurality of local regions from a plurality of images taken at different times; The cropping region determining means determines size information of the cropping region based on size information of at least one of the plurality of local regions.

2. The information processing apparatus according to claim 1, wherein:

8. the local area specifying means specifies a plurality of local areas including at least a part of a plurality of types of detection targets included in the tracking target; The crop region determination means determines size information of the crop region based on size information of a local region of a detection target selected from the plurality of types of detection targets.

8. The information processing apparatus according to claim 7,

9. The crop region determination means determines size information of the crop region based on size information of a local region of a detection target selected from the plurality of types of detection targets, based on a change in size information of the plurality of local regions.

9. The information processing apparatus according to claim 8,

10. the local area specifying means specifies a plurality of local areas including at least a part of a plurality of types of detection targets included in the tracking target; The crop region determination means determines size information of the crop region based on size information of a local region of a detection target selected from the plurality of types of detection targets based on recognition accuracy of the detection target.

2. The information processing apparatus according to claim 1, wherein:

11. The crop region determination means uses at least one of the recognition rate of the detection object and the recognition score of the detection object as the recognition accuracy.

11. The information processing apparatus according to claim 10,

12. the local region specifying means specifies a plurality of local regions from a plurality of images taken at different times; The cropping region determining means determines whether or not to set a cropping region at a previous time as a current cropping region based on a change in size information of the plurality of local regions.

2. The information processing apparatus according to claim 1, wherein:

13. The crop region determination means sets the crop region of the previous time as the crop region of the current image when the local region used to determine the crop region of the image of the previous time is not detected in the current image.

13. The information processing apparatus according to claim 12.

14. The local area specifying means specifies the local area from the cropped image and an image larger than the cropped image.

2. The information processing apparatus according to claim 1, wherein:

15. The local area specifying means sets a local area based on a detection target set according to a category of the tracking target.

2. The information processing apparatus according to claim 1, wherein:

16. image input means for acquiring the image to be input; a tracking target setting means for setting the tracking target; a multitasking unit that executes a plurality of recognition tasks on the tracking target of the cropped image; Equipped with The tracking target identification means identifies the tracking target included in the image based on results of the plurality of recognition tasks.

2. The information processing apparatus according to claim 1, wherein:

17. a tracking target identification step of identifying a target to be tracked included in the image as the tracking target; a local area specifying step of specifying, within the image, a local area including at least a part of a detection target included in the tracking target; a crop region determination step of determining size information of a crop region for cutting out the tracking target from the image based on size information of the local region; a cropping step of cutting out the image based on the cropping area to generate a cropped image; An information processing method comprising:

18. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 16.

Images