Information processing device, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing method, and a program that can create a stable cropped image.SOLUTION: An information processing apparatus comprises: an image input unit that receives one or more images to be input as input data; a tracking target specification unit that specifies a target to be tracked included in the image as a tracking target; a local area specification unit that specifies, in the image, a local area including at least part of a detection target included in the tracking target; a cropping area determination unit that determines size information of a cropping area for cutting out the tracking target from the image on the basis of size information of the local area; and a cropping unit that cuts out an image on the basis of the cropping area to create a cropped image.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In recent years, a technique has been known in which a partial region including a target to be tracked (hereinafter, tracking target) included in an image is cut out (hereinafter, cropped), and the tracking target is tracked using an image of the partial region (hereinafter, cropped image).

[0003] Patent Document 1 discloses a technique for determining a single cropped image for executing a plurality of recognition tasks, using size information of the entire body of the main subject and information other than the main subject. The technique of Patent Document 1 generates a cropped image by cropping a partial region including the subject from the entire image by performing cropping on the image.

Prior Art Documents

Patent Documents

[0004] [[ID= twenty-seven ]] Japanese Patent Application Laid-Open No. 2021-141421

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technique of Patent Document 1, when determining a cropped image, since cropping is performed based on the overall size of the tracking target, which is easily affected by the posture change of the tracking target, the size of the region cropped due to the posture change of the tracking target is likely to change, and the cropped image is not stable.

[0006] Therefore, an object of the present invention is to provide an information processing apparatus, an information processing method, and a program capable of generating a stable cropped image.

Means for Solving the Problems

[0007] To solve this problem, for example, the information processing apparatus of the present invention has the following configuration. That is, First frame Included subject of In the first frame described above A means for identifying a target to be tracked, The local region including at least a portion of the detection target included in the tracking target is First frame Inside in Local area identification means, Based on the size information of the local region, The second frame was captured after the first frame. From the aforementioned target to Areas that include Cut did A means for determining the size information of the crop region, Based on the crop region whose size information has been determined in the crop region determination means From the second frame on the side A cropping method for generating a cropped image, Equipped with 、 The tracking target identification means identifies the tracking target in the second frame from the subject included in the cropped image. ru. [Effects of the Invention]

[0008] According to the present invention, stable cropped images can be generated. [Brief explanation of the drawing]

[0009] [Figure 1] Hardware configuration diagram of the information processing device in the embodiment. [Figure 2] Functional block diagrams illustrating the functions of the information processing device in the first to fourth embodiments. [Figure 3] A diagram illustrating a time series of images illustrating the first and second embodiments. [Figure 4] A diagram showing flowcharts of the recognition process in the first to fourth embodiments. [Figure 5] This figure shows flowcharts of the crop region determination process in the first and second embodiments. [Figure 6] Diagram showing pre-registration information for the first to fifth embodiments. [Figure 7] A diagram illustrating a time series of images illustrating the third and fourth embodiments. [Figure 8] A diagram showing a flowchart of the crop area determination process according to the third embodiment. [Figure 9] A diagram showing a flowchart of the crop area determination process according to the fourth embodiment. [Figure 10] A diagram for explaining the configuration of the information processing apparatus according to the fifth embodiment. [Figure 11] A diagram of time-series images for explaining the fifth embodiment. [Figure 12] A diagram showing a flowchart of the fifth embodiment.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0011] <First Embodiment: Crop Method Using a Local Region with a Low Size Variation Rate with Respect to Posture Change> FIG. 1 shows an example of a hardware configuration diagram of the information processing apparatus 200. The information processing apparatus 200 may be a so-called computer. The information processing apparatus 200 includes a CPU 100, a ROM 110, a RAM 120, an HDD 130, an input unit 14, a display unit 150, a communication unit 160, and a bus 170. The CPU 100, the ROM 110, the RAM 120, the HDD 130, the input unit 140, the display unit 150, and the communication unit 160 are connected to be able to transmit and receive information via the bus 170.

[0012] CPU100 stands for Central Processing Unit, and is a central processing unit. The information processing unit 200 may have other processors such as an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), and a QPU (Quantum Processing Unit) in place of or in addition to the CPU100. The CPU100 performs calculations and logical decisions for various processes. For example, the CPU100 reads programs stored in the ROM110 or HDD130 and loads them into the RAM120 to realize various functions and execute various processes. Furthermore, some or all of the functions of the information processing unit 200 may be realized by one or more circuits such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array).

[0013] ROM110 stands for Read-Only Memory and is a type of non-volatile memory. ROM110 stores control programs such as the OS (Operating System).

[0014] RAM120 stands for Random Access Memory. RAM120 is used as the main memory of the CPU100, as well as for temporary storage areas such as the work area.

[0015] HDD130 is an abbreviation for Hard Disk Drive and is a large-capacity non-volatile storage device. The HDD130 stores electronic data, programs, and data necessary for program execution according to this embodiment. The information processing device 200 may have an external storage device that performs a similar role in place of or in addition to the HDD130. Here, the external storage device can be realized, for example, by a media (recording medium) and an external storage drive for accessing the media. Examples of such media include flexible disks (FD), CD-ROMs, DVDs, USB memory, MOs, and flash memory. The external storage device may also be a server device connected via a network.

[0016] The input unit 140 receives input from the user and passes it on to the CPU 100. The input unit 140 includes a mouse, keyboard, touch panel, etc.

[0017] The display unit 150 displays various data and images of processing results to the user based on image data acquired from the CPU 100 and the like. The display unit 150 is composed of display devices such as liquid crystal displays and organic EL (Electro-Luminescence) displays. The term "image" may be used to include still images, videos, images of a single frame of a video, video footage, and the data thereof.

[0018] The communication unit 160 relays communication with other devices. As a result, the information processing device 200 communicates data with other devices via the communication unit 160. The information processing device 200 may also receive instructions from the user via the communication unit 160 from other devices, or output processing results to other devices.

[0019] Figure 2 is a functional block diagram illustrating the functions of the information processing device 200 according to this embodiment. The configuration of this embodiment will be explained using Figure 2. Only an overview will be explained here, and details will be described later. The information processing device 200 has the functions of an image input unit 220, a tracking target setting unit 230, a crop area determination unit 240, a crop unit 250, a multitasking unit 260, a tracking target identification unit 270, a local area identification unit 280, and an output unit 290. For example, the CPU 100 realizes the functions of the image input unit 220, the tracking target setting unit 230, the crop area determination unit 240, the crop unit 250, the multitasking unit 260, the tracking target identification unit 270, the local area identification unit 280, and the output unit 290 by reading and executing a program stored in the ROM 110 or HDD 130.

[0020] The input data 210 represents data from an image that has been captured or a group of images containing multiple images. The input data 210 is, for example, a series of images obtained from an imaging device such as a digital camera or a surveillance camera.

[0021] The image input unit 220 receives one or more images as input data 210. The input data 210 is, for example, a video in which multiple frames of images are sequentially arranged in time.

[0022] The tracking target setting unit 230 sets at least one of the following: the type of tracking target, its position in the image, and its size, indicated by the number of pixels, etc., in the initial frame of the input data 210. The type of tracking target may be, for example, a person, an animal including cats and dogs, and a car.

[0023] The crop region determination unit 240 determines a crop region for extracting the tracking target from the image passed from the image input unit 220. Specifically, the crop region determination unit 240 determines the size information of the crop region based on the size information of a local region that includes at least a part of the detection target included in the tracking target. The size information is information about the size of a region in the image (in this case, a local region), and may be at least one of the number of pixels in the region and the length and width of the region. The detection target is a part of the tracking target, and for example, in the case of an animal, it may include the head, face, etc. There may be multiple detection targets. For example, the detection target may include the head, face, feet, and hands, etc. If multiple types of detection targets are detected, the crop region determination unit 240 may select a detection target from the multiple types of detection targets to determine the size information of the crop region according to the priority order described later. In this case, the crop region determination unit 240 may determine the size information of the crop region based on the size information of the local region of the selected detection target. The crop area determination unit 240 may also determine the position of the crop area along with the size information.

[0024] The cropping unit 250 crops the image passed from the image input unit 220 based on the cropping area determined by the cropping area determination unit 240. As a result, the cropping unit 250 generates a cropped image that includes the tracking target to be used by the multitasking unit 260.

[0025] The multitasking unit 260 performs multiple recognition tasks on the cropped image generated by the cropping unit 250. In this embodiment, the multiple recognition tasks will be described using a whole-body cat detector, a head detector, and a face detector as examples. Here, the recognition models used for the recognition tasks performed can be various models, including, for example, convolutional neural networks, ViT (Vision Transformer), and SVM (Support Vector Machine) combined with a feature extractor. This embodiment is not limited to the above form, but in this description, the multitasking unit 260 will be described as a CNN.

[0026] The tracking target identification unit 270 identifies the target to be tracked within the image as the tracking target based on the detection results obtained from the multitasking unit 260. The tracking target identification unit 270 identifies the tracking target based on the similarity between the characteristic information of the tracking target obtained from the tracking target setting unit 230 and the characteristic information of the detection results obtained from the multitasking unit 260.

[0027] The local area identification unit 280 identifies local areas for determining the size information of the crop area. Specifically, the local area identification unit 280 identifies local areas within the image that include at least a portion of the detection targets included in the tracking target identified by the tracking target identification unit 270. The image in which the local areas are identified may be a cropped image. That is, the local area identification unit 280 may identify local areas within the image using either the image or the cropped image. For example, if the tracking target is a cat and the detection targets are the cat's whole body, head, and face, the local area identification unit 280 identifies local areas within the image for each detection target that include at least a portion of each detection target. The local area identification unit 280 passes the information of one or more identified local areas to the crop area determination unit 240. The local area information is used when determining the crop area for the next frame.

[0028] The output unit 290 outputs the results obtained from the multitasking unit 260, the tracking target identification unit 270, and the local area identification unit 280. In this way, the information processing device 200 can accurately track the tracking target by processing the input scene in a time series.

[0029] Figure 3 shows a time-series image of a cat being tracked as an example in this embodiment. Figure 4 is a flowchart of the recognition process in this embodiment. Hereafter, the flowchart will be assumed to be implemented by the CPU 100 executing a control program. Figure 6(a) shows the pre-registration information in this embodiment. The pre-registration information is stored in the HDD 130, and in this embodiment, the information processing device 200 can refer to the information as needed. The pre-registration information may be read from the HDD 130 and loaded into the RAM 120.

[0030] The details of the process in this embodiment will be explained below with reference to Figure 4.

[0031] In S401, the image input unit 220 acquires an image for one frame, which is input as input data 210. Here, the image input unit 220 acquires the image 301 shown in Figure 3(a). Image 301 in Figure 3(a) is the image of the initial frame at time t=0. In image 301, a cat is walking to the left.

[0032] In S402, the image input unit 220 determines whether the image of the acquired input data 210 is the initial frame. If image 301 is the initial frame at time t=0, the image input unit 220 determines that the image is the initial frame and proceeds to S403.

[0033] In S403, the tracking target setting unit 230 sets the tracking target based on the image of the initial frame. In S403, the tracking target setting unit 230 sets the tracking target to a cat and sets the position and size of the tracking target. Any method can be used to set the tracking target. For example, the tracking target setting unit 230 may set the tracking target by accepting touch and voice operations on the camera screen by the user, or it may set the tracking target using the recognition results of the recognition processing unit of the camera, and various other methods can be used. In this embodiment, the tracking target setting unit 230 sets the entire body of the cat as the tracking target area 302, as shown in Figure 3(a). After the tracking target setting unit 230 has finished setting the tracking target, the image input unit 220 performs the process of S401 again.

[0034] In S401, the image input unit 220 acquires the image 311 shown in Figure 3(b). Here, image 311 is an image of a cat at time t=1. In image 311, the cat, which was walking to the left, has stopped.

[0035] In S402, the image input unit 220 determines that image 311 is a frame later than the initial frame and therefore is not the initial frame, and proceeds to the crop area determination process in S404.

[0036] Figure 5(a) shows a detailed flow of the crop region determination process in S404 executed by the crop region determination unit 240 in the first embodiment. The crop region determination unit 240 performs the process in S404 and the processes in S501 to S507, which are detailed versions of S404.

[0037] In S501, the crop region determination unit 240 determines whether or not there is a local region detected in the previous frame (i.e., the frame at the previous time) that can be used for calculating the crop region. At time t=1, the local region identification process, which will be described later in S408, has not yet been performed, so the crop region determination unit 240 determines that there is no local region, the determination in S501 is No, and the process moves on to S503.

[0038] In S503, the crop area determination unit 240 sets the total number of pixels of the tracking target set in the tracking target setting unit 230 as the crop standard.

[0039] Here, the crop criterion refers to the size information of the tracking target area on the image, and is information used to calculate the size of the crop area (in this case, the number of pixels). The size and number of pixels are examples of size information. The crop area determination unit 240 can determine the crop area by using the crop magnification and crop criterion, which will be described later. In this embodiment, the crop criterion is described as the number of pixels, but this is not necessarily required, and any information related to the size of the tracking target may be used. For example, the crop area determination unit 240 may use size information such as the length of the long side, the length of the short side, and the length of the diagonal of the rectangular frame representing the target area as the crop criterion. At time t=1, the crop area determination unit 240 determines the crop area by taking the product of the number of pixels of the entire tracking target, which is the crop criterion, and the crop magnification.

[0040] In S504, the crop area determination unit 240 acquires the crop magnification corresponding to the crop criterion. The crop magnification is a multiplier applied to a pre-set crop criterion, and can be set for each type of crop criterion (for example, the detection target). Figure 6(a) shows the pre-registered information 601 that is set in advance in this embodiment. The pre-registered information 601 determines the crop magnification for each detection target, which is the type of recognition task. It shows that the crop magnification is 3.0 times when the whole body is selected as the crop criterion, 15.0 times when the head is selected, and 30.0 times when the face is selected. Therefore, since the number of pixels of the whole body of the tracking target is set as the crop criterion in S503, in S504, the crop area determination unit 240 selects and acquires 3.0 times, which is associated with the whole body, from the pre-registered information 601 shown in Figure 6(a) as the crop magnification. In this embodiment, the number of pixels of the entire body of the tracking target is the number of pixels of the tracking target area 302 set by the tracking target setting unit 230, but it is not limited to this, and the result of the whole-body detector of the multitasking unit 260 may also be used.

[0041] In S505, the crop area determination unit 240 calculates the number of pixels in the crop area from the crop criterion and the crop magnification. Specifically, the crop area determination unit 240 calculates the number of pixels in the crop area as the product of the number of pixels of the entire body of the tracking target set as the crop criterion and the crop magnification determined in S504.

[0042] In S506, the crop area determination unit 240 determines the aspect ratio of the crop area. In this embodiment, the aspect ratio is described as 4:3, but it is not limited to this. The crop area determination unit 240 may, for example, determine the aspect ratio according to the crop area, or it may adopt a predetermined aspect ratio.

[0043] In S507, the crop region determination unit 240 determines the position of the crop region. In this embodiment, the crop region determination unit 240 determines the center of the tracking target as the position of the crop region. Note that the position of the crop region does not have to be the center of the tracking target; any position determination method can be used. As a result, the crop region determination unit 240 determines the crop region 312 shown in Figure 3(b). Exit the detailed flow of S404 and proceed to S405.

[0044] In S405, the cropping unit 250 generates a cropped image 313. For example, the cropping unit 250 crops the image 311 using the cropping region 312 determined in S404 and resizes the image. In this embodiment, the resized image size is described as QVGA (320 pixels × 240 pixels), but the image size is not limited to this. Through the above process, the cropping unit 250 generates the cropped image 313 shown in Figure 3(c).

[0045] In S406, the multitasking unit 260 performs multitasking processing on the cropped image 313 in Figure 3(c). In this embodiment, the multitasking processing performed by the multitasking unit 260 is described using a whole-body detector, head detector, and face detector of a cat as examples, but is not limited to these. For example, the multitasking unit 260 may perform multitasking processing on a pupil detector and a function to track the target, in place of or in addition to the above detectors. The detection results of the multitasking unit 260 at time t=1 are the whole-body detection result 314, the head detection result 315, and the face detection result 316 shown in Figure 3(c).

[0046] In S407, the tracking target identification unit 270 identifies the tracking target from the detection results of the multitasking unit 260. In this embodiment, the tracking target identification unit 270 performs feature comparison between the tracking target region 302 and each detection result, and determines that the detection result with the closer feature is the tracking target. Identification of the tracking target does not necessarily have to be by feature comparison; any method that can determine that the detection result is the tracking target is acceptable. Let's assume that at time t=1, the tracking target identification unit 270 identified the whole-body detection result 314 as the tracking target.

[0047] In S408, the local area identification unit 280 identifies the local area to be tracked from the detection results of the multitasking unit 260. In this embodiment, the local area identification unit 280 identifies the local area using the distance from the center position of the whole body detection result 314 in Figure 3(c), which has been identified as a tracking target by the tracking target identification unit 270. Identifying the local area does not necessarily require the use of distance information. Let's assume that at time t=1, the local area identification unit 280 identified the head detection result 315 and the face detection result 316 as local areas. Here, the whole body detection result 314 is not used as a local area for determining the crop area because it has been identified as a tracking target. The areas of the head detection result 315 and the face detection result 316 are examples of multiple local areas of multiple different types of detection targets of the tracking target.

[0048] In S409, it is determined whether processing of all frames has finished. At time t=1, processing of all frames has not finished, so the process returns to S401 and processes the data for time t=2.

[0049] In S401, the image input unit 220 acquires image 321 in Figure 3(d). Image 321 is an image of a cat at time t=2. In image 321, the cat, which was stopped in the previous frame, has resumed moving to the left. After that, the image input unit 220 performs the processing in S402 again, and then the crop area determination unit 240 performs the processing in S404.

[0050] In S501 of Figure 5(a), which is a detailed flow of S404, the crop region determination unit 240 determines whether or not there is a local region detected in the previous frame that can be used for calculating the crop region. At time t=1, the crop region determination unit 240 has detected the head detection result 315 and the face detection result 316 in the previous frame. In this embodiment, the crop region determination unit 240 then proceeds to processing S502 so that all the information registered in the pre-registered information 601 can be used for calculating the crop region.

[0051] In S502, the crop region determination unit 240 sets the number of pixels in the local region as the crop criterion according to the priority order of the pre-registered information 601. If there are multiple types of local regions, the detection target is selected according to the priority order, and the local region of that detection target is set as the crop criterion. Specifically, there are two local regions, in this case the head detection result 315 and the face detection result 316, but referring to the pre-registered information 601, the head has a higher priority. Therefore, in S502, the crop region determination unit 240 sets the number of pixels in the head detection result 315 as the crop criterion.

[0052] Here, the priority order of each crop criterion in the pre-registered information 601 is determined in advance by considering the "rate of size change in response to changes in posture." In the example of a cat in this embodiment, it can be seen that the size change of the whole body is large when walking and when standing still, but the size change of the head is small. Therefore, by setting the crop area determination unit 240 to the head as the crop criterion, the change in the crop area in response to changes in posture can be reduced. The face has a larger size change in response to changes in posture compared to the head, but it is small compared to the whole body. Therefore, the priority of the head is set as 1st, the priority of the face as 2nd, and the priority of the whole body as 3rd.

[0053] In S504, the crop area determination unit 240 obtains a crop ratio of 15.0 from the pre-registered information 601, and in S505 calculates the number of pixels in the crop area. Subsequently, the crop area determination unit 240 performs the processes in S506 and S507 to determine the crop area 322 shown in Figure 3(d).

[0054] In S405, the cropping unit 250 generates a cropped image using the cropping region 322. Then, the processes from S406 to S409 are performed, and the series of processes is completed.

[0055] In this embodiment, the crop criterion and crop magnification can be determined by considering the "rate of size change in response to changes in posture" as described above. If an area that is easily affected by changes in posture is set as the crop criterion, the crop area will become extremely narrow, for example, with a curled-up cat. Since the crop area is determined using information from the previous frame, if the crop area is narrow, there is a possibility that part or all of the cat may extend beyond the crop area if the cat makes a sudden movement in the current frame. If this happens, the cat will extend beyond the cropped image, and multitasking processing cannot be performed correctly.

[0056] This embodiment processes data by setting prioritizing the knowledge that a cat's entire body exhibits significant size variations depending on its posture, while the size variation of its head is small, as a basis for pre-registered information.

[0057] In other words, this embodiment determines the size information of the crop region based on size information of the head and other parts of the tracked target, which is less affected by changes in the tracked target's posture. As a result, this embodiment can generate stable cropped images because the size of the crop region is less affected by changes in the tracked target's posture. Consequently, this embodiment can reduce the occurrence of the tracked target extending beyond the crop region, stabilizing the performance of multitasking processing that handles multiple recognition tasks, and stabilizing the accuracy of detection tasks for the detected target.

[0058] In this embodiment, a detection target is selected from multiple detection targets, including the whole body, head, and face, based on a predetermined priority order, to be used in determining the size information of the crop region. This embodiment then determines the size information of the crop region based on the size information of the local region of the detection target. As a result, this embodiment can more reliably stabilize the crop region. Furthermore, even if the detection target with the highest priority is not detected, this embodiment can determine the size information of the crop region using the size information of the local region of the next highest priority detection target, which is less affected by changes in the posture of the tracked target.

[0059] <Second Embodiment: Calculation of Crop Region Using Results from Multiple Recognition Tasks> In the second embodiment, a method for determining a crop region using the results of multiple recognition tasks will be described, using the case of a cat walking as an example, similar to the first embodiment. In this embodiment, the local region identification unit 280 identifies multiple local regions. For example, the local region identification unit 280 identifies multiple local regions by identifying local regions from each of multiple detection targets, including the head and face. The crop region determination unit 240 determines the size information of the crop region based on the size information of the multiple local regions. For example, the crop region determination unit 240 may determine the crop region based on the result of averaging the size information of the multiple local regions.

[0060] The hardware configuration example of this embodiment is the same as that shown in Figure 1 of the first embodiment, and the configuration diagram is also the same as that shown in Figure 2.

[0061] Figure 3 shows a time-series image of a cat being tracked as an example in this embodiment. Image 301 in Figure 3(a) is an image of the cat walking to the left in the initial frame at time t=0. In the second embodiment, the flowcharts in Figure 4 and Figure 5(b) are used. Figure 6(b) shows the pre-registration information in this embodiment.

[0062] In processing image 301 at time t=0, the processing is carried out in the same order as in the first embodiment: S401, S402, and S403. In S403, the tracking target setting unit 230 sets the entire body of the cat as the tracking target area 302, as shown in Figure 3(a).

[0063] Next, in S401, the image input unit 220 acquires image 311 in Figure 3(b). Here, image 311 is an image of a cat at time t=1. In image 311, the cat, which was walking to the left, has stopped. The image input unit 220 determines in S402 that image 311 is a frame later than the initial frame, so the crop region determination unit 240 executes the crop region determination process in S404.

[0064] Figure 5(b) shows a detailed flow of the crop region determination process S404 executed by the crop region determination unit 240 in the second embodiment. Note that the explanation of the process in Figure 5(b) that is the same as that in Figure 5(a) will be simplified. Figure 6(b) shows the pre-registration information 602 in the second embodiment.

[0065] In S511, the crop region determination unit 240 determines whether or not there is a local region detected in the previous frame that can be used for calculating the crop region. At time t=1, the local region identification process, which will be described later in S408, has not yet been performed, so the determination in S511 is No, and the process moves on to S514. In S514, the crop region determination unit 240 sets the total number of pixels of the tracking target set in the tracking target setting unit 230 as the crop standard.

[0066] In step S515, the crop area determination unit 240 obtains a crop ratio of 3.0, which corresponds to the whole body and is the crop standard, from the pre-registered information 602 in Figure 6(b).

[0067] In S516, the crop area determination unit 240 calculates the number of pixels in the crop area by multiplying the total number of pixels of the tracking target set as the crop criterion by the crop magnification determined in S515.

[0068] In S517, the crop region determination unit 240 determines the aspect ratio of the crop region, and in S518, it determines the position of the crop region. As a result, the crop region determination unit 240 determines the crop region 312 shown in Figure 3(b).

[0069] In S405, the cropping unit 250 performs cropping using the cropping region 312 determined in S404 and resizes the image to QVGA to generate the cropped image 313 shown in Figure 3(c). In S406, the multitasking unit 260 performs multitasking and obtains the whole body detection result 314, the head detection result 315, and the face detection result 316.

[0070] In S407, the tracking target identification unit 270 identifies the tracking target from the detection results of the multitasking unit 260. At time t=1, the tracking target identification unit 270 identifies the whole-body detection result 314 as the tracking target.

[0071] In S408, the local area identification unit 280 identifies the local area to be tracked from the detection results of the multitasking unit 260. At time t=1, the tracking target identification unit 270 identifies the head detection result 315 and the face detection result 316 as local areas. Here, the whole body detection result 314 is identified as a tracking target and is therefore not used as a local area for determining the crop area.

[0072] In S409, it is determined whether processing of all frames has finished. At time t=1, processing of all frames has not finished, so the process returns to S401 and processes the data for time t=2.

[0073] In S401, the image input unit 220 acquires image 331 shown in Figure 3(e). Image 331 is an image of the cat at time t=2. In image 331, the cat, which was stopped in the previous frame, has resumed moving to the left. After that, the processes from S402 to S404 are performed again.

[0074] In S511, the crop region determination unit 240 determines whether there is a local region detected in the previous frame that can be used for calculating the crop region. At time t=1, the head detection result 315 and the face detection result 316 have been detected. In this embodiment, all information registered in the pre-registration information 602 can be set as crop criteria, so the process proceeds to S512.

[0075] In S512, the crop region determination unit 240 obtains the crop magnification and weight coefficients for all local regions identified from multiple detection targets from the pre-registered information 602. In S513, the crop region determination unit 240 calculates the number of pixels in the crop region from the number of pixels of the crop criterion, the crop magnification, and the weight coefficients.

[0076] Here, the weight coefficient indicates the importance of each recognition task, and by using the weight coefficient, it is possible to calculate a more stable number of pixels in the cropped area. In this embodiment, the "size variation rate with respect to posture changes" was taken into consideration, and the pre-registered information 602 was set so that the weight coefficient of detection results with a small size variation rate is large.

[0077] The method for calculating the number of pixels C in the cropped area in this embodiment is shown below.

[0078]

number

[0079] In equation (1.1), W is the weighting coefficient, P is the number of pixels in the local region, and R is the cropping ratio. N represents the number of identified local regions, which in this embodiment is two in total: head detection result 315 and face detection result 316. In this embodiment, the number of pixels in head detection result 315 is assumed to be 60 pixels, and the number of pixels in face detection result 316 is assumed to be 20 pixels.

[0080] Equation (1.1) above calculates the number of pixels in the crop region for each recognition task and then calculates its weighted average. When the head detection result 315 is used as the cropping criterion, the number of pixels in the crop region is 900 pixels, which is the result of multiplying the head detection result's 60 pixels by the head's cropping magnification of 15.0. When the face detection result 316 is used as the cropping criterion, the number of pixels in the crop region is 600 pixels, which is the result of multiplying the face detection result's 20 pixels by the head's cropping magnification of 30.0. By multiplying each of these by a weight coefficient of 5.0 for the head and a weight coefficient of 3.0 for the face and calculating the weighted average, the number of pixels in the crop region C becomes 787.5 pixels. It can be seen that the crop region determination unit 240 calculates the number of pixels in the crop region using multiple detection results, so that even if the size of the head detection result 315 is incorrectly detected, the number of pixels in the crop region remains stable.

[0081] In S517, the crop region determination unit 240 determines the aspect ratio of the crop region, and in S518, it determines the position of the crop region. As a result, the crop region determination unit 240 determines the crop region 332 shown in Figure 3(e).

[0082] Subsequently, in S405, the cropping unit 250 generates a cropped image using the cropping region 332. After that, the processes from S406 to S409 are performed, and the processing of this embodiment is completed.

[0083] As described above, this embodiment calculates the size information of the crop region based on the size information of multiple detection targets detected by multiple recognition tasks, thereby enabling the calculation of a stable crop region even if the size of some of the detection results is incorrect. For example, in this embodiment, the size information of the crop region is calculated based on the result of averaging the size information of multiple local regions. This makes it possible to stabilize the crop region even if the size information of multiple local regions contains outliers. In this embodiment, a weighted average was used as an example, but this is not necessarily required, and the crop region may be determined using averaging processes such as a simple average or a moving average.

[0084] Furthermore, the crop region determination unit 240 may determine at least one of the upper and lower limits of the size information of the crop region using the detection results of multiple recognition tasks. For example, if a priority is set in the pre-registered information, the crop region determination unit 240 may calculate the crop region from the detection results of the recognition task with the highest priority and set the upper limit as 2.0 times the number of pixels in the crop region and the lower limit as 0.5 times the number of pixels. Subsequently, if the number of pixels in the crop region exceeds the upper limit or falls below the lower limit when the crop region is calculated using a weighted average as in this embodiment, the crop region determination unit 240 sets the number of pixels in the crop region to fall within the set upper and lower limits. By setting the upper and lower limits as described above, the crop region can be set within a range of 0.5 to 2.0 times the number of pixels in the crop region calculated from the recognition task with the highest priority.

[0085] As a result, the performance of the multitasking mechanism can be stabilized by using the detection results of multiple recognition tasks.

[0086] <Third Embodiment: Using time-series information, determine the cropping criteria used to calculate the cropped area.> In this embodiment, a method for determining crop criteria using time-series information is described, using the case of a cat walking as an example. For example, in this embodiment, the local area identification unit 280 identifies multiple local areas from multiple images taken at different times. The crop area determination unit 240 determines the size information of the crop area using the size information of at least one of the multiple local areas. Here, the local area identification unit 280 may identify multiple local areas of multiple types of detection targets, for example, multiple local areas of the head and face. In other words, the local area identification unit 280 may identify multiple local areas of different types of detection targets from each of multiple images taken at different times. The crop area determination unit 240 may determine the size information of the crop information based on the size information of the local areas of the detection targets selected from the multiple types of detection targets. For example, the crop area determination unit 240 may determine the size information of the crop area based on the size information of the local areas of the selected detection targets based on the change in the size information of the local areas. The change in the size information of the local areas may be, for example, the rate of change in the size of the local areas identified from images taken at different times. The size referred to here may be the number of pixels in the local region, or the product of the length and width of the local region, or any other such representation.

[0087] The hardware configuration example of this embodiment is the same as that shown in Figure 1 of the first embodiment, and the configuration diagram is also the same as that shown in Figure 2.

[0088] Figure 7 shows a time-series image of a cat being tracked as an example in this embodiment. Image 701 in Figure 7(a) is an image of the cat walking to the left in the initial frame at time t=0. In the third embodiment, the flowcharts in Figure 4 and Figure 8 are used. Figure 6(c) shows the pre-registration information in this embodiment.

[0089] In processing image 701 at time t=0, the processing is carried out in the same order as in the first embodiment: S401, S402, and S403. In S403, the tracking target setting unit 230 sets the entire body of the cat as the tracking target area 702, as shown in Figure 7(a).

[0090] Next, in S401, the image input unit 220 acquires image 711, shown in Figure 7(b). Here, image 711 is an image of the cat at time t=1. In image 711, the cat continues to move to the left. In S402, the image input unit 220 determines that image 711 is a frame later than the initial frame, and proceeds to the crop region determination process in S404.

[0091] Figure 8 shows a detailed flow of the crop region determination process in S404 executed by the crop region determination unit 240 of the third embodiment. The explanation of the processes in Figure 8 that are the same as those in the embodiments described above will be simplified.

[0092] In S801, the crop region determination unit 240 determines whether or not there is a local region detected in the previous frame that can be used for calculating the crop region. At time t=1, the local region identification process, which will be performed in S408 later, has not yet been performed, so there is no local region. Therefore, the crop region determination unit 240 determines No in the determination of S801 and proceeds to the process in S807.

[0093] In the S807, the crop area determination unit 240 sets the total number of pixels of the tracking target set in the tracking target setting unit 230 as the crop standard.

[0094] In S808, the crop area determination unit 240 obtains a crop ratio of 3.0, which corresponds to the whole body and is the crop standard, from the pre-registered information 603 in Figure 6(c).

[0095] In S809, the crop area determination unit 240 calculates the number of pixels in the crop area by multiplying the total number of pixels of the tracking target set as the crop criterion by the crop magnification determined in S808.

[0096] The crop region determination unit 240 determines the aspect ratio of the crop region in S810 and the position of the crop region in S811. As a result, the crop region determination unit 240 determines the crop region 712 shown in Figure 7(b).

[0097] In S405, the cropping unit 250 performs cropping using the cropping region 712 determined in S404 and resizes the image to QVGA, generating the cropped image 713 shown in Figure 3(c).

[0098] In S406, the multitasking unit 260 performs multitasking and obtains the whole body detection result 714, the head detection result 715, and the face detection result 716.

[0099] In S407, the tracking target identification unit 270 identifies the tracking target from the detection results of the multitasking unit 260. At time t=1, the tracking target identification unit 270 identifies the whole body detection result 714 as the tracking target.

[0100] In S408, the local area identification unit 280 identifies the local area to be tracked from the detection results of the multitasking unit 260. At time t=1, the local area identification unit 280 identifies the head detection result 715 and the face detection result 716 as local areas.

[0101] In S409, it is determined whether processing of all frames has finished. At time t=1, processing of all frames has not finished, so the process returns to S401 and processing at time t=2 is performed.

[0102] In S401, the image input unit 220 acquires image 721, shown in Figure 7(d). Here, image 721 is an image of the cat at time t=2. In image 721, the cat continues to move to the left. After that, the processes from S402 to S404 are performed again.

[0103] In S801, the crop region determination unit 240 determines whether there is a local region detected in the previous frame that can be used to calculate the crop region. At time t=1, the head detection result 715 and the face detection result 716 were detected in the previous frame. In this embodiment, all information registered in the pre-registered information 603 is considered usable for calculating the crop region, so the crop region determination unit 240 determines that there is a local region in the previous frame and proceeds to processing in S802.

[0104] In S802, the crop region determination unit 240 determines whether there are multiple local regions that can be used to calculate the crop region. At time t=1, a head detection result 715 and a face detection result 716 have been detected, so the crop region determination unit 240 determines that there are multiple local regions that can be used and proceeds to processing in S803.

[0105] In S803, the crop region determination unit 240 determines whether the time-series information of the local region detection results is available. At time t=2, only the detection results from time t=1 are available, so the information such as detection results or local regions identified from images at different time points, which is necessary to determine the temporal changes of the local region, is not available. Therefore, the crop region determination unit 240 determines that the time-series information is not available and proceeds to processing in S806.

[0106] In S806, the crop area determination unit 240 sets the detected local area as the crop criterion according to priority. Since it is known from the pre-registered information 603 that the face detection result has the highest priority, the crop area determination unit 240 sets the number of pixels in the face area as the crop criterion.

[0107] In S808, the crop area determination unit 240 obtains and sets a crop magnification of 30.0x, which is associated with the face, from the pre-registered information 603 in Figure 6(c).

[0108] In S809, the crop area determination unit 240 calculates the number of pixels in the crop area by multiplying the number of pixels of the face set as the crop criterion by the crop magnification determined in S808. In S810, the crop area determination unit 240 determines the aspect ratio of the crop area, and in S811, it determines the position of the crop area. As a result, the crop area determination unit 240 determines the crop area 722 shown in Figure 7(d).

[0109] In S405, the cropping unit 250 performs cropping using the cropping region 722 determined in S404 and resizes the image to QVGA, generating the cropped image 723 shown in Figure 3(e).

[0110] In S406, the multitasking unit 260 performs multitasking and obtains the whole body detection result 724, the head detection result 725, and the face detection result 726.

[0111] In S407, the tracking target identification unit 270 identifies the tracking target from the detection results of the multitasking unit 260. At time t=2, the tracking target identification unit 270 identifies the whole body detection result 724 as the tracking target.

[0112] In S408, the local area identification unit 280 identifies the local area to be tracked from the detection results of the multitasking unit 260. At time t=2, the local area identification unit 280 identifies the head detection result 725 and the face detection result 726 as local areas.

[0113] In S409, it is determined whether processing of all frames has finished. At time t=2, processing of all frames has not finished, so the process returns to S401 and performs the processing at time t=3.

[0114] In S401, the image input unit 220 acquires image 731, shown in Figure 7(g). Image 731 is an image of a cat at time t=3. In image 731, the cat continues to move to the left. Subsequently, the processes from S402 to S404 are performed again.

[0115] In S801, the crop region determination unit 240 determines whether there is a local region detected in the previous frame that can be used for calculating the crop region. At time t=2, the head detection result 725 and the face detection result 726 were detected in the previous frame. Therefore, the crop region determination unit 240 determines that there is a local region and proceeds to processing in S802.

[0116] In S802, the crop region determination unit 240 determines whether there are multiple local regions that can be used for cropping. At time t=2, the head detection result 725 and the face detection result 726 have been detected. Therefore, the crop region determination unit 240 determines that there are multiple local regions and proceeds to processing in S803.

[0117] In S803, the crop region determination unit 240 determines whether the time-series information of the local region detection results is available. At time t=3, since the detection results for time t=1 and time t=2 are available, all the information necessary to determine the temporal changes of the local region is available. Therefore, the crop region determination unit 240 determines that the time-series information is available and proceeds to processing in S804. In this embodiment, it is determined that the time-series information is available if the detection results for the past two frames of the local region are available, but this is not the only condition.

[0118] In S804, the crop region determination unit 240 calculates the size change rate from the time-series information of each local region.

[0119] The size change rate is a value calculated from the size information of the detection results of a local region in past frames to determine how much the size (in this case, the number of pixels) has changed. In this embodiment, the crop region determination unit 240 calculates how much the size has changed from the original detection size of the local region by comparing the size information of the detection results from two frames ago with the size information of the detection results from one frame ago, but this is not necessarily required. For example, the crop region determination unit 240 may calculate the size change rate by calculating the size variance and standard deviation of the time-series information of the local region. In whole-body detection result 714 and whole-body detection result 724, the orientation of the cat's body has not changed, so the size change rate of the whole body is small. In face detection result 716 and face detection result 726, the cat's face has changed from facing forward to facing sideways, so the size change rate of the face is large. In head detection result 715 and head detection result 725, it can be seen that even when the cat turns sideways, the change in the size detection result of the head is small.

[0120] In S805, the crop region determination unit 240 sets the number of pixels in the local region with the smallest size change rate as the crop criterion. In this embodiment, the crop region determination unit 240 determines that the region with the smallest size change rate is the head and sets the number of pixels in the head as the crop criterion. However, if the method of the first embodiment were used, the number of pixels in the face would be set as the crop criterion according to priority, which could result in a narrower crop region compared to time t=2.

[0121] In S808, the crop region determination unit 240 obtains a crop ratio of 15.0x, associated with the head, from the pre-registered information 603 in Figure 6(c).

[0122] In S809, the crop area determination unit 240 calculates the number of pixels in the crop area by multiplying the number of pixels in the head, which is set as the crop criterion, by the crop magnification determined in S808.

[0123] In S810, the crop region determination unit 240 determines the aspect ratio of the crop region, and in S811, it determines the position of the crop region. As a result, the crop region determination unit 240 determines the crop region 732 shown in Figure 7(g).

[0124] Subsequently, in S405, the cropping unit 250 generates a cropped image using the cropping region 732. After that, the processes from S406 to S409 are performed, and the processing of this embodiment is completed.

[0125] As described above, the information processing device of the third embodiment can set a stable crop region by calculating a crop region using one of the local regions identified by using multiple images taken at different times as time-series information. For example, this embodiment selects a local region for calculating the crop region based on the rate of change of the size information of the local region. Specifically, this embodiment determines the crop region using a local region with small changes in size information. As a result, this embodiment can appropriately calculate a stable crop region even when local regions unsuitable for a stable crop region have a high priority, or when no priority is set.

[0126] Furthermore, although this embodiment describes a method for determining crop criteria from different types of local regions such as the face and head, it is not limited to this. For example, even if there are multiple candidates for the same type of local region (for example, two in the case of the pupil), one local region suitable for calculating the crop region may be determined using the method described above.

[0127] <Fourth embodiment: When the recognition task is not detected, the crop area is determined using time-series information.> In this embodiment, using the case of a cat walking as an example, we will explain how to determine the crop region when a high-priority recognition task has not been detected. Specifically, in this embodiment, the local region identification unit 280 identifies multiple local regions from each of the images of multiple frames taken at different times. The crop region determination unit 240 sets the crop region of the current image based on the crop region of the image of the previous frame (i.e., the image at the previous time) based on the changes in the multiple local regions taken at different times. In addition, if there is no local region at the previous time, the crop region determination unit 240 determines whether or not to set the current crop region based on the previous crop region based on the changes in the local region.

[0128] Figure 7 shows a time-series image of a cat being tracked as an example in the fourth embodiment. Image 701 in Figure 7(a) is an image of the cat walking to the left in the initial frame at time t=0. In the fourth embodiment, the flowcharts in Figure 4 and Figure 9 are used. Figure 6(d) shows the pre-registration information in the fourth embodiment.

[0129] In processing image 701 at time t=0, the processing is carried out in the same order as in the third embodiment: S401, S402, and S403. In S403, the tracking target setting unit 230 sets the entire body of the cat as the tracking target area 702, as shown in Figure 7(a).

[0130] Next, in S401, image 711 of Figure 7(b) is acquired. Here, image 711 is an image of the cat at time t=1. In image 711, the cat continues to move to the left. The image input unit 220 determines in S402 that image 711 is a frame later than the initial frame, and proceeds to the crop region determination process in S404.

[0131] Figure 9 shows a detailed flow of the crop region determination process in S404 executed by the crop region determination unit 240 of the fourth embodiment.

[0132] In S901, the crop region determination unit 240 determines whether or not there is a local region detected in the previous frame that can be used to calculate the crop region. At time t=1, the multitasking unit 260 has not yet performed its processing, so the crop region determination unit 240 determines that there is no local region detected and proceeds to processing in S903.

[0133] In S903, the crop region determination unit 240 determines whether there is a detection result for the previous frame. At time t=1, the processing of the multitasking unit 260 has not yet been performed, so the crop region determination unit 240 determines that there is no detection result for the previous frame and proceeds to processing in S904.

[0134] In the S904, the crop area determination unit 240 sets the total number of pixels of the tracking target set in the tracking target setting unit 230 as the crop standard.

[0135] In S907, the crop region determination unit 240 obtains a crop ratio of 3.0, which corresponds to the whole body and is the crop standard, from the pre-registered information 604 in Figure 6(d).

[0136] In S908, the crop area determination unit 240 calculates the number of pixels in the crop area by multiplying the total number of pixels of the tracking target set as the crop criterion by the crop magnification determined in S907.

[0137] In S909, the crop region determination unit 240 determines the aspect ratio of the crop region, and in S911, it determines the position of the crop region. As a result, the crop region determination unit 240 determines the crop region 712 shown in Figure 7(b).

[0138] In S405, the cropping unit 250 performs cropping using the cropping region 712 determined in S404 and resizes the image to QVGA, generating the cropped image 713 shown in Figure 7(c).

[0139] In S406, the multitasking unit 260 performs multitasking and obtains the whole body detection result 714, the head detection result 715, and the face detection result 716.

[0140] In S407, the tracking target identification unit 270 identifies the tracking target from the detection results of the multitasking unit 260. At time t=1, the tracking target identification unit 270 identifies the whole body detection result 714 as the tracking target.

[0141] In S408, the local area identification unit 280 identifies the local area to be tracked from the detection results of the multitasking unit 260. At time t=1, the local area identification unit 280 identifies the head detection result 715 and the face detection result 716 as local areas.

[0142] In S409, it is determined whether processing of all frames has finished. At time t=1, processing of all frames has not finished, so the process returns to S401 and processes the data for time t=2.

[0143] In S401, the image input unit 220 acquires image 721 in Figure 7(d). Here, image 721 is an image of the cat at time t=2. In image 721, the cat continues to move to the left. After that, the processes from S402 to S404 are performed again.

[0144] In S901, the crop region determination unit 240 determines whether there is a local region detected in the previous frame that can be used to calculate the crop region. At time t=1, the head detection result 715 and the face detection result 716 were detected in the previous frame. In this embodiment, all information registered in the pre-registered information 604 can be set as crop criteria, so the process proceeds to S902.

[0145] In S902, the crop region determination unit 240 sets the detected local region as the crop criterion according to priority. Since the pre-registered information 604 indicates that the head has the highest priority, the crop region determination unit 240 sets the number of pixels in the head region as the crop criterion.

[0146] In S907, the crop area determination unit 240 obtains a crop multiplier of 15.0, which corresponds to the head and is the crop criterion, from the pre-registered information 604 in Figure 6(d).

[0147] In S908, the crop area determination unit 240 calculates the number of pixels in the crop area by multiplying the number of pixels in the head, which is set as the crop criterion, by the crop magnification determined in S907.

[0148] In S909, the crop region determination unit 240 determines the aspect ratio of the crop region, and in S911, it determines the position of the crop region. As a result, the crop region determination unit 240 determines the crop region 722 shown in Figure 7(d).

[0149] In S405, the cropping unit 250 performs cropping using the cropping region 722 determined in S404 and resizes the image to QVGA, generating the cropped image 727 shown in Figure 7(f).

[0150] In S406, the multitasking unit 260 performs multitasking processing and obtains only the whole body detection result 728, leaving the head and face undetected.

[0151] In S407, the tracking target identification unit 270 identifies the tracking target from the detection results of the multitasking unit 260. At time t=2, the tracking target identification unit 270 identifies the whole body detection result 728 as the tracking target.

[0152] In S408, the local area identification unit 280 identifies the local area to be tracked from the detection results of the multitasking unit 260. At time t=2, the local area identification unit 280 does not detect a local area.

[0153] In S409, it is determined whether processing of all frames has finished. At time t=2, processing of all frames has not finished, so the process returns to S401 and processing at time t=3 is performed.

[0154] In S401, the image input unit 220 acquires image 733 in Figure 7(h). Image 733 is an image of a cat at time t=3. In image 733, the cat continues to move to the left. After that, processing from S402 to S404 is performed again.

[0155] In S901, the crop region determination unit 240 determines whether or not there is a local region detected in the previous frame that can be used for calculating the crop region. At time t=2, the head and face have not been detected, so it is determined that there is no detected local region and the process moves on to S903.

[0156] In S903, the crop region determination unit 240 determines whether there is a detection result for the previous frame. At time t=2, the whole body detection result 728 has been detected, so the crop region determination unit 240 determines that there is a detection result for the previous frame and proceeds to processing in S905.

[0157] In S905, the crop region determination unit 240 calculates the size change rate from the time-series information of the detection results. The size change rate is a value calculated from the size information (in this case, the number of pixels) of the detection results in past frames, as described in the third embodiment. For example, the crop region determination unit 240 may calculate the size change rate as the change between the size of a local region at a certain time and the size of a local region immediately after a certain time.

[0158] In S906, the crop region determination unit 240 determines whether the size change rate is below a threshold. If the size change rates of whole-body detection result 714 and whole-body detection result 728 are low and the crop region determination unit 240 determines that the size change rate is below the threshold, the process proceeds to S910. On the other hand, if the crop region determination unit 240 determines that the size change rate is greater than the threshold, the process proceeds to S904.

[0159] In S910, the crop region determination unit 240 sets the crop region of the previous frame as the crop region of the current frame. In other words, if the local region of the previous time is not detected and the size change rate is below a threshold, the crop region determination unit 240 determines the crop region of the previous time as the current crop region.

[0160] In this third embodiment, for example, the local region of the previous frame that can be used to calculate the crop region is not detected in the current frame, so there is a possibility that the entire body of the tracked subject with a large "size change rate in response to posture change" will be set as the crop criterion. In this embodiment, the crop region determination unit 240 determines that there is no significant change in the posture of the tracked subject from the size change rate of the local region in the current frame of the region with a large "size change rate in response to posture change". If the crop region determination unit 240 determines that the size change rate is small and there is no significant change in the shooting state of the tracked subject, the crop region of the previous frame can be used as the crop region of the current frame.

[0161] Subsequently, in S911, the crop region determination unit 240 determines the position of the crop region. As a result, the crop region determination unit 240 determines the crop region 734 shown in Figure 7(h).

[0162] Subsequently, in S405, the cropping unit 250 generates a cropped image using the cropping region 734. After that, the processes from S406 to S409 are performed, and the processing of this embodiment is completed.

[0163] As described above, the information processing device of the fourth embodiment can perform cropping by using the cropped area of ​​the previous frame as the cropped area of ​​the current frame, by confirming that there is no significant change in the shooting state of the tracked object from the size change rate of the region where the "size change rate with respect to posture change" is large. However, the position of the cropped area does not need to use the coordinates of the previous frame and may be changed according to the movement of the subject. In addition, in this embodiment, confirmation that there is no significant change in the shooting state of the tracked object was made using the size change rate, but confirmation may also be made using the recognition accuracy of the tracked object. Here, recognition accuracy refers to the recognition rate calculated from the number of frames in which the tracked object was recognized, or the recognition score that indicates the likelihood of the recognized object.

[0164] Furthermore, in this embodiment, we determined whether cropping could be performed using the crop region of the previous frame, but we may also determine whether the size information of the local region of the previous frame can be used in the current frame. By performing the above determination, even if the local region used to calculate the crop region in the previous frame is not detected in the current frame, it becomes possible to calculate the crop region using the local region of the previous frame.

[0165] As a result, even if high-priority recognition tasks are not detected, a stable crop region can be calculated.

[0166] <Fifth Embodiment: Multitasking performed on images of different sizes> Figure 10 is a diagram illustrating the first information processing device 200 and the second information processing device 1000 according to the fifth embodiment. The image input unit 220, tracking target setting unit 230, crop area determination unit 240, crop unit 250, first multitasking unit 260, tracking target identification unit 270, local area identification unit 280, and output unit 290 in Figure 10 perform the same operations as the corresponding configurations in the first embodiment, so their explanation is omitted.

[0167] The second information processing device 1000 receives input data 210 from the image input unit 220. The second multitasking unit 1010 performs multiple recognition tasks on the image data of the input data 210. In this embodiment, the second multitasking unit 1010 processes an image that has not been cropped, but it is not limited to this, and it may perform recognition tasks on an image larger than a cropped image. For example, the second multitasking unit 1010 may perform recognition tasks on a cropped image whose size is halved, and on a cropped image whose aspect ratio is 1:1. Therefore, the local region identification unit 280 identifies a local region from the cropped image and an image larger than the cropped image. The second multitasking unit 1010 may also be provided in the first information processing device 200. In this case, the first information processing device 200 and the second information processing device 1000 are integrated into a single information processing device.

[0168] Figure 11 shows a time-series image of a cat being tracked as an example of the fifth embodiment. Figure 12 shows a flowchart of the processing in this embodiment. Figure 12(a) shows the overall flowchart. Figure 12(b) shows a detailed flow of the crop area determination process in S1204. Figure 6(a) shows the pre-registration information in this embodiment.

[0169] First, in S1201, the image input unit 220 acquires an image for one frame that has been input as input data 210. Here, the image input unit 220 acquires image 1101 in Figure 11(a). Image 1101 in Figure 11(a) is the image of the initial frame at time t=0. In image 1101, a cat is walking to the left.

[0170] In S1202, the image input unit 220 determines whether the acquired frame is the initial frame of the input data 210. If the image of the initial frame at time t=0 is acquired, the image input unit 220 determines that it is the initial frame and proceeds to S1203.

[0171] In S1203, the tracking target setting unit 230 sets the tracking target. Here, in S1203, the tracking target setting unit 230 sets the cat as the tracking target and sets the position and size of the tracking target. In this embodiment, as shown in Figure 11(a), the tracking target setting unit 230 sets the entire body of the cat as the tracking target area 1102.

[0172] In S1207, after the setting of the tracking target is completed, the second multitasking unit 1010 performs the second multitasking process. Figure 11(b) shows the result of the second multitasking process. Image 1103 in Figure 11(b) shows the image after the second multitasking process, and in this embodiment, the same image as image 1101 is used. Based on image 1103, the second multitasking unit 1010 detects the whole body detection result 1104 and the head detection result 1105.

[0173] In S1208, the tracking target identification unit 270 identifies the tracking target from the detection results of the second multitasking unit 1010. At time t=0, the tracking target identification unit 270 identifies the whole body detection result 1104 as the tracking target.

[0174] In S1209, the local region identification unit 280 identifies the local region to be tracked from the detection results of the second multitasking unit 1010. At time t=0, the local region identification unit 280 identifies the head detection result 1105 as the local region.

[0175] In S1210, it is determined whether processing of all frames has finished. At time t=0, processing of all frames has not finished, so the process returns to S1201 and processes the data for time t=1.

[0176] In S1201, the image input unit 220 acquires image 1111, shown in Figure 11(c). Here, image 1111 is an image of the cat at time t=1. In image 1111, the cat continues to move to the left.

[0177] In S1202, the image input unit 220 determines that image 1111 is a frame later than the initial frame, and proceeds to the crop area determination process in S1204.

[0178] Figure 12(b) shows a detailed flow of the crop region determination process in S1204.

[0179] In S1211, the crop region determination unit 240 determines whether or not there is a local region detected in the previous frame that can be used to calculate the crop region. At time t=1, there is a head detection result 1105 obtained from the second multitasking unit 1010, so the crop region determination unit 240 determines that there is a local region detected in the previous frame and proceeds to processing S1212. On the other hand, if there is no local region in the previous frame that can be used to calculate the crop region, such as the head detection result 1105, the crop region determination unit 240 determines that there is no local region and proceeds to S1214, where it sets the total number of pixels of the tracking target set in the tracking target setting unit 230 as the crop standard. After that, the crop region determination unit 240 proceeds to S1215, which will be described later.

[0180] In S1212, the crop region determination unit 240 integrates the results of the first multitasking process and the second multitasking process. For example, if a head was detected in each multitasking process, the crop region determination unit 240 determines the number of pixels in the head region as a local region. In this embodiment, the crop region determination unit 240 determines the number of pixels in the head region using the average of the number of pixels in the head region detected by the two processes. However, the determination of the number of pixels is not limited to this, and it may also be determined by using a weighted average of the two regions or by using either one of them. Furthermore, if there is only one detection result, the crop region determination unit 240 may use the number of pixels in the single head detection result.

[0181] In S1213, the crop region determination unit 240 sets the number of pixels of the head detection result 1105 as the crop criterion according to the priority order of the pre-registered information 601 in Figure 6(a).

[0182] In S1215, the crop area determination unit 240 obtains a crop ratio of 15.0 times, corresponding to the head, from the pre-registered information 601 in Figure 6(a).

[0183] In S1216, the crop area determination unit 240 calculates the number of pixels in the crop area by multiplying the number of pixels in the head, which is set as the crop criterion, by the crop magnification determined in S1215.

[0184] The crop region determination unit 240 determines the aspect ratio of the crop region in S1217 and the position of the crop region in S1218. As a result, the crop region determination unit 240 determines the crop region 1112 shown in Figure 11(c).

[0185] In S1205, the cropping unit 250 generates the cropped image 1116. For example, the cropping unit 250 crops the image 1111 using the cropping region 1112 determined in S1204 and resizes the image. Through the above process, the cropping unit 250 generates the cropped image 1116 shown in Figure 11(e). At time t=1, the cat's movement was rapid, so part of the cat's body and head extend beyond the cropping region.

[0186] In S1206, the first multitasking unit 260 performs the first multitasking process on the cropped image 1116 in Figure 11(e). At time t=1, the result of the first multitasking unit 260 is only the whole-body detection result 1117, as shown in Figure 11(e), because the entire body of the cat extends beyond the cropped area.

[0187] In S1207, the second multitasking unit 1010 performs the second multitasking process. Figure 11(d) shows the result of the second multitasking process. Image 1113 in Figure 11(d) shows the image after the second multitasking process, and is the result of processing using the same image as image 1111. The second multitasking unit 1010 performs the second multitasking process on image 1111 and detects the whole body detection result 1114 and the head detection result 1115 as shown in image 1113.

[0188] In S1208, the tracking target identification unit 270 identifies the tracking target from the detection results of the first multitasking unit 260 and the second multitasking unit 1010. At time t=1, the tracking target identification unit 270 identifies the whole body detection result 1114 as the tracking target, as shown in Figure 11(d).

[0189] In S1209, the local area identification unit 280 identifies the local area to be tracked from the detection results of the first multitasking unit 260 and the second multitasking unit 1010. At time t=1, the local area identification unit 280 identifies the head detection result 1115 as the local area, as shown in Figure 11(d).

[0190] In S1210, it is determined whether processing of all frames has finished. At time t=1, processing of all frames has not finished, so the process returns to S1201 and processes the data for time t=2.

[0191] In S1201, the image input unit 220 acquires image 1121 in Figure 11(f). Image 1121 is an image of a cat at time t=2. In image 1121, the cat continues to move to the left. After that, the processes of S1202 and S1204 are performed again.

[0192] In S1211, the crop region determination unit 240 determines whether or not there is a local region detected in the previous frame that can be used for calculating the crop region. At time t=2, the head detection result 1115 has been detected, so the crop region determination unit 240 determines that there is a local region and proceeds to processing in S1212.

[0193] In S1212, the crop region determination unit 240 integrates the results of the first multitasking process and the second multitasking process. Here, since there is only one detection result for the head, the crop region determination unit 240 integrates the processing results by using the result of the second multitasking unit 1010 as is for the head. For the whole body, the crop region determination unit 240 integrates the processing results by taking the average of the number of pixels of the whole body detection result 1114 and the whole body detection result 1117 as the number of pixels for the whole body.

[0194] In S1213, the crop area determination unit 240 sets the number of pixels in the head, which was set in S1212, as the cropping criterion, according to the priority order of the pre-registered information 601.

[0195] Subsequently, the crop region determination unit 240 performs the processing from S1215 to S1218, similar to the process at time t=1, to determine the crop region 1122 shown in Figure 11(f).

[0196] In S1205, the cropping unit 250 generates a cropped image using the cropping region 1122.

[0197] Subsequently, processes S1206 through S1210 are executed, and the series of processes is completed.

[0198] As described above, the fifth embodiment can make the calculation of the cropped area more stable by performing the first and second multitasking processes using the cropped image and an image larger than the cropped image. In this embodiment, even if the entire body of the cat extends beyond the cropped image, cropping can be performed using a local region with a small "size fluctuation rate in response to changes in posture" by using the detection result of the second multitasking process performed on an image larger than the cropped image. In addition, although processing is performed on an image that has not been cropped in this embodiment, processing may be performed on any image that is different from the cropped image generated by the cropping unit 250.

[0199] <Other Embodiments> In the above embodiment, a cat was used as the tracking target, but other categories such as people or motorcycles may also be used as tracking targets. When applying this embodiment to other categories, a new local area to be used as the cropping criterion may be set using the "rate of size change in response to changes in posture" as an indicator. For example, for a person, the area with a large rate of size change is the whole body, and the area with a small rate is the head. For a motorcycle, the area with a large rate of size change is the entire vehicle, and the area with a small rate is the length of the tires. Therefore, in the above embodiment, the cropping criterion was determined by priority and rate of size change, but the cropping criterion may also be determined by the category classification result of the tracking target. For example, the cropping criterion may be the head when tracking a person, and the cropping criterion may be the tires when tracking a motorcycle, and so on. A combination of category and cropping criterion may be decided in advance.

[0200] The above-described embodiment illustrates an example of determining a crop region based on a local region selected from multiple local regions, but the method of determining the crop region is not limited to this method. For example, priorities may be set for body parts such as the whole body, head, and face, and local regions may be set only for the parts with the highest priority among the detected parts, and the crop region may be determined based on those local regions.

[0201] Although the above embodiment was described assuming multitasking, multitasking is not required. In this case, the local region identification unit 280 can identify the local region by detecting a predetermined part or the like.

[0202] In the embodiments described above, an example was given in which the input data 210 is a video, but the invention is not limited to this. For example, the input data 210 may be a series of still images taken at regular time intervals, or a series of time-lapse images.

[0203] The embodiments described above may be combined as appropriate.

[0204] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0205] The disclosures herein include the following information processing devices, information processing methods, and programs. (Item 1) A means for identifying tracking targets included in an image, Local region identification means for identifying a local region within the image that includes at least a portion of the detection target included in the tracking target, A crop region determination means that determines the size information of a crop region for cutting out the tracking target from the image based on the size information of the local region, A cropping means that generates a cropped image by cutting out the image based on the cropped region, An information processing device characterized by comprising: (Item 2) The local region identification means identifies a plurality of local regions that include at least a portion of a plurality of types of detection targets included in the tracking target, The crop region determination means determines the size information of the crop region based on the size information of one of the plurality of local regions. The information processing device described in item 1, characterized by the features described herein. (Item 3) The crop region determination means determines the size information of the crop region based on the size information of the local region of the detection target selected from the plurality of types of detection targets, based on a predetermined priority order of detection targets. The information processing device described in item 2, characterized in that it is an information processing device. (Item 4) The local area identification means identifies a plurality of local areas that include at least a portion of the detection target of the tracking target, The crop region determination means determines the size information of the crop region based on the size information of the plurality of local regions. An information processing device according to any one of items 1 to 3, characterized by the features described herein. (Item 5) The crop region determination means determines the size information of the crop region based on the result of averaging the size information of the plurality of local regions. The information processing device described in item 4, characterized by the features described herein. (Item 6) The crop region determination means determines the size information of the crop region based on at least one of an upper limit and a lower limit set based on the size information of the crop region. An information processing device according to item 4 or item 5, characterized in that it is an information processing device. (Item 7) The local region identification means identifies multiple local regions from multiple images taken at different times, The crop region determination means determines the size information of the crop region based on the size information of at least one of the plurality of local regions. An information processing device according to any one of items 1 to 6, characterized by the features described herein. (Item 8) The local region identification means identifies a plurality of local regions that include at least a portion of a plurality of types of detection targets included in the tracking target, The crop region determination means determines the size information of the crop region based on the size information of the local region of the detection target selected from the plurality of types of detection targets. The information processing device described in item 7, characterized by the features described herein. (Item 9) The crop region determination means determines the size information of the crop region based on the size information of the local region of the detection target selected from the plurality of types of detection targets, based on the change in the size information of the plurality of local regions. The information processing device described in item 8, characterized by the features described herein. (Item 10) The local region identification means identifies a plurality of local regions that include at least a portion of a plurality of types of detection targets included in the tracking target, The crop region determination means determines the size information of the crop region based on the size information of the local region of the detection target selected from the plurality of types of detection targets, based on the recognition accuracy of the detection target. An information processing device according to any one of items 1 to 9, characterized in that it is an information processing device. (Item 11) The crop region determination means uses at least one of the recognition rate of the detection target and the recognition score of the detection target as the recognition accuracy. The information processing device according to item 10, characterized in that it is a processing device. (Item 12) The local region identification means identifies multiple local regions from multiple images taken at different times, The crop region determination means determines whether or not to set the crop region from the previous time as the current crop region based on the change in the size information of the plurality of local regions. An information processing device according to any one of items 1 to 11, characterized by the features described herein. (Item 13) The crop region determination means sets the crop region of the previous time as the crop region of the current image if the local region used to determine the crop region of the image at the previous time is not detected in the current image. The information processing device described in item 12, characterized by the features described herein. (Item 14) The local region identification means identifies the local region from the cropped image and an image larger than the cropped image. An information processing device according to any one of items 1 to 13, characterized by the features described herein. (Item 15) The local area identification means sets a local area based on the detection target set according to the category of the tracking target. An information processing device according to any one of items 1 to 14, characterized by the features described in item 1 to 14. (Item 16) Image input means for acquiring the input image, Tracking target setting means for setting the tracking target, A multitasking means for performing multiple recognition tasks on the target being tracked in the cropped image, Equipped with, The tracking target identification means identifies the tracking target included in the image based on the results of the plurality of recognition tasks. An information processing device according to any one of items 1 to 15, characterized by the features described herein. (Item 17) A tracking target identification step in which the object to be tracked included in the image is identified as the tracking target, A local region identification step of identifying a local region within the image that includes at least a portion of the detection target included in the tracking target, A crop region determination step, which determines the size information of a crop region for cutting out the tracking target from the image based on the size information of the local region, A cropping step of generating a cropped image by cutting out the image based on the cropped region, An information processing method comprising the following: (Item 18) A program to cause a computer to function as one of the means of an information processing device described in any one of items 1 through 16.

[0206] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0207] 200... Information processing unit, 220... Image input unit, 230... Tracking target setting unit, 240... Crop area determination unit, 250... Crop unit, 260... Multitasking unit, 270... Tracking target identification unit, 280... Local area identification unit, 1000... Second information processing unit, 1010... Second multitasking unit.

Claims

1. Tracking target identification means for identifying a subject included in the first frame as the tracking target in the first frame, Local region identification means for identifying a local region within the first frame that includes at least a portion of the detection target included in the tracking target, A crop region determination means that determines the size information of a crop region for cutting out a region including the tracking target from a second frame captured after the first frame, based on the size information of the local region, A cropping means that generates a cropped image from the second frame based on the cropped region whose size information has been determined in the cropped region determination means, Equipped with, The tracking target identification means is characterized by identifying the tracking target in the second frame from the subject included in the cropped image.

2. The local region identification means identifies a plurality of local regions that include at least a portion of a plurality of types of detection targets included in the tracking target, The crop region determination means determines the size information of the crop region based on the size information of one of the plurality of local regions. The information processing apparatus according to feature 1.

3. The crop region determination means determines the size information of the crop region based on the size information of the local region of the detection target selected from the plurality of types of detection targets, based on a predetermined priority order of detection targets. The information processing apparatus according to feature 2.

4. The local area identification means identifies a plurality of local areas that include at least a portion of the detection target of the tracking target, The crop region determination means determines the size information of the crop region based on the size information of the plurality of local regions. The information processing apparatus according to feature 1.

5. The crop region determination means determines the size information of the crop region based on the result of averaging the size information of the plurality of local regions. The information processing apparatus according to feature 4.

6. The information processing apparatus according to claim 4, wherein the crop region determination means determines at least one of the upper limit and lower limit of the size information of the crop region based on the size information of the local region.

7. The local region identification means identifies multiple local regions from multiple frames at different times, The crop region determination means determines the size information of the crop region based on the size information of at least one of the plurality of local regions. The information processing apparatus according to feature 1.

8. The local region identification means identifies a plurality of local regions that include at least a portion of a plurality of types of detection targets included in the tracking target, The crop region determination means determines the size information of the crop region based on the size information of the local region of the detection target selected from the plurality of types of detection targets. The information processing apparatus according to feature 7.

9. The crop region determination means determines the size information of the crop region based on the size information of the local region of the detection target selected from the plurality of types of detection targets, based on the change in the size information of the plurality of local regions. The information processing apparatus according to feature 8.

10. The local region identification means identifies a plurality of local regions that include at least a portion of a plurality of types of detection targets included in the tracking target, The crop region determination means determines the size information of the crop region based on the size information of the local region of the detection target selected from the plurality of types of detection targets, based on the recognition accuracy of the detection target. The information processing apparatus according to feature 1.

11. The crop region determination means uses at least one of the recognition rate of the detection target and the recognition score of the detection target as the recognition accuracy. The information processing apparatus according to feature 10.

12. The local region identification means identifies multiple local regions from multiple frames at different times, The crop region determination means determines whether or not to set the crop region from the previous time as the current crop region based on the change in the size information of the plurality of local regions. The information processing apparatus according to feature 1.

13. The crop region determination means sets the crop region of the previous time as the crop region of the current frame if the local region used to determine the crop region of the previous time frame is not detected in the current frame. The information processing apparatus according to feature 12.

14. The local area identification means sets a local area based on the detection target set according to the category of the tracking target. The information processing apparatus according to feature 1.

15. A multitasking means for performing multiple recognition tasks on the target being tracked in the cropped image, Equipped with, The tracking target identification means identifies the tracking target included in the second frame based on the results of the plurality of recognition tasks. The information processing apparatus according to feature 1.

16. A program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 15.

17. Tracking target identification means includes a tracking target identification step of identifying a subject included in the first frame as the tracking target in the first frame, Local area identification means includes a local area identification step of identifying a local area within the first frame that includes at least a portion of the detection target included in the tracking target, The crop region determination means includes a crop region determination step in which it determines the size information of a crop region for cutting out a region including the tracking target from a second frame captured after the first frame, based on the size information of the local region, The cropping means includes a cropping step of generating a cropped image from the second frame based on the cropped region whose size information was determined in the cropped region determination step, Equipped with, The tracking target identification step is characterized by identifying the tracking target in the second frame from the subject included in the cropped image.

18. A program for causing a computer to perform each step of the information processing method described in claim 17.

Images