Detection device, detection method, and storage medium

Abstract

To provide an information processing apparatus capable of determining the best scene suitable for extracting a feature amount of an object captured in a frame image.SOLUTION: An information processing apparatus for detecting an object from a plurality of frame images is provided with an input circuit 106 for obtaining the plurality of frame images, a recognition processing unit 124 for subjecting each of the obtained frame images to object recognition processing and when the object is detected, performing recognition processing of a feature amount of the detected object, a determination unit 116 for using a recognition result to determine whether or not one of the plurality of frame images is the best scene suitable for extracting the feature amount of the object, and an output unit 117 for outputting a feature amount of the object recognized in the frame image if it is determined to be the best scene.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a technique for detecting an object from an image and extracting a feature amount from the detected object. [Background technology]

[0002] Technologies are becoming increasingly used that detect objects such as people that appear in frame images obtained by taking pictures with a camera or the like, distinguish them from the background, and then detect the object's features, such as its size, shape, position, and color, from the detected objects.

[0003] For example, if an object is detected from each of two frame images taken at different times, and then the features of that object are detected, and the two features detected from each of the two frame images match, it can be assumed that the entities (e.g., people) represented by the two objects in the two frame images are the same. In this way, it is possible to detect the features of an object from each of multiple frame images taken at different times and use the detected features to track how a single entity moves over time. Tracking the same object in different scenes is called ReID.

[0004] According to Patent Document 1, an optimal face image recording device records image data captured by a camera, detects faces from the recorded image data using a predetermined method, and determines that the detected face images are the same face if they exist within a certain range within a certain time period. As long as this determination continues, it is considered to be one same face image tracking period, and sequentially selects and updates the best shot image during one same face image tracking period using a predetermined evaluation method. It collects information to be recorded in association with the best shot image during one same face image tracking period, and records the collected information. With this configuration, it is possible to always select and save the objectively most balanced frontal face image (i.e., the best shot image) from a series of consecutive images (video). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2005-227957 A Summary of the Invention [Problem to be solved by the invention]

[0006] As described above, according to Patent Document 1, it is possible to objectively select the most balanced frontal face image from a series of consecutive images, but if a frame image does not include a balanced frontal face image, for example, if a person in the frame image is facing away from the camera, the frame image will not be saved. This causes a problem in that it is not possible to extract the feature amount of a person in the frame image even if the person is wearing distinctive clothing or carrying distinctive baggage.

[0007] The present disclosure aims to solve the above problem and provide a detection device, a detection method, and a storage medium storing a computer program that can determine the best scene from multiple frame images that is suitable for extracting features of an object captured in the frame images. [Means for solving the problem]

[0008] One aspect of the present disclosure is a detection device that detects an object from a plurality of frame images, the detection device comprising: an acquisition means for acquiring a plurality of frame images; a recognition means for performing an object recognition process on each of the acquired frame images and, when an object is detected, performing a recognition process on a feature amount of the detected object; a determination means for determining whether or not one of the plurality of frame images is a best scene suitable for extracting a feature amount of the object using the recognition result by the recognition means; and an output means for outputting the feature amount of the object recognized in the frame image when the frame image is determined to be the best scene. If the one frame image is determined not to be the best scene, the output means suppresses output of the feature amount of the object recognized in the frame image. It is characterized by:

[0010] In another aspect of the present disclosure, a detection device for detecting an object from a plurality of frame images includes an acquisition means for acquiring the plurality of frame images, a recognition means for performing an object recognition process on each of the acquired frame images and, when an object is detected, performing a recognition process on a feature of the detected object, a determination means for determining whether or not one of the plurality of frame images is the best scene suitable for extracting a feature of the object using the recognition result by the recognition means, and an output means for outputting the feature of the object recognized in the frame image when it is determined to be the best scene; The determining means sets a weight using the recognition result by the recognizing means, and determines whether the one frame image is the best scene using the set weight. It is characterized by .

[0011] Here, the judgment means may judge that the frame image is the best scene if the set weight is equal to or greater than a predetermined threshold, and may judge that the frame image is not the best scene if the set weight is less than the predetermined threshold.

[0012] Here, when two objects are detected by the recognition means, the judgment means may judge whether or not at least a portion of the first object is occluded by the second object, and set the weight according to the judgment result indicating whether or not it is occluded.

[0013] Here, the determination means may set the weight using a first weight value when it determines that at least a portion of the first object is occluded, and may set the weight using a second weight value that is greater than the first weight value when it determines that the first object is not occluded.

[0014] Here, the determination means may determine whether or not there is an overlapping area between a rectangular area surrounding the first object and a rectangular area surrounding the second object, and if it determines that an overlapping area exists, it may determine that a part of the first object is occluded in the overlapping area.

[0015] Here, the determination means may determine whether the distance between the rectangular area surrounding the first object and the rectangular area surrounding the second object is within a predetermined value, and if it determines that the distance is within the predetermined value, it may determine that a portion of the first object is occluded in the overlapping area.

[0016] Here, the method may further include a storage means for storing skeleton definition information that defines the shape of the skeleton of a specific type of object, and when an object is detected, the recognition means uses the skeleton definition information to detect skeleton information indicating the skeleton of the object from each detected object, and the determination means refers to the skeleton definition information to determine whether there is a gap or a deformation greater than a predetermined level in the skeleton indicated by the skeleton information detected from the first object, and if it determines that there is a gap or a deformation greater than a predetermined level, it may determine that at least a part of the first object is occluded.

[0017] Here, the recognition means may perform object recognition processing on each frame image to detect multiple partial objects that make up the object, and detect the direction in which the main surface of the partial object is facing and the attributes of the object as features of the object, and the judgment means may set the weight according to the relationship between the detected direction and the attributes.

[0018] Here, the judgment means may determine whether the detected direction is the front direction of the frame image, determine whether there is a correlation between the detected partial object and the attributes of the detected object, and if it determines that the detected direction is not the front direction or if it determines that there is no correlation between the partial object and the attributes of the detected object, set the weight using a first weight value, and if it determines that the detected direction is the front direction and that there is a correlation between the partial object and the attributes of the detected object, set the weight using a second weight value that is greater than the first weight value.

[0019] Here, the system may further include a storage means for storing skeleton definition information that defines the shape of the skeleton of a specific type of object, and when an object is detected, the recognition means may use the skeleton definition information to detect skeleton information indicating the skeleton of each detected object, and use the detected skeleton information to detect the direction in which the main surface of the partial object is facing.

[0020] Here, the recognition means may detect the size of the object as a feature of the object, and the determination means may set the weight in accordance with a relationship between the detected size and a size threshold value.

[0021] Here, the judgment means may compare the detected size with the size threshold, and if the size is less than the size threshold, set the weight using a first weight value, and if the size is equal to or greater than the size threshold, set the weight using a second weight value greater than the first weight value.

[0022] Here, the recognition means may further include a storage means for storing skeleton definition information that defines the shape of the skeleton of a specific type of object, and when an object is detected, the recognition means may use the skeleton definition information to detect skeleton information indicating the skeleton of the object from each detected object, and may use the detected skeleton information to detect the size of the object.

[0023] Here, the plurality of frame images may be generated in chronological order, the recognition means may detect a change in the position or size of the object in chronological order as a feature of the object, and the determination means may set the weight in accordance with the relationship between the detected amount of change in position or size of the object and an acceptable range.

[0024] Here, the judgment means may compare the detected amount of change in position or size of the object with the allowable range, and if the amount of change in position or size of the object is outside the allowable range, set the weight using a first weight value, and if the amount of change in position or size of the object is within the allowable range, set the weight using a second weight value greater than the first weight value.

[0025] Here, the recognition means may further include a storage means for storing skeleton definition information that defines the shape of the skeleton of a specific type of object, and when an object is detected, the recognition means may use the skeleton definition information to detect skeleton information indicating the skeleton of the object from each detected object, and use the detected skeleton information to detect the amount of change in position or size of the object.

[0026] Here, the recognition means may detect a position occupied by the object within the frame image as a feature of the object, and the determination means may set the weight in accordance with the detected position and position range.

[0027] Here, the determination means may compare the position of the detected object with the position range, and if the position of the object is within the position range, set the weight using a first weight value, and if the position of the object is outside the position range, set the weight using a second weight value that is greater than the first weight value.

[0028] Here, the recognition means may further include a storage means for storing skeleton definition information that defines the shape of the skeleton of a specific type of object, and when an object is detected, the recognition means may use the skeleton definition information to detect skeleton information indicating the skeleton of the object from each detected object, and use the detected skeleton information to detect the position occupied by the object within the frame image.

[0029] Here, the plurality of frame images may be generated in chronological order, and each of the plurality of time-series sections may include one or more frame images, and the determination means may determine the frame image that is the best scene for each section.

[0030] Here, the recognition means may detect features of an object from each frame image as the recognition result, and different features may be detected for each section, and the judgment means may set a weight for each section using the detected features, and judge whether or not it is the best scene using the set weight.

[0031] Here, the determining means may determine that the frame image in the section corresponding to the largest weight among the weights in each of the plurality of sections is the best scene.

[0032] Here, the recognition means may detect object attributes from each frame image as the recognition result, and the attributes may be detected for each section, and the judgment means may add weights for each attribute in each of the multiple sections, and judge the frame image in the section in which the attribute with the largest added value is detected to be the best scene.

[0033] In yet another aspect of the present disclosure, a detection device for detecting an object from a plurality of frame images includes an acquisition means for acquiring the plurality of frame images, a recognition means for performing an object recognition process on each of the acquired frame images and, when an object is detected, performing a recognition process on a feature of the detected object, a determination means for determining whether or not one of the plurality of frame images is the best scene suitable for extracting a feature of the object using the recognition result by the recognition means, and an output means for outputting the feature of the object recognized in the frame image when it is determined to be the best scene, The recognition means detects the shape of the object as a feature of the object, and the determination means determines whether the scene is the best scene depending on whether the detected shape is appropriate. It is characterized by .

[0034] Here, the recognition means may recognize an object from an image and recognize a feature amount from the object using artificial intelligence.

[0035] Another aspect of the present disclosure is a detection method used in a detection device that detects an object from a plurality of frame images, the method including: an acquisition step of acquiring a plurality of frame images; a recognition step of performing object recognition processing on each of the acquired frame images and, when an object is detected, performing recognition processing on a feature amount of the detected object; a determination step of using the recognition result from the recognition step to determine whether or not one of the plurality of frame images is a best scene suitable for extracting a feature amount of the object; and an output step of outputting the feature amount of the object recognized in the frame image when it is determined to be the best scene. and when it is determined that the one frame image is not the best scene, the output step suppresses output of the feature amount of the object recognized in the frame image. It is characterized by: Another aspect of the present disclosure is a detection method used in a detection device that detects objects from a plurality of frame images, the method including: an acquisition step of acquiring a plurality of frame images; a recognition step of performing object recognition processing on each of the acquired frame images and, when an object is detected, performing recognition processing on the features of the detected object; a judgment step of using the recognition result from the recognition step to determine whether or not one of the plurality of frame images is the best scene suitable for extracting features of the object; and an output step of, if it is determined to be the best scene, outputting the features of the object recognized in the frame image, wherein the judgment step sets a weight using the recognition result from the recognition step and determines whether or not the one frame image is the best scene using the set weight. Yet another aspect of the present disclosure is a detection method used in a detection device that detects objects from a plurality of frame images, the method including: an acquisition step of acquiring a plurality of frame images; a recognition step of performing object recognition processing on each of the acquired frame images and, when an object is detected, performing recognition processing on features of the detected object; a judgment step of using the recognition result from the recognition step to determine whether one of the plurality of frame images is the best scene suitable for extracting features of the object; and an output step of, if it is determined to be the best scene, outputting the features of the object recognized in that frame image; wherein the recognition step detects the shape of the object as the feature of the object, and the judgment step determines whether or not the scene is the best depending on whether the detected shape is appropriate.

[0036] Another aspect of the present disclosure is a computer-readable storage medium storing a computer program used in a detection device that detects objects from a plurality of frame images, the computer program being for causing the detection device, which is a computer, to execute an acquisition step of acquiring a plurality of frame images, a recognition step of performing object recognition processing on each of the acquired frame images and, when an object is detected, performing recognition processing on a feature amount of the detected object, a determination step of using the recognition result from the recognition step to determine whether or not one of the plurality of frame images is the best scene suitable for extracting a feature amount of the object, and, when it is determined to be the best scene, an output step of outputting the feature amount of the object recognized in that frame image. and when it is determined that the one frame image is not the best scene, the output step suppresses output of the feature amount of the object recognized in the frame image. It is characterized by: Another aspect of the present disclosure is a computer-readable storage medium storing a computer program used in a detection device that detects objects from a plurality of frame images, the storage medium storing the computer program causing the detection device, which is a computer, to execute an acquisition step of acquiring a plurality of frame images, a recognition step of performing object recognition processing on each of the acquired frame images and, when an object is detected, performing recognition processing on the features of the detected object, a judgment step of using the recognition result from the recognition step to determine whether or not one of the plurality of frame images is the best scene suitable for extracting the features of the object, and an output step of outputting the features of the object recognized in the frame image if it is determined to be the best scene, wherein the judgment step sets a weight using the recognition result from the recognition step and determines whether or not the one frame image is the best scene using the set weight. Yet another aspect of the present disclosure is a computer-readable storage medium storing a computer program used in a detection device that detects objects from a plurality of frame images, the computer program storing therein causing the detection device, which is a computer, to execute an acquisition step of acquiring a plurality of frame images, a recognition step of performing object recognition processing on each of the acquired frame images and, when an object is detected, performing recognition processing on the features of the detected object, a judgment step of using the recognition result from the recognition step to determine whether one of the plurality of frame images is the best scene suitable for extracting features of the object, and an output step of outputting the features of the object recognized in the frame image if it is determined to be the best scene, wherein the recognition step detects the shape of the object as the feature of the object, and the judgment step determines whether the scene is the best depending on whether the detected shape is appropriate. [Effects of the Invention]

[0037] According to the aspects of the present disclosure, it is possible to determine the best scene from multiple frame images that is suitable for extracting features of an object captured in the frame images, thereby achieving the excellent effect of improving the accuracy of the features extracted from the object. [Brief explanation of the drawings]

[0038] [Figure 1] 1 shows the configuration of a monitoring system 1 as a first embodiment. [Figure 2] In the frame image 200, a part of a person object 203 is shown to be occluded by a person object 202. [Figure 3] 1 is a block diagram showing a configuration of an information processing device 10. FIG. [Figure 4] 2 shows the data structure of a tracking table 261. [Figure 5] FIG. 2 is a block diagram showing the configuration of a neural network 50. [Figure 6] 1A is a schematic diagram showing one neuron U of the neural network 50. FIG. 1B shows the data structure of a neuron setting table 60 set in the neural network 50. FIG. [Figure 7] 1A is a diagram schematically illustrating a data propagation model during pre-learning (training) in a neural network 50. FIG. 1B is a diagram schematically illustrating a data propagation model during practical inference in a neural network 50. [Figure 8] (a) shows a person object 221 and a rectangular frame 222 surrounding the person object 221. (b) shows a person object 231, a rectangular frame 235 surrounding the person object 231, and another object 232 that shields the person object 231. [Figure 9] Two human objects are shown moving within the frame images over time. [Figure 10](a) Shows a person object 301 and a person object 302 that occludes the person object 301. (b) Shows a rectangular overlapping area 305 where a rectangular frame 303 and a rectangular frame 304 overlap. (c) Shows a rectangular frame 304 that exists within a certain range from the center 306 of the rectangular frame 303. [Figure 11] 4 is a flowchart showing the operation of the information processing device 10. [Figure 12] (a) Shows a person object 331 and its multiple joint points 332 to 346. (b) Shows a person object 331a and its multiple joint points 332a to 336a, ..., and another object 348a that shields the person object 331a. (c) Shows a person object 331b and its multiple joint points 332b to 346b, and another object 348b that shields the person object 331b. (d) Shows a person object 331c and its multiple joint points 332c to 346c, and another object 348c that shields the person object 331c. [Figure 13] (a) Example 1 of a single person object moving within a frame image over time. (b) Example 2 of a single person object moving within a frame image over time. [Figure 14] (a) Example (3) of a single person object moving within a frame image over time. (b) Example (4) of a single person object moving within a frame image over time. [Figure 15] (a) Example (5) of a single person object moving within a frame image over time. (b) Example (6) of a single person object moving within a frame image over time. [Figure 16] (a) shows a person object 431a facing sideways relative to the plane of the frame image, (b) shows a person object 432a facing forward relative to the plane of the frame image, and (c) shows an example of the angle of the detected direction. [Figure 17]In the frame image 435, a part of the person object 436b is shown to be occluded by the person object 440. [Figure 18] 5 shows the data structure of an evaluation table 501. [Figure 19] 5 shows the data structure of a weight table 511. [Figure 20] 5 shows the data structure of an evaluation table 531. DETAILED DESCRIPTION OF THE INVENTION

[0039] 1 Example 1 1.1 Surveillance System 1 A monitoring system 1 according to a first embodiment will be described with reference to FIG.

[0040] The monitoring system 1 constitutes a part of a security control system, and is composed of cameras 5a, 5b, and 5c and an information processing device 10.

[0041] The cameras 5a, 5b, and 5c are fixed at predetermined positions and installed facing predetermined directions, and are connected to the information processing device 10 via cables, respectively.

[0042] Cameras 5a, 5b, and 5c (acquisition means) each capture an image of a person or the like passing through passage 6 to generate a frame image. In this way, cameras 5a, 5b, and 5c acquire frame images. Furthermore, cameras 5a, 5b, and 5c continuously acquire a plurality of frame images (video) over time. Each generated frame image is marked with the time (year, month, date, hour, and minute) at which the frame image was generated. Cameras 5a, 5b, and 5c transmit the generated frame images to information processing device 10. Information processing device 10 receives the frame images from cameras 5a, 5b, and 5c.

[0043] The information processing device 10 (detection device) tracks the behavior of a single person by associating multiple person objects representing the same person among multiple person objects captured in multiple frame images acquired at different times. Specifically, the information processing device 10 uses a neural network to detect multiple person objects from the multiple frame images and recognizes and extracts attributes or features, such as the person's gender, clothing, and age, from each of the detected multiple person objects. The information processing device 10 determines whether the attributes or features extracted from a first person object detected from a first frame image match the attributes or features extracted from a second person object detected from a second frame image. If they match, the first person object and the second person object are considered to represent the same person, and the information processing device 10 has successfully tracked the person's behavior.

[0044] When at least a part of a person object is occluded by another person object or the like in a frame image, the information processing device 10 excludes the attributes or features of the object extracted from the occluded part and does not use them in determining whether the attributes or features match. In other words, the attributes or features are not used when tracking the behavior of a single person.

[0045] 1, images 4a, 4b, and 4c represent frame images generated by cameras 5a, 5b, and 5c, respectively. Images 4a, 4b, and 4c are not shown as being projected onto the wall surface of passage 6. The same applies to other frame images in this embodiment.

[0046] 1.2 Occlusion of human objects An example of a situation in which a person object is occluded by another person object or the like will be described with reference to FIG.

[0047] Here, other person objects etc. include other person objects, plant objects, furniture objects, signboard objects, building objects, vehicle objects etc. In other words, person objects may be occluded not only by other person objects but also by plant objects, furniture objects, signboard objects, building objects, vehicle objects etc.

[0048] The frame image 200 shown in FIG. 2 is a frame image generated by the camera 5a, 5b or 5c at a certain point in time.

[0049] In the frame image 200, person objects 201, 202, and 203 are photographed.

[0050] The frame image 200 shows a path 201a along which a person object 201 is to move, a path 202a along which a person object 202 is to move, and paths 203a, 203b, and 203c along which a person object 203 is to move.

[0051] It should be noted that the route 201a, route 202a, routes 203a, 203b, and 203c are not all visible in the frame image 200.

[0052] According to the frame image 200, the person object 201 and the person object 202 are moving in the same direction with a time gap between them. Therefore, the person object 201 does not occlude the person object 202. Furthermore, the person object 201 and the person object 203 are moving in opposite directions as if they are passing each other, but since the person object 201 and the person object 203 are separated from each other, the person object 201 does not occlude the person object 203.

[0053] On the other hand, the person object 202 and the person object 203 are moving in opposite directions so as to pass each other, and on the path 203b, the person object 202 and the person object 203 are close to each other, so a situation occurs in which the person object 202 shields the person object 203.

[0054] An xy coordinate system is defined in the frame image shown in Fig. 2. As shown in Fig. 2, the horizontal direction of the frame image 200 is the x-axis, and the vertical direction is the y-axis.

[0055] 1.3 Configuration of the information processing device 10 3, the information processing device 10 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random access memory) 103, an input / output circuit 104, an input circuit 106, and a network communication circuit 108, all of which are connected to a bus 107, and a GPU (Graphics Processing Unit) 121, a ROM 122, a RAM 123, and a memory circuit 125, all of which are connected to a bus 109. The bus 107 and the bus 109 are connected to each other. The memory circuit 105 is connected to the input / output circuit 104.

[0056] The RAM 103 is made up of semiconductor memory and provides a work area when the CPU 101 executes a program.

[0057] The ROM 102 is composed of a non-volatile semiconductor memory. The ROM 102 stores a control program for executing the functions of the information processing device 10. The control program may be stored in the storage circuit 105.

[0058] The CPU 101 operates according to a control program stored in the ROM 102 .

[0059] The network communication circuit 108 receives data from an external terminal device via the network, and also transmits data to an external terminal device via the network.

[0060] The input / output circuit 104 writes data to the memory circuit 105 or reads data from the memory circuit 105 .

[0061] The memory circuit 105 (memory means) is composed of a non-volatile semiconductor memory. The memory circuit 105 may be composed of a hard disk. The memory circuit 105 has an area for storing frame images received from the cameras 5a, 5b, and 5c. The memory circuit 105 also has an area for storing a tracking table 261.

[0062] The tracking table 261 has an area containing multiple pieces of tracking information 262 (tracking information), as shown in an example data structure in Figure 4. Each piece of tracking information 262 includes time 263, center coordinate x (264), center coordinate y (265), upper body attribute 266, lower body attribute 267, rectangular frame size 268, and ID (269, object identifier). The rectangular frame size 268 consists of an x ​​length 268a and a y length 268b.

[0063] Time 263 is the time when the frame image was generated. The frame image is assigned the time when the image was generated as ancillary information, and the time assigned to the image is written as time 263 in tracking information 262. In the example shown in FIG. 4, for simplicity, times are indicated as "0," "1," "2," etc. Of course, the year, month, day, hour, and minute may also be written as the time.

[0064] The center coordinate x (264) and center coordinate y (265) are the x and y coordinate values ​​that indicate the position of the center of the rectangular frame that surrounds the object.

[0065] The upper body attribute 266 and the lower body attribute 267 are attributes extracted from a person object. The upper body attribute 266 is, for example, the color of clothing on the upper half of the person object, and the lower body attribute 267 is, for example, the color of clothing on the lower half of the person object.

[0066] The rectangular frame size 268 is made up of an x ​​length 268a, which is the length in the x direction of the rectangular frame of the area surrounding the object, and a y length 268b, which is the length in the y direction of the rectangular frame.

[0067] The ID (269) is an identifier for identifying an object.

[0068] In the tracking table 261, the object is assumed to be a person object, for example, and therefore the type of object is omitted in the tracking table 261.

[0069] Furthermore, the attributes or feature amounts of objects included in the tracking table 261 are not limited to upper body attributes and lower body attributes.

[0070] For example, if the type of object is a person, tracking table 261 may include, as object attributes, the gender of the person, the color of the clothing, the classification of the clothing design, etc. Furthermore, for example, if the type of object is a person, tracking table 261 may include, as object feature amounts, the height, age, etc. of the person.

[0071] 3, the input circuit 106 is connected to the cameras 5a, 5b, and 5c by cables. The input circuit 106 receives frame images from each of the cameras 5a, 5b, and 5c, and controls the input / output circuit 104 to write the received frame images into the memory circuit 105 via the input / output circuit 104.

[0072] The RAM 123 is made up of non-volatile semiconductor memory and provides a work area when the GPU 121 executes a program.

[0073] The ROM 122 stores a control program, which is a computer program for executing processing in a recognition processing unit 124 (described later).

[0074] The GPU 121 operates according to a control program stored in the ROM 122 .

[0075] The GPU 121 operates according to a control program stored in the ROM 122 using the RAM 123 as a work area, and the GPU 121, the ROM 122, and the RAM 123 constitute a recognition processing unit .

[0076] The recognition processing unit 124 (recognition means) incorporates a neural network 50, which will be described later.

[0077] The neural network 50 incorporated in the recognition processing unit 124 performs its functions by the GPU 121 operating in accordance with a control program stored in the ROM 122 .

[0078] The recognition processing unit 124 performs object recognition processing on the acquired frame image using a neural network 50 (artificial intelligence) and extracts the position of the area occupied by the object. When an object is detected, the recognition processing unit 124 performs recognition processing on the attributes or feature amounts of the detected object and detects the attributes or feature amounts of the object.

[0079] 1.4 Neural Networks 50 The neural network 50 will be described with reference to FIG.

[0080] (1) Structure of Neural Network 50 As shown in this figure, the neural network 50 (artificial intelligence) is a hierarchical neural network having an input layer 50a, a feature extraction layer 50b, and a recognition layer 50c.

[0081] Here, a neural network is an information processing system that mimics the human neural network. In the neural network 50, an engineered neuron model, which corresponds to a nerve cell, is called a neuron U. The input layer 50a, the feature extraction layer 50b, and the recognition layer 50c each have multiple neurons U.

[0082] The input layer 50a typically consists of one layer. Each neuron U in the input layer 50a receives, for example, the pixel values ​​of each pixel constituting an image. The received image values ​​are output directly from each neuron U in the input layer 50a to the feature extraction layer 50b.

[0083] The feature extraction layer 50b extracts features from the data received from the input layer 50a (all pixel values ​​constituting an image) and outputs them to the recognition layer 50c. This feature extraction layer 50b extracts areas occupied by objects (the position and size of the center of a rectangular frame surrounding the object) from the received image through calculations in each neuron U. Here, the objects include a person's entire body, a person's head, a person's face, a person's upper body, a person's lower body, a person's left arm, a person's right arm, a person's left leg, a person's right leg, plants, furniture, a vehicle, etc., and the areas occupied by objects such as a person's entire body, a person's head, a person's face, a person's upper body, a person's lower body, a person's left arm, a person's right arm, a person's left leg, a person's right leg, plants, furniture, a vehicle, etc. are extracted.

[0084] An object of a person may be called a person object. An object of a person's entire body may be called a whole-body object. Furthermore, objects such as a person's head, face, upper body, lower body, left arm, right arm, left leg, and right leg may be called partial objects. Furthermore, for objects other than people, in order to distinguish between the whole and the parts, a whole object may be called a whole object, and a partial object may be called a partial object.

[0085] The recognition layer 50c performs classification using the features extracted by the feature extraction layer 50b. The recognition layer 50c, for example, identifies the type, attribute, or feature of an object from the region of the object extracted by the feature extraction layer 50b through calculations in each neuron U. Examples of object types include a person's entire body, a person's head, a person's face, a person's upper body, a person's lower body, a person's left arm, a person's right arm, a person's left leg, a person's right leg, plants, furniture, a vehicle, etc. If the object type is a person, examples of object attributes include the person's gender, clothing color, clothing design, etc. If the object type is a person, examples of object features include the person's height, age, etc.

[0086] The recognition layer 50c also recognizes and outputs object attributes or features such as object orientation, object size, object position, change in object position, change in object size, etc. The object orientation, object size, object position, change in object position, and change in object size will be described later.

[0087] Neuron U is usually a multi-input, single-output element, as shown in Figure 6(a). Signals propagate in only one direction, and the input signal xi (i = 1, 2, ..., n) is multiplied by a certain neuron weight (SUwi) before being input to neuron U. This neuron weight represents the strength of the connection between neurons U arranged hierarchically. Neuron weights can be changed through learning. Neuron U outputs a value X, which is the sum of each input value (SUwi × xi) multiplied by the neuron weight SUwi minus a neuron threshold θU, after being transformed by a response function f(X). In other words, the output value y of neuron U is expressed by the following formula:

[0088] y=f(X) where: X=Σ(SUwi×xi)-θU As the response function, for example, a sigmoid function can be used.

[0089] Each neuron U in the input layer 50a does not usually have a sigmoid characteristic or a neuron threshold. Therefore, the input value is directly reflected in the output. On the other hand, each neuron U in the final layer (output layer) of the recognition layer 50c outputs the classification result of the recognition layer 50c.

[0090] The learning algorithm for the neural network 50 is, for example, backpropagation, which uses the steepest descent method to sequentially change the neuron weights of the recognition layer 50c and the neuron weights of the feature extraction layer 50b so that the squared error between the value (data) indicating the correct answer and the output value (data) from the recognition layer 50c is minimized.

[0091] (2) Training process The training process in the neural network 50 will now be described.

[0092] The training step is a step of pre-learning the neural network 50. In the training step, pre-learning of the neural network 50 is performed using image data with correct answers (supervised and annotated) that has been obtained in advance.

[0093] Figure 7(a) shows a schematic diagram of the data propagation model during pre-learning.

[0094] Image data is input to the input layer 50a of the neural network 50 for each image, and is output from the input layer 50a to the feature extraction layer 50b. Each neuron U in the feature extraction layer 50b performs a computation with a neuron weight on the input data. Through this computation, the feature extraction layer 50b extracts features (e.g., the area occupied by an object) from the input data, and outputs data indicating the extracted features to the recognition layer 50c (step S51).

[0095] Each neuron U in the recognition layer 50c performs a computation with a neuron weight on the input data (step S52). This allows classification based on the above features (e.g., classification of object attributes). Data indicating the classification result is output from the recognition layer 50c.

[0096] The output value (data) of the recognition layer 50c is compared with a value indicating the correct answer, and the error (loss) between them is calculated (step S53). To reduce this error, the neuron weights of the recognition layer 50c and the neuron weights of the feature extraction layer 50b are sequentially changed (backpropagation) (step S54). This allows the recognition layer 50c and the feature extraction layer 50b to train.

[0097] The above training process results in the following neuron configuration table 60:

[0098] The training process described here is performed in the training device, not in the information processing device 10. The neuron setting table 60 generated in the training device is transmitted to the information processing device 10, and the information processing device 10 receives the neuron setting table 60 and writes the received neuron setting table 60 into the memory circuitry 125 (FIG. 3).

[0099] (3) Neuron setting table 60 As a result of the above training process, the neuron setting table 60 shown in Fig. 6(b) is generated. As described above, the neuron setting table 60 is stored in the memory circuit 125 shown in Fig. 3.

[0100] 6(b), the neuron setting table 60 is made up of a plurality of pieces of neuron information 61. Each piece of neuron information 61 corresponds to each neuron U described above.

[0101] Each neuron information 61 includes a neuron number 62 , a neuron weight 63 , and a neuron threshold 64 .

[0102] The neuron number 62 is a number that identifies each neuron U.

[0103] The neuron weights 63 and neuron thresholds 64 are the neuron weights and neuron thresholds described above, respectively.

[0104] (4) On-site recognition process The actual recognition process in the neural network 50 will now be described.

[0105] FIG. 7(b) shows a data propagation model for when the neural network 50 learned by the above-described training process is used to perform actual recognition (e.g., identifying the area occupied by an object and recognizing the object's attributes) using data obtained in the field as input.

[0106] In the on-the-spot recognition process in the neural network 50, feature extraction and recognition are performed using the trained feature extraction layer 50b and the trained recognition layer 50c (step S55).

[0107] 1.5 Main control unit 111 3, the main control unit 111 is made up of a CPU 101, a ROM 102, and a RAM 103. The main control unit 111 performs its functions by the CPU 101 operating in accordance with a control program stored in the ROM 102.

[0108] The main control unit 111 is functionally composed of an overall control unit 112 , a recognition result acquisition unit 113 , an object tracking unit 115 , a determination unit 116 , and an output unit 117 .

[0109] (1) General control unit 112 The CPU 101 operates in accordance with the control program, and the overall control unit 112 comprehensively controls the input / output circuit 104, the input circuit 106, the network communication circuit 108, etc. The overall control unit 112 also comprehensively controls the recognition result acquisition unit 113, the object tracking unit 115, the determination unit 116, and the output unit 117.

[0110] The overall control unit 112 reads all frame images from the memory circuit 105 in the order of their generation, outputs each read frame image to the recognition processing unit 124, and instructs the recognition processing unit 124 to detect human objects from the images and detect their attributes, their features, or their attributes and features.

[0111] 9 shows, as an example, frame images 250a, 250b, ..., 250e that have been read out in the order of their generation time. Frame image 250a includes two person objects 251a and 252a. The other frame images 250b, ..., 250e also include two person objects in a similar manner. The two person objects included in frame images 250a, 250b, ..., 250e move in the same direction as time passes.

[0112] Here, person objects 251a, 251b, ..., 251e included in frame images 250a, 250b, ..., 250e represent the same first person. The original upper body attribute of the first person is "green" and the lower body attribute is also "green". Note that the upper body attribute being "green" indicates, for example, that the first person is wearing a green jacket, and the lower body attribute being "green" indicates, for example, that the first person is wearing green pants or a skirt.

[0113] Furthermore, person objects 252a, 252b, ..., 252e included in frame images 250a, 250b, ..., 250e represent the same second person, different from the first person. The original upper body attribute of the second person is "purple," and the original lower body attribute is also "purple." Note that the upper body attribute being "purple" indicates, for example, that the second person is wearing a purple jacket, and the lower body attribute being "purple" indicates, for example, that the second person is wearing green pants or a skirt.

[0114] In this way, the color of the jacket and the color of the pants or skirt are called attributes from the viewpoint that they belong to green (or purple).

[0115] It should be noted that frame images 250a, 250b, ..., 250e respectively show paths 253a, 253b, 254 along which the person object is about to move. Note that the paths 253a, 253b, 254 are not actually reflected in the frame images, as explained in FIG.

[0116] (2) Recognition result acquisition unit 113 When an object is detected in the recognition processing unit 124 for an image output from the overall control unit 112 to the recognition processing unit 124, the recognition result acquisition unit 113 receives from the recognition processing unit 124 the area occupied by the object, the position and size of the center of the rectangular frame surrounding the area, the type of object, the attribute of the object, and the feature amount or the attribute and the feature amount.

[0117] In the following, for the sake of simplicity, the type of object is assumed to be a person, and the object is assumed to be a person object (including the whole body of the person, the head of the person, the face of the person, the upper body of the person, the lower body of the person, the left arm of the person, the right arm of the person, the left leg of the person, the right leg of the person, etc.). As described above, examples of object attributes, when the type of object is a person, include the gender of the person, the color of the clothing, and the classification of the clothing design. Examples of object features, when the type of object is a person, include the height, age, etc. of the person.

[0118] 8(a) shows, as an example, an area occupied by a person object 221 and a rectangular frame 222 surrounding the area, received from the recognition processing unit 124. The center position 223 of the rectangular frame 222 is also shown.

[0119] 8(b) shows, as an example, an area occupied by the person object 231 and a rectangular frame 235 surrounding the area, received from the recognition processing unit 124, when the person object 231 is occluded by another object 232. Also shown is a center position 234 of the rectangular frame 235. Here, one arm portion 231a and one foot portion 231b of the person object 231 are occluded by the other object 232. In such a case, the recognition processing unit 124 imagines the existence of the occluded arm portion 231a and foot portion 231b of the person object 231, and recognizes the rectangular frame 235 by imagining sides 235a and 235b of the rectangular frame that should surround the person object 231 from the unoccluded portion of the person object 231, the occluded arm portion 231a, and the occluded foot portion 231b.

[0120] When the presence of multiple person objects is recognized in a frame image, the recognition result acquisition unit 113 receives from the recognition processing unit 124, for each of the multiple person objects, the area occupied by the person object, the center position and size of the rectangular frame surrounding the area, the type of object (indicating a person), and the attributes, feature amounts, or attributes and feature amounts of the person object.

[0121] As an example, Figure 9 shows feature information 255a for one person object 251a out of two person objects included in frame image 250a, which includes the time when frame image 205a was generated, the position of the center of the rectangular frame (center coordinate x, center coordinate y), and attributes (upper body attribute, lower body attribute).

[0122] Similarly, Figure 9 also shows, for each of the frame images 250b, ..., 250e, feature information 255b, ..., 255e for each person object 251b, ..., 251e, consisting of the time the frame image was generated, the position of the center of the rectangular frame (center coordinate x, center coordinate y), and attributes (upper body attribute, lower body attribute).

[0123] 9, in frame image 250a, person object 251a and person object 252a do not overlap. That is, person object 251a is not occluded by person object 252a. Furthermore, person object 252a is not occluded by person object 251a. The upper body attribute of person object 251a is "green", and the lower body attribute of person object 251a is also "green".

[0124] Furthermore, in frame image 250b, similar to frame image 250a, person object 251b and person object 252b do not overlap. That is, person object 251b is not occluded by person object 252b. Furthermore, person object 252b is not occluded by person object 251b. The upper body attribute of person object 251b is "green", and the lower body attribute of person object 251b is also "green".

[0125] On the other hand, in the frame image 250c, the lower body part of the person object 251c is occluded by the person object 252c. Therefore, although the lower body attribute of the person object 251c is originally "green", "purple", which is the upper body attribute of the person object 252c, is mistakenly recognized as the lower body attribute of the person object 251c, and thus "purple" is displayed.

[0126] Also, in frame image 250d, the lower body part of person object 251d is occluded by person object 252d. For this reason, although the lower body attribute of person object 251d is originally "green", "purple", which is the upper body attribute of person object 252d, is mistakenly recognized as the lower body attribute of person object 251d, resulting in "purple".

[0127] The frame image 250e is similar to the frame image 250d.

[0128] In this way, when a part of a person object is occluded by another person object, the attributes of the person object on the occluding side are erroneously recognized as the attributes of part of the person object on the occluded side.

[0129] Therefore, for the person objects 251a, . . . , 251e, if the lower body attribute of the feature information 255a, . . . , 255e is used and, for example, "purple" is adopted by majority vote, an incorrect result will be obtained.

[0130] The recognition result acquisition unit (writing means) 113 writes, for each object, the time 263 when the frame image was generated, the center coordinate x (264) and center coordinate y (265) which are the center position of the rectangular frame surrounding the object, upper body attribute 266, lower body attribute 267 which are the attributes of the object, and the x length (268a) and y length (268b) of the rectangular frame size 268 as tracking information into the tracking table 261.

[0131] The recognition result acquisition unit 113 may write tracking information, which includes the position of the area occupied by the object and the attribute or feature amount of the object, into the tracking table 261 for each object and for each time when an image is acquired.

[0132] Here, since the ID (269) for identifying the object has not yet been determined, the recognition result acquisition unit 113 does not write the ID (269) for identifying the object into the tracking table 261 as tracking information.

[0133] (3) Object Tracking Unit 115 The object tracking unit 115 assigns the same ID (object identifier) ​​to tracking information indicating the same object in the tracking table 261 as follows.

[0134] The object tracking unit 115 reads all the tracking information from the tracking table 261 .

[0135] The object tracking unit 115 uses the pair of center coordinates x and y contained in each of all the read tracking information to virtually draw a path of successive pairs of center coordinates x and y.

[0136] Here, the object tracking unit 115 determines the continuity as follows.

[0137] The object tracking unit 115 selects, from the read tracking information, first tracking information at a first time and second tracking information at a second time following the first time. Here, the first time and the second time are adjacent to each other. Next, the object tracking unit 115 calculates the distance between a first coordinate value indicated by the center coordinates x and y included in the selected first tracking information and a second coordinate value indicated by the center coordinates x and y included in the selected second tracking information.

[0138] Next, if the calculated distance is equal to or less than a predetermined threshold, the object tracking unit 115 determines that the first coordinate value and the second coordinate value exist consecutively on two frame images generated consecutively over time. On the other hand, if the calculated distance exceeds the predetermined threshold, the object tracking unit 115 determines that the first coordinate value and the second coordinate value do not exist consecutively on two frame images generated consecutively over time. Here, the predetermined threshold is counted as the number of pixels on a frame image, and is, for example, 10 pixels.

[0139] The object tracking unit 115 calculates the distance and compares it with a predetermined threshold value for all combinations of two pieces of tracking information at two adjacent times out of all the tracking information read from the tracking table 261, and performs judgment.

[0140] In this way, the object tracking unit 115 virtually draws a route 201a, a route 202a, and routes 203a, 203b, and 203c as a single route, as shown in Fig. 2, for example. Here, routes 203a, 203b, and 203c constitute a single route and are therefore distinguishable from other routes. Meanwhile, route 201a and route 202a intersect. In other words, at the intersection where route 201a and route 202a intersect, the four routes are joined together.

[0141] When two routes intersect, such as route 201a and route 202a, resulting in a shape where four routes are joined at the intersection of the two routes, the object tracking unit 115 calculates the amount of change in the y direction relative to the x direction for each of the four routes before and after the intersection point. For a pair of the four calculated amounts of change where the difference between two calculated amounts of change is equal to or less than a predetermined value, the object tracking unit 115 recognizes the two routes that were the basis for calculating these two amounts of change as a single route.

[0142] In this way, the object tracking unit 115 recognizes the path 201a and the path 202a as separate paths, for example, as shown in FIG.

[0143] In this manner, the object tracking unit 115 uses the tracking information read from the tracking table 261 to identify different routes.

[0144] The object tracking unit 115 determines that each path indicates the movement of an object entity, and assigns one object identifier (ID) to each path, that is, to the object entity.

[0145] The object tracking unit 115 writes the assigned IDs for the entities of the same object into the tracking table 261 as the IDs of the multiple pieces of tracking information that served as the basis for drawing the route.

[0146] In this way, as an example, as shown in FIG. 4, the tracking table 261 includes a plurality of tracking information 271 to 275 including the ID "1" and a plurality of tracking information 276 to 280 including the ID "2".

[0147] It can be assumed that the plurality of tracking information 271 to 275 including the ID "1" represent one person object, and the plurality of tracking information 276 to 280 including the ID "2" represent another person object.

[0148] 4, the upper body attribute and the lower body attribute of the person object with ID "1" in the tracking information 271 and 272 are all "green." On the other hand, the upper body attribute of the person object with ID "1" in the tracking information 273, 274, and 275 is "green" and the lower body attribute is "purple." In this way, although the person object has the same ID "1," the attributes of the tracking information 271 and 272 and the tracking information 273, 274, and 275 are different.

[0149] This is because, as mentioned above, part of a person object is occluded by another person object, and the attributes of the occluding person object are mistakenly recognized as the attributes of part of the occluded person object.

[0150] As described above, the object tracking unit 115 uses the tracking information written in the tracking table 261 of the memory circuit 105 to track the entities of the same object, assigns an object identifier (ID) to the entities of the same object, and writes the assigned object identifier into the tracking information.

[0151] (4) Determination unit 116 The determination unit 116 determines, from among a plurality of frame images, a frame image that is suitable for extracting the feature amount of an object as the best scene, as follows.

[0152] The determining unit 116 refers to the tracking table 261 in which the IDs are written, and determines whether or not there are multiple pieces of tracking information assigned different IDs at the same time.

[0153] Next, when it is determined that there are multiple pieces of tracking information (first tracking information and second tracking information) assigned different IDs at the same time, the judgment unit 116 (judgment means) judges whether the person object indicated by the first tracking information is occluding the person object indicated by the second tracking information, or whether the person object indicated by the first tracking information is occluding the person object indicated by the second tracking information.

[0154] In other words, the judgment unit 116 judges whether two pieces of tracking information corresponding to the same time in the tracking table 261 of the memory circuit 105 contain different object identifiers (IDs), and if it is judged that they contain different object identifiers, it judges whether one object is occluded by the other object.

[0155] The determination unit 116 determines whether or not the object is blocked as follows.

[0156] 10(a) shows a person object 302 indicated by the first tracking information and a person object 301 indicated by the second tracking information. The person object 302 is surrounded by a rectangular frame 304, and the person object 301 is surrounded by a rectangular frame 303.

[0157] The determining unit 116 determines whether or not there is a rectangular overlapping area 305 where the rectangular frame 304 and the rectangular frame 303 overlap, as shown in FIG. 10(b).

[0158] When it is determined that an overlapping area 305 exists, the determining unit 116 determines that the person object indicated by the second tracking information is occluded by the person object indicated by the first tracking information (or vice versa).

[0159] In this way, the determination unit 116 determines whether or not there is an overlapping area between the rectangular area surrounding the first object and the rectangular area surrounding the second object, and if it determines that an overlapping area exists, it determines that a part of the first object is occluded in the overlapping area.

[0160] As shown in FIG. 10(c), the determination unit 116 may draw a circle 308 with its center at the center 306 of the rectangular frame 303 and its radius equal to the distance r (a predetermined value) from the center 306 to one vertex 309 of the rectangular frame 303, and if the center 307 of the rectangular frame 304 is inside the circle 308, determine that the person object indicated by the second tracking information is occluded by the person object indicated by the first tracking information (or vice versa).

[0161] In other words, the determination unit 116 determines whether the distance between the rectangular area surrounding the first object and the rectangular area surrounding the second object is within a predetermined value, and if it determines that the distance is within the predetermined value, it may determine that a portion of the first object is occluded in the overlapping area.

[0162] When the determination unit 116 determines that the person object indicated by the second tracking information is occluded by the person object indicated by the first tracking information, the determination unit 116 excludes (deletes) the second tracking information from the tracking table 261.

[0163] As an example, in the case of the tracking table 261 shown in FIG.

[0164] In this way, by excluding (deleting) tracking information relating to objects that are occluded by other objects, the frame image including the object relating to the tracking information that was not excluded (deleted) is determined to be the best scene suitable for extracting the object's features.

[0165] (5) Output unit 117 When it is determined that a part of an object is occluded, the output unit 117 (output means) outputs the attributes or features of the object detected from each object, excluding the occluded part, as described below.

[0166] The output unit 117 extracts and outputs the attributes of the objects (for example, upper body attributes and lower body attributes) from the tracking table 261 from which the tracking information corresponding to the occluded human object has been deleted.

[0167] In the case of tracking table 261 from which tracking information 273, 274, and 275 have been deleted, the attributes (upper body attributes and lower body attributes) of the person object with ID "1" are all "green," and the attributes (upper body attributes and lower body attributes) of the person object with ID "2" are all "purple."

[0168] In this way, the output unit 117 can output the correct attributes for the person object.

[0169] As described above, when it is determined that one object is occluded by another object, the output unit 117 excludes the tracking information corresponding to the occluded object from the tracking table 261 of the memory circuitry 105, and outputs the attributes or features of the object included in the remaining tracking information.

[0170] 1.6 Operation of the Information Processing Device 10 The operation of the information processing device 10 will be described with reference to the flowchart shown in FIG.

[0171] The recognition processing unit 124 detects a person object from each frame image (step S101), and recognizes and extracts attributes from each person object (step S102).

[0172] The recognition result acquisition unit 113 writes, for each object, the time 263 when the frame image was generated, the center coordinate x (264) and center coordinate y (265) which are the center position of the rectangular frame surrounding the object, upper body attribute 266, lower body attribute 267 which are the attributes of the object, and the x length (268a) and y length (268b) of the rectangular frame size 268 as tracking information in the tracking table 261 (step S103).

[0173] Next, the object tracking unit 115 tracks the same person object based on the object's center position (center coordinates x and x) and attributes, and writes the same ID into the tracking table 216 (step S104).

[0174] The determination unit 116 determines whether the human object is occluded for each frame image, detects the occlusion (step S105), and excludes (deletes) the tracking information corresponding to the human object on the occluded side from the tracking table 216 (step S106).

[0175] The output unit 117 extracts and outputs the attributes (upper body attributes and lower body attributes) of the human object from the tracking table 261 from which the tracking information corresponding to the occluded human object has been deleted (step S107).

[0176] 1.7 Variations Here, a modification of the first embodiment will be described.

[0177] (a) The determination unit 116 may set the weight to "0" if the IoU of two objects (an index indicating how much the rectangle surrounding the first object and the rectangle surrounding the second object overlap, hereinafter referred to as index I) is greater than or equal to a preset threshold value t (I≥t). On the other hand, if the index I is less than the threshold value t (I<t), the weight may be set to "1".

[0178] In addition, when I>0.5 and it is determined that there is an overlap, the determination unit 116 may change the weight within the range of "0" to "0.5".

[0179] In addition, the determination unit 116 may set the weight w as shown in (Equation 2).

[0180]

Equation

[0181] In addition, since a person object can be divided into an upper body object and a lower body object, for example, for the rectangle surrounding the upper body object, the weight Wupper may be set according to (Equation 3).

[0182]

Equation

[0183] In (Equation 3), when the upper body object is blocked by another object, the weight Wupper is set to "0".

[0184] Note that as described later (see Fig. 12(a)), the joints of a person can be extracted from a person object, and using the extracted joints, rectangles surrounding the upper body object and rectangles surrounding the lower body object can be set.

[0185] (b) The recognition processing unit 124 may output a recognition result in the form of a numerical value. For example, the recognition processing unit 124 may output, as a feature, a numerical value indicating the likelihood that the upper body attribute is "green," or may output, as a feature, a numerical value indicating the likelihood that the lower body attribute is "purple."

[0186] The determination unit 116 may set a weight using the recognition result (feature amount) output from the recognition processing unit 124, and may use the set weight to determine whether or not a frame image is the best scene. The setting of the weight will be described later.

[0187] Specifically, the determination unit 116 may determine that the frame image is the best scene if the set weight is equal to or greater than a predetermined threshold. On the other hand, if the set weight is less than the predetermined threshold, the determination unit 116 may determine that the frame image is not the best scene. Here, the predetermined threshold is, for example, "0.5." In other words, the determination unit 116 may determine that the frame image is the best scene if the weight is equal to or greater than "0.5." On the other hand, if the weight is less than "0.5," the determination unit 116 may determine that the frame image is not the best scene.

[0188] When two objects are detected by the recognition processing unit 124, the determination unit 116 may determine whether or not at least a part of the first object is occluded by the second object. The determination unit 116 may set a weight depending on the determination result indicating whether or not the first object is occluded.

[0189] Here, when the determination unit 116 determines that at least a portion of the first object is occluded, the determination unit 116 may set a weight based on a first weight value. For example, when the determination unit 116 determines that at least a portion of the first object is occluded, the determination unit 116 may set the first weight value to "0." On the other hand, when the determination unit 116 determines that the first object is not occluded, the determination unit 116 may set a weight based on a second weight value that is greater than the first weight value. For example, when the determination unit 116 determines that the first object is not occluded, the determination unit 116 may set the second weight value to "1" that is greater than the first weight value "0."

[0190] As described above, the recognition processing unit 124 outputs a recognition result (feature amount) in the form of a numerical value, and the determination unit 116 sets a weight using the feature amount output from the recognition processing unit 124. In this case, the determination unit 116 may calculate a weighted feature amount by multiplying the feature amount output from the recognition processing unit 124 by the set weight using the following formula.

[0191] (weighted feature amount)=(feature amount output from the recognition processing unit 124)×(weight set by the determination unit 116) The determination unit 116 may use the weighted feature amount to determine whether or not a frame image is the best scene.

[0192] Specifically, the determination unit 116 may determine that a frame image is the best scene if the weighted feature amount is equal to or greater than a predetermined threshold. On the other hand, if the weighted feature amount is less than the predetermined threshold, the determination unit 116 may determine that the frame image is not the best scene. Here, the predetermined threshold is, for example, "0.5." That is, the determination unit 116 may determine that a frame image is the best scene if the weighted feature amount is equal to or greater than "0.5." On the other hand, if the weighted feature amount is less than "0.5," the determination unit 116 may determine that the frame image is not the best scene.

[0193] 1.8 Summary As explained above, for each frame image, occlusion of an object is detected, tracking information corresponding to the occluded object is excluded (deleted), and the frame image including the object corresponding to the tracking information that was not excluded (deleted) is determined to be the best scene suitable for extracting the feature amounts (attributes) of the object. If it is determined to be the best scene, the feature amounts of the object recognized in that frame image are output. In this way, the feature amounts (attributes) of the object included in the frame image determined to be the best scene are output, thereby improving the accuracy of the feature amounts (attributes) extracted from the object.

[0194] Here, if it is determined that one frame image is not the best scene, the output section 117 may suppress output of the feature amount of the object recognized in that frame image.

[0195] 2. Example 2 In the first embodiment described above, the determination unit 116 determines whether a person object is occluded by another person object based on whether an overlapping area 305 exists, as shown in Fig. 10(b). Also, the determination unit 116 determines whether a center 307 of a rectangular frame 304 exists inside a circle 308, as shown in Fig. 10(c). However, the present invention is not limited to this.

[0196] The memory circuitry 125 (memory means) may further store information relating to the skeleton of the object to be detected, in other words, skeleton definition information that defines the shape of the skeleton of a particular type of object.

[0197] Furthermore, the recognition processing unit 124 may have a two-dimensional skeleton detection unit 127 (not shown). When an object is detected by the recognition processing unit 124, the two-dimensional skeleton detection unit 127 uses skeleton definition information to detect skeleton information indicating the skeleton of each detected object from the object.

[0198] The two-dimensional skeleton detection unit 127 is a convolutional neural network that has undergone machine learning to detect the two-dimensional skeleton of the object to be detected. The two-dimensional skeleton detection unit 127 can be realized using an existing library such as OpenPose.

[0199] The two-dimensional skeleton detection unit 127 generates, from the input image, two-dimensional skeleton detection information including a two-dimensional joint likelihood map, a two-dimensional joint connection likelihood map, and two-dimensional skeleton information.

[0200] The two-dimensional joint likelihood map is map information that exists for each type of joint defined by the skeleton definition information, and has pixel values ​​that indicate the likelihood that a joint of that type exists at each pixel of the input image.

[0201] The two-dimensional joint connection likelihood map exists for each type of connection between joints defined in the skeleton definition information, and is map information having, as pixel values, likelihoods indicating the likelihood that a connection of that type exists at each pixel of the input image. Note that the two-dimensional joint connection likelihood map is sometimes called "Part Affinity Fields," etc.

[0202] The two-dimensional skeleton information is the skeleton information for each person detected from the input image. This skeleton information includes the position (x-coordinate and y-coordinate) of each joint defined by the skeleton definition information on the image and the likelihood indicating the probability of its existence.

[0203] 12(a) is a diagram showing a model of two-dimensional skeletal information for an input image. As shown in this figure, the two-dimensional skeleton detection unit 127 detects joints 332, 333, 334, . . . , 346 from one person object 331, and connects these joints 332, 333, 334, . . . , 346 in accordance with the skeleton definition information to detect skeleton information 347.

[0204] (b) FIG. 12(b) shows a case where a person object 331a is occluded by another object 348a.

[0205] The two-dimensional skeleton detection unit 127 detects joints 332a, 333a, 334a, . . . , 346a from the person object 331a, and connects these joints 332a, 333a, 334a, . . . , 346a in accordance with the skeleton definition information to detect skeleton information 347a.

[0206] When the skeleton information 347a shown in FIG. 12(b) is compared with the skeleton information 347 shown in FIG. 12(a), the joints 339, 343, and 345 included in the skeleton information 347 are missing from the skeleton information 347a.

[0207] The determination unit 116 determines whether or not there is a missing joint from the skeleton information 347a in light of the skeleton definition information. If there is a missing joint, it determines that the person object 331a, from which the skeleton information 347a was detected, is occluded by another object.

[0208] (c) FIG. 12(c) shows a case where a person object 331b is occluded by another object 348b.

[0209] The two-dimensional skeleton detection unit 127 detects joints 332b, 333b, 334b, . . . , 346b from the person object 331b, and connects these joints 332b, 333b, 334b, . . . , 346b in accordance with the skeleton definition information to detect skeleton information 347b.

[0210] Comparing the skeletal information 347b shown in Figure 12(c) with the skeletal information 347 shown in Figure 12(a), in the skeletal information 347b, the length of the skeleton between joint 339b and joint 343b is shorter, and the length of the skeleton between joint 343b and joint 345b is also shorter. Therefore, according to the skeletal information 347b shown in Figure 12(c), the ratio of the length of the left leg to the length of the right leg is not appropriate, with one leg being significantly shorter than the other. In other words, according to the skeletal information 347b shown in Figure 12(c), there is a deformation of the skeleton beyond a predetermined level.

[0211] The determination unit 116 determines whether the length of each skeleton is constant from the skeleton information 347b in light of the skeleton definition information. If there is a skeleton whose length is not constant, in other words, if there is a deformation of a skeleton that is greater than a predetermined value, it determines that the person object 331b that was the basis for detecting the skeleton information 347b is occluded by another object.

[0212] Furthermore, the determination unit 116 may determine whether the ratio between the length of the left leg and the length of the right leg is appropriate from the skeleton information 347b in light of the skeleton definition information. If it is determined that the ratio between the length of the left leg and the length of the right leg is not appropriate, it determines that the person object 331b, which was the basis for detecting the skeleton information 347b, is occluded by another object.

[0213] (d) FIG. 12(d) shows a case where a person object 331c is occluded by another object 348c.

[0214] The two-dimensional skeleton detection unit 127 detects joints 332c, 333c, 334c, . . . , 346c from the person object 331c, and connects these joints 332c, 333c, 334c, . . . , 346c in accordance with the skeleton definition information to detect skeleton information 347c.

[0215] Comparing the skeletal information 347c shown in Fig. 12(d) with the skeletal information 347 shown in Fig. 12(a), in the skeletal information 347c, the length of the skeleton between the joint 335c and the joint 337c is shorter, and the length of the skeleton between the joint 337c and the joint 341c is also shorter. Therefore, according to the skeletal information 347c shown in Fig. 12(d), the ratio of the length of the left hand to the length of the right hand is not appropriate, with one hand being significantly shorter than the other. In other words, according to the skeletal information 347c shown in Fig. 12(d), there is a deformation of the skeleton beyond a predetermined level.

[0216] In addition, the length of the skeleton between joint 333c and joint 346c is also shortened. Therefore, according to skeleton information 347c shown in Fig. 12(d), the ratio between the length of the upper body and the length of the upper body is not appropriate, with one side being significantly shorter than the other. In other words, according to skeleton information 347c shown in Fig. 12(d), there is a deformation of the skeleton that is greater than a predetermined value.

[0217] Furthermore, the rectangle formed by shoulder joint 334c, shoulder joint 335c, waist joint 338c, and waist joint 339c is extremely small compared to the example shown in Fig. 12(a), and is also extremely small compared to the whole body shown by 347c. In other words, according to skeletal information 347c shown in Fig. 12(d), there is a deformation of the skeleton that is greater than a predetermined value.

[0218] The determination unit 116 determines whether the length of each skeleton is constant from the skeleton information 347c in light of the skeleton definition information. If there is a skeleton whose length is not constant, it determines that the person object 331c from which the skeleton information 347c was detected is occluded by another object.

[0219] Furthermore, the determination unit 116 may determine whether the ratio between the length of the left hand and the length of the right hand is appropriate from the skeleton information 347c in light of the skeleton definition information. If it is determined that the ratio between the length of the left hand and the length of the right hand is not appropriate, it determines that the person object 331c from which the skeleton information 347c was detected is occluded by another object.

[0220] Furthermore, the determination unit 116 may determine whether the ratio of the upper body length to the upper body length is appropriate from the skeleton information 347c in light of the skeleton definition information. If the ratio of the upper body length to the upper body length is not appropriate, and one is extremely shorter than the other, it is determined that the person object 331c from which the skeleton information 347c was detected is occluded by another object.

[0221] Furthermore, the determination unit 116 may determine whether the rectangle formed by the four joints of the shoulders and waist has an appropriate shape and an appropriate size from the skeleton information 347c in light of the skeleton definition information. If the rectangle formed by the four joints of the shoulders and waist is not an appropriate shape or an appropriate size, it is determined that the person object 331c from which the skeleton information 347c was detected is occluded by another object.

[0222] As described above, the determination unit 116 refers to the skeleton definition information to determine whether there is a missing part or a deformation greater than a predetermined value in the skeleton indicated by the skeleton information detected from one object, and if it is determined that there is a missing part or a deformation greater than a predetermined value, it determines that a part of the object is occluded.

[0223] As described above, skeletal information can be used to determine whether an object is occluded. In this way, the determination unit 116 excludes (deletes) the occluded object, and determines a frame image including an object that has not been excluded (deleted) as the best scene suitable for extracting the feature amounts (attributes) of the object. The output unit 117 outputs the feature amounts (attributes) of the object included in the frame image determined to be the best scene. As a result, the accuracy of the feature amounts (attributes) extracted from the object can be improved.

[0224] (Variation (1)) As shown in FIG. 12(a), when skeleton information 347 is detected by the two-dimensional skeleton detection unit 127, the recognition processing unit 124 may use the skeleton information 347 to detect the size of the person object 331 (the size of the rectangle surrounding the person object).

[0225] Specifically, the recognition processing unit 124 may virtually generate a rectangular frame 350 (FIG. 12(a)) so as to pass through the position of the head joint 332, the positions of the joints 340 and 341 of both hands, and the positions of the joints 334 and 343 of both feet, and detect and output the size (width and height) of the rectangular frame 350.

[0226] (Variation (2)) As shown in FIG. 12( a ), when skeleton information 347 is detected by the two-dimensional skeleton detection unit 127 , the recognition processing unit 124 may use the skeleton information 347 to detect the position of the person object 331 .

[0227] Specifically, the recognition processing unit 124 may detect the position of a joint 346 in the center of the human body as the position of the person object 331 and output it.

[0228] (Variation (3)) The recognition processing unit 124 may detect the shape of the object as the feature of the object. The determination unit 116 may determine whether the scene is the best scene depending on whether the detected shape is appropriate.

[0229] Specifically, the two-dimensional skeleton detection unit 127 of the recognition processing unit 124 detects skeleton information 347 in accordance with the skeleton definition information (FIG. 12(a)). Here, the skeleton information 347 is the shape of the object.

[0230] The determination unit 116 determines whether the shape indicated by the skeleton information 347 is appropriate in light of the skeleton definition information. For example, as described above, the determination unit 116 determines whether the skeleton indicated by the skeleton information 347 is missing any parts or has a deformation of a predetermined magnitude or more.

[0231] If the skeleton indicated by the skeleton information 347 has no missing parts and has not been deformed beyond a predetermined level, the determination unit 116 determines that the frame image is the best scene. On the other hand, if the skeleton indicated by the skeleton information 347 has missing parts or has been deformed beyond a predetermined level, the determination unit 116 determines that the frame image is not the best scene.

[0232] (summary) As described above, it is determined for each frame image whether it is the best scene. In this way, the feature amounts (attributes) of the objects included in the frame image determined to be the best scene are output, thereby improving the accuracy of the feature amounts (attributes) extracted from the objects.

[0233] 3 Example 3 According to the above-described first embodiment, when there are multiple pieces of tracking information (first tracking information and second tracking information) assigned different IDs at the same time, and a person object indicated by the second tracking information is occluded by a person object indicated by the first tracking information, the determination unit 116 excludes (deletes) the second tracking information from the tracking table 261. In this case, the output unit 117 extracts and outputs the attributes of the object from the tracking table 261 from which the tracking information corresponding to the occluded person object has been deleted.

[0234] However, this is not limiting.

[0235] (a) When first tracking information and second tracking information assigned different IDs exist at the same time, and a person object indicated by the second tracking information is occluded by a person object indicated by the first tracking information, the determination unit 116 may exclude (delete) all tracking information at the same time from the tracking table 261. For example, in the example shown in Fig. 9 , in frame image 250c at time "2," person object 252c occludes person object 251c, in frame image 250d at time "3," person object 252d occludes person object 251d, and in frame image 250e at time "4," person object 252e occludes person object 251e. Therefore, in the tracking table 261 shown in Fig. 4, tracking information 273, 278, 274, 279, 275, and 280 at times "2," "3," and "4" may be excluded (deleted). In this case, the output unit 117 extracts and outputs the object attributes from the tracking table 261 from which the tracking information has been deleted. For example, in the example shown in Fig. 9, the output unit 117 outputs the object attributes extracted from the person objects 251a, 252a, 251b, and 252b that appear in the frame images 250a and 250b.

[0236] (b) When it is determined that first tracking information and second tracking information assigned different IDs exist at the same time and it is determined that the person object indicated by the second tracking information is occluded by the person object indicated by the first tracking information, the determination unit 116 may exclude (delete) the frame images that were used to detect the person object indicated by the second tracking information. For example, in the example shown in FIG. 9, the determination unit 116 excludes (deletes) frame images 250a to 250e. In this case, the output unit 117 excludes tracking information generated based on the excluded frame images, extracts object attributes from the remaining tracking table 261, and outputs the object attributes. For example, in the example shown in FIG. 9, the output unit 117 excludes (deletes) tracking information 271 to 280 generated based on the excluded frame images 250a to 250e from the tracking table 261, and extracts object attributes from the remaining tracking information and outputs the object attributes. In other words, when it is determined that a portion of an object is occluded, the output unit 117 outputs the attributes of the object detected from other frame images, excluding the frame image from which the object was detected.

[0237] (summary) As explained above, occlusion of an object is detected for each frame image, frame images containing occluded objects are excluded (deleted), and frame images that have not been excluded (deleted) are determined to be the best scenes suitable for extracting feature amounts (attributes) of the objects. In this way, the feature amounts (attributes) of the objects contained in the frame images determined to be the best scenes are output, thereby improving the accuracy of the feature amounts (attributes) extracted from the objects.

[0238] 4. Example 4 Here, a case where the best scene is determined using the size of an object will be described.

[0239] FIG. 13(a) shows an example (part 1) of a single person object moving within a frame image over time.

[0240] 13(a) shows, as an example, frame images 401a, 401b, ..., 401e arranged in chronological order of their generation. Time passes from frame image 401a to frame image 401e. Frame image 401a includes one person object 402a. Similarly, the other frame images 401b, ..., 401e also include one person object 402b, ..., 402e. The person objects 402a, ..., 402e included in frame images 401a, 402b, ..., 402e represent the same person, and they move along a path 403 as time passes.

[0241] Here, as shown in FIG. 13(a), it is assumed that the sizes of the person objects 402a, . . . , 402e (the sizes of the rectangular frames surrounding the person objects) are all the same.

[0242] The recognition processing unit 124 recognizes and detects person objects 402a, ..., 402e from frame images 401a, 401b, ..., 401e, and outputs the position and size of each person object within the frame image (for example, a value greater than or equal to "1") as features of the person object.

[0243] FIG. 13(b) shows an example (part 2) of a single person object moving within a frame image over time.

[0244] 13(b) shows, as an example, frame images 405a, 405b, ..., 405e arranged in chronological order of their generation. Time passes from frame image 405a to frame image 405e. Frame image 405a includes one person object 406a. Similarly, the other frame images 405b, ..., 405e also include one person object 406b, ..., 406e. The person objects 406a, ..., 406e included in frame images 405a, 405b, ..., 405e represent the same person, and they move along a path 407 as time passes.

[0245] Here, as shown in FIG. 13(b), it is assumed that the person objects 406a, . . . , 406e all have the same size.

[0246] However, as can be seen by comparing FIGS. 13(a) and (b), the size of the person object 402a is smaller than the size of the person object 406a.

[0247] The recognition processing unit 124 recognizes and detects person objects 406a, ..., 406e from frame images 405a, 405b, ..., 405e, and outputs the position and size of each person object within the frame image (for example, a value greater than or equal to "1") as features of the person object.

[0248] In this way, when the sizes of objects appearing in two frame images are different, comparing the feature amounts extracted from the larger object with the feature amounts extracted from the smaller object, the feature amounts extracted from the larger object will be more accurate than the feature amounts extracted from the smaller object, because the larger object contains more information.

[0249] Using this characteristic, it is possible to determine whether or not a frame image containing an object is the best scene, depending on the size of the object.

[0250] The recognition processing unit 124 detects the size of an object as a feature of the object.

[0251] (a) The determination unit 116 compares the detected size with a preset size threshold. If the detected size is larger than the size threshold, the frame image containing the object whose size is detected may be determined to be the best scene.

[0252] (b) When the recognition processing unit 124 detects an object and the size of the object for each of a plurality of frame images, the determination unit 116 sorts the frame images (or image identifiers that identify the frame images) containing the object whose size has been detected in descending order of size. The determination unit 116 may determine, as the best scene, a frame image that falls within the top 25% of the largest sizes, for example.

[0253] (c) The determination unit 116 may set a weight for the size of the object detected by the recognition processing unit 124 in accordance with the relationship between the size and a predetermined size threshold.

[0254] Specifically, the determination unit 116 compares the size of the object extracted by the recognition processing unit 124 with a predetermined size threshold. Next, if the size of the object is less than the predetermined size threshold, the determination unit 116 assigns a first weight value (e.g., "0") to the size of the object. On the other hand, if the size of the object is equal to or greater than the predetermined size threshold, the determination unit 116 assigns a second weight value (e.g., "1") that is greater than the first weight value.

[0255] The determination unit 116 sets a weight using the size that is the recognition result by the recognition processing unit 124, and determines whether or not the frame image is the best scene using the set weight.

[0256] Specifically, if the set weight is equal to or greater than a predetermined threshold (e.g., 0.5), the determination unit 116 determines that the frame image including this object is the best scene. On the other hand, if the set weight is less than the predetermined threshold (e.g., 0.5), the determination unit 116 determines that the frame image including this object is not the best scene.

[0257] (summary) As explained above, when an object is detected in a frame image and its size is detected, a weight is set using the object size, and the set weight is used to determine whether the frame image is the best scene suitable for extracting the feature amounts (attributes) of the object. In this way, the feature amounts (attributes) of the object included in the frame image determined to be the best scene are output, thereby improving the accuracy of the feature amounts (attributes) extracted from the object.

[0258] 5 Example 5 Here, a case will be described in which the best scene is determined using changes in the position of an object over time.

[0259] FIG. 14(a) shows an example (part 3) of a single person object moving within a frame image over time.

[0260] As an example, FIG. 14(a) shows frame images 411a, 411b, ..., 411f arranged in chronological order of generation time. Time passes from frame image 411a to frame image 411f. Frame image 411a includes one person object 412a. Similarly, the other frame images 411b, ..., 411f also include one person object 412b, ..., 412f. The person objects 412a, ..., 412f included in frame images 411a, 411b, ..., 411f represent the same person, and they move along a path 413 as time passes. Here, the time from frame image 411a to frame image 411f is defined as t1.

[0261] The recognition processing unit 124 recognizes and detects person objects 412a, . . . , 412f from frame images 411a, 411b, . . . , 411f, and outputs the position of each person object within the frame image as the feature amount of the person object.

[0262] FIG. 14(b) shows an example (No. 4) of a single person object moving within a frame image over time.

[0263] 14(b) shows, as an example, frame images 415a, 415b, ..., 415f arranged in chronological order of their generation. Time passes from frame image 415a to frame image 415f. Frame image 415a includes one person object 416a. Similarly, the other frame images 415b, ..., 415d also include one person object 416b, ..., 416d. On the other hand, frame images 415e and 415f do not include any person objects. Person objects 416a, ..., 416d included in frame images 415a, 415b, ..., 415d represent the same person, and they move along a path 417 as time passes. Here, if the time from frame image 415a to frame image 415d is t2, and the time from frame image 415a to frame image 415f is t3, then t1=t3, and t2 <t1である。

[0264] The recognition processing unit 124 recognizes and detects person objects 416a, . . . , 416d from frame images 415a, 415b, . . . , 415d, and outputs the position of each person object within the frame image as the feature amount of the person object.

[0265] Comparing the frame image 411a and the frame image 415a, the position of the person object 412a in the direction of the path 413 and the position of the person object 416a in the direction of the path 417 are substantially the same.

[0266] In contrast, when comparing frame image 411d and frame image 415d, the position of person object 412d in the direction of the path 413 is further behind the position of person object 416d in the direction of the path 417 in the direction of the path 413.

[0267] Furthermore, person objects 412e and 412f are depicted in frame images 411e and 411f, but no person objects are present in frame images 415e and 415f.

[0268] In other words, the speed of movement of the human objects appearing in the frame images 411a to 411f is slower than the speed of movement of the human objects appearing in the frame images 415a to 415d.

[0269] When the moving speeds of objects appearing in two frame image groups (in the example of FIGS. 14(a) and 14(b) , the frame image group of frame images 411a to 411f and the frame image group of frame images 415a to 415d) are different, comparing the feature amount extracted from the object moving faster with the feature amount extracted from the object moving slower, the feature amount extracted from the object moving slower will be more accurate than the feature amount extracted from the object moving fast. This is because the object moving slower contains more information.

[0270] Using this characteristic, it is possible to determine whether a frame image containing an object is the best scene based on the speed of the object's movement over time, that is, the change in the object's position over time.

[0271] (a) The determination unit 116 calculates the amount of change using the following (Equation 1).

[0272]

number

[0273] The determination unit 116 may compare consecutive frame images from time k-1 and time k using (Equation 1), and if the amount of change is smaller than a preset threshold t, determine that the consecutive frame images from time k-1 and time k are the best scene.

[0274] This may also be applied when determining the best scene using the change in size of an object over time, which will be described later.

[0275] It may also be applied when determining the best scene using the change in the aspect ratio of a rectangle surrounding an object over time.

[0276] (b) The determination unit 116 also determines whether or not the amount of change smaller than the threshold value t continues for a predetermined number of frame images or more. If it determines that the amount of change smaller than the threshold value t continues for a predetermined number of frame images or more, the determination unit 116 may determine that these multiple frame images are the best scene.

[0277] This may also be applied when determining the best scene using the change in size of an object over time, which will be described later.

[0278] It may also be applied when determining the best scene using the change in the aspect ratio of a rectangle surrounding an object over time.

[0279] (c) The determination unit 116 also determines whether the total amount of change in a pre-specified number of frame images is smaller than another threshold. If it determines that the total amount of change is smaller than another threshold, the determination unit 116 may determine that these multiple frame images are the best scene.

[0280] This may also be applied when determining the best scene using the change in size of an object over time, which will be described later.

[0281] It may also be applied when determining the best scene using the change in the aspect ratio of a rectangle surrounding an object over time.

[0282] (d) The plurality of frame images are generated in chronological order. The recognition processing unit 124 detects a change in the position of the object within the frame images in chronological order (for example, a value equal to or greater than "1") as a feature of the object. The determination unit 116 may set a weight for the amount of change in the object position detected by the recognition processing unit 124, depending on the relationship between the amount of change in the object position and a predetermined tolerance range.

[0283] Specifically, the determination unit 116 compares the amount of change in the position of the object detected by the recognition processing unit 124 with a predetermined allowable range. Next, if the amount of change in the position of the object is outside the predetermined allowable range, the determination unit 116 sets a first weight value (e.g., "0"). On the other hand, if the amount of change in the position of the object is within the predetermined allowable range, the determination unit 116 sets a second weight value (e.g., "1") that is greater than the first weight value (e.g., "0").

[0284] The determination unit 116 sets a weight using the amount of change in position, which is the recognition result of the recognition processing unit 124, and determines whether or not the frame image is the best scene using the set weight.

[0285] Specifically, if the set weight is equal to or greater than a predetermined threshold (e.g., 0.5), the determination unit 116 determines that the frame image including this object is the best scene. On the other hand, if the set weight is less than the predetermined threshold (e.g., 0.5), the determination unit 116 determines that the frame image including this object is not the best scene.

[0286] (summary) As explained above, when an object is detected in a frame image and a change in the object's position is detected, a weight is set using the amount of change in the object's position, and the set weight is used to determine whether the frame image is the best scene suitable for extracting the feature amounts (attributes) of the object. In this way, the feature amounts (attributes) of the object included in the frame image determined to be the best scene are output, thereby improving the accuracy of the feature amounts (attributes) extracted from the object.

[0287] 6 Example 6 Here, a case will be described in which the best scene is determined using the change in size of an object over time.

[0288] FIG. 15(a) shows an example (No. 5) of a single person object moving within a frame image over time.

[0289] 15(a) shows, as an example, frame images 421a, 421b, ..., 421e arranged in chronological order of generation time. Time passes from frame image 421a to frame image 421e. Frame image 421a includes one person object 422a. Similarly, the other frame images 421b, ..., 421e also include one person object 422b, ..., 422e. Person objects 422a, ..., 422e included in frame images 421a, 421b, ..., 421e represent the same person, and they move along a path 423 as time passes.

[0290] Here, the person objects 422a, . . . , 422e have the same size.

[0291] For example, the person object shown in FIG. 15(a) moves across in front of the camera.

[0292] The recognition processing unit 124 recognizes and detects person objects 422a, . . . , 422e from frame images 421a, 421b, . . . , 421e, and outputs the size of each person object in the frame image as the feature amount of the person object.

[0293] FIG. 15(b) shows an example (No. 6) of a single person object moving within a frame image over time.

[0294] 15(b) shows, as an example, frame images 425a, 425b, ..., 425e arranged in chronological order of generation time. Time passes from frame image 425a to frame image 425e. Frame image 425a includes one person object 426a. Similarly, the other frame images 425b, ..., 425e also include one person object 426b, ..., 426e. The person objects 426a, ..., 426e included in frame images 425a, 425b, ..., 425e represent the same person, and they move along a path 427 as time passes.

[0295] Here, the sizes of the person objects 426a, ..., 426e are different. The size of the person object 426a is the smallest. The sizes of the person objects increase in the order of 426a, 426b, 426c, 426d, and 426e. The size of the person object 426e is the largest.

[0296] For example, the person object shown in FIG. 15(b) moves from a position far from the camera to a position close to the camera.

[0297] The recognition processing unit 124 recognizes and detects person objects 426a, . . . , 426e from frame images 425a, 425b, . . . , 425e, and outputs the size of each person object in the frame image as a feature amount of the person object.

[0298] In this way, the sizes of the person objects appearing in frame images 421a to 421e are the same and do not change, whereas the sizes of the person objects appearing in frame images 425a to 425e are not the same and do change.

[0299] Comparing the case where the size of an object displayed in one frame image group (frame images 421a to 421e in FIG. 15(a)) does not change with the case where the size of an object displayed in one frame image group (frame images 425a to 425e in FIG. 15(b)) changes, the accuracy of the feature amount extracted from the object is higher when the object size does not change than when the object size changes. This is because the amount of information is greater when the object size does not change.

[0300] Using this characteristic, it is possible to determine whether a frame image containing an object is the best scene, depending on whether the size of the object changes.

[0301] The plurality of frame images are generated in chronological order. The recognition processing unit 124 detects a change in the size of an object within the frame images in chronological order (for example, a value equal to or greater than "1") as a feature of the object. The determination unit 116 sets a weight for the change in the size of the object detected by the recognition processing unit 124, depending on the relationship between the amount of change in the size of the object and a predetermined tolerance range.

[0302] Specifically, the determination unit 116 compares the amount of change in the size of the object detected by the recognition processing unit 124 with a predetermined allowable range. Next, if the amount of change in the size of the object is outside the predetermined allowable range, the determination unit 116 sets a first weight value (e.g., "0") to the object. On the other hand, if the amount of change in the size of the object is within the predetermined allowable range, the determination unit 116 sets a second weight value (e.g., "1") that is greater than the first weight value (e.g., "0").

[0303] The determination unit 116 sets a weight using the amount of change in size, which is the recognition result of the recognition processing unit 124, and determines whether or not the frame image is the best scene using the set weight.

[0304] Specifically, if the weight is equal to or greater than a predetermined threshold (e.g., 0.5), the determination unit 116 determines that the frame image containing this object is the best scene. On the other hand, if the weight is less than a predetermined threshold (e.g., 0.5), the determination unit 116 determines that the frame image containing this object is not the best scene.

[0305] (summary) As explained above, when an object is detected in a frame image and a change in the size of the object is detected, a weight is set using the amount of change in the size of the object, and the set weight is used to determine whether the frame image is the best scene suitable for extracting the feature amounts (attributes) of the object. In this way, the feature amounts (attributes) of the object included in the frame image determined to be the best scene are output, thereby improving the accuracy of the feature amounts (attributes) extracted from the object.

[0306] 7 Example 7 Here, a case where the best scene is determined using the orientation of an object will be described.

[0307] (a) Determining the orientation of a person object (a-1) FIG. 16(a) shows a person object 431a facing sideways, and FIG. 16(b) shows a person object 432a facing forward.

[0308] Here, the horizontal direction is a direction parallel to the surface of the frame image, and the front direction is a direction perpendicular to the surface of the frame image, facing forward of the surface of the frame image.

[0309] In addition to the above-mentioned sideways and frontward orientations, there are various other orientations for the object, such as backwards (facing backwards), diagonally forward left, and diagonally forward right.

[0310] A rectangular frame 431 surrounding a person object 431a shown in FIG. 16(a) is a vertically long rectangle, and a rectangular frame 432 surrounding a person object 432a shown in FIG. 16(b) is a rectangle that is close to a square.

[0311] The orientation of a human object included in a frame image can be recognized by the recognition processing unit 124, that is, by artificial intelligence.

[0312] Furthermore, it is not limited to this, and it is possible to determine whether a human object included in a frame image is facing sideways or forward, based on the shape of the rectangular frame surrounding the human object.

[0313] Generally, the shoulder width of a human body is longer than the distance between the abdomen and the back of the human body, so that when the human body is viewed from the front, the rectangular frame surrounding the human body object is nearly square, and when the human body is viewed from the side, the rectangular frame surrounding the human body object is vertically long.

[0314] If it is determined that the rectangular frame surrounding the person object is stably shaped like a square, the determination unit 116 determines that the person object is facing forward. Here, stably shaped like a square means that the shaped frame remains shaped like a square for a predetermined period of time, for example, five seconds.

[0315] On the other hand, if it is determined that the rectangular frame surrounding the person object is vertically long, the determining unit 116 determines that the person object is oriented horizontally.

[0316] Furthermore, if the rectangular frame surrounding the person object alternates between a nearly square shape and a vertically elongated shape in a short cycle, the determination unit 116 determines that the person object is oriented sideways. Here, a short cycle is, for example, 1 to 2 seconds. When a person walks while swinging both arms, if both arms are on the sides of the body, the rectangular frame surrounding the person object will have a vertically elongated shape, and if one arm is in front of the body and the other arm is behind the body, the rectangular frame surrounding the person object will have a nearly square shape.

[0317] That is, the orientation of a person object can be determined based on the aspect ratio of the rectangular frame surrounding the person object.

[0318] In this way, it is possible to determine whether the person object is facing sideways or forward based on the shape of the rectangular frame surrounding the person object.

[0319] In this case, if the person object faces forward (or backward), the weight w may be set to "1", and if the person object faces sideways, the weight w may be set to "0.5".

[0320] (a-2) As described above, when joints are obtained from a human object, the obtained joints include shoulder joints 334, 335 and waist joints 338, 339, as shown in FIG. 12( a). The determination unit 116 forms a rectangle by sequentially connecting the shoulder joints 334, 335 and the waist joints 338, 339, and determines the midpoint of the formed rectangle. Using the determined center, the determination unit 116 determines the aspect ratio of the rectangle formed by the shoulder joints 334, 335 and the waist joints 338, 339. Next, using the aspect ratio and the positional relationship between the shoulder joints 334, 335, the determination unit 116 determines whether the human object is facing forward, sideways, or backward.

[0321] (a) Using the aspect ratio, if it is determined that the horizontal length of the rectangle is longer than the vertical length of the rectangle, or if it is determined that the rectangle is close to a square, the person object is determined to be facing forward or backward.

[0322] (a) On the other hand, if it is determined using the aspect ratio that the horizontal length of the rectangle is shorter than the vertical length of the rectangle, the human object is determined to be oriented horizontally.

[0323] (c) As described above, when joints are obtained from a human object, the obtained joints include shoulder joints 334 and 335, as shown in Fig. 12(a). In this way, it is possible to determine whether the human object is facing forward or backward based on the positional relationship between the obtained left shoulder joint and right shoulder joint.

[0324] 12(a) is facing forward, the right shoulder joint 334 is located to the left of the left shoulder joint 335 on the frame image. On the other hand, if the person object is facing backward, the right shoulder joint is located to the right of the left shoulder joint.

[0325] An xy coordinate system is defined for a frame image including a person object shown in Fig. 12(a). As shown in Fig. 12(a), the horizontal direction is the x-axis and the vertical direction is the y-axis.

[0326] When the person object 331 shown in FIG. 12(a) is facing forward, xRshoulder≦xLshoulder.

[0327] Here, xRshoulder is the x coordinate value of the right shoulder joint 334, and xLshoulder is the x coordinate value of the left shoulder joint 335.

[0328] When the person object 331 shown in FIG. 12(a) is facing backward, xRshoulder>xLshoulder.

[0329] Here, xRshoulder is the x coordinate value of the right shoulder joint, and xLshoulder is the x coordinate value of the left shoulder joint.

[0330] In this case, the weight w may be set as shown in (Equation 4).

[0331]

number

[0332] (b) The recognition processing unit 124 performs object recognition processing on each frame image to detect multiple partial objects that make up the object, and detects the direction in which the main surface of the partial object is facing and the attributes of the object as features of the object.

[0333] As described above, the object is, for example, a whole-body object of a person, and the partial objects are objects such as the head, face, back, upper body, lower body, left arm, right arm, left leg, and right leg of the person.

[0334] Furthermore, if the partial object is a person's head, the main surface of the partial object is, for example, the surface of the person's face. Furthermore, if the partial object is a person's upper body, the main surface of the partial object is, for example, the surface of the person's back.

[0335] Frame image 435 shown in Figure 17 shows a person represented by person object 436b wearing glasses and carrying a backpack, walking from the position of person object 436a along path 439a, then along path 439b, and then along path 439c.

[0336] Here, examples of object attributes are "wearing glasses on face" and "carrying a backpack on back" (see FIG. 17).

[0337] As an example, the angles of the detected directions are "90°" for a frontal direction, "0°" or "180°" for a sideways direction, and "270°" for a rearward direction, as shown in Fig. 16(c). The diagonally forward left direction, diagonally forward right direction, etc. are determined according to Fig. 16(c).

[0338] The determination unit 116 sets a weight for the angle indicating the detected direction in accordance with the relationship between the detected direction and the detected attribute.

[0339] Specifically, the determination unit 116 determines whether the detected direction is the front direction of the frame image. The determination unit 116 also determines whether there is a relationship between the detected partial object and the attribute of the detected object.

[0340] For example, "glasses" are worn on the "face," so there is a relationship between the attribute "wearing glasses" and the partial object "face." Also, a "backpack" is carried on the "back," so there is a relationship between the attribute "carrying a backpack on one's back" and the partial object "back."

[0341] Here, as an example, the attribute "wearing glasses" and the partial object "face" will be described.

[0342] When the detected orientation is forward, the partial object "face" is facing forward, and the "glasses" worn on the partial object "face" are entirely reflected in the frame image (see Figure 17, 437a), and the accuracy of the attribute "wearing glasses" is high.

[0343] On the other hand, if the detected orientation is landscape, the partial object "face" is facing landscape, and only a portion of the "glasses" worn on the partial object "face" is captured in the frame image (see Figure 17, 437b), and the accuracy of the attribute "wearing glasses" is somewhat low.

[0344] Furthermore, when the detected orientation is rearward facing, the partial object "face" is facing rearward, and the "glasses" worn on the partial object "face" are not reflected in the frame image (see person object 436c in Figure 17), and the accuracy of the attribute "wearing glasses" is extremely low.

[0345] Also, as an example, the attribute "carrying a backpack on the back" and the partial object "back" will be described.

[0346] When the detected orientation is rearward facing, the partial object "back" faces forward, and the "backpack" carried on the partial object "back" is entirely reflected in the frame image (see Figure 17, 438c), and the accuracy of the attribute "carrying a backpack on one's back" is high.

[0347] On the other hand, if the detected orientation is landscape, the partial object "back" is facing landscape, and only a portion of the "backpack" carried on the partial object "back" is captured in the frame image (see Figure 17, 438b), and the accuracy of the attribute "carrying a backpack on the back" is somewhat low.

[0348] Furthermore, when the detected orientation is forward, the partial object “back” faces backward, and only a small part of the “backpack” carried on the partial object “back” is reflected in the frame image (see Figure 17, 438a), and the accuracy of the attribute “carrying a backpack on the back” is extremely low.

[0349] If it is determined that the detected direction is the front and that there is an association between the partial object and the attribute of the detected object, the determination unit 116 sets a weight based on a second weight value that is greater than the first weight value for the value (angle) indicating the direction.

[0350] In this case, the determining unit 116 sets, for example, the value "1" as the second weight value to the value (angle) indicating the direction.

[0351] On the other hand, if it is determined that the detected direction is not the front, or if it is determined that there is no association between the partial object and the attributes of the detected object, the determination unit 116 sets a weight based on a first weight value.

[0352] In this case, the determining unit 116 sets, for example, the value indicating the direction (angle) to the value "0" as the first weight value.

[0353] In addition, when an object is detected, the recognition processing unit 124 may use skeleton definition information to detect skeleton information indicating the skeleton of each detected object, and use the detected skeleton information to detect the direction in which the main surface of the partial object is facing.

[0354] For example, as described above (see FIG. 12(a)), the recognition processing unit 124 virtually generates a rectangular frame 350 so as to pass through the position of the head joint 332, the positions of the joints 340 and 341 of both hands, and the positions of the joints 334 and 343 of both feet. The recognition processing unit 124 may detect the direction in which the object is facing by determining whether the rectangular frame 350 is a vertically long rectangle or a rectangle that is close to a square (see FIGS. 16(a) and 16(b)).

[0355] The determination unit 116 sets a weight using a value (angle) indicating a direction as a recognition result from the recognition processing unit 124, and determines whether or not a frame image is the best scene using the set weight.

[0356] If the set weight is equal to or greater than a predetermined threshold, the determination unit 116 determines that the frame image is the best scene, and if the set weight is less than the predetermined threshold, the determination unit 116 determines that the frame image is not the best scene.

[0357] For example, if the set weight is equal to or greater than a predetermined threshold (e.g., "0.5"), the determination unit 116 determines that the frame image is the best scene, and if the set weight is less than the predetermined threshold (e.g., "0.5"), the determination unit 116 determines that the frame image is not the best scene.

[0358] (summary) As explained above, when an object is detected in a frame image, the direction in which the main surface of the partial object faces and the attributes of the object are detected as the feature amounts of the object, a weight is set using the detected direction and attribute, and the set weight is used to determine whether the frame image is the best scene suitable for extracting the feature amounts (attributes) of the object. In this way, the feature amounts (attributes) of the object included in the frame image determined to be the best scene are output, thereby improving the accuracy of the feature amounts (attributes) extracted from the object.

[0359] 8 Example 8 Here, a case where the best scene is determined using the position of an object will be described.

[0360] As shown in FIG. 1, the cameras 5a, 5b, and 5c are installed fixedly at predetermined positions, and therefore the cameras 5a, 5b, and 5c always capture images of a fixed range.

[0361] When fixed objects such as plants, furniture, and signs are installed within the shooting range without moving, a moving person object may be occluded by these fixed objects. Therefore, by storing the areas occupied by these fixed objects in each frame image, it is possible to detect occlusion of the person object by determining whether the area occupied by the person object is included in the area occupied by the fixed objects.

[0362] The memory circuitry 105 stores the positions, sizes, and shapes (positional ranges) of the areas occupied by these fixed portions.

[0363] Furthermore, as described above, since the cameras 5a, 5b, and 5c always capture images of a fixed range, a point in a frame image (a point in a two-dimensional plane) indicates a point in three-dimensional space within the capture range. Therefore, if the area occupied by a person object in a frame image is known, the three-dimensional area occupied by the person object in the three-dimensional space within the capture range can be identified. As a result, the position and size of the person object can be known from the area occupied by the person object in the frame image.

[0364] The recognition processing unit 124 detects the position of the object within the frame image as the feature amount of the object.

[0365] The determination unit 116 sets a weight according to the position detected by the recognition processing unit 124 and the position range stored in the storage circuit 105 .

[0366] Specifically, the determination unit 116 compares the position of the detected object with the position range stored in the memory circuitry 105. In other words, the determination unit 116 determines whether the position of the detected object is outside the position range stored in the memory circuitry 105 or inside the position range.

[0367] If it is determined that the position of the detected object is within the position range, the determining unit 116 assigns a weight based on the first weight value to the feature amount extracted from the object.

[0368] In this case, for example, the determining unit 116 sets a value of "0" as the first weight value for the feature amount extracted from the object.

[0369] On the other hand, if it is determined that the object position is outside the position range, the determining unit 116 sets a weight based on a second weight value that is greater than the first weight value.

[0370] In this case, for example, the determining unit 116 sets a value of "1" as the second weight value for the feature amount extracted from the object.

[0371] Furthermore, when an object is detected, the recognition processing unit 124 may use the skeleton definition information to detect skeleton information indicating the skeleton of each detected object, and may use the detected skeleton information to detect the position occupied by the object within the frame image.

[0372] Furthermore, the determination unit 116 may set a weight using the feature amount of the object that is the recognition result by the recognition processing unit 124, and use the set weight to determine whether or not one frame image is the best scene.

[0373] In addition, the determination unit 116 may determine that the frame image is the best scene if the set weight is greater than or equal to a predetermined threshold (e.g., "0.5"), and may determine that the frame image is not the best scene if the set weight is less than the predetermined threshold (e.g., "0.5").

[0374] (summary) As explained above, when an object is detected in a frame image and its position is detected, a weight is set using the object's position, and the set weight is used to determine whether the frame image is the best scene suitable for extracting the feature amounts (attributes) of the object. In this way, the feature amounts (attributes) of the object included in the frame image determined to be the best scene are output, thereby improving the accuracy of the feature amounts (attributes) extracted from the object.

[0375] 9 Example 9 Here, a method for determining whether a frame image is the best scene or not will be described using the relationship between orientation and attribute and occlusion by an object, which will be described later.

[0376] (Frame image 435) FIG. 17 shows a state in which a part of a person object 436b is occluded by a person object 440 in a frame image 435.

[0377] The frame image 435 is a frame image captured and generated by the camera 5a, 5b or 5c at a certain point in time.

[0378] The frame image 435 includes person objects 436b and 440.

[0379] Note that person object 436a shown in frame image 435 does not indicate that person object 436a appears in frame image 435, but indicates the person object at a point in time shortly before (for example, 5 seconds before) the photographing time of the person represented by person object 436b. Similarly, person object 436c shown in frame image 435 does not indicate that person object 436c appears in frame image 435, but indicates the person object at a point in time shortly after (for example, 5 seconds after) the photographing time of the person represented by person object 436b.

[0380] As described above, the person represented by person object 436b is wearing glasses, carrying a backpack, and walking from the position of person object 436a along path 439a, then along path 439b, and then along path 439c.

[0381] The person represented by the person object 436b faces forward (436a) on the path 439a, faces sideways (436b) on the path 439b, and has its back to the front (436c) on the path 439c.

[0382] Furthermore, the person represented by the person object 440 is walking along a path 441 .

[0383] In the path 439b, the upper body of the person object 440 occludes the lower body of the person object 436b.

[0384] On the other hand, in the paths 439a and 439c, no person object occludes any other person object.

[0385] The times of photography on path 439a are "0," "1," and "2," the times of photography on path 439b are "3," "4," and "5," and the times of photography on path 439c are "6," "7," and "8." A frame image exists at each time. In this way, multiple frame images are generated in chronological order.

[0386] Here, if shooting times "0," "1," and "2" are considered as one section, shooting times "3," "4," and "5" are considered as one section, and shooting times "6," "7," and "8" are considered as one section, then each of the multiple sections arranged in chronological order will contain multiple frame images. Alternatively, each shooting time may be considered as one section. In this case, each of the multiple sections arranged in chronological order will contain one frame image.

[0387] (Evaluation Table 501) The storage circuitry 105 includes an area for storing, for example, an evaluation table 501 shown in FIG.

[0388] The evaluation table 501 is a data table for storing attributes detected from frame images and the contents of those attributes.

[0389] The evaluation table 501 is made up of an attribute group 502, a plurality of attribute content groups 503, 504, 505, . . . , and a best scene group 506.

[0390] Attribute group 502 includes multiple attributes. The number of attributes included in attribute group 502 is equal to the number of attributes detected by recognition processing unit 124. The attributes included in attribute group 502 correspond to the attributes detected by recognition processing unit 124. The attributes included in attribute group 502 include, for example, time of day, object orientation, object occlusion, whether glasses are worn or a backpack is carried, upper body clothing, and lower body clothing.

[0391] The number of attribute content groups 503, 504, 505, ... is equal to the number of times at which images were taken by the camera. In other words, the number of attribute content groups 503, 504, 505, ... is equal to the number of frame images generated by the camera. Each of the attribute content groups 503, 504, 505, ... corresponds to a plurality of frame images generated by the camera.

[0392] Each attribute content group includes multiple attribute contents. The number of attribute contents included in each attribute content group is equal to the number of attributes detected by the recognition processing unit 124. The attribute contents included in each attribute content group correspond to the attributes detected by the recognition processing unit 124.

[0393] The attribute contents included in each attribute content group are "0", "1", "2", etc. for the attribute "time", "forward", "sideways", "backwards", etc. for the attribute "object orientation", and "yes" or "no" for the attribute "object occlusion".

[0394] On the other hand, in the case of the attribute "whether the person wears glasses," the attribute contents included in each attribute content group are "○ (yes)," "△ (yes)," "× (no)," etc. Here, "○," "△," and "×" indicate the accuracy of the attribute "whether the person wears glasses." In other words, they indicate the reliability of the attribute "whether the person wears glasses," which is the recognition result by the recognition processing unit 124. In the case of "○," the accuracy is the highest. In the case of "△," the accuracy is lower than "○." In the case of "×," the accuracy is extremely low. Furthermore, "yes" indicates that the person wears glasses, and "no" indicates that the person does not wear glasses.

[0395] In the case of the attribute "whether the person is carrying a backpack," the attribute content included in each attribute content group is "○ (yes)," "△ (yes)," "× (no)," etc. Here, "○," "△," and "×" indicate the accuracy of the attribute "whether the person is carrying a backpack," as in the case above. In other words, they indicate the reliability of the attribute "whether the person is carrying a backpack," which is the recognition result by the recognition processing unit 124. "○" indicates the highest accuracy. "△" indicates the accuracy is lower than "○." "×" indicates the accuracy is extremely low. Furthermore, "yes" indicates that the person is carrying a backpack, and "no" indicates that the person is not carrying a backpack.

[0396] In the case of the attribute "upper body clothing," the attribute contents included in each attribute content group are "○ (green)," "○ (purple)," "△ (green)," "×," etc. Here, "○," "△," and "×" are as described above. "Green," "purple," etc. indicate colors that are characteristic of the attribute "upper body clothing."

[0397] In the case of the attribute "lower body clothing," as in the case of the attribute "upper body clothing," the attribute contents included in each attribute content group are "○ (green)," "○ (purple)," "△ (green)," "×," etc. Here, "○," "△," and "×" are as described above. "Green," "purple," etc. indicate colors that are characteristic of the attribute "lower body clothing."

[0398] The best scene group 506 includes a plurality of best scenes 506d, 506e, 506f, and 506g.

[0399] Best scenes 506d, 506e, 506f, and 506g correspond to the attribute "whether glasses are worn," the attribute "whether a backpack is carried," the attribute "clothing for the upper body," and the attribute "clothing for the lower body," respectively.

[0400] Best scene 506d is the best scene for the attribute "whether glasses are worn" among times "0" to "8", best scene 506e is the best scene for the attribute "whether a backpack is carried" among times "0" to "8", best scene 506f is the best scene for the attribute "upper body clothing" among times "0" to "8", and best scene 506g is the best scene for the attribute "lower body clothing" among times "0" to "8".

[0401] (Weight Table 511) The storage circuitry 105 includes an area for storing, for example, a weight table 511 shown in FIG.

[0402] The weight table 511 is a data table that stores weights for determining the reliability of attributes in the evaluation table 501. The weight table 511 has a data structure similar to that of the evaluation table 501.

[0403] The weight table 511 consists of an attribute group 512 and a plurality of attribute content groups 513, 514, 515, . . .

[0404] The attribute group 512 is the same as the attribute group 502 in the evaluation table 501, and therefore a detailed description thereof will be omitted.

[0405] The number of attribute content groups 513, 514, 515, ... is equal to the number of frame images, and each of the attribute content groups 513, 514, 515, ... corresponds to a plurality of frame images.

[0406] Each attribute content group includes multiple attribute contents. The number of attribute contents included in each attribute content group is equal to the number of attributes detected by the recognition processing unit 124. The attribute contents included in each attribute content group correspond to the attributes detected by the recognition processing unit 124.

[0407] The attribute contents included in each attribute content group are "0", "1", "2", etc. for the attribute "time", "forward", "sideways", "backwards", etc. for the attribute "object orientation", and "yes" or "no" for the attribute "object occlusion". In this respect, too, it is the same as evaluation table 501.

[0408] On the other hand, for the attributes "whether the person is wearing glasses," "whether the person is carrying a backpack," "upper body clothing," and "lower body clothing," the attribute content included in each attribute content group is a weight (weight value) for determining the reliability of the attribute.

[0409] These weights are determined by the determination unit 116 based on the attribute content of the attribute "object orientation" ("forward", "sideways", "backwards", etc.) and the attribute content of the attribute "object occlusion" ("present" or "absent" occlusion) at the same time.

[0410] (Recognition processing unit 124) As described above, the recognition processing unit 124 performs object recognition processing on the frame images at each of times "0" to "8", and if an object is detected, detects the attribute and feature amount of the object.

[0411] The recognition processing unit 124 detects object attributes such as the time, the object's orientation, whether the object is occluded, whether the person is wearing glasses or a backpack, the clothing worn by the upper body, and the clothing worn by the lower body.

[0412] The attribute "time" can have the values ​​"0", "1", "2", ...

[0413] The attribute "object orientation" can have values ​​such as "forward facing", "sideways facing", and "backwards facing".

[0414] The attribute "object occlusion" can have the values ​​"yes" or "no".

[0415] The attribute "Does the person wear glasses?" can have the values ​​"Yes" or "No."

[0416] The attribute "Does the person carry a backpack?" can have the values ​​"Yes" or "No."

[0417] The attribute "upper body clothing" or the attribute "lower body clothing" can have values ​​such as "green" or "purple."

[0418] The recognition processing unit 124 outputs the detected multiple attributes and the contents of each attribute to the recognition result acquisition unit 113 at each time, that is, for each frame image.

[0419] (Recognition result acquisition unit 113) The recognition result acquisition unit 113 receives the detected multiple attributes and their respective attribute contents from the recognition processing unit 124 at each time, that is, for each frame image.

[0420] When receiving multiple attributes, the recognition result acquisition unit 113 writes multiple attributes, for example, the attribute "time", the attribute "object orientation", the attribute "object occlusion", the attribute "whether glasses are worn", the attribute "whether a backpack is carried", the attribute "upper body clothing", and the attribute "lower body clothing", into the evaluation table 501 and the weight table 511 of the memory circuitry 105 as one attribute group 502 for each attribute, which is time.

[0421] Furthermore, upon receiving the attribute content of each of the multiple attributes, the recognition result acquisition unit 113 writes the multiple attribute contents, for each attribute time, which is one of the attributes, such as the attribute content "0" of the attribute "time", the attribute content "forward" of the attribute "object orientation", the attribute content "no" of the attribute "object occlusion", the attribute content "yes" of the attribute "whether glasses are worn", the attribute content "no" of the attribute "whether a backpack is carried", the attribute content "green" of the attribute "upper body clothing", and the attribute content "green" of the attribute "lower body clothing", into the evaluation table 501 of the memory circuitry 105 as one attribute content group 503.

[0422] In this way, the recognition result acquisition unit 113 writes the detected multiple attributes and the contents of each attribute for each of all frame images into the evaluation table 501 in the storage circuitry 105.

[0423] Furthermore, upon receiving the attribute contents of each of the multiple attributes, the recognition result acquisition unit 113 writes the multiple attribute contents, for example, the attribute content "0" of the attribute "time", the attribute content "forward" of the attribute "object orientation", and the attribute content "none" of the attribute "object occlusion", into the weight table 511 of the memory circuitry 105 as one attribute content group 513 for each attribute, which is time.

[0424] In this way, the recognition result acquisition unit 113 writes the detected multiple attributes and the attribute contents of the attribute "time", the attribute contents of the attribute "object orientation", and the attribute contents of the attribute "object occlusion" for each of all frame images into the weight table 511 of the memory circuitry 105. Therefore, at this point in time, the attribute contents of the attribute "whether the person is wearing glasses", the attribute contents of the attribute "whether the person is carrying a backpack", the attribute contents of the attribute "upper body clothing", and the attribute contents of the attribute "lower body clothing" in the weight table 511 are blank.

[0425] When the recognition result acquisition unit 113 has finished receiving the detected multiple attributes and their respective attribute contents from the recognition processing unit 124 and has finished writing to the evaluation table 501 and weight table 511, it outputs an end notification to the judgment unit 116.

[0426] (Judgment unit 116) The determination unit 116 stores in advance the relationship between each attribute and the attribute of the direction (hereinafter referred to as the relationship between the direction and the attribute).

[0427] The determination unit 116 stores in advance, for example, the relationship between the attribute "whether the person wears glasses" and the attribute "object orientation."

[0428] When the attribute "object orientation" (hereinafter referred to as "orientation") is forward-facing, the entire surface of the glasses, which is the feature indicated by the attribute "whether glasses are worn," is captured in the frame image, so there is a large amount of information about the glasses, and the reliability of the attribute "whether glasses are worn," which is the recognition result, is high. For this reason, the determination unit 116 sets a large weight (for example, "1"). Furthermore, when the orientation is landscape, the side of the glasses is captured in the frame image, so there is less information about the glasses compared to when the orientation is forward-facing, and the reliability of the attribute "whether glasses are worn," which is the recognition result, is somewhat low. For this reason, the determination unit 116 sets a somewhat small weight (for example, "0.5" or "0.25"). Furthermore, when the orientation is backward-facing, the glasses are not captured in the frame image, and there is no information about the glasses in the frame image, so the reliability of the attribute "whether glasses are worn," which is the recognition result, is extremely low. For this reason, the determination unit 116 sets an extremely small weight (for example, "0").

[0429] Furthermore, the determination unit 116 stores in advance, for example, the relationship between the attribute "whether a backpack is being carried" and the attribute "object orientation."

[0430] When the attribute "object orientation" (hereinafter, "orientation") is backward, the entire surface of the backpack, which is the feature indicated by the attribute "whether the object is carrying a backpack," is captured in the frame image, so there is a lot of information about the backpack, and the reliability of the attribute "whether the object is carrying a backpack," which is the recognition result, is high. For this reason, the determination unit 116 sets a large weight (for example, "1"). Furthermore, when the orientation is landscape, the side of the backpack is captured in the frame image, so there is less information about the backpack compared to when the orientation is backward, and the reliability of the attribute "whether the object is carrying a backpack," which is the recognition result, is slightly lower. For this reason, the determination unit 116 sets a slightly smaller weight (for example, "0.5" or "0.25"). Furthermore, when the orientation is forward, the backpack is barely captured in the frame image, and there is very little information about the backpack in the frame image, so the reliability of the attribute "whether the object is carrying a backpack," which is the recognition result, is extremely low. For this reason, the determination unit 116 sets an extremely small weight (for example, "0").

[0431] Furthermore, as described above, the determining unit 116 determines whether or not at least a part of one object is occluded by another object in each frame image.

[0432] When determining that one object is not occluded by another object, the determination unit 116 assigns a high weight (for example, "1") to the object. On the other hand, when determining that at least a part of one object is occluded by another object in each frame image, the determination unit 116 assigns an extremely low weight (for example, "0").

[0433] Furthermore, the determination unit 116 receives a notification of completion from the recognition result acquisition unit 113 indicating that writing to the evaluation table 501 and the weight table 511 has been completed.

[0434] Upon receiving the notification of completion, the determining unit 116 reads all attributes included in the attribute group 512 from the weight table 511. The determining unit 116 also reads attribute content groups from the weight table 511 at each time point.

[0435] (Determination based on the relationship between orientation and attributes) The determination unit 116 determines whether or not each attribute has the above-described relationship between the orientation and the attribute. If it is determined that the attribute has a relationship between the orientation and the attribute, the determination unit 116 determines and sets the weight of the attribute using the relationship between the orientation and the attribute.

[0436] The weight set by the judgment is assigned to the attribute content in the evaluation table 501 and written. As described above, weights can be, for example, "1," "0.5," "0.25," "0," etc., so the judgment unit 116 writes the set weight into the attribute content of the attribute "whether the person is wearing glasses," the attribute content of the attribute "whether the person is carrying a backpack," the attribute content of the attribute "clothing for the upper body," and the attribute content of the attribute "clothing for the lower body" in the weight table 511.

[0437] (Determined by shielding) Furthermore, the determination unit 116 writes the set weight based on the determination result of whether or not at least a part of one object (or partial object) is occluded by another object into the attribute contents of the attribute "whether the object is wearing glasses," the attribute contents of the attribute "whether the object is carrying a backpack," the attribute contents of the attribute "clothing for the upper body," and the attribute contents of the attribute "clothing for the lower body" in the weight table 511.

[0438] In addition, when different weights are set for the judgment based on the relationship between orientation and attribute and the judgment based on occlusion for the same time and the same attribute, the judgment unit 116 selects the smaller weight and writes the selected weight into the attribute content of the attribute in the weight table 511.

[0439] (Judgment based on weight) The determination unit 116 reads out weights, which are attribute contents, from the weight table 511 for each time and attribute (attribute "whether the person is wearing glasses," attribute "whether the person is carrying a backpack," attribute "clothing for the upper body," and attribute "clothing for the lower body"). Next, the determination unit 116 determines whether the read weights are large (for example, "1"), somewhat small (for example, "0.5" or "0.25"), or extremely small (for example, "0").

[0440] If the read weight is judged to be large, the determining unit 116 writes a "◯" indicating that the attribute content is highly reliable to the attribute content of the corresponding time and attribute in the evaluation table 501.

[0441] If the judgment unit 116 determines that the read weight is somewhat small, it writes a "△" indicating that the reliability of the attribute content is somewhat low, to the attribute content of the corresponding time and attribute in the evaluation table 501.

[0442] If the judgment unit 116 determines that the read weight is extremely small, it writes an "X" indicating that the reliability of the attribute content is extremely low, to the attribute content of the corresponding time and attribute in the evaluation table 501.

[0443] (Judgment of the best scene) Next, the determining unit 116 selects the attribute content that has been assigned a "◯" indicating the highest reliability in the evaluation table 501 to which the reliability has been assigned.

[0444] 18, the reliability of "○" is assigned to the attribute "whether glasses are worn" from time "0" to time "2." Therefore, the determination unit 116 writes "0 to 2," indicating the time from time "0" to time "2," in the best scene corresponding to the attribute "whether glasses are worn" in the evaluation table 501.

[0445] 18, the reliability of the attribute "upper body clothing" is assigned with "○" from time "0" to time "2." Therefore, the determining unit 116 writes "0 to 2" indicating the time "0" to time "2" in the best scene corresponding to the attribute "upper body clothing" in the evaluation table 501.

[0446] 18, the reliability of the attribute "clothing for lower body" is assigned with "○" from time "0" to time "2." Therefore, the determining unit 116 writes "0 to 2" indicating the time "0" to time "2" in the best scene corresponding to the attribute "clothing for lower body" in the evaluation table 501.

[0447] 18, the reliability of the attribute "whether the user is carrying a backpack" is assigned a "yes" rating from time "6" to time "8." Therefore, the determining unit 116 writes "6-8," indicating the period from time "6" to time "8," in the best scene corresponding to the attribute "whether the user is carrying a backpack" in the evaluation table 501.

[0448] (Best scene selection) Next, after the selection of the attribute content with the highest reliability "○" is completed and the writing of the best scene for each attribute is completed, the determination unit 116 performs an OR operation on the multiple best scenes written in the best scene group 506 in the evaluation table 501 to determine the best scene.

[0449] According to the evaluation table 501 shown in FIG. 18, the best scenes "0-2," "6-8," "0-2," and "0-2" are written for the attributes "whether glasses are worn," "whether a backpack is carried," "upper body clothing," and "lower body clothing," respectively. Therefore, the determination unit 116 performs an OR operation on the best scenes "0-2," "6-8," "0-2," and "0-2," and determines the best scenes to be "0-2" and "6-8."

[0450] In this way, the determination unit 116 determines the best scene.

[0451] In this way, the determination unit 116 determines the frame image that is the best scene for each section.

[0452] (summary) As described above, weights can be set using the relationship between orientation and attribute and occlusion by objects, and the set weights can be used to determine whether a frame image is the best scene suitable for extracting feature amounts (attributes) of an object. In this way, the feature amounts (attributes) of objects included in a frame image determined to be the best scene are output, thereby improving the accuracy of feature amounts (attributes) extracted from objects.

[0453] 10 Example 10 Here, a tenth embodiment will be described as a modification of the ninth embodiment.

[0454] In the tenth embodiment, the weight table 511 shown in FIG. 19 is generated in the same manner as in the ninth embodiment, and the generated weight table 511 is used to select the best scene.

[0455] (Judgment unit 116) Using the generated weight table 511, the determination unit 116 reads the weights of the attribute contents of the attribute "whether the person wears glasses," the attribute "whether the person carries a backpack," the attribute "clothing for the upper body," and the attribute "clothing for the lower body" from the weight table 511, and determines whether the read weights are equal to or greater than a predetermined threshold value or less than a predetermined threshold value. Here, the predetermined threshold value is, for example, "0.5."

[0456] If it is determined that the weight value of each of the attribute contents of the attributes "whether glasses are worn," "whether a backpack is carried," "upper body clothing," and "lower body clothing" is equal to or greater than a predetermined threshold, the determination unit 116 writes the time in question as the best scene for that attribute content into the best scene group 506 in the evaluation table 501.

[0457] The determination unit 116 performs an OR operation on the best scenes written in the best scene group 506 in the evaluation table 501 to determine the best scene.

[0458] (summary) As explained above, when it is determined that the weight of the attribute content is equal to or greater than a predetermined threshold, it is possible to determine whether the frame image is the best scene suitable for extracting the feature amounts (attributes) of the object. In this way, the feature amounts (attributes) of the object included in the frame image determined to be the best scene are output, thereby improving the accuracy of the feature amounts (attributes) extracted from the object.

[0459] (Variation) A modification of the tenth embodiment will now be described.

[0460] In this modification, as in the tenth embodiment, the weight table 511 shown in FIG. 19 is generated.

[0461] Here, in weight table 511, time "0" to time "2" constitute one interval (first interval), time "3" to time "5" constitute one interval (second interval), and time "6" to time "8" constitute one interval (third interval).

[0462] The determination unit 116 may determine that the frame image in the section corresponding to the maximum weight among the weights of the attribute contents in the weight table 511 in each of the multiple sections (first, second, and third sections) is the best scene.

[0463] Specifically, according to evaluation table 501, for the attribute "whether glasses are worn," the attribute content weight of "1" at time "0" is the maximum value within the first interval. (Note that there are multiple images tied for first place within the first interval.) Therefore, the determination unit 116 determines that the frame image corresponding to time "0" is the best scene within the first interval.

[0464] Furthermore, according to evaluation table 501, for the attribute "whether the person is carrying a backpack," the attribute content weight of "1" at time "6" is the maximum value within the third interval. (Note that there are multiple scenes tied for first place within the third interval.) Therefore, the determination unit 116 determines that the frame image corresponding to time "6" is the best scene within the third interval.

[0465] In this way, it is possible to determine whether the frame image is the best scene suitable for extracting the feature amount (attribute) of the object.

[0466] 11 Example 11 Here, an eleventh embodiment will be described as a modification of the ninth embodiment.

[0467] In the ninth embodiment, the recognition processing unit 124, the recognition result acquisition unit 113, and the determination unit 116 generate an evaluation table 501 shown in FIG. 18, and the determination unit 116 uses the evaluation table 501 to select the best scene.

[0468] In Example 11, instead of detecting "yes," "no," "green," "purple," etc. as the attribute contents of the attributes "whether glasses are worn," "whether a backpack is carried," "upper body clothing," and "lower body clothing" in the evaluation table 501, the recognition processing unit 124 detects quantified values ​​of "yes," "no," "green," "purple," etc. as feature quantities.

[0469] For example, for the attribute "Does the person wear glasses?", the recognition processing unit 124 detects feature amounts such as "0.9", "0.8", ..., "0.2", and "0.1" instead of "yes" and "no". These values ​​indicate the likelihood of "wearing glasses". The larger the value, the stronger the likelihood, and the smaller the value, the weaker the likelihood.

[0470] Furthermore, for the attribute "Is the person carrying a backpack?", the recognition processing unit 124 detects feature amounts such as "0.9", "0.8", ..., "0.2", and "0.1" instead of "yes" and "no". These values ​​also indicate the likelihood of "carrying a backpack".

[0471] The same applies to the attribute "upper body clothing" and the attribute "lower body clothing."

[0472] In this way, the recognition processing unit 124 detects the feature amount of the object as the recognition result from each frame image.

[0473] In this way, evaluation table 531 shown in Fig. 20 is generated. Evaluation table 531 has almost the same data structure as evaluation table 501 shown in Fig. 18. Evaluation table 531 differs from evaluation table 501 in that the attribute contents of the attributes "whether glasses are worn," "whether a backpack is carried," "upper body clothing," and "lower body clothing" are quantified values.

[0474] In this case, different features are detected for each section. For example, for the attribute "clothing for the lower body," a feature indicating "green" is detected in the section of photographing time "0 to 1," a feature indicating "purple" is detected in the section of photographing time "3 to 5," and a feature indicating "green" is detected in the section of photographing time "6 to 8."

[0475] Next, the determination unit 116 sets weights for the detected features for each time and attribute content (attribute content of the attribute "whether the person is wearing glasses," the attribute "whether the person is carrying a backpack," the attribute "clothing for the upper body," and the attribute "clothing for the lower body"), as shown in the weight table 511 in FIG. 19.

[0476] In this way, the determination unit 116 sets a weight for each section using the detected feature amount.

[0477] Next, the determination unit 116 determines whether the weight set for each time and attribute content is equal to or greater than a predetermined threshold, or less than a predetermined threshold, where the predetermined threshold is, for example, "0.5."

[0478] If it is determined that the set weight is equal to or greater than a predetermined threshold, the determining unit 116 writes the time in question into the best scene group 536 as the best scene in the attribute content.

[0479] In this way, the determination unit 116 sets the weight using the weighted feature amount, and determines whether or not the frame image is the best scene using the set weight.

[0480] Next, the determination unit 116 performs an OR operation on the best scenes written in the best scene group 536 in the evaluation table 531 to determine the best scene.

[0481] (Variation 1) In the above, weights are set using feature amounts, and when it is determined that the set weights are equal to or greater than a predetermined threshold, it is determined that the scene is the best scene.

[0482] However, this is not limiting.

[0483] Whether to adopt or reject a detected feature may be determined based on a corresponding weight. For example, the determination unit 116 may adopt the detected feature if the weight corresponding to the detected feature is equal to or greater than a threshold value (e.g., 0.5), and may reject the detected feature if the corresponding weight is less than the threshold value (e.g., 0.5).

[0484] The determination unit 116 may determine that the frame image corresponding to the adopted feature amount is the best scene.

[0485] (Variation 2) You can also do it like this:

[0486] The determination unit 116 sets the weight wn(t) as shown in (Equation 5) by referring to the weight table 511 shown in FIG.

[0487]

number

[0488] According to (Equation 5), if each weight wn in the weight table 511 is less than the threshold value t, the weight wn is set to "0", and if each weight wn is equal to or greater than the threshold value t, the value of the weight wn is not changed.

[0489] Here, for example, assuming that the threshold value t is 0.8, the judgment unit 116 applies (Equation 5) to the weight table 511, and keeps only the weight "1" in the weight table 511 as it is, and changes the other weights such as "0.5", "0.25", etc. to "0".

[0490] Next, the judgment unit 116 selects a feature from the evaluation table 531 corresponding to a weight greater than or equal to a threshold value t (e.g., "0.8") in the weight table 511 to which (Equation 5) has been applied, and determines the time corresponding to the selected feature, i.e., the frame image, as the best scene.

[0491] In this case, the determination unit 116 determines the frame images corresponding to times "0" to "2" and "6" to "8" as the best scenes.

[0492] (Variation 3) In Example 11, the recognition processing unit 124 detects the quantified values ​​of the attribute contents of the attribute "whether the person is wearing glasses," the attribute "whether the person is carrying a backpack," the attribute "clothing for the upper body," and the attribute "clothing for the lower body" in the evaluation table 501, such as "yes," "no," "green," and "purple," as feature quantities.

[0493] However, the present invention is not limited to this method.

[0494] The recognition processing unit 124 may output a representation of a plurality of attributes as a feature of an object in the form of a multidimensional vector (for example, a 128-dimensional vector).

[0495] (summary) As described above, it is possible to determine whether a frame image is the best scene suitable for extracting the feature amount (attribute) of an object.

[0496] 12 Example 12 Here, a twelfth embodiment, which is a modification of the tenth embodiment, will be described.

[0497] In the twelfth embodiment, the recognition processing unit 124, the recognition result acquisition unit 113, and the determination unit 116 generate an evaluation table 501 shown in FIG. 18 and a weight table 511 shown in FIG.

[0498] The judgment unit 116 refers to the evaluation table 501 shown in FIG. 18, acquires the attribute content for the attribute "clothing for the lower body," for example, and sums up the weights of the attribute content in the weight table 511 shown in FIG. 19 for each attribute content having the same attribute content.

[0499] Here, according to the evaluation table 501, the attribute contents of the attribute "clothing for the lower body" are "green" and "purple."

[0500] For the attribute "clothing for the lower body", the weights of the attribute contents in the weight table 511 corresponding to the attribute contents of "green" in the evaluation table 501 are "1" (time "0"), "1" (time "1"), "1" (time "2"), "0.5" (time "6"), "0.5" (time "7"), and "0.5" (time "8"), so the determination unit 116 adds up the weights (527a, 527c) of these attribute contents and determines We get 1+1+1+0.5+0.5+0.5=4.5 (first voting result).

[0501] On the other hand, for the attribute "clothing for the lower body", the weights of the attribute contents in the weight table 511 corresponding to the attribute contents of "purple" in the evaluation table 501 are "0" (time "3"), "0" (time "4"), and "0" (time "5"), so the determination unit 116 adds up the weights (527b) of these attribute contents and determines We get 0+0+0=0 (second voting result).

[0502] The determination unit 116 compares the first voting result with the second voting result and selects the larger one.

[0503] Here, the first voting result is "4.5" and the second voting result is "0", so the determining unit 116 selects the first voting result.

[0504] As a result, the determining unit 116 determines the frame images corresponding to the times "0 to 2" and "6 to 8" that are the basis of the attribute content "green" corresponding to the first voting result as the best scenes.

[0505] (Variation) The determining unit 116 refers to the weight table 511, acquires the weight of the attribute content for the attribute "clothing for the lower body", for example, and sums up the weights of the attribute content for each section.

[0506] Here, in weight table 511, time "0" to time "2" constitute one interval (first interval), time "3" to time "5" constitute one interval (second interval), and time "6" to time "8" constitute one interval (third interval).

[0507] The determination unit 116 sums up the weights of the attribute content of the attribute "lower body clothing" for the first interval. Specifically, for the attribute "lower body clothing," the weights of the attribute content in weight table 511 are "1" (time "0"), "1" (time "1"), and "1" (time "2"), so the determination unit 116 adds up the values ​​of these attribute content to obtain 1+1+1=3 (first voting result).

[0508] Furthermore, the determination unit 116 sums up the values ​​of the attribute content of the attribute "lower body clothing" for the second interval. Specifically, for the attribute "lower body clothing," the weights of the attribute content in weight table 511 are "0" (time "3"), "0" (time "4"), and "0" (time "5"), so the determination unit 116 adds up the values ​​of these attribute content to obtain 0+0+0=0 (second voting result).

[0509] Furthermore, the determination unit 116 sums up the values ​​of the attribute content of the attribute "lower body clothing" for the third interval. Specifically, for the attribute "lower body clothing," the weights of the attribute content in weight table 511 are "0.5" (time "6"), "0.5" (time "7"), and "0.5" (time "8"), so the determination unit 116 adds up the values ​​of these attribute content to obtain 0.5+0.5+0.5=1.5 (third voting result).

[0510] The determination unit 116 compares the first, second, and third voting results and selects the largest voting result.

[0511] Here, the first voting result is "3", the second voting result is "0", and the third voting result is "1.5", so the determining unit 116 selects the first voting result.

[0512] As a result, the determining unit 116 determines the frame image corresponding to the time period "0 to 2" that corresponds to the first voting result as the best scene.

[0513] In this way, the determining unit 116 adds up the weights for each attribute in each of the multiple sections, and determines that the frame image in the section in which the attribute with the largest added value is detected is the best scene.

[0514] (summary) As described above, by comparing a plurality of voting results, it is possible to determine whether a frame image is the best scene suitable for extracting the feature amount (attribute) of an object.

[0515] 14 Other variations Although the present disclosure has been described based on the above-described embodiment, it goes without saying that the present disclosure is not limited to the above-described embodiment.

[0516] (1) The above-described embodiments and modifications may be combined with each other. [Industrial Applicability]

[0517] The detection device according to the present disclosure can determine the best scene from multiple frame images that is suitable for extracting features of an object captured in the frame images, thereby achieving the excellent effect of improving the accuracy of features extracted from objects, and is useful as a technology for extracting features from objects. [Explanation of symbols]

[0518] 1. Surveillance System 5a~5c Camera 10. Information processing equipment 50 Neural Networks 50a Input layer 50b Feature extraction layer 50c recognition layer 101 CPU 102 ROM 103 RAM 104 Input / Output Circuit 105 Memory circuit 106 Input circuit 107 Bus 108 Network Communication Circuit 109 Bus 111 Main control unit 112 General control section 113 Recognition result acquisition unit 115 Object Tracking Unit 116 Judgment section 117 Output section 121 GPU 122 ROM 123 RAM 124 Recognition processing section 125 Memory circuit 127 2D skeleton detection unit

Claims

1. A detection device for detecting an object from a plurality of frame images, an acquisition means for acquiring a plurality of frame images; a recognition means for performing object recognition processing on each acquired frame image, and when an object is detected, performing recognition processing on the feature amount of the detected object; a determination means for determining whether one of the plurality of frame images is the best scene suitable for extracting a feature amount of an object, using the recognition result by the recognition means; an output means for outputting, when the frame image is determined to be the best scene, the feature amount of the object recognized in the frame image; Equipped with If the one frame image is determined not to be the best scene, the output means suppresses output of the feature amount of the object recognized in the frame image. A detection device characterized by:

2. A detection device for detecting an object from a plurality of frame images, an acquisition means for acquiring a plurality of frame images; a recognition means for performing object recognition processing on each acquired frame image, and when an object is detected, performing recognition processing on the feature amount of the detected object; a determination means for determining whether one of the plurality of frame images is the best scene suitable for extracting a feature amount of an object, using the recognition result by the recognition means; an output means for outputting, when the frame image is determined to be the best scene, the feature amount of the object recognized in the frame image; Equipped with The determining means sets a weight using the recognition result by the recognizing means, and determines whether the one frame image is the best scene using the set weight. A detection device characterized by:

3. The determining means determines that the frame image is a best scene when the set weight is equal to or greater than a predetermined threshold, and determines that the frame image is not a best scene when the set weight is less than the predetermined threshold.

3. The detection device according to claim 2.

4. When two objects are detected by the recognition means, the determination means determines whether or not at least a part of the first object is occluded by the second object, and sets the weight in accordance with a determination result indicating whether or not the first object is occluded.

4. The detection device according to claim 3.

5. The determining means sets the weight by a first weight value when determining that at least a part of the first object is occluded, and sets the weight by a second weight value greater than the first weight value when determining that the first object is not occluded.

5. The detection device according to claim 4.

6. The determining means determines whether or not there is an overlapping area between a rectangular area surrounding the first object and a rectangular area surrounding the second object, and if it determines that there is an overlapping area, determines that a part of the first object is occluded in the overlapping area.

5. The detection device according to claim 4.

7. The determining means determines whether a distance between a rectangular area surrounding the first object and a rectangular area surrounding the second object is within a predetermined value, and if it is determined that the distance is within the predetermined value, determines that a part of the first object is occluded in the overlapping area.

5. The detection device according to claim 4.

8. Further, a storage means is provided for storing skeleton definition information that defines the shape of the skeleton of a specific type of object, When an object is detected, the recognition means detects skeleton information indicating a skeleton of each detected object from the object using skeleton definition information; The determining means determines whether or not there is a missing part or a predetermined or greater deformation in the skeleton indicated by the skeleton information detected from the first object, with reference to the skeleton definition information, and determines that at least a part of the first object is occluded if it is determined that there is a missing part or a predetermined or greater deformation.

5. The detection device according to claim 4.

9. the recognition means performs object recognition processing on each frame image to detect a plurality of partial objects constituting the object, and detects, as feature quantities of the object, the direction in which a main surface of the partial object faces and attributes of the object; The determining means sets the weight in accordance with the relationship between the detected direction and the attribute.

4. The detection device according to claim 3.

10. The determination means determining whether the detected direction is a front direction of the frame image, and determining whether there is a relationship between the detected partial object and an attribute of the detected object; If it is determined that the detected direction is not the front, or if it is determined that there is no association between the partial object and the attribute of the detected object, it sets the weight by a first weight value; If it is determined that the detected direction is the front and that there is a relationship between the partial object and the attribute of the detected object, the weight is set to a second weight value that is greater than the first weight value.

10. The detection device according to claim 9.

11. Further, a storage means is provided for storing skeleton definition information that defines the shape of the skeleton of a specific type of object, When an object is detected, the recognition means uses skeleton definition information to detect skeleton information indicating the skeleton of each detected object, and uses the detected skeleton information to detect the direction in which the main surface of the partial object is facing.

10. The detection device according to claim 9.

12. the recognition means detects the size of the object as a feature of the object; The determining means sets the weight in accordance with a relationship between the detected size and a size threshold.

4. The detection device according to claim 3.

13. The determination means comparing the detected size with the size threshold; if the size is less than the size threshold, setting the weight by a first weight value; If the size is equal to or greater than the size threshold, set the weight by a second weight value greater than the first weight value.

13. The detection device according to claim 12.

14. Further, a storage means is provided for storing skeleton definition information that defines the shape of the skeleton of a specific type of object, When an object is detected, the recognition means uses the skeleton definition information to detect skeleton information indicating the skeleton of each detected object, and detects the size of the object using the detected skeleton information.

13. The detection device according to claim 12.

15. the plurality of frame images are generated in chronological order, the recognition means detects a change in position or a change in size of the object over time as a feature of the object; The determining means sets the weight in accordance with a relationship between an amount of change in position or an amount of change in size of the detected object and an allowable range.

4. The detection device according to claim 3.

16. The determination means comparing the detected change in position or size of the object with the tolerance; If the amount of change in the position or size of the object is outside the allowable range, set the weight by a first weight value; If the amount of change in the position or size of the object is within the allowable range, the weight is set to a second weight value that is greater than the first weight value.

16. The detection device according to claim 15.

17. Further, a storage means is provided for storing skeleton definition information that defines the shape of the skeleton of a specific type of object, When an object is detected, the recognition means uses the skeleton definition information to detect skeleton information indicating the skeleton of each detected object, and uses the detected skeleton information to detect the amount of change in position or size of the object.

16. The detection device according to claim 15.

18. the recognition means detects a position occupied by the object within the frame image as a feature of the object; The determining means sets the weight in accordance with the detected position and position range.

4. The detection device according to claim 3.

19. The determination means comparing the detected object position with the position range; If the object's position is within the range of positions, setting the weight by a first weight value; If the object's position is outside the position range, set the weight by a second weight value that is greater than the first weight value.

20. The detection device according to claim 18.

20. Further, a storage means is provided for storing skeleton definition information that defines the shape of the skeleton of a specific type of object, When an object is detected, the recognition means uses the skeleton definition information to detect skeleton information indicating the skeleton of each detected object, and uses the detected skeleton information to detect the position occupied by the object within the frame image.

20. The detection device according to claim 18.

21. the plurality of frame images are generated in chronological order, Each of a plurality of time-series sections includes one or a plurality of frame images, The determining means determines the frame image that is the best scene for each section.

4. The detection device according to claim 3.

22. the recognition means detects a feature amount of an object from each frame image as the recognition result; Different features are detected for each section, The determining means sets a weight for each section using the detected feature amount, and determines whether or not the scene is the best scene using the set weight.

22. The detection device according to claim 21 .

23. The determining means determines that the frame image in the section corresponding to the largest weight among the weights in each of the plurality of sections is the best scene.

23. The detection device according to claim 22.

24. the recognition means detects an attribute of an object from each frame image as the recognition result; For each section, attributes are detected, The determining means adds up weights for each attribute in each of the plurality of sections, and determines that the frame image in the section in which the attribute with the largest added value is detected is the best scene.

23. The detection device according to claim 22.

25. A detection device for detecting an object from a plurality of frame images, an acquisition means for acquiring a plurality of frame images; a recognition means for performing object recognition processing on each acquired frame image, and when an object is detected, performing recognition processing on the feature amount of the detected object; a determination means for determining whether one of the plurality of frame images is the best scene suitable for extracting a feature amount of an object, using the recognition result by the recognition means; an output means for outputting, when the frame image is determined to be the best scene, the feature amount of the object recognized in the frame image; Equipped with the recognition means detects a shape of the object as a feature of the object; The determining means determines whether the scene is the best scene depending on whether the detected shape is appropriate. A detection device characterized by:

26. The recognition means recognizes an object from an image and recognizes a feature from the object using artificial intelligence.

26. The detection device of claim 1, 2 or 25.

27. A detection method used in a detection device that detects an object from a plurality of frame images, comprising: an acquisition step of acquiring a plurality of frame images; a recognition step of performing object recognition processing on each acquired frame image, and when an object is detected, performing recognition processing on the feature amount of the detected object; a determination step of determining whether or not one frame image among the plurality of frame images is a best scene suitable for extracting a feature amount of an object, using a recognition result from the recognition step; an output step of outputting, when the frame image is determined to be the best scene, a feature amount of an object recognized in the frame image; Including, If it is determined that the one frame image is not the best scene, the output step suppresses output of the feature amount of the object recognized in the frame image. A detection method characterized by:

28. A detection method used in a detection device that detects an object from a plurality of frame images, comprising: an acquisition step of acquiring a plurality of frame images; a recognition step of performing object recognition processing on each acquired frame image, and when an object is detected, performing recognition processing on the feature amount of the detected object; a determination step of determining whether or not one frame image among the plurality of frame images is a best scene suitable for extracting a feature amount of an object, using a recognition result from the recognition step; an output step of outputting, when the frame image is determined to be the best scene, a feature amount of an object recognized in the frame image; Including, The determining step sets a weight using the recognition result of the recognizing step, and determines whether the one frame image is the best scene using the set weight. A detection method characterized by:

29. A detection method used in a detection device that detects an object from a plurality of frame images, comprising: an acquisition step of acquiring a plurality of frame images; a recognition step of performing object recognition processing on each acquired frame image, and when an object is detected, performing recognition processing on the feature amount of the detected object; a determination step of determining whether or not one frame image among the plurality of frame images is a best scene suitable for extracting a feature amount of an object, using a recognition result from the recognition step; an output step of outputting, when the frame image is determined to be the best scene, a feature amount of an object recognized in the frame image; Including, the recognition step detects a shape of the object as a feature of the object; The determining step determines whether the scene is the best scene depending on whether the detected shape is appropriate. A detection method characterized by:

30. A computer-readable storage medium storing a computer program used in a detection device that detects an object from a plurality of frame images, The detection device is a computer. an acquisition step of acquiring a plurality of frame images; a recognition step of performing object recognition processing on each acquired frame image, and when an object is detected, performing recognition processing on the feature amount of the detected object; a determination step of determining whether or not one frame image among the plurality of frame images is a best scene suitable for extracting a feature amount of an object, using a recognition result from the recognition step; an output step of outputting, when the frame image is determined to be the best scene, a feature amount of an object recognized in the frame image; It stores a computer program to execute If it is determined that the one frame image is not the best scene, the output step suppresses output of the feature amount of the object recognized in the frame image. A storage medium characterized by:

31. A computer-readable storage medium storing a computer program used in a detection device that detects an object from a plurality of frame images, The detection device is a computer. an acquisition step of acquiring a plurality of frame images; a recognition step of performing object recognition processing on each acquired frame image, and when an object is detected, performing recognition processing on the feature amount of the detected object; a determination step of determining whether or not one frame image among the plurality of frame images is a best scene suitable for extracting a feature amount of an object, using a recognition result from the recognition step; an output step of outputting, when the frame image is determined to be the best scene, a feature amount of an object recognized in the frame image; It stores a computer program to execute The determining step sets a weight using the recognition result of the recognizing step, and determines whether the one frame image is the best scene using the set weight. A storage medium characterized by:

32. A computer-readable storage medium storing a computer program used in a detection device that detects an object from a plurality of frame images, comprising: The detection device is a computer. an acquisition step of acquiring a plurality of frame images; a recognition step of performing object recognition processing on each acquired frame image, and when an object is detected, performing recognition processing on the feature amount of the detected object; a determination step of determining whether or not one frame image among the plurality of frame images is a best scene suitable for extracting a feature amount of an object, using a recognition result from the recognition step; an output step of outputting, when the frame image is determined to be the best scene, a feature amount of an object recognized in the frame image; It stores a computer program to execute the recognition step detects a shape of the object as a feature of the object; The determining step determines whether the scene is the best scene depending on whether the detected shape is appropriate. A storage medium characterized by:

Images