Information processing device, information processing method, and program

Abstract

To detect belongings of a person from a captured image with high accuracy.SOLUTION: An information processing apparatus of the present invention includes a first detection unit, a second detection unit, and a specifying unit. The first detection unit detects a plurality of image regions including a target object that is a detection target from a captured image using a detector that has learned images. The second detection unit detects, in the captured image, body parts of a person appearing in the captured image. The specifying unit specifies a part of the plurality of body parts in which the target object is present, based on the image regions and the body parts. The target object is belongings of the person. The type of target object differs for each image region.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to object recognition.

Background Art

[0002] Techniques for detecting an object from a captured image generated by a camera have been developed. For example, Patent Document 1 discloses an apparatus that detects a plurality of objects from a captured image and associates the detected objects. Specifically, when the captured image includes an object (e.g., a bag) and a plurality of persons, the apparatus of Patent Document 1 associates the object with the owner person.

[0003] For this purpose, the apparatus of Patent Document 1 uses a predefined connection relationship and connects while sequentially recognizing the parts of a person. For example, recognition and connection are performed in the order of face -> neck -> torso -> arm. Further, the apparatus of Patent Document 1 recognizes an object predefined as an object that frequently exists around the recognized part. For example, a bag is defined as an object that frequently exists around an arm. Therefore, when the arm of a person is recognized as described above, the bag is recognized. As a result, it can be seen that they are connected as "face -> neck -> torso -> arm -> bag". Therefore, the apparatus of Patent Document 1 associates the connected face and bag (that is, associates the person and the bag).

[0004] Here, in Patent Document 1, information for estimating approximately where around the part an object that frequently exists around the part of a person is located is defined. And Patent Document 1 describes that this information may be used to limit the image area for recognizing the object. For example, when the apparatus of Patent Document 1 detects the arm of a person in the above-described flow, it uses information indicating approximately where a bag exists relative to the arm of the person to limit the image area for recognizing the bag. Then, the bag is recognized for the limited image area.

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2010-086482 [Non-patent literature]

[0006] [Non-Patent Document 1] Zhe Cao, 3 others, "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", CoRR, November 24, 2016 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] In the technology described in Patent Document 1, the recognition of objects that frequently exist around a person's body part is performed on the premise that a body part of that person has been recognized. Therefore, if a body part of a person is not recognized, objects that frequently exist around that body part will not be recognized. For example, in the above example, if the person's arm is not recognized, the bag will not be recognized.

[0008] In this regard, not all parts of a person are necessarily included in the captured image. For example, if there is an obstacle in front of a person's arm, or if the person's arm is outside the camera's field of view, the person's arm will not be recognized in the captured image, and therefore the bag will not be recognized either.

[0009] The present invention has been made in view of the above-mentioned problems, and aims to provide a technology for detecting a person's belongings from captured images with high accuracy. [Means for solving the problem]

[0010] The information processing apparatus of the present invention comprises: 1) a candidate region detection unit that detects one or more candidate regions, which are image regions estimated to represent a target object, from an captured image based on the image features of the target object to be detected; 2) an estimated position detection unit that detects a person region representing a person from an captured image and detects the estimated position of the target object in the captured image based on the person region; and 3) an object region, which is an image region representing a target object, from among one or more candidate regions based on one or more candidate regions and the estimated position.

[0011] The control method of the present invention is a control method performed by a computer. The control method comprises: 1) a candidate region detection step of detecting one or more candidate regions, which are image regions estimated to represent the target object, from an captured image based on the image features of the target object to be detected; 2) an estimated position detection step of detecting a person region representing a person from an captured image and detecting the estimated position of the target object in the captured image based on the person region; and 3) an object region, which is an image region representing the target object, from among one or more candidate regions based on one or more candidate regions and the estimated position.

[0012] The program of the present invention causes a computer to perform each step of the control method of the present invention. [Effects of the Invention]

[0013] According to the present invention, a technology is provided for detecting a person's belongings from captured images with high accuracy. [Brief explanation of the drawing]

[0014] The aforementioned objectives, as well as other objectives, features, and advantages, will become even clearer from the preferred embodiments described below and the accompanying drawings.

[0015] [Figure 1] This diagram conceptually illustrates the processing performed by the information processing device of this embodiment. [Figure 2]This is a diagram illustrating the functional configuration of the information processing apparatus according to Embodiment 1. [Figure 3] This is a diagram illustrating a computer for realizing the information processing apparatus. [Figure 4] This is a flowchart illustrating the flow of processing executed by the information processing apparatus according to Embodiment 1. [Figure 5] This is a diagram illustrating a candidate region including an estimated position. [Figure 6] This is a diagram illustrating a first score calculated based on the number of estimated positions included in the candidate region. [Figure 7] This is a diagram illustrating a first score calculated in consideration of the existence probability of the target object calculated for the estimated position. [Figure 8] This is a block diagram illustrating the functional configuration of the information processing apparatus according to Embodiment 2. [Figure 9] This is a flowchart illustrating the flow of processing executed by the information processing apparatus according to Embodiment 2.

Embodiments of the Invention

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same reference numerals are assigned to the same components, and the description will be omitted as appropriate. Also, unless otherwise specified, in each block diagram, each block represents a configuration of a functional unit, not a configuration of a hardware unit.

[0017] [Embodiment 1] [Overview] FIG. 1 is a diagram conceptually showing the processing performed by the information processing apparatus 2000 of the present embodiment. The information processing apparatus 2000 detects an object region 30, which is an image region representing a target object, from the captured image 20 generated by the camera 10. The target object is an item carried by a person. Here, the "item carried by a person" is not limited to an item held in the hand by a person (such as a handbag or a cane), but generally includes any item possessed by a person in some form. For example, items carried by a person include an item hung on the shoulder by a person (such as a shoulder bag), an item held by the person from the neck (such as an identity card), an item carried on the back by a person (such as a backpack), an item worn on the head by a person (such as a hat or a helmet), an item worn on the face by a person (such as glasses or sunglasses), and an item worn on the hand by a person (such as a watch), etc.

[0018] The information processing apparatus 2000 detects one or more candidate regions 22 from the captured image 20 based on the image features of the target object. The candidate region 22 is an image region estimated to represent the target object. For example, if the target object is a hat, the information processing apparatus 2000 detects an image region estimated to represent the hat based on the image features of the hat, and sets the detected image region as the candidate region 22. The candidate region 22 is, for example, an image region where the probability of representing the target object is estimated to be a predetermined value or more.

[0019] Also, the information processing apparatus 2000 detects a person region 26 from the captured image 20, and detects an estimated position 24 based on the detected person region 26. The person region 26 is a region estimated to represent a person. The estimated position 24 is a position in the captured image 20 where the target object is estimated to exist.

[0020] Here, for an item carried by a person, the relative positional relationship with the person can be predicted according to the type of the item. For example, the position of a hat is highly likely to be on the head of a person. In addition, for example, the position of sunglasses is highly likely to be on the face part of a person. In addition, for example, the position of a backpack is highly likely to be on the back part of a person.

[0021] The information processing device 2000 then detects the estimated position 24 based on the relative positional relationship between the predictable target object and the person. For example, if the target object is a hat, the information processing device 2000 detects the estimated position where the hat is located based on the relative positional relationship between the person represented by the person region 26 and the hat, and sets the detected position as the estimated position 24.

[0022] The information processing device 2000 then identifies the object region 30 based on the candidate region 22 and the estimated position 24. For example, the information processing device 2000 identifies the candidate region 22 containing the estimated position 24 as the object region 30 from among the multiple detected candidate regions 22. However, as will be described later, the object region 30 identified based on the candidate region 22 and the estimated position 24 is not limited to the candidate region 22 containing the estimated position 24.

[0023] <Effects and Actions> According to the information processing device 2000 of this embodiment, an object region 30 representing a target object is identified using candidate regions 22 detected based on the image features of the target object and estimated positions 24 detected based on the person region 26. In this way, not all candidate regions 22 detected based on the image features of the target object are identified as object regions 30 (image regions representing the target object). Instead, the candidate regions 22 identified as object regions 30 are limited by the estimated positions 24 detected based on the person region 26. For example, candidate regions 22 at locations where the probability of the target object being present is low will not be identified as object regions 30. In this way, by identifying the image region representing the target object using two criteria—the image features of the target object and the image region representing the person—the image region representing the target object can be identified with higher accuracy compared to the case where only the image features of the target object are used as the criterion.

[0024] Here, the estimated position 24 of the target object is detected using the image region representing the person. Therefore, even if some parts of the person (for example, an arm) are not detected in the captured image 20, the estimated position 24 can still be detected. Thus, according to the information processing device 2000, even if some parts of the person are not included in the captured image 20, the object region 30 can be identified.

[0025] The above explanation, with reference to Figure 1, is merely an example to facilitate understanding of the information processing device 2000 and does not limit its functions. The information processing device 2000 of this embodiment will now be described in more detail.

[0026] <Example of the functional configuration of the information processing device 2000> Figure 2 illustrates the functional configuration of the information processing device 2000 of Embodiment 1. The information processing device 2000 includes a candidate region detection unit 2020, an estimated position detection unit 2040, and a identification unit 2060. The candidate region detection unit 2020 detects one or more candidate regions 22 from the captured image 20 based on the image features of the target object to be detected. The estimated position detection unit 2040 detects a person region 26 from the captured image 20. Furthermore, the estimated position detection unit 2040 detects an estimated position 24 based on the detected person region 26. The identification unit 2060 identifies an object region 30 based on the candidate regions 22 and the estimated position 24.

[0027] <Hardware configuration of the information processing device 2000> Each functional component of the information processing device 2000 may be implemented by hardware (e.g., hardwired electronic circuits) or by a combination of hardware and software (e.g., a combination of electronic circuits and programs that control them). The following will further explain the case where each functional component of the information processing device 2000 is implemented by a combination of hardware and software.

[0028] Figure 3 illustrates a computer 1000 for realizing the information processing device 2000. Computer 1000 is any computer. For example, computer 1000 is a stationary computer such as a personal computer (PC) or a server machine. Alternatively, computer 1000 could be a portable computer such as a smartphone or a tablet terminal. Alternatively, computer 1000 could be a camera 10 that generates the captured image 20. Computer 1000 may be a dedicated computer designed to realize the information processing device 2000, or it may be a general-purpose computer.

[0029] Computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, memory 1060, storage device 1080, input / output interface 1100, and network interface 1120 to send and receive data from each other. However, the method of connecting the processor 1040 and other components is not limited to bus connection.

[0030] Processor 1040 is a variety of processors such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), and FPGA (Field-Programmable Gate Array). Memory 1060 is main memory implemented using RAM (Random Access Memory), etc. Storage device 1080 is auxiliary storage implemented using hard disk, SSD (Solid State Drive), memory card, or ROM (Read Only Memory), etc.

[0031] The input / output interface 1100 is an interface for connecting the computer 1000 to input / output devices. For example, input devices such as keyboards and output devices such as display devices are connected to the input / output interface 1100. The network interface 1120 is an interface for connecting the computer 1000 to a communication network. This communication network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method by which the network interface 1120 connects to the communication network may be wireless or wired.

[0032] The storage device 1080 stores program modules that realize each functional component of the information processing device 2000. The processor 1040 reads these program modules into the memory 1060 and executes them to realize the functions corresponding to each program module.

[0033] <About Camera 10> Camera 10 is any imaging device that takes images and generates image data as a result. For example, camera 10 is a surveillance camera installed at a monitoring location.

[0034] As mentioned above, the computer 1000 that implements the information processing device 2000 may be a camera 10. In this case, the camera 10 identifies the object region 30 by analyzing the captured image 20 that it generates. As a camera 10 with such functionality, for example, an intelligent camera, a network camera, or an IP (Internet Protocol) camera can be used.

[0035] <Examples of using the Information Processing System 2000> The information processing device 2000 can be used in any situation where the process of "detecting a predetermined object from image data generated by a camera" is useful. For example, the information processing device 2000 can be used to analyze surveillance video generated by a surveillance camera. In this case, camera 10 is a surveillance camera that generates surveillance video, and captured image 20 is a video frame that constitutes the surveillance video.

[0036] The information processing device 2000 identifies an image region representing a predetermined object (i.e., an object region 30 representing the target object) from the video frames that make up the surveillance video. In this way, the presence of the target object at the surveillance location can be determined. It can also detect a person possessing the target object.

[0037] More specifically, according to the information processing device 2000, surveillance footage can be used to identify the presence of dangerous objects or suspicious individuals (such as people carrying dangerous objects or people concealing their faces with sunglasses or helmets). Furthermore, if an abandoned object is found at a monitored facility, the information processing device 2000 can analyze past surveillance footage generated by surveillance cameras installed in various locations within the facility to identify the route the abandoned object was transported on and to detect the person who transported it.

[0038] <Processing flow> Figure 4 is a flowchart illustrating the processing flow performed by the information processing device 2000 of Embodiment 1. The information processing device 2000 acquires an image 20 (S102). The candidate region detection unit 2020 detects one or more candidate regions 22 from the image 20 based on the image features of the target object (S104). The estimated position detection unit 2040 detects a person region 26 from the image 20 (S106). The estimated position detection unit 2040 detects an estimated position 24 based on the detected person region 26 (S108). The identification unit 2060 identifies an object region 30 based on the candidate regions 22 and the estimated position 24 (S110).

[0039] It should be noted that not all processes must necessarily be executed sequentially, as shown in Figure 4. For example, the process executed by the candidate region detection unit 2020 (S104) and the processes executed by the estimated position detection unit 2040 (S106 and S108) may be executed in parallel.

[0040] The timing at which the information processing device 2000 executes the series of processes shown in Figure 4 varies. For example, each time an image 20 is generated by the camera 10, the information processing device 2000 acquires the image 20 and executes the series of processes shown in Figure 4. Alternatively, for example, the information processing device 2000 may acquire multiple images 20 generated by the camera 10 at a predetermined timing and execute the series of processes shown in Figure 4 for each image 20 (so-called batch processing). Alternatively, for example, the information processing device 2000 may accept an input operation to specify an image 20 and execute a series of processes for the specified image 20.

[0041] <Acquisition of captured image 20: S102> The information processing device 2000 acquires the captured image 20 (S102). The captured image 20 may be the image data generated by the camera 10 itself, or it may be the image data generated by the camera 10 to which some processing (for example, color correction or cropping) has been applied.

[0042] The method by which the information processing device 2000 acquires the captured image 20 is arbitrary. For example, the information processing device 2000 acquires the captured image 20 by accessing the storage device in which the captured image 20 is stored. The storage device in which the captured image 20 is stored may be located inside the camera 10 or outside the camera. Alternatively, for example, the information processing device 2000 may acquire the captured image 20 by receiving the captured image 20 transmitted from the camera 10. As mentioned above, if the information processing device 2000 is implemented as the camera 10, the information processing device 2000 acquires the captured image 20 that it has generated itself.

[0043] <Detection of candidate region 22: S104> The candidate region detection unit 2020 detects candidate regions 22 from the captured image 20 based on the image features of the target object (S104). Here, existing techniques can be used for detecting image regions (i.e., candidate regions 22) that are inferred to represent the object from the image data based on the image features of the object to be detected. For example, a detector that has been pre-trained to detect image regions estimated to represent the target object from the image data can be used to detect candidate regions 22. Any model can be used for the detector model, such as a neural network (e.g., a convolutional neural network) or an SVM (support vector machine).

[0044] Here, the candidate region detection unit 2020 detects image regions as candidate regions 22 in which the probability of representing the target object is greater than or equal to a threshold. If this threshold is increased, false negatives (failed detections) are more likely to occur. On the other hand, if this threshold is decreased, false positives (false detections) are more likely to occur.

[0045] In this regard, in the information processing device 2000, the object region 30 is not identified solely by the candidate region detection unit 2020, but also by using the estimated position 24 detected by the estimated position detection unit 2040. Therefore, it is preferable to set the threshold low and cause false detections rather than setting it high and causing detection failures. This is because by setting the threshold low and detecting a large number of candidate regions 22, and then using the estimated position 24 detected by the estimated position detection unit 2040 to narrow down the candidate regions 22, the object region 30 representing the target object can be identified with high accuracy.

[0046] Therefore, it is preferable that the threshold used by the candidate region detection unit 2020 be a value less than or equal to the threshold set when identifying the object region 30 based solely on the image features of the target object (i.e., when not using the estimated position detection unit 2040).

[0047] The candidate region detection unit 2020 generates data representing the detection result of the candidate region 22. This data identifies the detected candidate region 22 and indicates, for example, the specific position (e.g., the coordinates of the upper left corner) and size (e.g., width and height) of the candidate region 22.

[0048] <Detection of person region 26: S106> The estimated position detection unit 2040 detects a person region 26 from the captured image 20 (S106). Here, existing techniques can be used for detecting image regions representing people from image data. For example, a detector that has been pre-trained to detect image regions representing people from image data can be used. Any model, such as a neural network, can be used for this detector model.

[0049] In this case, to detect the estimated position 24, it is preferable to detect parts of the human body (head, face, torso, hands, and feet, etc.) from the human region 26. Parts of the human body can also be detected using the aforementioned detectors, or by using template images or local features.

[0050] In addition, the estimated position detection unit 2040 may also detect a set of characteristic points of a person, such as the positions of a person's joints, as the person region 26. For example, the technology shown in Non-Patent Document 1 can be used to detect the positions of characteristic points of a person, such as joints.

[0051] <Detection of estimated position 24: S106> The estimated position detection unit 2040 detects the estimated position 24 based on the person region 26. As described above, the estimated position 24 is the position in the captured image 20 where the target object is estimated to exist. The estimated position 24 may be represented as a single point on the captured image 20 or as an image region.

[0052] For example, to detect the estimated position 24, a detector that has been pre-trained to detect the estimated position where the target object exists in the image data can be used, based on the input image data in which the position of the image region representing a person is specified. Any model, such as a neural network, can be used for the detector model.

[0053] The detector is trained using training data consisting of, for example, combinations of "image data, the human region within that image data, and the position of the target object within that image data." By using such training data, the detector can learn the relative positional relationship between the target object and the person. Furthermore, it is preferable that the training data includes information indicating the position of each part of the person within the human region.

[0054] The estimated position detection unit 2040 detects a position where the probability of the target object being present is greater than or equal to a predetermined value, and defines that position as the estimated position 24. In this case, the estimated position detection unit 2040 may also output the probability of the target object being present at the estimated position 24, along with the estimated position 24.

[0055] For example, the estimated position detection unit 2040 generates matrix data of the same size as the captured image 20 as data representing the detection result of the estimated position 24. This matrix data, for example, shows 1 at the location of the estimated position 24 and 0 at other locations. Also, when outputting the probability that the target object exists at the estimated position 24, this matrix data shows the probability that the target object exists at each location. However, the data representing the detection result of the estimated position 24 can be in any format and is not limited to matrix data.

[0056] <<Limiting the detection range>> The estimated position detection unit 2040 may limit the image region in which it detects the estimated position 24 using the candidate region 22. That is, it detects the estimated position 24 from a portion of the image region limited based on the candidate region 22, rather than from the entire captured image 20. This reduces the time and computing resources required to detect the estimated position 24.

[0057] For example, the estimated position detection unit 2040 may define the detection range of the estimated position 24 as only the interior of the candidate region 22. Alternatively, the estimated position detection unit 2040 may detect the estimated position 24 from a predetermined range that includes the candidate region 22. For example, this predetermined range may be obtained by enlarging the candidate region 22 by a predetermined magnification greater than 1.

[0058] Furthermore, the estimated position detection unit 2040 may also limit the image area for detecting the person area 26 using the candidate area 22. For example, the estimated position detection unit 2040 detects the person area 26 from a predetermined range including the candidate area 22 (for example, a range obtained by enlarging the candidate area 22).

[0059] <Identifying object area 30> The identification unit 2060 identifies the object region 30 based on the candidate region 22 and the estimated position 24. Conceptually, the identification unit 2060 uses the estimated position 24 to identify a candidate region 22, which is an image region presumed to contain the target object, that has a particularly high probability of containing the target object, and identifies the identified candidate region 22 as the object region 30. However, as will be described later, the object region 30 does not need to perfectly match any one of the candidate regions 22, and may be a part of the image region of the candidate region 22.

[0060] The identification unit 2060 identifies the object region 30 by focusing on the overlap between the candidate region 22 and the estimated position 24. Various methods can be employed for this identification. Examples of such methods are given below.

[0061] <<Specific Method 1>> The identification unit 2060 identifies the candidate region 22 containing the estimated position 24 as the object region 30. Figure 5 is an example of a candidate region 22 containing the estimated position 24. In Figure 5, multiple candidate regions 22 are detected from the captured image 20. Also, one estimated position 24 is detected. Here, the estimated position 24 is included in candidate region 22-1. Therefore, the identification unit 2060 identifies candidate region 22-1 as the object region 30.

[0062] <<Specific Method 2>> Here, it is assumed that multiple estimated positions 24 are calculated. The identification unit 2060 then calculates a score (hereinafter referred to as the first score) for each candidate region 22 that represents the degree to which it contains the estimated position 24. Based on the first score, the identification unit 2060 identifies the object region 30.

[0063] There are various ways to identify the object region 30 based on the first score. For example, the identification unit 2060 identifies the candidate region 22 with the highest first score as the object region 30. Alternatively, the identification unit 2060 may identify the candidate region 22 with a first score equal to or greater than a predetermined value as the object region 30. In the latter case, multiple object regions 30 may be identified.

[0064] There are various ways to determine the first score. For example, the identification unit 2060 calculates the number of estimated positions 24 included in the candidate region 22 as the first score for that candidate region 22. Alternatively, the identification unit 2060 may calculate the number of estimated positions 24 included in the candidate region 22 as the first score for that candidate region 22, normalized by the size of the candidate region 22 (for example, the number of estimated positions 24 divided by the area of ​​the candidate region 22).

[0065] Figure 6 illustrates a first score calculated based on the number of estimated positions 24 included in the candidate region 22. The candidate region 22 contains three estimated positions 24. For example, the identification unit 2060 sets the first score of the candidate region 22 to 3. Here, let's assume the area of ​​the candidate region 22 is S. In this case, the identification unit 2060 may also use 3 / S as the first score, which is the first score of the candidate region 22 normalized by the area of ​​the candidate region 22.

[0066] The method for calculating the first score is not limited to the example described above. For example, suppose the probability of the target object existing for each estimated position 24 has been calculated. In this case, the identification unit 2060 may calculate the sum of the existence probabilities calculated for each estimated position 24 included in the candidate region 22 as the first score for that candidate region 22.

[0067] Figure 7 illustrates the first score calculated considering the probability of existence of the target object calculated for each estimated position 24. Candidate region 22 contains three estimated positions 24, and the probability of existence calculated for each is p1, p2, and p3. Therefore, the first score for candidate region 22 is p1 + p2 + p3.

[0068] In this way, by calculating the first score while considering the probability that the target object exists at the estimated position 24, the object region 30 representing the target object can be identified with higher accuracy. For example, a candidate region 22 containing one estimated position 24 with a probability of the target object's existence of 0.6 is more likely to be an image region representing the target object than a candidate region 22 containing three estimated positions 24 with a probability of the target object's existence of 0.1. According to the calculation method that uses the sum of existence probabilities as the first score, the first score of the latter candidate region 22 will be larger than the first score of the former candidate region 22. Therefore, the latter candidate region 22 is more likely to be identified as an object region 30.

[0069] <<Specific Method 3>> Here, the candidate region detection unit 2020 calculates the probability that each candidate region 22 represents the target object. The identification unit 2060 calculates the aforementioned first score for each candidate region 22. The identification unit 2060 calculates a second score as the product of the probability that the candidate region 22 represents the target object and the first score. The identification unit 2060 then identifies the object region 30 based on the second score.

[0070] There are various ways to identify the object region 30 based on the second score. For example, the identification unit 2060 identifies the candidate region 22 with the highest second score as the object region 30. Alternatively, the identification unit 2060 may identify the candidate region 22 with a second score equal to or greater than a predetermined value as the object region 30.

[0071] <<Specific Method 4>> The identification unit 2060 calculates a third score based on the distance between a representative point of the candidate region 22 and the estimated position 24, and identifies the object region 30 based on the calculated third score. There are various methods for identifying the object region 30 based on the third score. For example, the identification unit 2060 identifies the candidate region 22 for which the smallest third score was calculated as the object region 30. Alternatively, for example, the identification unit 2060 may identify the candidate region 22 for which a third score less than or equal to a predetermined value was calculated as the object region 30.

[0072] The representative point of candidate region 22 can be any point included in candidate region 22. For example, the representative point of candidate region 22 is the center of candidate region 22.

[0073] If there are multiple estimated positions 24, the identification unit 2060 may calculate the distance between each of the multiple estimated positions 24 and a representative point of the candidate region 22, or it may calculate the distance between any one of the estimated positions 24 and a representative point of the candidate region 22. In the former case, for example, the identification unit 2060 calculates a third score based on the statistical values ​​(minimum value, mode, or mean value, etc.) of the multiple distances calculated. In the latter case, the identification unit 2060 calculates the distance between one estimated position 24 and a representative point of the candidate region 22, and calculates a third score based on that distance.

[0074] In this case, when calculating the distance between only one estimated position 24 and a representative point of the candidate region 22, there are various methods for identifying that estimated position 24. For example, the identification unit 2060 calculates the center of an image region composed of multiple estimated positions 24, identifies the estimated position 24 closest to that center, and calculates the distance between the identified estimated position 24 and the representative point of the candidate region 22. Alternatively, for example, if the probability of the target object's existence is calculated for each estimated position 24, the identification unit 2060 calculates the distance between the estimated position 24 with the highest probability of the target object's existence and the representative point of the candidate region 22.

[0075] Furthermore, if the estimated position 24 is represented as an image region, the identification unit 2060 calculates a third score based on the distance between the representative point of that image region and the representative point of the candidate region 22. The representative point of the estimated position 24 represented as an image region is, for example, the center position of that image region.

[0076] There are various methods for calculating the third score based on the distance between the estimated position 24 and the representative point of the candidate region 22. For example, the identification unit 2060 uses the distance between the representative point of the candidate region 22 and the estimated position 24 as the third score.

[0077] In addition, for example, the identification unit 2060 uses a value obtained by multiplying the distance between the representative point of the candidate region 22 and the estimated position 24 by a correction coefficient based on the probability that the target object exists at the estimated position 24 as the third score. The correction coefficient is set to decrease as the probability that the target object exists at the estimated position 24 increases. For example, the correction coefficient is the reciprocal of the probability that the target object exists at the estimated position 24.

[0078] By considering the probability that the target object exists at the estimated position 24 in this way, the object region 30 representing the target object can be identified with higher accuracy. For example, a candidate region 22 that is 1 distance from the estimated position 24 where the probability of the target object exists is 0.6 is considered to have a higher probability of being an image region representing the target object than a candidate region 22 that is 1 distance from the estimated position 24 where the probability of the target object exists is 0.1. According to the method using the correction coefficient described above, the latter candidate region 22 will have a higher third score than the former candidate region 22. Therefore, the latter candidate region 22 is more likely to be identified as the object region 30.

[0079] <Output of results> The information processing device 2000 outputs information that identifies the object area 30 (hereinafter referred to as output information). There are various methods for outputting the output information. For example, the information processing device 2000 stores the output information in any storage device. Alternatively, for example, the information processing device 2000 stores the output information in a display device.

[0080] For example, the output information indicates the identifier of the captured image 20, a specific position of the object region 30 (for example, the coordinates of the upper left corner of the object region 30), and the size of the object region 30 (for example, width and height). If the object region 30 is identified from the captured image 20, the output information indicates the position and size for each of the multiple object regions 30. Alternatively, for example, the output information may be the captured image 20 with information indicating the object region 30 (for example, a frame) superimposed on it.

[0081] [Embodiment 2] Figure 8 is a block diagram illustrating the functional configuration of the information processing device 2000 of Embodiment 2. Except for the points described below, the information processing device 2000 of Embodiment 2 has the same functions as the information processing device 2000 of Embodiment 1.

[0082] The information processing device 2000 of Embodiment 2 handles multiple types of target objects. Specifically, the information processing device 2000 acquires type information indicating the type of object to be detected, and targets objects of the type indicated in the type information. To this end, the information processing device 2000 of Embodiment 2 has a type information acquisition unit 2080 that acquires type information.

[0083] The type information may indicate one or more types of objects. If the type information indicates multiple types of objects, the information processing device 2000 identifies an object region 30 for each target object, treating each type of object as a target object. For example, if the type information indicates three types, "hat, sunglasses, and white cane," the information processing device 2000 identifies an object region 30 representing a hat, an object region 30 representing sunglasses, and an object region 30 representing a white cane from the captured image 20.

[0084] There are various ways in which the type information acquisition unit 2080 can acquire type information. For example, the type information acquisition unit 2080 can acquire type information from a storage device where type information is stored. Alternatively, for example, the type information acquisition unit 2080 can acquire type information by receiving type information transmitted from other devices. Another possible method is for the type information acquisition unit 2080 to accept type information input from a user.

[0085] The candidate region detection unit 2020 of Embodiment 2 detects candidate regions 22 for objects of the type indicated in the type information. Here, existing technologies can be used for detecting specific types of objects from image data. For example, a detector trained to detect objects of that type from image data can be prepared for each type of object. The candidate region detection unit 2020 detects candidate regions 22 for objects of that type by inputting the captured image 20 to the detector trained to detect candidate regions 22 for objects of the type indicated in the type information.

[0086] The estimated position detection unit 2040 of Embodiment 2 detects the estimated position 24 for an object of the type indicated in the type information, based on the person region 26. For example, the estimated position detection unit 2040 also provides a detector for detecting the estimated position 24 for each type of object. That is, the positional relationship between the object and the person is learned for each type of object. The estimated position detection unit 2040 detects the estimated position 24 for an object of that type by inputting the captured image 20 and information identifying the person region 26 to a detector that has been trained to detect the estimated position 24 for an object of the type indicated in the type information.

[0087] The identification unit 2060 of Embodiment 2 identifies the object region 30 based on the detected candidate region 22 and estimated position 24 for the target object of the type indicated by the type information as described above. Output information is generated for each type of object.

[0088] <Effects and Effects> According to the information processing device 2000 of this embodiment, the object area 30 is identified for each type of object indicated by the type information. This allows the information processing device 2000 to be configured to detect a specified object from among multiple types of objects in the captured image 20. Therefore, it is possible to detect each of multiple types of objects from the captured image 20, or to change the type of object to be detected at any given time. This improves the usability of the information processing device 2000.

[0089] For example, if information about the belongings of a person of interest is obtained, the captured image 20 can be configured to detect the belongings of that person of interest. Also, if an abandoned object is found, the information processing device 2000 can be configured to detect that abandoned object.

[0090] <Example of hardware configuration> The hardware configuration of the computer implementing the information processing device 2000 of Embodiment 2 is shown, for example, in Figure 3, similar to Embodiment 1. However, the storage device 1080 of the computer 1000 implementing the information processing device 2000 of this embodiment further stores program modules that realize the functions of the information processing device 2000 of this embodiment.

[0091] <Processing flow> Figure 9 is a flowchart illustrating the processing flow performed by the information processing device 2000 of Embodiment 2. The type information acquisition unit 2080 acquires type information (S202). The information processing device 2000 acquires the captured image 20 (S204). The candidate area detection unit 2020 detects candidate areas 22 for objects of the type indicated in the type information (S206). The estimated position detection unit 2040 detects a person area 26 (S208). Based on the person area 26, the estimated position detection unit 2040 detects an estimated position 24 for objects of the type indicated in the type information (S210). The identification unit 2060 identifies an object area 30 based on the detected candidate areas 22 and estimated positions 24.

[0092] The embodiments of the present invention have been described above with reference to the drawings, but these are merely examples of the present invention, and various other configurations can also be adopted.

Claims

1. A first detection unit uses a detector trained on images to detect multiple image regions containing the target object from an captured image, The captured image includes a second detection unit that detects body parts of a person visible in the captured image, A identifying unit that identifies the part of a plurality of body parts in which the target object exists, based on the image region and the body part, It has, The specified unit determines the probability that the target object exists in the body part based on the image area and the body part, The aforementioned object is a person's possession, The type of target object differs for each image region in the information processing device.

2. The first detection unit detects the image region using a detector that has been trained with an image representing the target object. The information processing apparatus according to claim 1.

3. The first detection unit detects the image region using a detector that has been trained to recognize an image representing the target object for each type of target object. The information processing apparatus according to claim 1.

4. The information processing apparatus according to any one of claims 1 to 3, wherein the person's possession is an item worn on the body.

5. A method of information processing performed by a computer, Using a detector trained on images, multiple image regions containing the target object are detected from the captured image. In the captured image, body parts of a person appearing in the captured image are detected. Based on the image region and the body part, the probability that the target object exists in the body part is determined. Based on the image region and the body part, the part of the body where the target object exists is identified from among the multiple body parts. The aforementioned object is a person's possession, The type of the target object differs for each image region. Information processing methods.

6. A method of information processing performed by a computer, In the captured image, the body parts of the person shown in the captured image are detected. Using a detector trained on images, multiple image regions containing the target object to be detected are detected from the captured image. Based on the image region and the body part, the probability that the target object exists in the body part is determined. Based on the image region and the body part, the part of the body where the target object exists is identified from among the multiple body parts. The aforementioned object is a person's possession, The type of the target object differs for each image region. Information processing methods.

7. The information processing method according to claim 5 or 6, wherein the person's possession is an item worn on the body.

8. On the computer, A process that uses a detector trained on images to detect multiple image regions containing the target object from an captured image. In the captured image, a process is performed to detect the body parts of the person shown in the captured image. A process for determining the probability that the target object exists in the body part based on the image region and the body part. A process of identifying the part of a plurality of body parts in which the target object exists, based on the image region and the body part. Make it run, The aforementioned object is a person's possession, The type of the target object differs for each image region. program.

9. The program according to claim 8, wherein the person's possession is a wearable item.

Images