Object of interest determination system

Abstract

To provide an object of interest determination system capable of determining an appropriate object of interest of a robot.SOLUTION: An object of interest determination system determines an object of interest of a robot in the next image from the current and next images. The object of interest determination system comprises an object recognition part 11 and an object of interest determination unit 12. The object recognition unit 11 recognizes a plurality of objects included in the current and next images from the respective images. The object of interest determination unit 12 determines an object of interest of the robot in the next image based upon information representing the positions of the respective objects included in the next image and information representing times for which the robot is interested in the respective objects included in the next image, the times being calculated based upon the current and next images.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a target determination system.

Background Art

[0002] Patent Document 1 discloses an autonomous robot that enables natural communication and interaction. The autonomous robot disclosed in Patent Document 1 creates a saliency map from a camera image and turns toward the position with the maximum saliency.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the autonomous robot disclosed in Patent Document 1 described above, among the objects around the robot, there is a problem that a specific object becomes the target of attention and there are few opportunities to pay attention to other objects.

[0005] The present disclosure has been made in view of such circumstances, and provides a target determination system that can determine an appropriate target of attention for a robot.

Means for Solving the Problems

[0006] The target determination system according to the present disclosure is a target determination system that determines the target of attention of the robot in the later image from the front and rear images, an object recognition unit that recognizes a plurality of objects included in each image from the front and rear images, An attention target determination unit that determines an attention target of the robot in a later image based on information indicating the position of each of the targets included in the later image and information indicating a time calculated based on the earlier and later images and related to the attention of the robot to each of the targets included in the later image. is provided.

[0007] The attention target determination system according to the present disclosure determines an attention target of the robot in a later image based on information indicating the position of each target included in the later image and information indicating a time calculated based on the earlier and later images and related to the attention of the robot to each target included in the later image. By adopting such a configuration, an appropriate attention target of the robot can be determined in consideration of the time related to the attention of the target and the position of the target.

[0008] The attention target determination unit may calculate a position score for each target based on information indicating the position of the target included in the later image, calculate a time score related to the attention of the robot for each target included in the later image based at least on information indicating the time calculated from the earlier and later images, and determine an attention target from among the targets according to the priority based on the position score and the time score. With such a configuration, since scoring considering time and position is used, an appropriate attention target of the robot can be accurately determined.

[0009] When the target in the later image is an attention target in the earlier image, the attention target determination unit may calculate the time score of the target based on information indicating the cumulative attention time of the target, and when the target in the later image is not an attention target in the earlier image, calculate the time score of the target based on a predetermined time score. With such a configuration, it is possible to suppress the robot from setting a specific target as an attention target.

[0010] The target attention determination unit further includes a type score calculation unit that determines a type score for each of the targets based on the type information of the targets included in the post-image, and may further determine the attention target from among the targets according to the priority based on the type score. With such a configuration, an appropriate attention target of the robot can be determined in consideration of the time related to the attention of the target, the position of the target, and the type of the target.

[0011] The attention target determination system according to the present disclosure is an attention target determination system that determines the attention person of the robot in the post-image from the front and rear images, a target recognition unit that recognizes persons and non-persons included in each of the front and rear images and associates and recognizes the persons and the non-persons, an attention target determination unit that determines the attention person of the robot from among the persons in the post-image, and includes The attention target determination unit information indicating the position of each of the persons included in the post-image, time calculated based on the front and rear images, the information indicating the time related to the attention of the robot to each of the persons included in the post-image, information indicating the behavior of the person regarding the non-person associated with each of the persons included in the post-image, Based on this, the attention person of the robot in the post-image is determined.

[0012] The attention target determination system according to the present disclosure determines the attention target of the robot in the post-image based on the information indicating the position of each person included in the post-image, the time calculated based on the front and rear images, the information indicating the time related to the attention of the robot to each person included in the post-image, and the behavior information of the person regarding the non-person associated with each person included in the post-image. By adopting such a configuration, an appropriate attention target of the robot can be determined in consideration of the position of the person, the time related to the attention of the robot to the person, and the behavior of the person regarding the non-person associated with the person. [Advantages of the Invention]

[0013] According to the present disclosure, it is possible to provide an attention target determination system that can determine an appropriate attention target of a robot. [Brief Description of the Drawings]

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

[0015] Hereinafter, the present disclosure will be described through embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. Also, not all of the configurations described in the embodiments are essential as means for solving the problems. For clarity of explanation, the following description and drawings have been appropriately omitted and simplified. In each drawing, the same elements are denoted by the same reference numerals, and redundant explanations are omitted as necessary.

[0016] (Embodiment 1) <Target Determination System> Hereinafter, with reference to FIG. 1, the configuration of the target determination system according to Embodiment 1 will be described. FIG. 1 is a block diagram illustrating the target determination system according to Embodiment 1. As shown in FIG. 1, the target determination system 10 includes a target recognition unit 11 and a target determination unit 12.

[0017] The target determination system 10 is a target determination system that determines the target of the robot in the subsequent image from the previous and subsequent images. The previous and subsequent images indicate a previous image and a subsequent image. The previous image is the image with an earlier shooting time among images with different shooting times. The subsequent image is the image with a later shooting time among images with different shooting times.

[0018] The targets of the previous image and the subsequent image include a person (human face), a part of a person, and an object. For example, when the subsequent image includes person A holding a fan in the left hand and waving the right hand, the targets are person A, the fan, and the right hand.

[0019] The previous and subsequent images are acquired by a camera installed on the robot or a camera installed outside the robot. The camera is, for example, an infrared camera or a stereo camera capable of calculating the distance to the target. Also, the camera may be a camera capable of measuring depth information.

[0020] <Target Recognition Unit> The target recognition unit 11 recognizes a plurality of targets included in each of the front and rear images. More specifically, the target recognition unit 11 uses image recognition technology to recognize the targets. For example, the target recognition unit 11 recognizes the targets based on whether the targets included in the rear image match the pre-learned targets. The target recognition unit 11 similarly recognizes the targets in the front image. For example, when the target recognition unit 11 recognizes a person's face from the targets included in the image, it recognizes the target as a person. For example, when the target recognition unit 11 recognizes a part other than a person's face from the targets included in the image, it recognizes the target as a part of a person. The target recognition unit 11 may list up the plurality of targets recognized from the front image and the rear image as recognition results and store them in a storage unit (not shown in FIG. 1).

[0021] In addition, the target recognition unit 11 acquires identification information of the target, type information of the target, and position information of the target for each recognized target. The target recognition unit 11 attaches identification information of the target to the recognition targets in the front and rear images. The identification information of the target is unique. The same identification information of the target is attached to the same target. Then, the target recognition unit 11 uses a tracking method to determine whether the rear image includes the same target as the front image.

[0022] The target recognition unit 11 can calculate the three-dimensional position of the target based on, for example, the depth information of the target and the position information of the target. When the front and rear images are acquired by cameras installed outside the robot, the target recognition unit 11 converts the three-dimensional position of the target into the three-dimensional position from the robot based on, for example, the viewing angle of the camera and the relative position from the robot to the camera.

[0023] <Attention target determination unit> The attention target determination unit 12 determines the attention target of the robot in the rear image based on information indicating the position of each target included in the rear image and information indicating the time calculated based on the front and rear images and related to the attention of the robot to each target included in the rear image.

[0024] The time related to the robot's attention includes the attention time and the non-attention time. The attention time is the time when the robot is paying attention to the target. The non-attention time is the time when the robot is not paying attention to the target. The attention time and the non-attention time will be described in detail in the calculation of the time score described later. The attention target determination unit 12 determines the attention target using, for example, the scoring described later. However, it is not limited to this, and the attention target determination unit 12 may determine using the learning model described later. Hereinafter, the attention target determination unit 12 will be described in detail.

[0025] <Example of Determining Attention Target Using Scoring> First, an example in which the attention target determination unit 12 determines the attention target of the robot using scoring will be described. FIG. 2 is a block diagram illustrating the attention target determination unit. As shown in FIG. 2, the attention target determination unit 12 includes a position score calculation unit 21, a time score calculation unit 22, and a control unit 23.

[0026] <Calculation of Position Score> The position score calculation unit 21 calculates a position score for each target based on the information indicating the position of the target included in the post-image. More specifically, the position score calculation unit 21 calculates the position score from the information indicating the three-dimensional position of the target included in the post-image using Equation (1). In Equation (1), the position score indicates a larger value as the position of the target relative to the robot is closer.

[0027] [Number]

[0028] However, Sp is the position score. Cdis is the distance coefficient and is less than 0. x is the position of the target in the left-right direction relative to the robot. y is the position of the target in the depth direction relative to the robot. z is the position of the target in the up-down direction relative to the robot. Sp0 is the initial position score and is an arbitrarily set score.

[0029] <Calculation of Time Score> The time score calculation unit 22 calculates a time score regarding the robot's attention for each target included in the later image based on information indicating the time calculated from at least the previous and later images. More specifically, when the target in the later image was the target of attention in the previous image, the time score calculation unit 22 calculates the time score using Equation (2). In Equation (2), the time score indicates a small value as time elapses. Hereinafter, when the target in the later image was the target of attention in the previous image, the time score calculated by the time score calculation unit 22 is referred to as the attention time score.

[0030] [Number]

[0031] However, St is the time score. Ct is the time coefficient, which is less than 0. t is the cumulative attention time. St0 is the initial time score, which is an arbitrarily set score.

[0032] In addition, when the recognized target in the later image was not the target of attention in the previous image, the time score calculation unit 22 calculates a predetermined time score. Hereinafter, when the recognized target in the later image was not the target of attention in the previous image, the predetermined time score calculated by the time score calculation unit 22 is referred to as the non-attention time score. The time score calculation unit 22 calculates the non-attention time score as a value in a range that is smaller than the maximum value of the attention time score calculated using Equation (2) and larger than the minimum value of the attention time score.

[0033] [Cumulative attention time] Here, while referring to FIG. 3, the attention time, cumulative attention time, and non-attention time will be described. FIG. 3 is a diagram showing an example of a pre-image and a post-image. The pre-image G1 and the post-image G2 include objects M1 and M2. The shooting date and time of the pre-image G1 is 10:00, and the shooting date and time of the post-image G2 is 10:03. In the pre-image G1, since the attention target of the robot is the object M1, it is shown using hatching as shown in FIG. 3. Further, at 10:00, which is the shooting date and time of the pre-image G1, the object M1 has already been the attention target of the robot for 5 minutes. Information indicating that the object M1 has already been the attention target for 5 minutes is stored in a storage unit (not shown), and as will be described later in Embodiment 2, the object recognition unit 11 acquires this information, and the time score calculation unit 22 uses this information.

[0034] As shown in FIG. 3, the attention time of the object M1 in the post-image G2 is 3 minutes obtained by subtracting the shooting date and time of the pre-image G1, 10:00, from the shooting date and time of the post-image G2, 10:03. Also, the cumulative attention time of the object M1 in the post-image G2 is 8 minutes obtained by adding the attention time of the object M1 in the pre-image G1 (5 minutes) to the attention time of the object M1 (3 minutes). That is, in the example shown in FIG. 3, the time score calculation unit 22 calculates the attention time score for the object M1 using Equation (2) with the cumulative attention time being 8 minutes.

[0035] On the other hand, the object M2 is not regarded as an attention target. Therefore, the non-attention time of the object M2 is 3 minutes obtained by subtracting the shooting date and time of the pre-image G1, 10:00, from the shooting date and time of the post-image G2, 10:03. The time score calculation unit 22 is set, for example, to increase the non-attention time score as the non-attention time becomes longer.

[0036] <Calculation of Priority Score> The control unit 23 determines an attention target from among the objects according to the priority based on the position score and the time score. More specifically, the control unit 23 calculates a priority score using Equation (3). The larger the priority score obtained by Equation (3), the higher the priority for the robot to regard it as an attention target.

[0037]

Number

[0038] However, S is the priority score. Wp is the weight of the position score. Wt is the weight of the time score.

[0039] For example, assume that the post-image includes object A and object B. When the priority score S of object A is 5 and the priority score S of object B is 3, the control unit 23 determines that the priority score of object A is higher than that of object B. From this, the control unit 23 determines that object A is a higher-priority object than object B as the object of the robot's attention. Therefore, the control unit 23 determines object A as the object of the robot's attention. When the control unit 23 determines that object A is the object of the robot's attention, it transmits information regarding object A, which is the object of attention, to the motion control unit of the robot (not shown in FIGS. 1 and 2). The motion control unit (not shown in FIGS. 1 and 2) controls the orientation and motion of the robot based on the received information regarding object A.

[0040] Here, from Equation (1), the position score indicates a larger value as the position of the object relative to the robot is closer. From Equation (3), the priority score indicates a larger value as the position score is a larger value. That is, the priority score indicates a larger value as the position of the object relative to the robot is closer. In this way, by using Equation (1), the attention target determination unit 12 can preferentially set the object located near the robot as the attention target.

[0041] Also, from Equation (2), the time score indicates a smaller value because the cumulative attention time increases as time elapses. From Equation (3), the priority score indicates a smaller value as the time score becomes a smaller value. That is, the priority score indicates a smaller value as time elapses. In this way, by using Equation (2), the attention target determination unit 12 can preferentially set an object different from the attention target of the pre-image as the attention target of the post-image as time elapses. In other words, by using Equation (2), the attention target determination unit 12 can suppress the robot from setting a specific object as the attention target.

[0042] Furthermore, the non-attention time score is a value within a range that is smaller than the maximum value of the attention time score and larger than the minimum value of the attention time score. Here, an example will be described in which the attention target determination unit 12 determines the attention target of the subsequent image with target C as the attention target of the previous image among targets C and D.

[0043] Immediately after the robot pays attention to target C, the attention time score of target C exceeds the non-attention time score of target D. Therefore, the robot regards target C as the attention target for a certain period of time. However, as time passes, from Equation (2), the value of the attention time score of target C becomes smaller. From this, the attention time score of target C falls below the non-attention time score of target D. Therefore, the robot changes the attention target from target C to target D. In this way, the attention target determination system 10 can suppress the robot from regarding a specific target as the attention target by using the attention time score and the non-attention time score for the target.

[0044] Note that the position score calculation unit 21 is not limited to using the above-described Equation (1), and for example, the position score may be calculated using Equation (4). In Equation (4), weights are assigned to the planar distance and the height respectively. Thereby, the attention target determination unit 12 can preferentially set children or elderly people who tend to have a low height (the height of the target) as the attention target of the robot. Also, the attention target determination unit 12 can preferentially set a target with a low height as the attention target of the robot with a low-cost configuration without using complex image recognition.

[0045]

Number

[0046] However, Sp is the position score. Wdis is the weight of the planar distance score in the position score. Wheight is the weight of the height score in the position score. Cdis is the distance coefficient and is less than 0. Cheight is the height coefficient and is less than 0. x is the left - right position of the target relative to the robot. y is the depth position of the target relative to the robot. z is the up - down position of the target relative to the robot. Sp0 is the initial position score in the xy coordinates and is an arbitrarily settable score. Sh0 is the initial position score in the z coordinate and is an arbitrarily settable score.

[0047] <Example of Determining the Target of Interest Using the Learning Model> Next, an example of the target - of - interest determination unit 12 determining the target of interest using the learning model will be described. The target - of - interest determination unit 12 uses, for example, the following learning model. Information indicating the positions of a plurality of targets in the later image and information indicating the time calculated based on the front and rear images, which is the time related to the robot's attention to each target included in the later image, are input into the learning model. Then, based on each input, the learning model determines the target of interest of the robot in the later image. The learning model is generated, for example, by learning through a neural network or the like.

[0048] In this way, the target - of - interest determination unit 12 determines the target of interest of the robot in the later image based on information indicating the positions of each target included in the later image and information indicating the time calculated based on the front and rear images, which is the time related to the robot's attention to each target included in the later image.

[0049] By adopting such a configuration, the target determination system 10 can determine an appropriate target of attention for the robot in consideration of the position of the target and the time related to the attention to the target. As a result, the target determination system 10 can suppress the robot from setting a specific target as the target of attention. That is, the target determination system 10 can determine an appropriate target of attention for the robot. Also, since the target determination system 10 can suppress a specific target from becoming the target of attention of the robot, it can be said that it is highly fair.

[0050] Although not shown in the drawings, the target determination system 10 includes, for example, an arithmetic unit such as a CPU (Central Processing Unit), and a storage unit such as a RAM (Random Access Memory) and a ROM (Read Only Memory) in which various control programs, data, etc. are stored. That is, the target determination system 10 has the function of a computer and performs various processes based on the above-mentioned various control programs and the like.

[0051] Therefore, each functional block of the target recognition unit 11 and the target determination unit 12 in the target determination system 10 shown in FIG. 1 can be composed of the above-mentioned CPU, storage unit, and other circuits in terms of hardware. Also, each functional block can be realized by a program stored in the storage unit or the like in terms of software. That is, each functional block can be realized in various forms by hardware, software, or a combination of both.

[0052] <Target Determination Method> Subsequently, the target determination method according to Embodiment 1 will be described. FIG. 4 is a flowchart illustrating the target determination method according to Embodiment 1. The target determination method according to Embodiment 1 is a target determination method for determining the target of attention of the robot in the subsequent image from the front and rear images.

[0053] First, the target recognition unit 11 recognizes a plurality of targets included in each of the front and rear images (step ST111). More specifically, the target recognition unit 11 uses image recognition technology to recognize a plurality of targets.

[0054] Next, the target of interest determination unit 12 determines the target of interest of the robot in the rear image based on information indicating the position of each target included in the rear image and information indicating the time calculated based on the front and rear images and related to the robot's attention to each target included in the rear image (step ST112). The target of interest determination unit 12 determines the target of interest of the robot in the rear image using scoring or a learning model.

[0055] Thus, in the method for determining the target of interest, the target of interest of the robot in the rear image is determined based on information indicating the position of each target included in the rear image and information indicating the time calculated based on the front and rear images and related to the robot's attention to each target included in the rear image.

[0056] By adopting such a configuration, in the method for determining the target of interest, an appropriate target of interest of the robot can be determined in consideration of the time related to the attention to the target and the position of the target. Thereby, in the method for determining the target of interest, it is possible to suppress the robot from setting a specific target as the target of interest. That is, in the method for determining the target of interest, an appropriate target of interest of the robot can be determined. Also, since the method for determining the target of interest can suppress a specific target from becoming the target of interest of the robot, it can be said that it is highly fair.

[0057] (Embodiment 2) <Target of Interest Determination System> Hereinafter, the configuration of the target of interest determination system according to Embodiment 2 will be described. Hereinafter, the target of interest determination system according to Embodiment 2 is denoted as the target of interest determination system 20. The target of interest determination system 20 has a different configuration of the target of interest determination unit compared to the target of interest determination system 10 according to Embodiment 1. Since the target recognition unit is the same as that in Embodiment 1, the description thereof is omitted.

[0058] FIG. 5 is a block diagram illustrating the target determination unit according to Embodiment 2. As shown in FIG. 5, the target determination unit 122 includes a position score calculation unit 31, a time score calculation unit 32, a type score calculation unit 34, and a control unit 33. Since the position score calculation unit 31 and the time score calculation unit 32 are the same as those in Embodiment 1, the description thereof is omitted. Here, the type score calculation unit 34 and the control unit 33 will be described.

[0059] <Determination of Type Score> The type score calculation unit 34 determines a type score for each target based on the type information of the targets included in the post-image. More specifically, the type score calculation unit 34 determines a type score for each target included in the post-image using a type score predetermined for each type of target. Hereinafter, the type score will be denoted as type score So.

[0060] An example of the type score So will be described with reference to Table 1. Table 1 shows an example of the type score So for each target. As shown in Table 1, the type score calculation unit 34 determines the type score So to be 4 for persons A and B. The type score calculation unit 34 determines the type score So to be 4 for the stuffed animal. The type score calculation unit 34 determines the type score So to be 4 for a human hand. In this way, the type score calculation unit 34 determines a type score for each of the persons, body parts of the persons, and objects that are the targets in the post-image.

[0061]

Table 1

[0062] <Calculation of Priority Score> The control unit 33 determines a target of interest from among the targets according to the priority based on the position score, the time score, and the type score. More specifically, the control unit 33 calculates a priority score using Equation (5). The larger the priority score obtained by Equation (5), the higher the priority for the robot to target as the target of interest.

[0063]

Number

[0064] However, S is the priority score. Wp is the weight of the position score. Wt is the weight of the time score. Wo is the weight of the type score.

[0065] In this way, the target determination unit 12 determines a target of interest from among the targets according to the priority based on the type score in addition to the position score and the time score. By adopting such a configuration, the target determination system according to the second embodiment can determine an appropriate target of interest for the robot in consideration of the position of the target, the time related to the attention of the target, and the type of the target.

[0066] Here, for example, when the targets include a person and an object, the target determination system can preferentially set the person as the target of interest by setting the type score So of the person to 2 and the type score So of the object to 1. Further, the target determination system may preferentially set the person as the target of interest by increasing the weight Wo of the type score So of the person and decreasing the weight Wo of the type score of the object compared to that weight. That is, the target determination system according to the second embodiment can change the priority of the target of interest of the robot according to the type and determine an appropriate target of interest.

[0067] <Method for Determining Target of Interest> Subsequently, the method for determining the target of interest according to the second embodiment will be described. FIG. 6 is a flowchart illustrating the method for determining the target of interest according to the second embodiment. The method for determining the target of interest according to the second embodiment is a method for determining the target of interest of the robot from the front and rear images. Here, although not shown in FIG. 4, the target determination system will be described assuming that it includes a storage unit that stores the recognition target, the target of interest, the attention time of the target of interest, the cumulative attention time of the target of interest, and the non-attention time of the front and rear images.

[0068] First, the target recognition unit recognizes a plurality of targets included in each of the front and rear images (step ST1). More specifically, the target recognition unit 11 recognizes a plurality of targets using image recognition technology. This process is the same as step ST111 shown in FIG. 4.

[0069] Next, the target recognition unit acquires identification information of the target, information regarding the type of the target, and position information of the target for each recognized target (step ST2).

[0070] Next, when the front and rear images are acquired by cameras installed outside the robot, the target recognition unit converts the three-dimensional position of the target into the three-dimensional position from the robot (step ST3). In the method for determining a target of interest according to Embodiment 2, this process is omitted when the front and rear images are acquired by cameras installed on the robot.

[0071] Next, the target recognition unit compares the recognition target in the front image with the recognition target in the rear image (step ST4). More specifically, the target recognition unit attaches unique identification information to the recognition targets in the front and rear images, and uses a tracking method to determine whether the rear image includes the same target as the front image. When the rear image includes the same target as the front image, the identification information of the target included in the rear image matches the identification information of the target included in the front image. Therefore, the target recognition unit acquires the attention time up to the front image from a storage unit (not shown) for the targets included in the front and rear images.

[0072] Next, the position score calculation unit 31 calculates a position score for each target based on the information indicating the position of the target included in the rear image (step ST5). More specifically, the position score calculation unit 31 calculates the position score using the above-described formula (1).

[0073] The control unit 33 determines whether the object for which the position score has been calculated is the object of interest in the previous image (step ST6). If the object for which the position score has been calculated is the object of interest in the previous image (step ST6 YES), the time score calculation unit 32 calculates the time score regarding the robot's attention for the object based on the previous and subsequent images. More specifically, the time score calculation unit 32 calculates the attention time score using the above-described formula (2) (step ST7).

[0074] On the other hand, if the object for which the position score has been calculated is not the object of interest in the previous image (step ST6 NO), the time score calculation unit 32 calculates a predetermined non-attention time score (step ST8).

[0075] Next, the type score calculation unit 34 determines the type score for the object based on the type information of the object included in the subsequent image (step ST9). More specifically, the type score calculation unit 34 determines the type score for the object included in the subsequent image using the predetermined type score for each type of the object.

[0076] Next, the control unit 33 calculates the priority score based on the position score, the time score, and the type score (step ST10). More specifically, the control unit 33 calculates the priority score using the formula (5).

[0077] As shown in FIG. 6, in the method for determining the object of interest according to the second embodiment, the processes of steps ST5 to ST10 are executed for all the objects included in the subsequent image. Note that, in the method for determining the object of interest according to the second embodiment, a configuration may be adopted in which the repetitive processes of steps ST2 to ST10 are executed for each object.

[0078] Next, the control unit 33 determines, as the target of interest, the target with the highest score among the priority scores of each target (step ST11). Then, the control unit 33 stores information regarding the recognition result of the post-image in the storage unit in the storage unit (step ST12). The information regarding the recognition result is, for example, information including the target recognized in the post-image. The target recognition unit 11 may determine whether the next image (the image next to the post-image) includes the same target as the post-image using the information regarding the recognition result when determining the target of interest in the next image.

[0079] Next, the control unit 33 determines whether the target is the target with the highest score (target of interest) (step ST13). If the target is the target with the highest score (target of interest) (step ST13 YES), the control unit 33 determines whether the target is the target with the highest score (target of interest in the pre-image) in the pre-image (step ST14).

[0080] On the other hand, if the target is not the target with the highest score (target of interest) (step ST13 NO), the processes after step ST13 are repeatedly executed for other targets.

[0081] If the target is the target with the highest score (target of interest in the pre-image) in the previous image (step ST14 YES), the control unit 33 associates the attention time and the cumulative attention time of the target with the information regarding the recognition result and stores them in the storage unit (step ST15). Then, the control unit 33 stores the target as the target of interest in the post-image in the storage unit associated with the information regarding the recognition result (step ST16). When determining the target of interest in the next image (the image next to the post-image), the target recognition unit acquires the cumulative attention time of the target stored in the storage unit, and the time score calculation unit 32 uses that information.

[0082] On the other hand, when the target is not the target with the highest score in the previous image (the target of interest in the previous image) (NO in step ST14), the control unit 33 stores the target in the storage unit in association with the information regarding the recognition result as the target of interest in the subsequent image (step ST16). Note that the control unit 33 may store the non-attention time in the storage unit in association with the information regarding the recognition result.

[0083] In this way, it can be said that the control unit 33 is preparing to determine the target of interest in the next image (the image after the subsequent image) by executing steps ST12 to ST16.

[0084] After the control unit 33 executes the processes after step ST13 for all the targets, it transmits the information regarding the target of interest in the subsequent image to the motion control unit of the robot (step ST17). Thereby, the robot can face the appropriate target of interest and execute natural communication and interaction.

[0085] In FIG. 6, an example in which the control unit 33 executes the repetitive processes of steps ST13 to ST16 for all the targets has been described, but the present invention is not limited to this, and the repetitive processes of steps ST12 to ST16 may be executed. That is, the control unit 33 may perform the determination processes of steps ST13 and ST14 for all the targets and store the information regarding the target, the target of interest, the attention time, the cumulative attention time, and the non-attention time in the storage unit.

[0086] (Embodiment 3) <Target of Interest Determination System> Next, the configuration of the target determination system according to Embodiment 3 will be described. FIG. 7 is a block diagram illustrating the target determination system according to Embodiment 3. The target determination system 50 according to Embodiment 3 determines the target person of the robot in the rear image from the front and rear images. The target determination system 50 includes a target recognition unit 51 and a target determination unit 52. The target determination unit 52 includes a position score calculation unit 31, a time score calculation unit 32, an action score calculation unit 35, and a control unit 33. Since the position score calculation unit 31, the time score calculation unit 32, and the control unit 33 are the same as those in Embodiment 2, the target recognition unit 51 and the action score calculation unit 35 will be described.

[0087] The target recognition unit 51 recognizes the persons and non-persons included in each of the front and rear images, and associates and recognizes the persons and non-persons. The non-persons include human parts, for example, a human hand. In addition, the non-persons include objects. The action score calculation unit 35 determines an action score based on the action information of the person regarding the non-persons associated with each person included in the rear image. The target recognition unit 51 and the action score calculation unit 35 will be described in detail while using the flowchart described later.

[0088] <Target Determination Method> Subsequently, the target determination method according to Embodiment 3 will be described. FIG. 8 is a flowchart illustrating the target determination method according to Embodiment 3. In FIG. 8, the same processing as in FIG. 6 will be omitted as appropriate. Here, although not illustrated in FIG. 8, the target determination system 50 will be described assuming that it includes a storage unit that stores the persons in the front and rear images, the target persons, the target time of the target persons, the cumulative target time of the target persons, the non-target time, and the action history of the persons.

[0089] <Association between Persons and Non-Persons> Steps ST31 to ST33 shown in FIG. 8 are the same as steps ST11 to ST13 shown in FIG. 6, so the description thereof is omitted. As shown in FIG. 8, following step ST33, the target recognition unit 51 recognizes persons and non-persons included in each of the front and rear images from the front and rear images, and extracts the persons and non-persons (step ST34). Next, the target recognition unit 51 detects the nearest person for each recognized non-person, and associates and recognizes the person and the non-person (step ST35). The correspondence between the person and the non-person is not limited to a one-to-one correspondence, and a plurality of non-persons may be associated with one person.

[0090] Taking FIG. 9 as an example, the association between the person and the non-person will be described more specifically. FIG. 9 is a diagram showing an example of the rear image. In FIG. 9, the target recognition unit 51 recognizes the user U1, the user U2, and the user U3 as persons. The target recognition unit 51 recognizes the fan S1, the stuffed animal S2, the hands H1, H2, H3, and H4 as non-persons.

[0091] The target recognition unit 51 associates and recognizes the user U1 with the fan S1 and the hand H1. The target recognition unit 51 associates and recognizes the user U2 with the stuffed animal S2 and the hand H2. The target recognition unit 51 associates and recognizes the user U3 with the hands H3 and H4.

[0092] Next, the target recognition unit 51 stores, in the storage unit, the actions related to the non-persons associated with the person as the action history of that person (step ST36). For example, in the example shown in FIG. 9, when the user U1 is waving the fan S1, the target recognition unit 51 stores in the storage unit an action history such as the user U1 waving the fan S1. If the action already exists in the action history of the storage unit, the target recognition unit 51 may accumulate the action time.

[0093] <Action score> Next, the action score calculation unit 35 determines an action score based on the action information of the person regarding the non-person associated with each person included in the post-image (step ST37). The action score calculation unit 35 uses, for example, a preset action score for the action information. Hereinafter, the action score is denoted as action score SA.

[0094] Referring to Table 2, the action score SA will be described. Table 2 is a table showing an example of the action scores SA of the users U1 to U3 shown in FIG. As shown in Table 2, since the user U1 is waving a fan S1, the action score calculation unit 35 determines the action score SA of the user U1 as 4. Since the user U2 is holding a stuffed animal S2, the action score calculation unit 35 determines the action score SA of the user U2 as 3. Since the user U3 is waving hands H3 and H4, the action score calculation unit 35 determines the action score SA of the user U1 as 2. Note that when a plurality of non-people are associated with one person, the action score calculation unit 35 uses the maximum score among the action scores of the person regarding each non-person. However, the present invention is not limited to this, and the action score calculation unit 35 may use any one of the action scores of the person regarding each non-person, or may use the sum of all or some of them.

[0095]

Table 2

[0096] Steps ST38 to ST42 shown in FIG. 8 are the same as steps ST5 to ST8 and 10 shown in FIG. 6, and thus the description thereof is omitted. Note that in steps ST38 and ST40 to ST42, the attention target determination unit 52 calculates a score for the person included in the post-image.

[0097] Next, the control unit 33 determines, as the target person, the person with the highest priority score among the priority scores of each person (step ST43). Next, when the person is the person with the highest score in the post-image (the target person in the post-image) (step ST44 YES), the control unit 33 associates the information regarding the target person, the attention time, and the cumulative attention time with the action history in the storage unit (step ST45).

[0098] On the other hand, when the person is not the target with the highest score in the post-image (the target person in the pre-image) (step ST44 NO), the control unit 33 repeatedly executes the processes after step ST44 for other persons. Note that the control unit 33 may associate the non-attention time with the person and store it in the storage unit.

[0099] Next, after the control unit 33 executes the processes after step ST44 for all persons, it transmits the target person in the post-image and the information regarding the action history related to the target person to the motion control unit of the robot (step ST46). Thereby, the robot can face the appropriate target person and execute natural communication and interaction.

[0100] In this way, the target determination system 50 can determine the appropriate target of the robot's attention in consideration of the position of the person in the post-image, the time related to the robot's attention to the person, and the actions of the person associated with the non-person corresponding to the person.

[0101] <Effect of Association between Person and Non-Person> The differences between the target determination system 50 and the target determination system 20 will be described. FIG. 10 is a diagram showing an example of a pre-image and a post-image. The upper part of FIG. 10 shows an example of the determination of the target of attention in the case of the target determination system 20, and the lower part of FIG. 10 shows an example of the determination of the target of attention in the case of the target determination system 50. In FIG. 10, the hatched target indicates the target of attention or the target person.

[0102] In the front image G3, the person U11 holds a fan S3, and the person U12 holds a stuffed toy S4. In the rear image G4, the person U11 holds a stuffed toy S5, and the person U12 holds a stuffed toy SS4. That is, in FIG. 10, the person U11 holds a fan S3 in the front image G3, but has switched to a stuffed toy S5 in the rear image G4. Also, in FIG. 10, the person U12 holds a stuffed toy S4 in both the front image G3 and the rear image G4.

[0103] As shown in FIG. 10, the stuffed toy S5 is not included in the front image G3 but is included in the rear image G4. From this, as shown in the upper part of FIG. 10, in the target determination systems 10 and 20, the time score of the stuffed toy S5 becomes high, and the stuffed toy S5 becomes the target of attention. That is, in the target determination system 20, the person U11 side becomes the target of attention.

[0104] On the other hand, in the target determination system 50, in order to associate and recognize a person and a non-person, even if the person U11 switches from the fan S3 to the stuffed toy S5, the time score of the person U11 does not change. Therefore, as time passes, since the time score of the person U11 falls below the time score of the person U12, as shown in the lower part of FIG. 10, in the target determination system 50, the person U12 becomes the target of attention. In this way, in the target determination system 50, in order to associate and recognize a person and a non-person, the robot can pay more attention to other people.

[0105] <Reference to action history> Here, in the target determination system 50, the control unit 33 transmits information regarding the person of interest in the post-image and the action history related to the person of interest to the action control unit of the robot. Thereby, the robot refers to the action history of the person of interest and performs a predetermined action. For example, when the person of interest has already taken a photo of the robot using a mobile terminal and the person of interest is also taking a photo of the robot using a mobile terminal in the post-image, the robot makes a determination and performs an action as follows. The robot determines that the person of interest is about to take a photo again, focuses on the person of interest, and acts to pose differently from the previous photo-taking.

[0106] As described above, since the target determination system 50 is configured to transmit information regarding the action history related to the person of interest to the action control unit, the robot can refer to the action history of the person of interest and perform a predetermined action. By adopting such a configuration, it is possible to arouse the curiosity of the people around the robot.

[0107] Furthermore, part or all of the processing in the target determination system according to the above-described Embodiments 1 to 3 can be realized as a computer program. Such a program can be stored using various types of non-transitory computer-readable media and supplied to a computer. Non-transitory computer-readable media include various types of tangible recording media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R / W, semiconductor memories (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). Also, the program may be supplied to the computer by various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable media can supply the program to the computer via wired communication paths such as electric wires and optical fibers, or wireless communication paths.

[0108] Note that the present disclosure is not limited to the above-described embodiments and can be appropriately modified without departing from the gist thereof.

Description of Reference Numerals

[0109] 10, 20, 50 Target Determination System 11, 51 Target Recognition Unit 12, 52, 122 Target Determination Unit 21, 31 Position Score Calculation Unit 22, 32 Time Score Calculation Unit 23, 33 Control Unit 34 Type Score Calculation Unit 35 Action Score Calculation Unit G1 Previous Image G2 Next Image M1, M2 Target U1, U2, U3, U11, U12 Persons H1, H2, H3, H4 hands Among S1 and S3, fans S2, S4, S5 stuffed toys

Claims

1. A target attention determination system for determining a target of attention of a robot in a later image from front and rear images, comprising: a target recognition unit that recognizes a plurality of targets included in each of the front and rear images from the front and rear images; an attention target determination unit that determines an attention target of the robot in the later image based on information indicating the position of each of the targets included in the later image and information indicating a time calculated based on the front and rear images and indicating a time related to the attention of the robot to each of the targets included in the later image; characterized by comprising: an attention target determination system.

2. The attention target determination unit: calculates a position score for each of the targets based on information indicating the position of the targets included in the later image; calculates a time score related to the attention of the robot to each of the targets included in the later image based on at least information indicating the time calculated from the front and rear images; determines the attention target from among the targets according to a priority based on the position score and the time score. The attention target determination system according to Claim 1.

3. The attention target determination unit: calculates a time score of the target based on information indicating the cumulative attention time of the target when the target in the later image is an attention target in the front image; calculates a time score of the target based on a predetermined time score when the target in the later image is not an attention target in the front image. The attention target determination system according to Claim 2.

4. The attention target determination unit: further comprises a type score calculation unit that determines a type score for each of the targets based on type information of the targets included in the later image; further determines the attention target from among the targets according to a priority based on the type score. The attention target determination system according to Claim 2 or 3.

5. An attention target determination system for determining a person of attention of a robot in a later image from front and rear images, comprising: a target recognition unit that recognizes a person and a non-person included in each of the front and rear images from the front and rear images and associates and recognizes the person and the non-person; an attention target determination unit that determines a person of attention of the robot from among the persons in the later image; characterized by comprising: The attention target determination unit: information indicating the position of each of the persons included in the later image; Time calculated based on the front and rear images, which is information indicating the time regarding the robot's attention to each of the persons included in the rear image, and information indicating the actions of the person regarding the non-person associated with each of the persons included in the rear image, Based on this, determine the person the robot is paying attention to in the rear image. Attention target determination system.

Images