Communication robot, communication robot control method, and program

Abstract

This communication robot comprises: an auditory information processing unit which recognizes the magnitude of a sound collected by a sound collection unit and generates an auditory attention map by projecting the location of the sound in a three-dimensional space onto a two-dimensional attention map in which the robot is located at the center; a visual information processing unit which generates a visual attention map using a face detection result obtained by detecting the face of a person from an image captured by a photographic unit and a motion detection result obtained by detecting motion of the person; an attention map generation unit which generates an attention map by integrating the auditory attention map and the visual attention map; and a motion processing unit which controls eye movements and the motion of said communication robot using the attention map.

Description

[Technical field]

[0001] The present invention relates to a communication robot, a communication robot control method, and a program. This application claims priority based on Japanese Patent Application No. 2021-130726, filed on August 10, 2021, the contents of which are incorporated herein by reference. [Background technology]

[0002] The eye is a visual organ that receives visual information from the outside, and at the same time, it is also a cognitively special stimulus that provides information on the internal mental state. It has also been neurologically proven that faces are attractive and special stimuli that can be found in very early visual processing stages such as the primary visual cortex (V1) and the extrastriate cortex (V2, V3). The saliency map, which is a heat map that estimates the places where people are likely to pay attention when looking at an image, is an attention model that predicts attention shifts on an image, such as a visual search strategy (see, for example, Non-Patent Document 1).

[0003] In contrast to visual saliency maps, few models have been proposed to determine the saliency of audio signals. In human-robot interaction, there are some studies that consider audio signals and build bottom-up saliency maps (see, for example, Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] L. Itti, C. Koch and E. Niebur, “A model of saliency-based visual attention for rapid scene analysis”, in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 11, pp. 1254-1259, Nov. 1998, doi:10.1109 / 34.730558. [Non-Patent Document 2] Kae Nakajima, Tetsuto Minami, Shigeki Nakauchi, “Interaction between facial expression and color”, SCIENTIFIC REPORTS, 2017 Summary of the Invention [Problem to be solved by the invention]

[0005] However, previous studies have only considered visual cues and not actual audio sources from the 3D environment. Also, previous studies have only considered simple visual saliency features (intensity, color, direction, and motion). Also, previous studies have only considered face and hand features and have not considered auditory attention. Previous studies have not considered bottom-up attention based on stimuli, but top-down attention. Thus, it has been difficult to calculate both visual and auditory attention with conventional techniques.

[0006] The aspects of the present invention have been made in consideration of the above-mentioned problems, and aim to provide a communication robot, a communication robot control method, and a program that can control the robot by integrating both visual and auditory attention. [Means for solving the problem]

[0007] In order to solve the above problems, the present invention employs the following aspects. (1) A communication robot according to one embodiment of the present invention includes an auditory information processing unit that recognizes the volume of sound picked up by a sound collection unit and projects the position of the sound in three-dimensional space onto a two-dimensional attention map centered on the robot to generate an auditory attention map; a visual information processing unit that generates a visual attention map using a face detection result that detects a human face using an image captured by a capture unit and a motion detection result that detects the motion of the human; an attention map generation unit that integrates the auditory attention map and the visual attention map to generate an attention map; and a motion processing unit that uses the attention map to control eye movement and the motion of the communication robot.

[0008] (2) In the above aspect (1), the visual information processing unit may be configured to decompose a saliency map created by detecting faces and detecting movements from a captured image into two visual attention maps, one based on face detection and the other based on movement detection, to generate the visual attention map.

[0009] (3) In the above aspect (2), the visual attention map obtained by detecting a face may be a face attention map, and the detected face area may be emphasized by a face size value, and the visual attention map obtained by detecting movement may be a motion attention map, and the detected moving object may be emphasized by a movement speed value.

[0010] (4) In any one of the above aspects (1) to (3), the auditory information processing unit may calibrate the projection of the sound source with a binary image based on the facing direction of position, power, and duration, and create each circle to represent the sound source, and the center coordinates of each of the circles may be the position where the sound direction is three-dimensionally projected onto the two-dimensional image.

[0011] (5) In any one of the above aspects (1) to (4), the attention map generation unit may be configured to integrate a visual attention map and an auditory attention map from all normalized tension maps having different values ​​at each location on the same image size of each frame.

[0012] (6) In one embodiment of the communication robot control method of the present invention, an auditory information processing unit recognizes the volume of the sound picked up by the sound collection unit and generates an auditory attention map by projecting the position of the sound in three-dimensional space onto a two-dimensional attention map centered on the robot, a visual information processing unit generates a visual attention map using a face detection result that detects a human face using an image captured by the image capture unit and a motion detection result that detects the motion of the human, an attention map generation unit integrates the auditory attention map and the visual attention map to generate an attention map, and a motion processing unit uses the attention map to control eye movement and the motion of the communication robot.

[0013] (7) A program according to one embodiment of the present invention causes a computer to recognize the volume of sound picked up by a sound pickup unit, generate an auditory attention map by projecting the position of the sound in three-dimensional space onto a two-dimensional attention map centered on the robot, generate a visual attention map using a face detection result in which a human face is detected using an image captured by a capture unit and an action detection result in which the action of the human is detected, generate an attention map by integrating the auditory attention map and the visual attention map, and use the attention map to control eye movement and the action of the communication robot. Effect of the Invention

[0014] According to the above aspects (1) to (7), it is possible to control a robot by integrating both visual and auditory attention. [Brief description of the drawings]

[0015] [Figure 1] FIG. 2 is a diagram illustrating an example of communication by a communication robot according to an embodiment. [Diagram 2]1 is a diagram showing an example of the external shape of a communication robot according to an embodiment; [Diagram 3] FIG. 1 is a diagram illustrating an outline of a bottom-up attention map generation including visual and auditory perception according to an embodiment. [Figure 4] 1 is a block diagram showing an example of the configuration of a communication robot according to an embodiment. FIG. [Diagram 5] 11A to 11C are diagrams illustrating an example of processing performed by a visual information processing unit, an auditory information processing unit, and a bottom-up attention map generating unit according to the embodiment. [Figure 6] 6A to 6C are diagrams illustrating an example of a processing result of each unit according to the embodiment. [Figure 7] 6A to 6C are diagrams illustrating an example of a processing result of each unit according to the embodiment. [Figure 8] A diagram showing the flow of cognition, learning, and social abilities performed by a communication robot of an embodiment. [Figure 9] FIG. 11 is a diagram illustrating an example of data recognized by a recognition unit according to the embodiment. [Figure 10] FIG. 11 is a diagram illustrating an example of an agent creation method used by the action processing unit according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component is of a recognizable size.

[0017] <Summary> FIG. 1 is a diagram showing an example of communication by a communication robot 1 according to an embodiment. As shown in FIG. 1, the communication robot 1 communicates with an individual or multiple people 2. The communication is mainly dialogue g11 and gestures g12 (movements). The movements are expressed by actual movements as well as images displayed on the display unit. When an e-mail is sent to a user via an Internet line or the like, the communication robot 1 receives the e-mail and notifies the user of the arrival of the e-mail and its contents (g13). When a reply is required to the e-mail, for example, the communication robot 1 communicates with the user to ask if advice is necessary and makes a suggestion g14. The communication robot 1 transmits a reply (g15). The communication robot 1 also presents a weather forecast g16 for a location according to the scheduled date and time and location, for example, according to the user's schedule.

[0018] One aspect of the embodiment is an audio-visual bottom-up attention system for a communication robot 1. The system computes both visual and auditory attention and can be used to enable the communication robot 1 to exhibit a life-like initiative in reaching out to external stimuli.

[0019] <Example of the appearance of Communication Robot 1> Next, an example of the external shape of the communication robot 1 will be described. FIG. 2 is a diagram showing an example of the external shape of the communication robot 1 according to the embodiment. In the example of a front view g101 and a side view g102 in FIG. 2, the communication robot 1 is provided with three display units 111 (eye display unit 111a, eye display unit 111b, and mouth display unit 111c). In the example of FIG. 3, the photographing unit 102a is attached to the upper part of the eye display unit 111a, and the photographing unit 102b is attached to the upper part of the eye display unit 111b. The eye display units 111a and 111b correspond to human eyes and present image information. The speaker 114 is attached to the housing 120 near the mouth display unit 111c that displays an image corresponding to the human mouth. The sound collecting unit 103 is attached to the housing 120.

[0020] The communication robot 1 also includes a boom 121. The boom 121 is movably attached to the housing 120 via a movable part 131. A horizontal bar 122 is rotatably attached to the boom 121 via a movable part 132. Also, an eye display unit 111a is rotatably attached to the horizontal bar 122 via a movable part 133, and an eye display unit 111b is rotatably attached to the horizontal bar 122 via a movable part 134. The external shape of the communication robot 1 shown in FIG. 2 is an example and is not limited to this.

[0021] In the communication robot 1, the images displayed on the eye display units 111a and 111b correspond to images of human eyeballs, and the boom 121 corresponds to the human neck. The communication robot 1 performs eye movement by moving the positions of the eye images displayed on the eye display units 111a and 111b. The boom 121 is also configured to be tiltable forward and backward. When the eye images displayed on the eye display units 111a and 111b alone cannot accommodate the eye movement, the communication robot 1 controls the movement of the boom 121 to make the movement more natural.

[0022] <Bottom-up attention map including vision and hearing> Next, bottom-up attention will be described. Bottom-up attention is a sensory-driven selection mechanism that directs perception to a portion of a stimulus. The neural processes that control visual and auditory attention in robots are the same as in humans. This system adds the ability for the robot to automatically select the location of attention, including vision and hearing, in this embodiment.

[0023] 3 is a diagram showing an outline of generating a bottom-up attention map including vision and hearing according to an embodiment. As shown in FIG. 3, visual information is input to a visual attention map 501, and auditory information is input to an auditory attention map 502. In the embodiment, the visual attention map 501 and the auditory attention map 502 are integrated to generate a bottom-up attention map 511. In the embodiment, the line of sight and, for example, the neck movement of the communication robot 1 are controlled using this bottom-up attention map 511.

[0024] <Example of communication robot 1 configuration> Next, a configuration example of the communication robot 1 will be described. Fig. 4 is a block diagram showing an example of the configuration of a communication robot according to an embodiment. As shown in Fig. 4, the communication robot 1 includes a receiving unit 101, an image capturing unit 102, a sound collecting unit 103, a sensor 104, and a generating device 100. The generating device 100 includes, for example, a visual information processing unit 105, an auditory information processing unit 106, a bottom-up attention map generating unit 107, a memory unit 108, a model 109, a motion processing unit 110, an eye display unit 111 (first display unit), a mouth display unit 112 (second display unit), an actuator 113, a speaker 114, a transmitting unit 115, a recognition unit 150, and a learning unit 160.

[0025] The visual information processing unit 105 includes an image processing unit 1051 , a face detection unit 1052 , a motion detection unit 1053 , and a visual attention map generation unit 1054 . The auditory information processing unit 106 includes a power detection unit 1061 , a duration detection unit 1062 , and an auditory attention map generation unit 1063 . The movement processing unit 110 includes an eye image generating unit 1101 , a mouth image generating unit 1102 , a driving unit 1103 , a voice generating unit 1104 , and a transmission information generating unit 1105 .

[0026] The receiving unit 101 acquires information (eg, electronic blog information, news, weather forecast, etc.) from, for example, the Internet via a network, and outputs the acquired information to the operation processing unit 110.

[0027] The image capturing unit 102 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) image capturing element or a CCD (Charge Coupled Device) image capturing element. The image capturing unit 102 outputs the captured image to the visual information processing unit 105. The images are moving images or still images that are continuous in time. The communication robot 1 may be equipped with a plurality of image capturing units 102. In this case, the image capturing units 102 may be attached to, for example, the front and rear of the housing of the communication robot 1.

[0028] The sound collection unit 103 is, for example, a microphone array composed of a plurality of microphones. The sound collection unit 103 outputs the acoustic signals (human information) collected by the plurality of microphones to the auditory information processing unit 106. Note that the sound collection unit 103 may sample each of the acoustic signals collected by the microphones with the same sampling signal, convert the analog signals into digital signals, and then output the signals to the auditory information processing unit 106.

[0029] The sensor 104 is, for example, a temperature sensor that detects the temperature of the environment, an illuminance sensor that detects the illuminance of the environment, a gyro sensor that detects the inclination of the casing of the communication robot 1, an acceleration sensor that detects the movement of the casing of the communication robot 1, an air pressure sensor that detects the air pressure, etc. The sensor 104 outputs the detected value to the operation processing unit 110.

[0030] The visual information processing unit 105 uses the images captured by the image capturing unit 102 to generate a visual attention map.

[0031] The image processing unit 1051 performs well-known image processing on the captured image, such as feature detection, binarization, edge detection, contour detection, and clustering processing. The face detection unit 1052 uses the image-processed information to detect a human face, for example, in two dimensions. The movement detector 1053 detects, for example, the movement of a person's eyes in a two-dimensional image. The visual attention map generating unit 1054 uses the results detected by the face detecting unit 1052 and the results detected by the motion detecting unit 1053 to generate a visual attention map.

[0032] The auditory information processing unit 106 generates an auditory attention map by using the acoustic signal collected by the sound collection unit 103 . The power detection unit 1061 performs a fast Fourier transform on the collected acoustic signal to convert it into a signal in the frequency domain, and then detects the acoustic power by a known method. The duration detection unit 1062 detects, for example, the duration of one phrase using a known method. Auditory attention map generating section 1063 generates an auditory attention map using the result detected by power detecting section 1061 and the result detected by duration detecting section 1062 .

[0033] The bottom-up attention map generator 107 generates a bottom-up attention map by integrating the visual attention map and the auditory attention map.

[0034] The storage unit 108 stores programs, algorithms, predetermined values, thresholds, etc., required for various controls and processes of the communication robot 1. The storage unit 108 stores a visual attention map, an auditory attention map, and a bottom-up attention map. The storage unit 108 also stores a face attention map, an action attention map, and a saliency map. The storage unit 108 stores, for example, a language model database, an acoustic model database, a dialogue corpus database, and acoustic features used in speech recognition, a comparison image database and image features used in image recognition, etc. The storage unit 108 stores data related to relationships between people, such as social components, social norms, social customs, psychology, and humanities, which are used during learning. The storage unit 108 may be placed on a cloud or connected via a network.

[0035] Model 109 is a model for creating each attention map. The model for creating a visual attention map has visual information as input and a visual attention map as output. The model for creating an auditory attention map has auditory information as input and an auditory attention map as output. The model for creating a bottom-up attention map has a visual attention map and an auditory attention map as input and an output, respectively. The model for creating each attention map is created by learning using known information and teacher data.

[0036] The action processing unit 110 generates images to be displayed on the eye display unit 111 and the mouth display unit 112, generates a drive signal to drive the actuator 113, generates sound to be output from the speaker 114, and generates transmission information to be transmitted from the transmission unit 115.

[0037] The eye image generation unit 1101 uses the bottom-up attention map to generate an output image (a still image, a series of still images, or a video) to be displayed on the eye display unit 111, and causes the generated output image to be displayed on the eye display unit 111. The displayed image is an image that corresponds to the movement of a human eye.

[0038] The mouth image generation unit 1102 uses the bottom-up attention map to generate an output image (a still image, a series of still images, or a video) to be displayed on the mouth display unit 112, and causes the generated output image to be displayed on the mouth display unit 112. The displayed image is an image that corresponds to the movement of a person's mouth.

[0039] The driver 1103 uses the bottom-up attention map to generate a drive signal for driving at least the neck actuator 113, and drives the actuator 113 with the generated drive signal.

[0040] The audio generating unit 1104 generates an output audio signal to be output from the speaker 114 based on the information received by the receiving unit, and causes the speaker 114 to output the generated output audio signal.

[0041] The transmission information generating unit 1106 generates transmission information to be transmitted based on the received information, the captured image, and the collected audio signal, and causes the transmitting unit 115 to transmit the generated transmission information.

[0042] 2, there are two eye display units 111, one on the left and one on the right, and are, for example, a liquid crystal image display device, an organic EL (Electro Luminescence) image display device, etc. Eye display units 111 display the left and right eye images output by eye image generation unit 1101. Note that eye display units 111 are movable, for example, up and down, left and right, rotational directions, and forward and backward.

[0043] Mouth display unit 112 is, for example, an LED (light emitting diode), etc. Mouth display unit 112 displays the mouth image output by mouth image generation unit 1102.

[0044] The actuator 113 drives at least the movement section of the neck in response to the drive signal output by the drive section 1103. The neck is configured to be able to tilt forward and backward, for example.

[0045] The speaker 114 outputs the output audio signal output by the audio generation unit 1104 .

[0046] The transmitting section 115 transmits the transmission information output by the transmission information generating section 1105 to a destination via a network.

[0047] The recognition unit 150 recognizes interactions occurring between the communication robot 1 and a person, or interactions occurring between multiple people. The recognition unit 150 acquires images captured by the imaging unit 102, acoustic signals collected by the sound collection unit 103, and detection values ​​detected by the sensor 104. The recognition unit 150 may acquire information received by the receiving unit 101. The recognition unit 150 recognizes interactions occurring between the communication robot 1 and a person, or interactions occurring between multiple people, based on the acquired information and data stored in the memory unit 108. The recognition method will be described later. The recognition unit 150 outputs the recognition results (feature amounts related to sounds, feature information related to human behavior) to the learning unit 160 and the motion processing unit 110.

[0048] The learning unit 160 learns human emotional interactions using the recognition result output by the recognition unit 150 and the data stored in the storage unit 108. The learning unit 160 stores the model generated by the learning. The learning method will be described later.

[0049] <Examples of Processing Performed by the Visual Information Processing Unit, the Auditory Information Processing Unit, and the Bottom-Up Attention Map Generator> Next, the processing performed by the visual information processing unit, the auditory information processing unit, and the bottom-up attention map generator will be described. FIG. 5 is a diagram illustrating an example of processing performed by a visual information processing unit, an auditory information processing unit, and a bottom-up attention map generating unit according to the embodiment.

[0050] I. Visual Attention Map First, the visual attention map will be described. The visual information processing unit 105 uses the saliency map to decompose each capture frame from the image captured by the image capturing unit 102 into two visual attention maps (A(·)). The saliency map is created by detecting faces and detecting movements from the captured image. The visual information processing unit 105 also decomposes the saliency map into two: a visual attention map based on face detection and a visual attention map based on movement detection. The first visual attention map is the face attention map (A(F i)) and the detected face region i is highlighted by the face size value. The second visual attention map is the action attention map (A(M k )) and highlights the detected moving object (e.g., eye) k with the value of its motion speed.

[0051] The face detection unit 1052 detects faces using, for example, the Viola-Jones Haar Cascade Classifier method (see Reference 1). When a face is detected in the input image, the face detection unit 1052 returns the position of each detected face Fi as a rectangular coordinate and size (x, y, w, h). x is the length in the x-axis direction, y is the length in the y-axis direction, w is the width of the face, and h is the height of the face. The area of ​​one face is a circle whose center is at coordinates (x+w / 2, y+h / 2) and whose diameter is w. The center position Loc of the face and the size of the face size are expressed by the following formula (1).

[0052]

number

[0053] Loc is the coordinate position of the center of the face detected on each frame, and size is the pixel size of each face on the frame screen. These two values ​​represent the position and size of all faces detected on the frame. Therefore, if the position and size of the face are known, it is possible to create a face attention map A(Fi) in which the pixel value of the face area is 1 and the pixel value of other pixels is 0.

[0054] The motion detection unit 1053 generates a motion attention map A(M k ) is generated. For each moving object in the motion attention map, the value increases according to the speed. Therefore, the motion detection unit 1053 generates kEach pixel within the range is assigned a value from 0 to 1. The value of each pixel within the range of a moving object has a predetermined range. This value increases according to the position and speed of each moving object in the motion attention map. The motion detection unit 1053 detects a predetermined range, for example, the movement of a person's eyes, in a two-dimensional image. The predetermined range is, for example, a range from 0 to 1. The motion detection value Mj is expressed as in the following equation (2).

[0055]

number

[0056] Reference 1;Paul Viola and Michael J. Jones. “Robust real-time face detection”, International Journal of Computer Vision, 57(2):137-154, 2004. Reference 2; AB Godbehere, A. Matsukawa and K. Goldberg, “Visual tracking of human visitors under variable-lighting conditions for a responsive audio art installation” 2012 American Control Conference (ACC), Montreal, QC, 2012, pp. 4305-4312, doi: 10.1109 / ACC.2012.6315174.

[0057] The visual attention map generator 1054 integrates the face attention map and the action attention map, and generates one visual attention map (A v Note that the visual attention map is a two-dimensional binary image used in the embodiment, but is not limited to this, and may be, for example, three-dimensional with ternary or more values.

[0058]

number

[0059] In this manner, in the embodiment, a visual attention map is created that focuses on the features of the face and the movement. In each face region of each frame, a different pixel weight is set according to the speed of the face movement. The attention value of different faces can be calculated from the size of the pixels in the face region. Factors that affect the weight of the attention value are the size of the face region in addition to the speed of movement. Here, the larger the face region, the closer the face is to the image capture unit 102, and the higher the attention value.

[0060] II. Auditory Attention Map Next, the auditory attention map will be described. In the auditory attention model, the visual attention map is estimated as a two-dimensional binary image of the same size as the auditory attention map, which is a projection surface for capturing acoustic information in three-dimensional space.

[0061] The power detector 1061 performs a sound source localization process on the collected sound signal using, for example, MUSIC (Multiple Signal Classification). The power detector 1061 determines a valid location Φ j and the power p is calculated using the framewise method.

[0062] The duration detection unit 1062 detects the duration T of the sound event candidate j in a particular direction by detecting a section in which the speech is equal to or longer than a predetermined threshold value. The auditory attention map generator 1063 projects the sound source at a position Φ j , power p, duration T, and orientation-based binary image. Auditory attention map A a can be created with each circle representing a sound source. The center coordinate (L) of each circle is the position where the sound direction is projected three-dimensionally onto a two-dimensional image.

[0063] The power p of the sound source is set to 0 or 1 depending on the power level threshold. Then, calculation of the duration T' starts when the loudness of the sound exceeds the threshold T' = T + 1. The value of the duration T is calculated as the diameter of the sound source circle using a function such as the following equation (4).

[0064]

number

[0065] The bottom-up attention map generator 107 generates a visual attention map A v and Auditory Attention Map A a The bottom-up attention map generating unit 107 integrates the bottom-up attention map (A M ) is synthesized from all normalized attention maps, which have different values ​​at each location on the same image size for each frame, by the following equation (5):

[0066]

number

[0067] Thus, in the embodiment, the size of the face and the level of the sound source are changed to fuse visual and auditory information into a single attention map.

[0068] In the embodiment, the objective is to map the attention of the communication robot 1 to the location where the user pays the most attention using an attention map that integrates vision and hearing. For this purpose, in the embodiment, the “winner-take-all” mechanism (see Reference 3), which is a neural network model of selective visual attention provided by Koch and Ullman, is used.

[0069] Reference 3; Koch, C., Ullman, S. (1985)., “Shifts in selective visual attention: Towards the underlying neural circuitry”, Human Neurobiology, 4(4), 219-227.

[0070] In Koch and Ullman's model, feature maps are integrated by a bottom-up winner-take-all saliency map of stimuli. In contrast, in an embodiment, the visual and auditory attention maps are merged into a single attention map, and then a "winner-take-all" competition between the overlapping attention regions is activated, and one region is identified as the "focus of attention", leaving only the most attention-grabbing location and suppressing the rest.

[0071] Using this attention model, closer faces typically receive more attention than more distant faces. Moving faces receive more attention than still faces. Also, talking faces receive more attention than silent faces. These examples suggest that this attention model closely resembles humans in everyday interactions.

[0072] <Example of communication robot control using bottom-up attention map> Next, an example of control of the communication robot 1 using the bottom-up attention map will be described. In this embodiment, the movement of the eyes and neck is controlled to move the gaze of the communication robot 1 to the direction in which the person is most paying attention, based on the integrated bottom-up attention map.

[0073] First, eye movements will be described. The eye image generating unit 1101 moves the center position of the eye to match the waving point on the bottom-up attention map. The eye image generating unit 1101 regards the screen on which the eye is displayed as the retina, and crops the image obtained from the photographing unit 102 to the same size as the peak-to-peak amplitude of the eye (for example, 40°).

[0074] This eye image is seen by the user as an image on the retina of the communication robot 1. Note that the eye is never stationary, but is a small eye movement that occurs continuously during fixation. For this reason, in the embodiment, the state in which the eye is located at the center of the screen with a microsaccade whose amplitude is smaller than 0.5° (e.g., 9 pixels on the screen) is defined as the initial state of the eye.

[0075] In the above-mentioned eye, the range in which the eyeball can move is, for example, determined in advance. When a stimulus is projected within this range, the eye image generating unit 1101 moves the image of the eyeball to the projected position.

[0076] Next, the movement of the neck will be described. If the projection is out of range, eye image generator 1101 first moves the eye image to the projected position. Then driver 1103 starts head movement in the same direction to bring the eye position on the screen back within the central range. The neck control is controlled according to the following rule of eye-head coordination, Equation (6), while the eyes are moving beyond the eye-only range.

[0077]

number

[0078] In equation (6), α is the angular distance vector between the eye position and the center of the LCD screen. The neck moves when α is smaller than the range θ of the eyeballs alone. The angle β of the neck movement is the same magnitude and direction as α in order to return the eyeballs to the center position of the screen. θ is, for example, 10.

[0079] If there is an object that deserves attention in the vicinity of the communication robot 1, the eyes of the communication robot 1 first move to that object, and then the head of the communication robot 1 starts to move in the same direction and at the same angle. This is the same as the human vestibulo-ocular reflex (VOR), and is a behavior to focus the gaze on the object of the salivary gland.

[0080] In this way, in this embodiment, the gaze is automatically guided to a location that requires attention (the size and movement of the face are counted in the attention value).The bottom-up attention model combines different sensory modalities (vision and hearing).

[0081] 6 and 7 are diagrams showing examples of processing results of each unit according to this embodiment. Image g101 in FIG. 6 and image g111 in FIG. 7 are images showing how the gaze of the communication robot 1 automatically follows the location of attention. Image g102 in FIG. 6 and image g112 in FIG. 7 represent visual attention, and are the results of detecting a human face and its movement. Image g103 in FIG. 6 and image g113 in FIG. 7 are bottom-up attention maps, which are attention maps obtained by integrating visual information and auditory information.

[0082] For example, the visual attention map first detects faces and movements, the value of each face is calculated by its size and movement characteristics, and the eye follows the face with the maximum attention value. If there is sound, finally, all stimuli are compared in the bottom-up attention map, and the eye automatically follows the location of the maximum value.

[0083] In the audiovisual attention system of the prior art, only simple visual features were considered. In contrast, the audiovisual attention system of the present embodiment is a bottom-up system that recognizes the faces of people, moving objects, and sounds in the environment. In this embodiment, the loudness of the sound is recognized, and the position of the sound in three-dimensional space is projected onto a two-dimensional attention map centered on the robot. In addition, in this embodiment, the magnitude of the visual attention and the auditory attention are automatically calculated and integrated into the attention map, and finally the maximum attention value is selected as the attention position of the robot.

[0084] (Cognitive, learning, and social skills stream) In addition, the communication robot 1 may generate social capabilities for the robot so that an emotional connection can be formed between the robot and a person, and may be able to communicate with a person, for example, in response to the person's reactions or actions. Next, a description will be given of the flow of cognition and learning performed by the communication robot 1. Fig. 8 is a diagram showing the flow of cognition, learning, and social ability performed by the communication robot 1 of this embodiment.

[0085] The recognition result 201 is an example of a result recognized by the recognition unit 150. The recognition result 201 is, for example, an interpersonal relationship, an interpersonal reciprocity, or the like.

[0086] Multimodal learning and understanding 211 is an example of the learning content performed by the learning unit 160. Learning method 212 is machine learning, etc. Furthermore, learning subject 213 is social components, social models, psychology, humanities, etc.

[0087] Social competencies 221 are social skills such as empathy, individuation, adaptability, emotional ahodan- ces, etc.

[0088] (Data to be recognized) Next, an example of data recognized by the recognition unit 150 will be described. 9 is a diagram showing an example of data recognized by the recognition unit 150 according to this embodiment. In this embodiment, personal data 301 and interpersonal relationship data 351 are recognized as shown in FIG.

[0089] The personal data is behavior that occurs within one person, and includes data acquired by the image capturing unit 102 and the sound collecting unit 103, and data obtained by performing voice recognition processing, image recognition processing, etc. on the acquired data. The personal data includes, for example, voice data, meaning data resulting from voice processing, voice volume, voice intonation, spoken words, facial expression data, gesture data, head posture data, face direction data, gaze data, co-occurrence expression data, physiological information (body temperature, heart rate, pulse rate, etc.), etc. The type of data to be used may be selected, for example, by the designer of the communication robot 1. In this case, for example, for actual communication between two people or a demonstration, the designer of the communication robot 1 may set important features of the personal data in communication. The recognition unit 150 also recognizes the user's emotions as personal data based on information extracted from the acquired speech and image. In this case, the recognition unit 150 recognizes, for example, based on the voice volume and intonation, speech duration, facial expression, etc. The communication robot 1 of the embodiment controls the user's emotions to maintain positive emotions and to maintain a good relationship with the user.

[0090] Here, an example of a method for recognizing a user's social background will be described. The recognition unit 150 estimates the user's nationality, place of origin, etc., based on the acquired speech, image, and data stored in the storage unit 108. The recognition unit 150 extracts the user's life schedule, such as the time of waking up, the time of going out, the time of returning home, and the time of going to bed, based on the acquired speech, image, and data stored in the storage unit 108. The recognition unit 150 estimates the user's gender, age, occupation, hobbies, career, preferences, family structure, religion, and attachment level to the communication robot 1, etc., based on the acquired speech, image, life schedule, and data stored in the storage unit 108. Since the social background may change, the communication robot 1 updates information about the user's social background based on the conversation, image, and data stored in the storage unit 108. In order to enable emotional sharing, the social background and the attachment level to the communication robot 1 are recognized not only based on the inputtable levels of age, gender, career, etc., but also based on, for example, emotional fluctuations according to the time of day, the volume and intonation of the voice in response to the topic, etc. In this way, the recognition unit 150 learns things that the user himself does not realize based on daily conversations and facial expressions during conversations.

[0091] Interpersonal relationship data is data about the relationship between the user and other people. By using interpersonal relationship data in this way, social data can be used. Interpersonal relationship data includes, for example, the distance between people, whether or not the people talking make eye contact, the intonation of the voice, the volume of the voice, etc. The distance between people differs depending on the interpersonal relationship, as will be described later. For example, the interpersonal relationship between a husband and wife or friends is L1, while the interpersonal relationship between businessmen is L2, which is larger than L1.

[0092] For example, the designer of the communication robot 1 may set important features of the interpersonal data in communication for an actual communication between two people or a demonstration. Such personal data, interpersonal relationship data, and information on the social background of the user are stored in the storage unit 108.

[0093] Furthermore, in the case of multiple users, for example, a user and his / her family, the recognition unit 150 collects and learns personal data for each user, and estimates the social background for each person. Note that such social background may be acquired, for example, via a network and the receiving unit 101, and in this case, the user may input his / her social background or select items on, for example, a smartphone.

[0094] Here, an example of a method for recognizing interpersonal relationship data will be described. The recognition unit 150 estimates the distance (spacing) between people who are communicating based on the acquired utterances, images, and data stored in the memory unit 108. The recognition unit 150 detects whether or not the eyes of people who are communicating meet based on the acquired utterances, images, and data stored in the memory unit 108. The recognition unit 150 estimates friendships, work colleagues, and relatives / parent-child relationships based on the contents of the utterances, the volume of the voice, the intonation of the voice, and the recipients of received and sent e-mails and sent and received e-mails based on the acquired utterances and data stored in the memory unit 108.

[0095] Note that, in the initial state used, the recognition unit 150 may start communication by, for example, randomly selecting one combination from among several combinations of initial values ​​of social backgrounds and personal data stored in the storage unit 108. Then, if the behavior generated by the randomly selected combination makes it difficult to continue communication with the user, the recognition unit 150 may select another combination.

[0096] (Learning Procedure) In the embodiment, the learning unit 160 performs learning using the personal data 301 and interpersonal relationship data 351 recognized by the recognition unit 150 and the data stored in the storage unit 108.

[0097] Here, we will explain social structure and social norms. In spaces where people participate in social interactions, interpersonal relationships differ, for example, depending on the careers of the people. For example, a relationship where the distance between people is 0-50cm is an intimate relationship, and a relationship where the distance between people is 50-1m is a personal relationship. A relationship where the distance between people is 1-4m is a social relationship, and a relationship where the distance between people is 4m or more is a public relationship. During learning, such social norms are used as rewards (implicit rewards) based on whether or not gestures and speech conform to the social norms.

[0098] Also, the interpersonal relationship may be set according to the environment and the user in which the robot is used, by setting the reward feature during learning. Specifically, multiple intimacy settings may be provided, such as a rule to not talk much to people who do not like robots, and a rule to talk actively to people who like robots. Then, in the real environment, the recognition unit 150 may recognize which type the user is based on the results of processing the user's speech and images, and the learning unit 160 may select a rule.

[0099] Furthermore, a human trainer may evaluate the behavior of the communication robot 1 and provide rewards (implicit rewards) according to the social structure and norms that the human trainer is familiar with.

[0100] FIG. 10 is a diagram showing an example of an agent creation method used by the action processing unit 110 according to this embodiment. The area indicated by the reference numeral 300 shows the flow from input to agent creation and output (agent). The image captured by the image capturing unit 102 and the information 310 captured by the sound collecting unit 103 are information about people (users, people related to the user, and other people) and environmental information around the people. The raw data 302 acquired by the image capturing unit 102 and the sound collecting unit 103 are input to the recognition unit 150.

[0101] The recognition unit 150 extracts and recognizes a plurality of pieces of information (such as the volume of the voice, the intonation of the voice, the content of the speech, the words spoken, the user's gaze, the user's head posture, the user's face direction, the user's biological information, the distance between people, whether or not people are making eye contact, etc.) from the input raw data 302. The recognition unit 150 uses the plurality of pieces of information extracted and recognized to perform multimodal understanding using, for example, a neural network. The recognition unit 150 identifies an individual based on at least one of a voice signal and an image, for example, and assigns identification information (ID) to the identified individual. The recognition unit 150 recognizes the behavior of each identified person based on at least one of a voice signal and an image. The recognition unit 150 recognizes the gaze of the identified person, for example, by performing well-known image processing and tracking processing on the image. The recognition unit 150 recognizes the voice by performing voice recognition processing (sound source identification, sound source localization, sound source separation, speech period detection, noise suppression, etc.) on the voice signal, for example. The recognition unit 150 recognizes the head posture of the identified person, for example, by performing well-known image processing on the image. For example, when two people are captured in a captured image, the recognition unit 150 recognizes the interpersonal relationship based on the content of the utterance, the distance between the two people in the captured image, etc. The recognition unit 150 recognizes (estimates) the social distance between the communication robot 1 and the user, for example, according to the results of processing the captured image and the collected voice signal.

[0102] The learning unit 160 performs reinforcement learning 304, not deep learning. In reinforcement learning, learning is performed to select the most relevant feature (including social structure and social norms). In this case, multiple pieces of information used in multimodal understanding are used as features for input. The input to the learning unit 160 is, for example, raw data itself, a name ID (identification information), face influence, recognized gestures, keywords from voice, etc. The output of the learning unit 160 is the behavior of the communication robot 1. The output behavior may be anything that is desired to be defined according to the purpose, such as a voice response, a robot routine, and an orientation angle for the robot to rotate. Note that in multimodal understanding, a neural network or the like may be used for detection. In this case, different modalities of the body may be used to detect human activity. Also, which features to use may be selected in advance, for example, by the designer of the communication robot 1. Furthermore, in this embodiment, by using implicit rewards and explicit rewards during learning, social exemplars and social constructs can be incorporated. The result of reinforcement learning is the output, which is the agent 305. In this manner, in this embodiment, an agent to be used by the action processing unit 110 is created.

[0103] The area indicated by the reference numeral 350 shows how rewards can be used. The implicit reward 362 is used to learn the implicit response. In this case, the raw data 302 includes the user's response, and the raw data 302 is subjected to the above-mentioned multimodal understanding 303. The learning unit 160 generates an implicit response system 372 using the implicit reward 362 and the social exemplars stored in the memory unit 108. The implicit reward may be obtained by reinforcement learning or may be given by a human being. The implicit response system may also be a model acquired by learning.

[0104] In learning explicit responses, for example, a human trainer evaluates the behavior of the communication robot 1 and gives a reward 361 according to the social structure and social norms that the human trainer knows. The agent adopts the behavior that maximizes the reward for the input. This allows the agent to adopt behaviors (utterances, gestures) that maximize positive feelings toward the user.

[0105] The learning unit 160 uses this explicit reward 361 to generate an explicit response system 371. The explicit response system may be a model acquired by learning. The explicit reward may be given by the user after evaluating the behavior of the communication robot 1, or the reward may be estimated based on whether the communication robot 1 has performed the behavior desired by the user, for example, based on the user's utterances and behaviors (gestures, facial expressions, etc.). During operation, the learning unit 160 uses these learning models to output an agent 305 .

[0106] In the embodiment, for example, an explicit reward, which is a user's reaction, is prioritized over an implicit reward, because a user's reaction has a higher degree of reliability in communication.

[0107] As described above, in this embodiment, the destination of attention is determined based on sound and video. As a result, this embodiment can provide an efficient control strategy for deploying attention based on bottom-up cues including visual and auditory inputs in a centralized two-dimensional attention map.

[0108] In addition, a program for realizing all or part of the functions of the communication robot 1 of the present invention may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to perform all or part of the processes performed by the communication robot 1. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a homepage providing environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" also includes those that hold a program for a certain period of time, such as volatile memory (RAM) inside a computer system that becomes a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0109] The above program may also be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be for realizing part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0110] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0111] 1...communication robot, 101...receiving unit, 102...photographing unit, 103...sound collection unit, 104...sensor, 105...visual information processing unit, 106...auditory information processing unit, 107...bottom-up attention map generation unit, 108...memory unit, 109...model, 110...motion processing unit, 111...eye display unit, 112...mouth display unit 112, 113...actuator, 114...speaker, 115... Transmitting unit, 150...recognition unit, 160...learning unit, 1051...image processing unit, 1052...face detection unit, 1053...motion detection unit, 1054...visual attention map generation unit, 1061...power detection unit, 1062...duration length detection unit, 1063...auditory attention map generation unit, 1101...eye image generation unit, 1102...mouth image generation unit, 1103...drive unit, 1104...speech generation unit, 1105...transmission information generation unit

Claims

1. A communication robot, an auditory information processing unit that recognizes the volume of the sound picked up by the sound pickup unit and projects the position of the sound in a three-dimensional space onto a two-dimensional attention map centered on the robot to generate an auditory attention map; a visual information processing unit that generates a visual attention map using a face detection result that detects a human face using an image captured by the capture unit and a motion detection result that detects a motion of the human; an attention map generator that generates an attention map by integrating the auditory attention map and the visual attention map; a motion processing unit that controls eye movement and a motion of the communication robot using the attention map; Equipped with The attention map generator integrates the visual attention map and the auditory attention map from all normalized tension maps having different values ​​at each location on the same image size of each frame. Communication robot.

2. The visual information processing unit generates the visual attention map by decomposing a saliency map created by detecting a face and detecting a movement from a captured image into two visual attention maps, that is, the visual attention map created by detecting a face and the visual attention map created by detecting a movement. The communication robot according to claim 1.

3. The visual attention map by detecting the face is a face attention map, and the detected face area is highlighted by a face size value; The visual attention map based on motion detection is a motion attention map, which highlights the detected moving objects by the value of their motion speed. The communication robot according to claim 2.

4. The auditory information processor calibrates the projection of the sound source with a facing direction based binary image of position, power, and duration, and creates each circle as a representation of a sound source; The center coordinates of each circle are the positions where the sound direction is three-dimensionally projected onto a two-dimensional image. The communication robot according to claim 1.

5. an auditory information processing unit recognizes the volume of the sound picked up by the sound pickup unit, and projects the position of the sound in three-dimensional space onto a two-dimensional attention map centered on the robot to generate an auditory attention map; a visual information processing unit generating a visual attention map using a face detection result that detects a human face using the image captured by the image capturing unit and a motion detection result that detects a motion of the human; an attention map generating unit generating an attention map by integrating the auditory attention map and the visual attention map; a motion processing unit that uses the attention map to control eye movement and a motion of the communication robot; The attention map generator integrates the visual attention map and the auditory attention map from all normalized tension maps having different values ​​at each location on the same image size of each frame. A method for controlling a communication robot.

6. On the computer, The loudness of the sound picked up by the sound pickup unit is recognized, and an auditory attention map is generated by projecting the position of the sound in three-dimensional space onto a two-dimensional attention map centered on the robot. generating a visual attention map using a face detection result in which a person's face is detected using the image captured by the capture unit and a motion detection result in which a motion of the person is detected; merging the auditory attention map and the visual attention map to generate an attention map; Using the attention map, eye movements and the behavior of a communication robot are controlled; Integrating the visual attention map and the auditory attention map from all normalized tension maps that have different values ​​at each location on the same image size for each frame; program.

Images