Viewing state estimation device, robot system, viewing state estimation method, and viewing state estimation program
The viewing state estimation device uses panoramic imaging and distance detection to estimate viewer states without additional devices, addressing installation and comfort issues, enabling accurate robot interaction based on viewer behavior.
Patent Information
- Application Number
- JP2020162380
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-09-28
- Publication Date
- 2024-12-09
- Estimated Expiration
- 2040-09-28
AI Technical Summary
Conventional methods for estimating viewing conditions in a daily environment are limited by the need for additional devices like cameras or eyeglass-type gaze direction acquisition, which are cumbersome and difficult to install, and often fail to accurately capture the viewer's state due to installation constraints and discomfort.
A viewing state estimation device that uses a panoramic image unit to synthesize images around a robot, detects the television and viewer positions, calculates distances and angles, and determines the viewing direction and state without additional devices like cameras, using a robot system with imaging and distance detection units to estimate the viewer's state based on panoramic images.
Enables accurate estimation of the viewer's state in a daily environment without additional equipment, allowing the robot to adjust its behavior based on the viewer's viewing direction and state, improving user comfort and interaction accuracy.
Smart Images

Figure 0007596105000001 
Figure 0007596105000002 
Figure 0007596105000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an apparatus, a method and a program for estimating a state of a television viewer. [Background technology]
[0002] 2. Description of the Related Art Conventionally, for a robot that watches a television program or other video together with a viewer, technology has been researched for controlling the movement of the robot in accordance with the viewing state of the viewer. Techniques for estimating the viewing state include, for example, a method of installing a camera to observe in a room where a television is being watched in order to detect the direction in which the viewer is facing, or a method of having the viewer wear a glasses-type gaze direction acquisition device to acquire gaze direction data of the viewer.
[0003] Furthermore, as a control according to the estimated viewing state, for example, Patent Document 1 proposes an apparatus that detects the user's line of sight, determines the display position for projecting an image from an image projection device, and performs geometric correction on the displayed image to display it, thereby displaying an image that is easy for the user to see. Moreover, Patent Document 2 proposes a device that detects a viewing state from an image including a viewer viewing content, and associates the state with the elapsed time from the start of the content being viewed. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2017-55178 A [Patent Document 2] Patent No. 6614547 Summary of the Invention [Problem to be solved by the invention]
[0005] Among conventional methods for estimating the viewing state, a method for estimating the gaze direction from the viewer's video in a specific location, such as a laboratory with cameras installed on the ceiling, wall, or above the television, is only carried out experimentally for a certain period of time, and therefore it is difficult to estimate the viewer's gaze direction in an everyday viewing environment. In addition, the method of having the viewer wear a glasses-type gaze direction acquisition device on their head is different from everyday viewing conditions, and wearing the device creates a sense of discomfort and puts a strain on the viewer. Therefore, this method is also difficult to use in everyday viewing environments.
[0006] In the method of Patent Document 1, imaging devices are provided at the four corners of the ceiling in a room, and the user's gaze direction is estimated based on the captured images acquired from the imaging devices. In this case, it is necessary to install imaging devices in the room, but it is difficult to install them in a home. In addition, multiple imaging devices are required to capture the entire room.
[0007] In the method of Patent Document 2, the viewing state of a viewer is determined based on vital information of the viewer extracted from a camera image including the viewer, but the camera needs to be installed on the top of the display, etc., which differs from everyday viewing conditions. In addition, this method cannot obtain information such as when the viewer is facing someone else instead of the display and talking, and it also reflects vital information when the viewer is not actually watching, making it impossible to determine the appropriate viewing state.
[0008] An object of the present invention is to provide a viewing state estimation device, a viewing state estimation method, and a viewing state estimation program that can estimate the viewing state of a viewer in a robot without using an additional device such as a camera. [Means for solving the problem]
[0009] The viewing state estimation device of the present invention includes a panoramic image unit that acquires an omnidirectional panoramic image synthesized from images captured around a robot, a distance panoramic image unit that generates a distance panoramic image in which pixel values correspond to distance data corresponding to each pixel of the panoramic image, a television detection unit that detects a television position from the panoramic image, a viewer detection unit that detects a face position of a viewer from the panoramic image, a distance acquisition unit that acquires distances at the television position and the face position from the distance panoramic image, respectively, a viewing direction detection unit that specifies the positional relationship between the robot, the viewer, and the television by calculating the angle between the television and the viewer as seen by the robot based on the size of the panoramic image, the television position, and the face position, and acquires a viewing direction image at the viewing direction angle of the viewer from the panoramic image based on the face direction angle of the viewer obtained from the face position image, and a viewing state determination unit that detects an object included in the viewing direction image and determines the state of the viewer based on the type of the object.
[0010] The viewing direction detection unit may approximate a distance from the viewer to a viewing direction position by a sum of a distance from the robot to the viewer and a distance from the robot to the viewing direction position.
[0011] The viewing state determination unit may calculate the viewing state based on statistical information on the state of the viewer within a certain period of time.
[0012] The viewing state determination section may calculate, as the viewing state, a viewing degree indicating a proportion of the time the user is watching the television.
[0013] The viewing state determination section may determine a plurality of states as the viewing state, including a state of watching the television and a state of watching another person.
[0014] The robot system according to the present invention includes the viewing state estimation device, and an operation control unit that controls the operation of the robot based on the result of comparing the viewing degree output from the viewing state estimation device with a predetermined threshold value.
[0015] The operation control unit may gradually change control of the robot based on a plurality of threshold values.
[0016] A robot system according to the present invention includes the viewing state estimation device, and an operation control unit that controls an operation of the robot in accordance with the plurality of state classifications output from the viewing state estimation device.
[0017] The viewing state estimation method according to the present invention includes a panoramic image generation step of acquiring an omnidirectional panoramic image synthesized from images captured around a robot; a distance panoramic image generation step of generating a distance panoramic image in which distance data corresponding to each pixel of the panoramic image is used as a pixel value; a television detection step of detecting a television position from the panoramic image; a viewer detection step of detecting a face position of the viewer from the panoramic image; a distance acquisition step of acquiring distances at the television position and the face position from the distance panoramic image; a viewing direction detection step of calculating an angle between the television and the viewer as seen by the robot based on a size of the panoramic image, the television position, and the face position, thereby specifying a positional relationship between the robot, the viewer, and the television, and acquiring a viewing direction image at the viewing direction angle of the viewer from the panoramic image based on the face direction angle of the viewer obtained from the image of the face position; and a viewing state determination step of detecting an object included in the viewing direction image and determining the state of the viewer based on the type of the object.
[0018] A viewing state estimating program according to the present invention is for causing a computer to function as the viewing state estimating device. Effect of the Invention
[0019] According to the present invention, the viewing state of the viewer can be estimated in the robot without using any additional device such as a camera. [Brief description of the drawings]
[0020] [Figure 1] FIG. 1 is a diagram illustrating a usage scene of a robot incorporating a viewing state estimating device according to an embodiment. [Diagram 2] 1 is a block diagram showing a functional configuration of a viewing state estimation device according to an embodiment. [Diagram 3] 10A and 10B are diagrams illustrating example synthesis position data according to an embodiment. [Figure 4] FIG. 4 is a diagram illustrating a functional configuration of a distance panorama image unit in the embodiment. [Diagram 5] 6A to 6C are diagrams illustrating an example of the operation of a distance image synthesis unit in the embodiment. [Figure 6] 11 is a diagram illustrating an example of the operation of an overlapping section calculation unit in the embodiment. FIG. [Figure 7] FIG. 4 is a diagram illustrating an example of a distance image database in the embodiment. [Figure 8] FIG. 4 is a diagram illustrating a distance panorama image database in the embodiment. [Figure 9] FIG. 4 is a diagram illustrating a functional configuration of a viewer detection unit in the embodiment. [Figure 10] FIG. 2 is a diagram illustrating a functional configuration of a viewing direction detection unit in the embodiment. [Figure 11] FIG. 2 is a diagram showing the positional relationship between a robot, a television, and a viewer in an embodiment. [Figure 12] 10A and 10B are diagrams illustrating a method for calculating an angle between a television and a viewer in an embodiment. [Figure 13] 4A and 4B are diagrams illustrating a face direction angle according to an embodiment. [Figure 14] FIG. 2 is a diagram showing the positional relationship between a robot, a viewer, and a viewing direction position in the embodiment. [Figure 15] 10A and 10B are diagrams illustrating a method for acquiring a viewing direction image in an embodiment. [Figure 16] FIG. 2 is a diagram illustrating a functional configuration of a viewing state determination unit in the embodiment. [Figure 17] FIG. 11 is a diagram showing an example of calculation of a viewer rating in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0021] An example of an embodiment of the present invention will now be described. FIG. 1 is a diagram illustrating a usage scene of a robot 1 incorporating a viewing state estimating device 10 according to the present embodiment.
[0022] The robot 1 is placed, for example, on a table near a viewer watching television. In addition to the viewing state estimation device 10, the robot 1 is equipped with an imaging unit 20 and a distance detection unit 30, as well as an action control unit 40 that performs actions such as speaking in accordance with the viewing state estimated by the viewing state estimation device 10.
[0023] The viewing state estimation device 10 acquires image data of the robot's surroundings from the imaging unit 20 and distance data from the distance detection unit 30, and using a method described below, calculates the distance from the robot 1 to the television, the distance from the robot 1 to the viewer, and the viewer's viewing direction, and further estimates the viewing state by acquiring an image of the viewer's viewing direction.
[0024] The imaging unit 20 is a camera mounted on the robot 1 for acquiring images, and may be configured to rotate horizontally by a motor and capture images of the surroundings of the robot 1. The imaging unit 20 is not limited to a method of acquiring images while rotating, and may acquire images by mounting a camera array made up of a plurality of cameras on the robot 1, for example.
[0025] The distance detection unit 30 acquires distance data based on the infrared light emitted from the irradiation unit and the light reflected by the object and reaching the light receiving unit. The infrared irradiation method is, for example, a pattern method or a TOF (Time Of Flight) method. When an image is acquired by the imaging unit 20, the distance detection unit 30 acquires distance data in a direction corresponding to each pixel of the image in synchronization with time.
[0026] Here, the viewing state indicates a category of viewing direction, such as watching television, watching someone else, or watching something else. The action control unit 40 changes the action of the robot 1, such as speech, according to the viewing state estimated by the viewing state estimating device 10.
[0027] FIG. 2 is a block diagram showing a functional configuration of the viewing state estimating device 10 in this embodiment. The viewing state estimation device 10 is an information processing device (computer) equipped with a control unit, a storage unit, an input / output interface, etc., and operates as the following functional units by the control unit executing software (viewing state estimation program) stored in the storage unit. The viewing state estimation device 10 includes a panoramic image section 11 , a distance panoramic image section 12 , a viewer detection section 13 , a television detection section 14 , a distance acquisition section 15 , a viewing direction detection section 16 , and a viewing state determination section 17 .
[0028] The panoramic image unit 11 superimposes multiple images of the surroundings of the robot 1 acquired by the imaging unit 20 to generate a horizontal omnidirectional panoramic image seen from the robot 1. Note that the open source OpenCV Stitcher class can be used as software for generating the panoramic image, but the generation method is not limited to this.
[0029] Furthermore, the panoramic image unit 11 outputs synthesis position data Dc when a plurality of images are synthesized. FIG. 3 is a diagram illustrating the synthesis position data Dc in this embodiment. The synthesis position data Dc is composed of the numbers of the multiple images that are stitched together when generating a panoramic image, and the coordinates of the synthesis position. For example, the coordinates (150,10) of the first image are pasted onto the coordinates (0,0) of the second image, resulting in the coordinates (x n ,y n ) is pasted over with the coordinates (0,0) of the first image. In this example, the numbers of the distance images acquired synchronously are associated with each other.
[0030] The distance panorama image unit 12 uses the distance data acquired by the distance detection unit 30 to generate a distance panorama image in which the distance data is used as pixel values, similar to the panorama image generated by the panorama image unit 11 .
[0031] FIG. 4 is a diagram showing the functional configuration of the distance panoramic image unit 12 in this embodiment. The distance panorama image section 12 includes a distance image synthesis section 121 and an overlapping section calculation section 122, and receives the synthesis position data Dc and outputs distance panorama image data Dp.
[0032] The distance image synthesis unit 121 uses the synthesis position data Dc input from the panoramic image unit 11 to synthesize multiple distance images obtained by the distance detection unit 30 at the same positions as the multiple images synthesized by the panoramic image unit 11. Here, distance data corresponding to each pixel of the captured image is stored in distance image database 12 A. Note that distance image database 12 A may be stored in a storage unit of viewing state estimation device 10, or may be provided in a common storage unit accessible by each unit of robot 1.
[0033] An overlapping section calculation unit 122 averages a plurality of overlapping distance data in a section (pixel) where the combined distance images overlap, and determines distance data for each pixel. The method of determining the distance data is not limited to this, and data from any of the distance images may be determined as a representative.
[0034] FIG. 5 is a diagram illustrating the operation of distance image synthesis section 121 in this embodiment. For example, the image 1 and the image 2 are captured in order by the imaging unit 20, and the image 2 is captured at a synthesis position P c1 When combining at (150,0), distance image combination unit 121 also combines distance image 1 and distance image 2, which were acquired simultaneously with image 1 and image 2, and maps the distance data acquired from distance image database 12A to each pixel.
[0035] FIG. 6 is a diagram illustrating the operation of the overlapping section calculation unit 122 in this embodiment. 5, suppose that when distance image 1 and distance image 2 are combined, the distance data at coordinate P1 (150, 159) in distance image 1 is 1.50, and the distance data at coordinate P2 (0, 159) in distance image 2 is 1.60. In this case, the overlapping section calculation unit 122 determines the average value (1.50 + 1.60) / 2 = 1.55 as the distance data at position (150, 159) in the distance panorama image. Alternatively, the overlapping section calculation section 122 may adopt the distance data 1.60 of P2 (0,159) in the distance image 2 as the distance data of the distance panoramic image.
[0036] Similarly, the overlapping section calculation unit 122 determines distance data corresponding to each pixel for the entire overlapping section. The determined distance data is stored in the distance panorama image database 12B in the storage unit.
[0037] FIG. 7 is a diagram illustrating distance image database 12A in this embodiment. In the range image database 12A, range image number n d For each, the position (coordinate x d ,y d ) and distance data l d and distance data for each pixel in each distance image are stored.
[0038] FIG. 8 is a diagram illustrating an example of the distance panoramic image database 12B in this embodiment. The distance panoramic image database 12B stores image numbers n p , position (coordinate x p ,y p ), distance data l p The distance panorama image data composed of the above is stored.
[0039] The viewer detection unit 13 detects the face position of the viewer from the panoramic image generated by the panoramic image unit 11, and acquires the face position on the panoramic image.
[0040] FIG. 9 is a diagram showing the functional configuration of the viewer detection unit 13 in this embodiment. The viewer detection unit 13 includes a person detection unit 131 and a face detection unit 132 .
[0041] The human detection unit 131 detects people from the panoramic image acquired by the panoramic image unit 11. As a means for detecting people, for example, open source software such as OpenCV (Haarcascade detector fullbody model) or Faster-RCNN can be used, but the detection method is not limited to these. When the person detection unit 131 detects a person, the person detection unit 131 sets a viewer flag f h = 1, if no person is detected, the viewer flag f h =Output 0.
[0042] When the person detection unit 131 detects a person, i.e., a viewer (f h If the viewer is not detected (f = 1), face detection is performed from the panoramic image generated by the panoramic image unit 11, and the face position on the panoramic image is acquired. h =0), the face detection unit 132 does not perform face detection, and proceeds to the process of acquiring a new image from the imaging unit 20, thereby increasing the speed. For face detection, for example, open source software such as OpenCV (Haar-cascade detector) or OpenFace can be used, but the detection method is not limited to these.
[0043] Here, the face position is detected by detecting the face part in a rectangular frame, and the face detection unit 132 detects the start point P sf (x sf ,y sf ) and end point P ef (x ef ,y ef ) to center point P f (x f ,yf ) is calculated. f is used to obtain the distance to the face position. When a face is detected, the face detection unit 132 sets a face flag f f = 1, and if no face is detected, the face flag f f =Output 0.
[0044] The television detection unit 14 detects the television from the panoramic image generated by the panoramic image unit 11, and obtains the position of the television on the panoramic image. As a means for detecting a television, for example, open source software such as Faster-RCNN can be used, but the detection method is not limited to this.
[0045] Here, the television position is detected by detecting the television part as a rectangular frame, and the television detection unit 14 detects the start point P stv (x stv ,y stv ) and end point P etv (x etv ,y etv ) to center point P tv (x tv ,y tv ) is calculated. tv is used to obtain the distance to the TV position. When the television detection unit 14 detects a television, it sets a television flag f tv =1, if no TV is detected, set the TV flag f tv =Output 0.
[0046] The distance acquisition unit 15 acquires distance data for each position of the detected viewer and television from the distance panorama image. Specifically, the distance acquisition unit 15 detects the face center point P f The television detection unit 14 detects the television center point P tv and, among the distance data stored in the distance panorama image database 12B, the distance d from the robot 1 to the viewer's face is obtained. f and the distance d from robot 1 to the TV tv and get. For example, the face center point P f If the distance is (1000, 200), obtain the distance data ld at the point (1000, 200) in the distance panorama image, and f =l d The center point of the TV is P tv If the point is (3000, 150), the distance data l at the point (3000, 150) in the panoramic distance image is d Get d tv =l d Let us assume that.
[0047] The viewing direction detection unit 16 acquires the distance from the robot 1 to the viewer's face, the distance from the robot 1 to the television, and the angle between the viewer and the television as seen by the robot 1, thereby identifying the relative positions of the robot 1, the viewer, and the television, and acquires the viewing direction and viewed image from the viewer's face direction angle.
[0048] FIG. 10 is a diagram showing the functional configuration of the viewing direction detection unit 16 in this embodiment. The viewing direction detection unit 16 includes a television-viewer angle calculation unit 161, a television-viewer distance calculation unit 162, a robot-television angle calculation unit 163, a face direction angle acquisition unit 164, a viewing direction angle calculation unit 165, and a viewing direction image acquisition unit 166.
[0049] FIG. 11 is a diagram showing the positional relationship between the robot 1, the television, and the viewer in this embodiment. In the triangle consisting of robot 1 (point A), viewer (point B) and television (point C), BC=r, AB=r1 and AC=r2 are fixed. Also, the angle between the viewer and the TV as seen from Robot 1 (∠BAC) = θ r , the angle between the robot and the TV as seen by the viewer is ∠ABC=θ h , the angle between Robot 1 and the viewer as seen from the TV is ∠ACB=θ tv is fixed.
[0050] In this positional relationship, the viewer is facing the robot 1 at a face direction angle θ h'When the viewer faces in a different direction, it is assumed that the viewer is looking at the viewing direction position (point D). Here, it is assumed that AD=AC=r2. Also, BD=r'. In this case, the angle between the viewer and the viewing direction position as seen from the robot 1 is ∠BAD=θ r '
[0051] The television-viewer angle calculation unit 161 calculates the angle θ between the television and the viewer as seen by the robot 1. r Calculate. FIG. 12 shows the angle θ between the television and the viewer in this embodiment. r FIG.
[0052] First, the television-viewer angle calculation unit 161 calculates the face center point P f (x f ,y f ) and TV center point P tv (x tv ,y tv ), the number of pixels d1 between the television and the viewer in the panoramic image is calculated as follows: d1=|x tv -x f | Next, the television-viewer angle calculation unit 161 calculates the size of the panoramic image (X p ,Y p ), the number of pixels d2 between the television and the viewer in the panoramic image is calculated as follows: d2=|X p -d1|
[0053] Then, the television-viewer angle calculation unit 161 compares the number of pixels d1 and d2 between the television and the viewer, d = min(d1,d2) Let us assume that.
[0054] The television-viewer angle calculation unit 161 calculates the size X of the x-axis in the panoramic image. p is converted to the angle of 360 degrees around robot 1, and the angle between the TV and the viewer as seen by robot 1, θ ris calculated as follows: θ r = d × angle a Angle a=360 / X p
[0055] For example, in a panoramic image, the size of the x-axis is p =4320, x-axis value of TV position x tv =3541, x-axis value of face position x f = 713, the angle θ between the TV and the viewer as seen by Robot 1 r is calculated as follows: d1=3541-713=2828 d2=4320-2828=1492 d=d2=1492 θ r = 1492 × 360 / 4320 ≒ 124 degrees
[0056] The television-to-viewer distance calculation unit 162 calculates the angle θ between the television and the viewer as seen by the robot 1. r , the distance from robot 1 to the viewer is r1=d f , the distance from robot 1 to the TV is r2=d tv Using this, the distance r between the television and the viewer is calculated as follows: r = √(r1 2 +r2 2 -2r1r2cosθ r )
[0057] The robot-television angle calculation unit 163 calculates the angle θ between the robot 1 and the television. h is calculated using the law of cosines as follows: r2 2 =r 2 +r1 2 -2rr1cosθ h θ h =cos -1 [(r 2 +r1 2 -r2 2 ) / (2rr1)]
[0058] The face direction angle acquisition unit 164 acquires a face direction angle θ of the viewer as seen from the robot 1 based on the face image detected by the face detection unit 132. h ' is estimated and obtained along with the time T.
[0059] FIG. 13 shows the face direction angle θ h ' is a diagram for explaining '. Face direction angle θ h ' is based on the state (A) where the viewer's face is facing robot 1, and the direction in which the viewer's face is facing (y axis) is calculated from the reference direction (y axis) as shown in (B). h The angle between the axis of rotation and the In addition, the face direction angle θ h In order to estimate ', for example, open source software such as OpenFace can be used, but the estimation method is not limited to this.
[0060] The viewing direction angle calculation unit 165 calculates a viewing direction angle θ between the viewer as seen from the robot 1 and the direction in which the viewer is looking (viewing direction position). r ' is calculated using the cosine theorem as follows: r2 2 =r' 2 +r1 2 -2r'r1cosθ h ' r' 2 =r1 2 +r2 2 -2r1r2cosθ r ' θ r '=cos -1 [(r1 / r2)-(r' / r2)cosθ h '] Here, the distance r' from the viewer to the viewing direction position can be calculated, for example, by approximation as follows.
[0061] FIG. 14 is a diagram showing the positional relationship between the robot 1, the viewer, and the viewing direction position in this embodiment. Here, the robot 1 (point A) is placed, for example, on a table between the viewer (point B) and the television, and when the viewing direction position is point D, the distance between point A and the line segment BD is sufficiently close. In this case, let E be the foot of the perpendicular line drawn from point A to the line segment BD, and let BE=r1', DE=r2'. r'=|r1 2 -r2 2 | / |r1'-r2'| In contrast, r1'≒r1, r2'≒r2 can be approximated as follows: r'=r1+r2 It is calculated as follows.
[0062] The viewing direction image acquisition unit 166 detects the viewing direction angle θ r ', TV-to-viewer angle θ r , the TV center point P in the panoramic image tv (x tv ,y tv ) to determine the viewing direction position P v (x v ) is calculated as follows to obtain the viewing direction image: x v =x tv -(θ r '-θ r (Number of pixels equivalent to
[0063] At this time, the viewing direction image acquisition unit 166 acquires, for example, the horizontal angle of view θ c The number of pixels in the field of view is d c and the viewing direction position P v (x v ) at the center (x v -(d c / 2), x v +(d c / 2)), or (x v -(d c / 2), x v +(d c Obtain images in the range of / 2)-1) etc. d c =(X p / 360)×θ c Alternatively, when the size (number of pixels) of the image captured by the imaging unit 20 is known, the viewing direction image acquisition unit 166 calculates the number of pixels as d c It may also be used as.
[0064] FIG. 15 is a diagram illustrating a method for acquiring a viewing direction image in this embodiment. For example, the horizontal angle of view of the camera is θ c = 40, the x-axis size of the panoramic image X p = 4320, x coordinate of TV center point x tv = 3800, θ r '-θ r If = 0, d c =(4320 / 360)×40=480, x v = 3800 - 0 = 3800, so viewing direction image acquisition section 166 acquires images in the range of (3800 - 240, 3800 + 240) = (3560, 4040). Also, for example, θ r '-θ r For d = 35, c =(4320 / 360)×40=480, x v = 3800 - ((4320 / 360) x 35) = 3380, so the viewing direction image acquisition section 166 acquires images in the range of (3380 - 240, 3380 + 240) = (3140, 3620).
[0065] In addition, the viewing direction image acquisition unit 166 detects the television center point P tv Instead, the center point of the viewer's face P f (x f ,y f ) to determine the viewing direction position P v (x v ) may be calculated. x v =x f -(θ r ' equivalent pixel count)
[0066] The viewing state determination unit 17 obtains viewing direction images for a certain period of time and performs statistical processing to determine the viewing state, such as whether or not the viewer is watching television. In this embodiment, the viewing level is defined as the viewing state. The viewing intensity is an index of the degree to which a user watches television. A larger value indicates that the user is watching more television, and conversely, a smaller value indicates that the user is not watching television.
[0067] FIG. 16 is a diagram showing a functional configuration of the viewing state determination unit 17 in this embodiment. The viewing state determination unit 17 includes a viewing direction object detection unit 171 and a viewing degree calculation unit 172, and receives a viewing direction image as input and calculates a viewing degree I w Output.
[0068] The viewing direction object detection unit 171 detects objects from the input viewing direction image and extracts keywords. As a means for object detection, for example, open source software such as Faster-RCNN can be used, but the detection method is not limited to this.
[0069] The viewing degree calculation unit 172 uses the keywords extracted by the viewing direction object detection unit 171 to calculate a viewing degree I as an index representing the degree to which the viewer is actually watching television. w Calculate (T).
[0070] FIG. 17 shows the viewer rating I w FIG. 13 is a diagram showing a calculation example of (T). At time T, when the viewing direction object detection unit 171 extracts a keyword indicating a type of video viewing device such as "television", "television", or "monitor", the viewing degree calculation unit 172 determines the viewing state as "television". Also, when a person watching television together is detected, the viewing degree calculation unit 172 determines the viewing state as "others". The viewing state in other cases is defined as "other".
[0071] In this embodiment, the constant time T f The ratio of the number of times the TV was viewed to the number of times it was detected is called the viewing intensity I w In the example of FIG. 17, the viewing intensity I w=0.6, which shows that the subject is paying attention to the television even while shifting their gaze to other people or other objects.
[0072] In this case, the operation control unit 40 sets the threshold value for the robot to talk to the viewer to 0.5, for example, and sets the viewing degree I w is higher than a predetermined level, the robot 1 may be controlled to talk to the viewer. Also, for example, the calculated viewing degree I w When the viewing state ratio of “television” is particularly high, such as 0.8, it is highly likely that the viewer is concentrating on watching television, so the operation control unit 40 may control the robot 1 to refrain from talking to the viewer.
[0073] Conversely, I w When the ratio of the television viewing state is low, such as 0.3, that is, the time spent watching the television direction is about 3 minutes per 10 minutes, the operation control unit 40 may control the robot 1 to talk to the viewer or perform actions using gestures in order to arouse the viewer's interest in television. For example, when the viewer is not looking in the direction of the television, the robot 1 looks around the viewer and the surroundings and says things like, "I'd really like to go to this place," or "Would you like to change the channel?" to encourage the viewer to pay attention to the television. In this way, the threshold is set stepwise, for example, I w When the value is high, only speech is heard. w The movement control unit 40 may appropriately control the robot 1 for each viewer, for example, to perform movements using gestures along with speech when the level of the viewer is low.
[0074] Furthermore, when multiple types of classifications indicating the viewing direction, such as "television," "others," and "other," are acquired as the viewing state, the operation control unit 40 may change the operation of the robot 1 according to the classification. For example, regarding the content of a television program, the frequency of each utterance type, such as disclosure, question, confirmation, information, and response, may be adjusted as follows.
[0075] When the viewing direction is often toward the television, it is assumed that the viewer is watching the television, so the operation control unit 40 reduces the frequency of questions requesting answers or confirmations that would disturb the viewer in a paused state. When the viewing direction is often toward "other people," it is considered that the viewer is in a state of frequent communication with others, so the operation control unit 40 reduces the frequency of speech for all speech types so as not to interfere with communication with others. When the viewing direction is often "other," it is considered that the viewer is not paying attention to the television and is not communicating with others, so the operation control unit 40 increases the frequency of disclosures or questions to stimulate interest in television viewing.
[0076] According to this embodiment, the viewing state estimation device 10 detects the viewer's face and the television from a panoramic image in all directions around the robot 1, and calculates the angle between the television and the viewer as seen by the robot 1 from the distance between them on the image and the size of the panoramic image. Furthermore, the viewing state estimation device 10 specifies the positional relationship between the robot 1, the viewer, and the television by measuring the distance from the robot 1 to the detected television and viewer. Then, the viewing state estimation device 10 detects an object included in the viewing direction image at the viewing direction angle of the viewer from the panoramic image based on the viewer's face direction angle obtained from the face position image, and judges the viewer's state based on the type of this object.
[0077] Therefore, the viewing state estimation device 10 can estimate the viewing state of the viewer by detecting objects in the viewing direction image based on the panoramic image using the robot 1 installed on a table or the like, without installing an imaging device such as a camera on the ceiling or the like in the home, and without having the viewer wear a gaze direction acquisition device. As a result, the robot 1 can behave in accordance with the viewing state, such as being considerate not to talk to the viewer when the viewer is watching television, and making speech and gestures to encourage the viewer to participate in watching television when the viewer is not watching television.
[0078] The viewing state estimation device 10 can easily identify the viewing direction position and estimate the viewing state by approximating the distance from the viewer to the viewing direction position by the sum of the distance from the robot 1 to the viewer and the distance from the robot 1 to the viewing direction position.
[0079] The viewing state estimation device 10 calculates the viewing state based on statistical information on the viewer's state within a certain period of time, and therefore can reduce calculation errors in the viewing direction position and improve the reliability of the determined viewing state.
[0080] The viewing state estimation device 10 calculates the viewing degree indicating the proportion of television viewing as the viewing state, so that the operation control unit 40 can grasp the degree to which the viewer is actually watching television and appropriately adjust the operation of the robot 1 to the state of the viewer according to this degree. At this time, the operation control unit 40 can easily control the operation of the robot 1 based on the result of comparing the calculated viewing rate with a predetermined threshold value. Moreover, the movement control unit 40 can appropriately set the movement variations of the robot 1 in accordance with the state of the viewer by gradually changing the control based on a plurality of threshold values.
[0081] The viewing state estimation device 10 determines a plurality of viewing states, including a state of watching television and a state of watching others, so that the operation control unit 40 can grasp the type of object the viewer is looking at and appropriately adjust the operation of the robot 1 to the viewer's state according to the type.
[0082] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the above-described embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.
[0083] In the above-described embodiment, the viewing state estimation device 10 estimated the type of object the viewer is looking at by object detection in the viewing direction image, but whether or not the viewer is watching television may be determined based on the viewer's face direction angle. Specifically, for example, the face direction angle θ h 'but, θ h -α<θ h '<θ h +α If the above condition is satisfied, it may be determined that the viewer is facing the television. Note that α is an adjustment angle (e.g., half the viewing angle of the television) that is set based on the size of the television and the distance from the viewer.
[0084] In the above embodiment, the face direction angle θ h 'Based on the viewing direction angle θ r However, the calculation method is not limited to this. For example, θ h If ' cannot be obtained properly, θ r '=cos -1 [(r1 2 +r2 2 -r' 2 ) / (2r'r1)] As in, θ h θ without using ' r ' may be required.
[0085] In the above embodiment, the distance from the robot 1 to the viewing direction position is assumed to be equal to the distance r2 from the robot 1 to the television, but the assumed condition is not limited to this. For example, θ h Depending on the positive or negative value of ', or the range of values, r1, r2, or another value may be used for the distance, and may be set appropriately according to the situation.
[0086] In this embodiment, the configuration and operation of the viewing state estimation device 10 have been mainly described, but the present invention is not limited to this, and may be configured as a method or program for estimating the viewing state by including each component.
[0087] Furthermore, the functions of the viewing state estimation device 10 may be realized by recording a program for realizing the functions of the viewing state estimation device 10 on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system.
[0088] The term "computer system" here includes hardware such as the OS and peripheral devices. Additionally, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems.
[0089] Furthermore, the term "computer-readable recording medium" may include a recording medium that dynamically stores a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and a recording medium that stores a program for a certain period of time, such as a volatile memory in a computer system that serves as a server or client in such a case. The above program may be a program for realizing part of the above-mentioned functions, or may be a program that can realize the above-mentioned functions in combination with a program already recorded in the computer system. [Explanation of symbols]
[0090] 1. Robot 10. Viewing state estimation device 11 Panoramic Image Section 12 Distance panoramic image section 12A Range Image Database 12B Range panoramic image database 13 Viewer detection unit 14 TV detector 15 Distance acquisition part 16 Viewing direction detection section 17 Viewing state determination unit 20 Imaging unit 30 Distance detection unit 40 Motion control section 121 Distance image synthesis unit 122 Overlapped section calculation unit 131 Human detection unit 132 Face detection unit 161 Television-to-viewer angle calculation unit 162 Television-to-viewer distance calculation unit 163 Robot-TV angle calculation unit 164 Face direction angle acquisition unit 165 Viewing direction angle calculation unit 166 Viewing direction image acquisition unit 171 Viewing direction object detection unit 172 Viewing Calculation Unit
Claims
1. a panoramic image unit for acquiring an omnidirectional panoramic image synthesized from images captured around the robot; a distance panorama image unit for generating a distance panorama image in which distance data corresponding to each pixel of the panorama image is used as a pixel value; a television detection unit that detects a television position from the panoramic image; a viewer detection unit that detects a face position of a viewer from the panoramic image; a distance acquisition unit that acquires distances at the television position and the face position from the distance panoramic image; a viewing direction detection unit that specifies a positional relationship between the robot, the viewer, and the television by calculating an angle between the television and the viewer as seen by the robot based on a size of the panoramic image, the television position, and the face position, and obtains a viewing direction image at the viewing direction angle of the viewer from the panoramic image based on a face direction angle of the viewer obtained from the image of the face position; a viewing state determination unit that detects an object included in the viewing direction image and determines a state of the viewer based on a type of the object.
2. 2. The viewing state estimation device according to claim 1, wherein the viewing direction detection unit approximates a distance from the viewer to a viewing direction position by a sum of a distance from the robot to the viewer and a distance from the robot to the viewing direction position.
3. 3. The viewing state estimation device according to claim 1, wherein the viewing state determination unit calculates the viewing state based on statistical information of the state of the viewer within a certain period of time.
4. The viewing state estimation device according to claim 3 , wherein the viewing state determination unit calculates, as the viewing state, a viewing degree indicating a proportion of the time the user is watching the television.
5. The viewing state estimation device according to claim 3 , wherein the viewing state determination unit determines, as the viewing state, whether the viewing state is one of a plurality of states including a state in which the viewing state is watching the television and a state in which the viewing state is watching another person.
6. The viewing state estimating device according to claim 4 ; a motion control unit that controls the motion of the robot based on a result of comparing the viewing degree output from the viewing state estimation device with a predetermined threshold.
7. The robot system according to claim 6 , wherein the operation control unit provides a plurality of the predetermined thresholds, and changes the control of the robot in a stepwise manner depending on a result of comparing the viewing rate with the plurality of thresholds.
8. The viewing state estimating device according to claim 5 ; and an operation control unit that controls an operation of the robot in accordance with the classification of the plurality of states output from the viewing state estimation device.
9. a panoramic image generating step of acquiring an omnidirectional panoramic image synthesized from images captured around the robot; a distance panoramic image generating step of generating a distance panoramic image in which distance data corresponding to each pixel of the panoramic image is used as a pixel value; a television detection step of detecting a television position from the panoramic image; a viewer detection step of detecting a face position of a viewer from the panoramic image; a distance acquisition step of acquiring distances at the television position and the face position from the distance panoramic image; a viewing direction detection step of calculating an angle between the television and the viewer as seen by the robot based on the size of the panoramic image, the television position, and the face position, thereby specifying a positional relationship between the robot, the viewer, and the television, and acquiring a viewing direction image at the viewing direction angle of the viewer from the panoramic image based on the face direction angle of the viewer obtained from the image of the face position; a viewing state determination step of detecting an object included in the viewing direction image and determining a state of the viewer based on a type of the object, the viewing state determination method being executed by a computer.
10. A viewing state estimation program for causing a computer to function as the viewing state estimation device according to any one of claims 1 to 5.
Citation Information
Patent Citations
Electronic equipment, determination method, and program
JP2015082247A
Reproduction control apparatus
JP2016100033A
Information processor, information processing method, and program
JP2017055178A
Engagement value processing system and engagement value processing device
JP2018005892A
Control device, control method, and control program
JP2018180472A
