Information processing method and information processing system

Abstract

The information processing apparatus causes a display unit to display a content image, determines whether or not a user is directing a line of sight toward a service medium for the user to process a service, and, if it is determined that the user is directing the line of sight toward the service medium, changes the form of the content image displayed on the display unit so as to reduce the degree of gaze of the user.

Description

Technical Field

[0001] The present disclosure relates to a technology for presenting content images. Background Art

[0002] Patent document 1 discloses a technology that detects the amount of movement of a user's head from an image of the user captured by a camera, and determines whether the user is in a focused state based on the amount of movement of the head. If it is determined that the user is in a focused state, the liquid crystal controllable partition arranged around the user is changed to a non-transmissive state.

[0003] Patent document 2 discloses a technology that identifies a user's body movements from an image of the user captured by a camera, and determines whether the user is concentrating on business based on the body movement recognition results. If it is determined that the user is concentrating on business, the image quality of the display screen of the video conferencing software displayed on the monitor is reduced.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2012-160173

[0007] Patent Document 2: Japanese Patent Application Laid-Open No. 2001-186489 Summary of the Invention

[0008] However, the above-mentioned prior art does not determine whether the user's gaze is directed toward the user's business medium, and therefore cannot accurately determine whether the user is concentrating on the business using the business medium. Consequently, the above-mentioned prior art cannot display content images in an appropriate form based on whether the user is concentrating on the business using the business medium, and cannot prevent the user's concentration on the business from decreasing.

[0009] The present disclosure provides a technology for suppressing a decrease in a user's concentration on a business using a business medium.

[0010] An information processing method in one form of the present disclosure is an information processing method executed by a computer, wherein the method displays a content image on a display unit, determines whether a user is directing their gaze toward a business medium used by the user to process business, and when it is determined that the user is directing their gaze toward the business medium, changes the form of the content image displayed on the display unit to reduce the user's attention. This general or specific form can be implemented by a device, a system, an integrated circuit, a computer program, or a computer-readable recording medium, or by any combination of a device, a system, a method, an integrated circuit, a computer program, and a computer-readable recording medium. Computer-readable recording media include non-volatile recording media such as CD-ROMs (Compact Disc Read-Only Memory).

[0011] According to the present disclosure, it is possible to suppress a decrease in user's concentration on a business using a business medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a diagram showing an example of a booth to which the information processing method according to the embodiment of the present disclosure is applied.

[0013] Figure 2 This is a block diagram showing an example of the configuration of an information processing system in an embodiment of the present disclosure.

[0014] Figure 3 This is a diagram showing an example of processing by the information processing system in the embodiment of the present disclosure.

[0015] Figure 4 This is a diagram showing an example of a change in the form of a content image.

[0016] Figure 5 This is a diagram showing another example of a change in the form of a content image.

[0017] Figure 6 This is a diagram showing another example of a change in the form of a content image.

[0018] Figure 7 This is a diagram showing another example of a change in the form of a content image.

[0019] Figure 8 This is a diagram showing another example of a change in the form of a content image.

[0020] Figure 9 This is a diagram showing an example of a vehicle.

[0021] Figure 10 It schematically represents Figure 9The diagram shows an example of a scene when a front seat 710 is viewed from a rear seat 720 .

[0022] Figure 11 This is a diagram showing an example of a situation where a plurality of users utilize the booth 400 .

[0023] Figure 12 This is a diagram showing an example of processing of the information processing system 1 in a case where a plurality of users utilize the booth 400 . DETAILED DESCRIPTION

[0024] (The process of coming up with one aspect of the present disclosure)

[0025] Research is underway on a technology that displays user-favorite content, such as landscapes, as background images on a display installed on the inner wall of a private cubicle where the user conducts work. This technology aims to enhance the user's sense of excitement, relaxation, and concentration, thereby improving work efficiency.

[0026] However, when a user is concentrating on a business task using a business medium such as a personal computer, displaying a content image on a display may draw the user's attention to the content image, thereby reducing the user's concentration on the task.

[0027] When the user is focused on a service using a service medium, the user's line of sight will be directed toward the service medium. Therefore, whether the user is focused on the service can be determined based on whether the user's line of sight is directed toward the service medium.

[0028] In Patent Document 1, the user's concentration is determined based on the amount of head movement, but it does not determine whether the user's gaze is directed toward the service medium. In Patent Document 2, the user's concentration is determined based on body movement, but it does not determine whether the user's gaze is directed toward the service medium.

[0029] Therefore, Patent Documents 1 and 2 cannot accurately determine whether the user is focused on the business using the business medium. Therefore, it is impossible to display content images in an appropriate form according to whether the user is focused on the business using the business medium, and it is impossible to suppress the decline of the user's focus on the business.

[0030] Furthermore, Patent Document 1 discloses that when a user's concentration is high, the user is simply warned of their high concentration, or a third party's request to contact the user through a two-way communication device is restricted. However, no processing is performed to display content images intended to enhance the user's concentration, or to change the form of the content images. Patent Document 2 discloses that the image quality of images displayed on a display is reduced when the user is focused on their work. However, this image is intended to display a list of working scenes of team members within an organization, such as those working from home, rather than content images intended to enhance the user's concentration.

[0031] Therefore, the present inventors discovered that by determining whether the user's gaze is directed toward the business medium, it is possible to accurately determine whether the user is concentrating on the business using the business medium. Furthermore, the present inventors discovered that by changing the form of the content image to reduce the user's attention when the user's gaze is directed toward the business medium, it is possible to suppress the user's decreased concentration on the business using the business medium. This led them to conceive of the present disclosure.

[0032] (1) An information processing method in one form of the present disclosure is an information processing method executed by a computer, which displays a content image on a display unit, determines whether a user is directing their gaze toward a business medium used by the user to process business, and, if it is determined that the user is directing their gaze toward the business medium, changes the form of the content image displayed on the display unit to reduce the user's gaze.

[0033] This structure accurately determines whether the user's gaze is directed toward the business medium, thereby accurately determining whether the user is fully focused on the business using the business medium. If the user's gaze is directed toward the business medium, the content image is altered to reduce the degree of focus, thereby not disrupting the user's focus on the business. This directs the user's attention toward the business medium and suppresses a decrease in their concentration on the business. On the other hand, if the user's gaze is not directed toward the business medium, the content image can be directed toward the user, thereby increasing their sense of relaxation and improving work efficiency.

[0034] (2) In the information processing method described in (1) above, changing the form of the content image may include softening the content image.

[0035] According to this configuration, when it is determined that the user's gaze is directed toward the business medium, the content image displayed on the display unit is softened, thereby reducing the user's attention to the content image and directing the user's attention to the business.

[0036] (3) Alternatively, in the information processing method described in (2) above, when it is determined that the user is directing his or her gaze toward the business medium, the degree of softening of the content image is increased according to the length of time the user is directing his or her gaze toward the business medium.

[0037] According to this configuration, the degree of softening of the content image is increased according to the period during which the line of sight is directed toward the business medium, and thus the form of the content image can be changed without causing discomfort.

[0038] (4) Alternatively, in the information processing method described in any one of (1) to (3) above, the content image is a moving image, and changing the form of the content image refers to changing the area corresponding to the moving body contained in the content image to a background image corresponding to the area.

[0039] According to this configuration, the moving object included in the content image is partially hidden by the background image, and thus the user's attention to the content can be reduced.

[0040] (5) In the information processing method described in any one of (1) to (4) above, changing the form of the content image may include reducing the brightness of the content image.

[0041] According to this configuration, by reducing the brightness of the content image, it is possible to reduce the degree of user attention directed toward the content.

[0042] (6) Alternatively, in the information processing method described in any one of (1) to (5) above, when it is determined that the user is not directing his or her gaze toward the business medium after the form of the content image is changed, the form of the content image is restored to the form before the change.

[0043] According to this configuration, when the user's concentration on the task is interrupted, the form of the content image is restored to its original form, thereby making it possible to direct the user's consciousness toward the content image and enhance the user's sense of relaxation.

[0044] (7) Alternatively, in the information processing method described in any one of (1) to (6) above, the display unit includes a first display area opposite to the user and a second display area different from the first display area, displaying the content image on the display unit includes displaying the content image in the first display area and the second display area, a first content image included in the content image is displayed in the first display area, and a second content image included in the content image is displayed in the second display area, and changing the form of the content image includes changing the form of the first content image and not changing the form of the second content image.

[0045] With this structure, when the user's gaze is directed toward the business medium, the form of the content image displayed in the first display area facing the user changes, while the form of the content image displayed in the second display area remains unchanged, thereby reducing the processing burden of changing the form of the content image. In addition, because the first and second display areas facing the user are used to display the content image, a more immersive content image can be displayed when the user's gaze is not directed toward the business medium.

[0046] (8) Alternatively, in the information processing method described in any one of (1) to (7) above, the content image includes a first content motion image and a second content motion image having a smaller motion amount than the first content motion image, and when it is determined that the user is directing their gaze toward the business medium, the first content motion image displayed on the display unit is changed to the second content motion image.

[0047] According to this structure, when the user's gaze is directed toward the business medium, the first content motion image is changed to the second content motion image with a smaller amount of motion than the first content motion image, thereby guiding the user's consciousness to the business medium, and suppressing the decline in the user's concentration on the business.

[0048] (9) In the information processing method described in any one of (1) to (8), the content image may be an image that creates an atmosphere surrounding the user.

[0049] According to this configuration, the content image is an image that creates an atmosphere surrounding the user, and thus the user's sense of relaxation can be enhanced by the content image.

[0050] (10) Alternatively, in the information processing method described in any one of (1) to (9) above, the determination of whether the user's line of sight is directed toward the business medium includes making a determination using sensing data detected by a sensor that detects at least the state of the user's eyes.

[0051] According to this configuration, whether the user's gaze is directed toward the business medium is determined using sensing data detected by the sensor that detects the user's eye state, thereby accurately determining whether the user's gaze is directed toward the business medium.

[0052] (11) Another aspect of the present disclosure relates to an information processing system comprising a processor, wherein the processor displays a content image on a display unit, determines whether a user is directing their gaze toward a business medium used by the user to process business, and, if it is determined that the user is directing their gaze toward the business medium, changes the form of the content image displayed on the display unit to reduce the user's attention.

[0053] According to this configuration, it is possible to provide an information processing system that suppresses a decrease in user's concentration on a business using a business medium.

[0054] The present disclosure can also be implemented as an information processing program that causes a computer to execute the various features included in such an information processing method, or as an information processing device that operates according to such an information processing program. Furthermore, it goes without saying that such a computer program can be distributed via a computer-readable non-transitory recording medium such as a CD-ROM or a communication network such as the Internet.

[0055] In addition, the embodiments described below each illustrate a specific example of the present disclosure. The numerical values, shapes, constituent elements, steps, and the order of the steps shown in the following embodiments are merely examples, and the subject matter does not limit the present disclosure. In addition, among the constituent elements of the following embodiments, the constituent elements that are not recorded in the independent technical solutions representing the highest concepts are described as arbitrary constituent elements. In addition, in all embodiments, the various contents can be combined.

[0056] (Implementation Method)

[0057] Figure 1 This diagram illustrates an example of a cubicle 400 to which an information processing method according to an embodiment of the present disclosure is applied. The cubicle 400 is a space used by a user 401 to conduct business using a business medium 404. Examples of cubicle 400 include a room of the user 401 working from home, a room in an office, or a space separated by partitions within an office.

[0058] The booth 400 is provided with at least a camera 402 for observing the interior of the booth 400 and a screen 403 (an example of a display unit) for displaying content.

[0059] An example of screen 403 is a display device such as a liquid crystal panel or an organic EL panel. Screen 403 may be a screen that projects images from a projector. In addition, when a projector is used to project images, screen 403 may be a part of the wall surface of booth 400.

[0060] Camera 402 (an example of a sensor) is composed of, for example, a camera that captures images of the interior of booth 400 at a predetermined frame rate. Camera 402 is mounted on the ceiling of booth 400 with its optical axis facing business medium 404 and user 401 using business medium 404. Camera 402 is used to observe user 401, the direction of user 401's line of sight, and user 401's business medium 404. Furthermore, if a single camera is unable to capture all information, multiple cameras may be installed within booth 400.

[0061] Furthermore, camera 402 can be located anywhere within compartment 400, as long as it is positioned to capture images of business medium 404 and user 401. For example, if business medium 404 is a personal computer, camera 402 can be a camera included with the personal computer. Alternatively, camera 402 can be installed on or near a tabletop, supported by a tripod or fixed arm. Furthermore, camera 402 can be a camera included in a glasses-type wearable device worn by user 401.

[0062] Screen 403 displays a content image. This content image serves as a background for user 401, who is working in booth 400. The content image can be a favorite image that user 401 has previously selected. For example, the content image could be a beautiful natural landscape, an image that creates the atmosphere of a moderately noisy cafe, or an image that creates the atmosphere of a quiet library. Alternatively, the content image could be a soothing image, or an image of an animal or plant. In this way, the content image creates an atmosphere around user 401, who is working in booth 400, thereby enhancing user 401's sense of excitement, relaxation, or the like.

[0063] Business medium 404 is an object used by users to process business. Business medium 404 is, for example, a personal computer, a notebook, a book, a tablet computer, a smartphone, etc. A personal computer can be a desktop computer or a laptop computer.

[0064] The content image may be an image selected by user 401 when making a reservation for use of booth 400. Furthermore, if booth 400 is provided with an input device (not shown) for selecting a content image, user 401 can operate the input device and select a content image. Furthermore, a speaker (not shown) may be provided within booth 400 to reproduce sounds that match the content image.

[0065] Figure 2 This is a block diagram showing an example of the configuration of the information processing system 1 in the embodiment of the present disclosure. Figure 1 In addition to the camera 402 and screen 403 shown, the system also includes a database 200 and a processor 300 .

[0066] The database 200 is composed of a rewritable nonvolatile storage device such as a hard disk drive or a solid-state drive, and includes a content database 201 and a sensor database 202 .

[0067] Content database 201 stores content images displayed on screen 403. Content database 201 may store multiple types of content images as display candidates. In this case, the content image selected by user 401 may be displayed on screen 403. Furthermore, if the content image is associated with a reproduced sound, content database 201 may store the content image in association with the sound data of the reproduced sound. The content image may be a moving image composed of multiple frame images or a still image.

[0068] The sensor database 202 stores images captured by the camera 402. The image may be given a timestamp indicating the date and time of capture. In addition, the image may be given an identifier for identifying the compartment 400.

[0069] Processor 300 is comprised of, for example, a central processing unit (CPU). It includes a user determination unit 301, a line of sight estimation unit 302, a service medium detection unit 303, and a content processing unit 304. User determination unit 301 through content processing unit 304 can be implemented by the CPU executing an information processing program or by dedicated hardware circuitry.

[0070] User determination unit 301 reads an image from sensor database 202 and uses the read image to determine whether user 401 is present in booth 400. User determination unit 301 may use a method such as pattern matching to determine whether a human is included in the image, and if so, determines that user 401 is present in booth 400. Alternatively, user determination unit 301 may determine whether user 401 is present in booth 400 by applying facial recognition technology to the image. In addition, to achieve real-time processing, user determination unit 301 may execute the process of determining whether the user is present in booth 400 each time an image is stored in sensor database 202.

[0071] Gaze estimation unit 302 estimates the gaze direction of user 401 using an image containing user 401. Since techniques for estimating the gaze direction of user 401 from images are well known, gaze estimation unit 302 can use these well-known methods to estimate the gaze direction of user 401. For example, gaze estimation unit 302 detects the eye position, pupil position, and facial orientation of user 401 from an image containing user 401, and estimates the gaze direction of user 401 based on the detected eye position, pupil position, and facial orientation. For example, Japanese Patent Application Laid-Open No. 2004-21870 discloses a technique for estimating gaze using the eye position and pupil position detected from an image, and gaze estimation unit 302 can use this technique to estimate the gaze direction of user 401. The estimated gaze direction can be a two-dimensional vector or a three-dimensional vector.

[0072] The business medium detection unit 303 reads an image from the sensor database 202 and uses the read image to detect the business medium 404 placed in the compartment 400. The business medium detection unit 303 may, for example, use object detection technology to detect specific objects from an image to detect the business medium 404. The type of business medium 404 to be detected is predetermined. Therefore, the business medium detection unit 303 can detect the business medium 404 by detecting the type of business medium 404 to be detected from the image. There may be multiple types of business medium 404 to be detected. In this case, the business medium detection unit 303 can determine whether any one or more of all the types of business medium 404 to be detected are placed in the compartment 400. The business medium detection unit 303 can generate the location data and type of the detected business medium 404 as a detection result. The location data of the business medium 404 can be, for example, two-dimensional or three-dimensional. If the location data of the business medium 404 is two-dimensional, the location data may be, for example, two-dimensional coordinates representing the position of the business medium 404 in the image. When the position data of the business medium 404 is three-dimensional, the position data can be, for example, three-dimensional coordinates representing the position of the business medium 404 within the three-dimensional coordinate space representing the compartment 400. The position data of the business medium 404 can be composed of a set of coordinates. When the business medium 404 is two-dimensional, the set of coordinates represents an outline, and when the business medium 404 is three-dimensional, the set of coordinates represents a surface.

[0073] Furthermore, the service medium detection unit 303 determines whether the gaze direction of the user 401 estimated by the gaze estimation unit 302 is directly toward the service medium 404. Here, the service medium detection unit 303 may determine whether the gaze direction is directly toward the service medium 404 based on whether an extension line of the vector representing the gaze direction of the user 401 intersects with the service medium 404. For example, if the extension line of the vector representing the gaze direction intersects with the outline or surface of the service medium 404, the service medium detection unit 303 may determine that the gaze direction of the user 401 is directly toward the service medium 404; if the extension line of the gaze direction does not intersect with the outline or surface of the service medium 404, the service medium detection unit 303 may determine that the gaze direction of the user 401 is not directly toward the service medium 404.

[0074] Alternatively, when the business medium detection unit 303 detects multiple types of business media 404 , it may determine that the user 401 is looking toward the business medium 404 if the user 401 is looking toward at least one of the multiple types of business media 404 .

[0075] When the business medium detection unit 303 determines that the user 401 is looking toward the business medium 404 , the content processing unit 304 changes the form of the content image to reduce the user 401 's attention, and displays the changed content image on the screen 403 .

[0076] The content processing unit 304 may change the form of the content image by, for example, softening the content image. In this case, the content processing unit 304 may increase the degree of softening of the content image as the user 401 keeps their gaze directed toward the business medium 404 for an increasing period of time.

[0077] Alternatively, if it is determined that the user 401 is not looking toward the business medium 404 after the content image has been changed in form, the content processing unit 304 may restore the form of the content image to its form before the change, that is, to its unblurred form.

[0078] In addition, Figure 2 In the information processing system shown, the database 200 and processor 300 can be installed on a single computer or distributed across multiple computers. For example, the database 200 and processor 300 can be configured in a cloud server or an edge server. In the case where the database 200 and processor 300 are configured in a cloud server, the camera 402 and screen 403 can be connected to the database 200 and processor 300 via an edge computer located at the edge.

[0079] Figure 3 This is a diagram showing an example of the processing of the information processing system 1 in the embodiment of the present disclosure. For example, the process is repeatedly executed every time the camera 402 captures an image. Figure 3 The flowchart shown.

[0080] First, in step S101 , the camera 402 captures an image of the interior of the compartment 400 and stores the obtained image of the interior of the compartment 400 in the sensor database 202 .

[0081] Next, in step S102, the user determination unit 301 reads the latest image captured in step S101 from the sensor database 202, and uses the read image to determine whether the user 401 is present in the booth 400. Hereinafter, the image to be processed is referred to as a target image.

[0082] If it is determined that the user 401 is present in the booth 400 (YES in step S103 ), the process proceeds to step S104 . If it is determined that the user 401 is not present in the booth 400 (NO in step S103 ), the process ends.

[0083] Next, in step S104 , the gaze estimation unit 302 applies the above-described technique for estimating the gaze direction to the object image, and estimates the gaze direction of the user 401 .

[0084] Next, in step S105 , the business medium detection unit 303 applies the above-mentioned object detection technology to the target image and detects the business medium 404 .

[0085] Next, in step S106 , the business medium detection unit 303 determines whether the line of sight direction estimated in step S104 is directed toward the business medium 404 detected in step S105 .

[0086] If it is determined that the business medium 404 exists in the line of sight of the user 401 ("Yes" in step S107), the processing proceeds to step S108. If it is determined that the business medium 404 does not exist in the line of sight of the user 401 ("No" in step S107), the processing proceeds to step S109.

[0087] In step S108 , the content processing unit 304 reads out a content image to be displayed from the content database 201 , applies a softening process to the read content image, and displays the softened content image on the screen 403 .

[0088] In step S109 , the content processing unit 304 reads out a content image to be displayed from the content database 201 , and displays the read content image directly on the screen 403 .

[0089] Next, the change of the form of the content image will be described. Figure 4 This is a diagram showing an example of a change in the form of a content image. Figure 4 In FIG. 4 , the content image 501 is a content image displayed on the screen 403 when the user 401 is not looking toward the business medium 404 .

[0090] The content image 502 is a content image displayed on the screen 403 when the user 401 is looking directly toward the business medium 404. The content processing unit 304 applies a softening process to the content image 501 to generate the content image 502.

[0091] One example of a softening process is blurring. For example, the content processing unit 304 may divide the content image 501 into a plurality of blocks and soften the content image 501 by replacing the color of the pixels within each block with the average color of the pixels within the block. In this case, the larger the block size, the greater the degree of softening. Furthermore, the method of softening content is not limited to blurring. Therefore, the content processing unit 304 may increase the degree of softening by increasing the block size set in the content image 501 or by increasing the number of times the blurring process is repeated.

[0092] Here, the content processing unit 304 may gradually increase the degree of softening as the user 401's gaze direction becomes longer toward the business medium 404. By gradually increasing the degree of softening, the user 401's awareness can be directed toward the business medium 404 without feeling uncomfortable.

[0093] Figure 5 4 is a diagram showing another example of a change in the form of a content image. Content image 503 is displayed on screen 403 when user 401's line of sight is not directed toward business medium 404. Content image 503 is a moving image composed of a plurality of frame images.

[0094] Content image 504 is displayed on screen 403 when user 401 is looking toward business medium 404. Content processing unit 304 softens content image 503 and then masks the moving object region showing the prominent moving object with a mask image, thereby generating content image 504.

[0095] The following method can be used to determine the moving object region. First, the content processing unit 304 calculates the optical flow between the frame image of interest and the frame image immediately preceding the frame image of interest. Next, the content processing unit 304 extracts vectors with a magnitude greater than a threshold from the vectors representing the calculated optical flow. Next, the content processing unit 304 clusters the extracted vectors to determine the moving object region in the frame image of interest. Next, the content processing unit 304 overwrites the moving object region with a mask image.

[0096] Here, the content processing unit 304 identifies frame images in which the vector representing the optical flow is less than a threshold value from the frame images constituting the content image 503, and extracts a region corresponding to the moving object region from the identified frame images as a mask image. The content processing unit 304 then masks the identified moving object region by overwriting the extracted mask image onto the identified moving object region.

[0097] Alternatively, the content database 201 may store time period information indicating which time period the frame image includes the moving body region for each content image. In this case, the content processing unit 304 refers to the time period information to determine the frame image of the time period in which the vector representing the optical flow is less than the threshold.

[0098] Figure 6 This is another example of a change in the form of a content image. Figure 6 In FIG. 4 , the content image 505 is a content image displayed on the screen 403 when the user 401 is not looking toward the business medium 404 .

[0099] Content image 506 is a content image displayed on screen 403 when the user 401's line of sight is directed toward business medium 404. The amount of motion of content image 506 is smaller than that of content image 505. The amount of motion will be described later.

[0100] When the gaze estimation unit 302 determines that the gaze direction of the user 401 is toward the business medium 404, the content processing unit 304 switches the content image 505 displayed on the screen 403 to the content image 506. Thus, "changing the form of the content image" includes switching one content image to another.

[0101] In this case, the content processing unit 304 can switch content image 505 to content image 506 based on the amount of motion information previously stored in the content database 201. The amount of motion information is information indicating the magnitude (amount of motion) of the motion of the moving body contained in each content image. For example, the amount of motion is determined by the length of the time period of the frame image including the moving body region calculated according to the above process. In this case, the content processing unit 304 can determine content image 506 from the content database 201, which has motion information indicating a smaller amount of motion than that indicated by the motion information of content image 505.

[0102] The amount of motion can be either the maximum size of the moving object region included in the content image, the maximum magnitude of the motion of the moving object included in the content image, or a combination thereof. The size of the moving object region is, for example, the size of the region obtained by clustering vectors representing optical flow. The magnitude of the motion of the moving object is, for example, the average magnitude of the vectors representing optical flow included in the moving object region.

[0103] Furthermore, when switching content images, the content processing unit 304 may skip frame images that are to be displayed on the screen 403 until a frame image that does not include a moving body region is skipped among frame images constituting a certain content image.

[0104] In this case, while reproducing the content image, the content processing unit 304 refers to the aforementioned time period information to determine whether the time has arrived for reproducing a frame image that includes a moving body region. If it is then determined that the time has arrived for generating a frame image that includes a moving body region, the content processing unit 304 references the time period information to identify the most recent frame image among subsequent frames that does not include a moving body region. The content processing unit 304 can then display the frame image that does not include the identified moving body region on the screen 403. By repeating this process, the content processing unit 304 can display content images with minimal motion on the screen 403 when the user 401 is looking toward the service medium 404.

[0105] Figure 7 、 Figure 8 This is a diagram showing another example of a change in the form of a content image. Figure 7 This indicates that the user 401's line of sight is not directed toward the business medium 404 . Figure 8 This shows that the line of sight of the user 401 is directed toward the business medium 404 .

[0106] like Figure 7 、 Figure 8 As shown, the screen 601 includes a first display area 601a and a second display area 601b. The first display area 601a is configured in front of the user 401. The second display area 601b is configured on the left and right of the user 401. Figure 7 、 Figure 8 For ease of illustration, the second display area 601b located to the right of user 401 is omitted. Thus, screen 601 has a structure encompassing the front, left, and right sides of the user, capable of displaying a panoramic content image 602. Consequently, screen 601 can provide user 401 with a sense of immersion. Content image 602 may be a moving image.

[0107] exist Figure 7 、 Figure 8 In the example, the content processing unit 304 can change the form of the content image in the same manner as described above. Specifically, if the line of sight estimation unit 302 determines that the user 401's line of sight is not directed toward the service medium 404, the content processing unit 304 applies a softening process, a masking process, or performs a content image switching process on the content image 602.

[0108] Specifically, when the line of sight estimation unit 302 determines that the line of sight of the user 401 is directed toward the business medium 404, the content processing unit 304 changes the form of the content displayed in the first display area 601a, while the form of the content displayed in the second display area 601b remains unchanged. Figure 8In the example of FIG. 6 , the content image 602 displayed in the second display area 601 b is not softened, but the content image 602 displayed in the first display area 601 a is softened.

[0109] In this way, Figure 7 、 Figure 8 In the example shown in Figure 6, if user 401 is not looking toward business medium 404, screen 601 displays content image 602 over a wide area, spanning both first display area 601a and second display area 601b, thereby providing user 401 with a greater sense of immersion. On the other hand, if user 401 is looking toward business medium 404, screen 601 does not change the form of content image 602 displayed in second display area 601b, but changes the form of content image 602 displayed in first display area 601a. ​​This effectively directs the user's attention to business medium 404. In addition, the processing load of the processor 300 when the processor 300 does not change the form of the content image 602 displayed in the second display area 601b and the processor 300 changes the form of the content image 602 displayed in the first display area 601a is smaller than the processing load of the processor 300 when the processor 300 changes the form of the content image 602 displayed in the second display area 601b and the processor 300 changes the form of the content image 602 displayed in the first display area 601a.

[0110] As described above, according to this embodiment, by determining whether user 401's gaze is directed toward business medium 404, it is possible to accurately determine whether user 401 is focused on the business using business medium 404. If user 401's gaze is determined to be directed toward business medium 404, the form of the content image is modified to reduce the degree of fixation, thereby directing user 401's attention toward business medium 404 and preventing a decrease in the user's concentration on the business. On the other hand, if the user's gaze is not directed toward business medium 404, the user's attention can be directed toward the content image, thereby enhancing the user's sense of relaxation and improving work efficiency.

[0111] The present disclosure can adopt the following modified examples.

[0112] (1) Alternatively, the gaze estimation unit 302 may obtain a potential difference (ocular voltage) between the cornea and retina of the eyeball of the user 401, detected by an electrode (an example of a sensor) in contact with the periphery of the user's eye, and estimate the gaze direction of the user 401 based on the obtained potential difference. Details of this method are disclosed in, for example, Japanese Patent Application Laid-Open No. 2018-82727.

[0113] (2) Alternatively, the content processing unit 304 may determine the user 401's field of view on the screen 403 or the first display area 601a using the user's line of sight direction estimated by the line of sight estimation unit 302, and may soften the content images within the determined field of view, while not softening the content images in areas not included in the determined field of view. In this case, the content processing unit 304 may determine a viewing cone centered on the field of view direction and determine the plane where the viewing cone intersects the screen 403 or the first display area 601a as the field of view.

[0114] (3) Alternatively, when the sight estimation unit 302 determines that the sight line of the user 401 is directed toward the business medium 404, the content processing unit 304 may lower the brightness of the content displayed on the screen 403 to a lower value than the default brightness. This can guide the user's awareness toward the business medium. Figure 8 In the example of , the content processing unit 304 can make the brightness of the first display area 601a lower than the default brightness. In addition, in this case, as the time when the user 401's sight line is directed toward the business medium 404 increases, the brightness gradually decreases.

[0115] (4) If the business medium 404 is a computer equipped with a display, the content processing unit 304 obtains brightness information related to the brightness of the display equipped with the computer. The content processing unit 304 can then reduce the brightness of the content so that the brightness of the content displayed on the screen 403 is lower than the brightness of the display indicated by the obtained brightness information. If the business medium 404 and the information processing system 1 are connected via a network, the content processing unit 304 can obtain the brightness information of the display via the network.

[0116] (5) When restoring the changed form of the content image to its pre-change form, the content processing unit 304 may immediately restore the form. Alternatively, the content processing unit 304 may perform a process of gradually restoring the form as the time during which the user 401 is not looking toward the service medium 404 increases. For example, if the change in the form of the content image is a softening of the content image, if it is determined that the user 401 is not looking toward the service medium 404, the softening intensity may be gradually reduced, causing the content image to gradually change toward a non-softened content image. This can reduce the discomfort felt by the user 401.

[0117] (6) Figure 5 In the example, the content image 504 is softened and the moving body region showing the moving body with obvious motion is masked. However, the content image 504 may not be softened and the moving body region showing the moving body with obvious motion may be masked.

[0118] (7) Other 1

[0119] The space where the user 401 performs a service using the service medium 404 may be a space inside a mobile object.

[0120] During the period when the user 401 is processing services, the mobile object may be stopped or in motion.

[0121] For example, the mobile object may be a vehicle.

[0122] The processor 300 may be an ECU (Electronic Control Unit) that is standard equipment in a vehicle, or may be configured separately from the ECU.

[0123] Figure 9 is a diagram showing an example of a vehicle. Figure 9 As shown, a screen 703 displaying a content image may be configured in a vehicle.

[0124] The camera 402 may be a camera provided in a personal computer or may be a camera installed somewhere inside a vehicle where the camera can capture the location of the business medium and the user.

[0125] The user 401 may be handling business in the rear seat of the vehicle, and the camera may be configured in a position capable of photographing the rear seat 720 of the vehicle.

[0126] The screen 703 may be disposed between the front seats 710 of the vehicle and the rear seats 720 of the vehicle. The front seats 710 include a driver's seat and a passenger seat.

[0127] Figure 10 It schematically represents Figure 9 The figure shows an example of a scene where a user in rear seat 720 can see front seat 710. Screen 703 may include a light-adjusting sheet with controllable transmittance. By controlling this transmittance, the information processing system can be configured so that the user can visually recognize the vehicle's direction of travel through screen 703 from rear seat 720.

[0128] Alternatively, screen 703 may have a back surface and a display surface for displaying images, with the display surface located on the side of rear seat 720 and the back surface located on the side of front seat 710. In other words, screen 703 may be configured to present content images to users in rear seats conducting business or meetings using computers, etc. The distance between the display surface and rear seat 720 may be smaller than the distance between the back surface and rear seat 720.

[0129] When a projector is used to project an image onto a screen, the projector may be disposed inside the moving object.

[0130] Even if the space where the user handles the business is the space inside the mobile object, the information processing system 100 can also Figure 3 The same actions as shown in the flowchart.

[0131] (8) Other 2

[0132] The information processing method of the present disclosure may be applied to a plurality of users.

[0133] Figure 11 4 is a diagram showing an example of a situation where multiple users utilize the booth 400. Figure 11 In the example, multiple users include a first user 801 and a second user 802. The first user 801 uses a personal computer 803 as a business medium to process business, and the second user 802 uses a notebook 804 as a business medium to process business. Figure 11 Unlike the example shown, the plurality of users may include more than three users.

[0134] exist Figure 11 In the embodiment, there is a camera 402 in the observation booth 400. However, multiple cameras may be provided in the booth 400, or multiple cameras corresponding to the multiple users may be provided in the booth 400 for photographing the multiple users. The multiple cameras may be cameras equipped for the multiple business media used by the multiple users.

[0135] exist Figure 11 In the embodiment, there is only one screen 403 for displaying content images, but multiple screens may be arranged in the booth 400. Alternatively, multiple screens corresponding to multiple users may be arranged in the booth 400 for displaying content images to the multiple users. The multiple screens may be arranged facing the multiple users.

[0136] Figure 12 This is a diagram showing an example of processing of the information processing system 1 in a mode assuming a case where a plurality of users utilize the booth 400 .

[0137] Step S201 is the same operation as step S101.

[0138] In step S202, the user determination unit 301 reads the latest image captured in step S201 from the sensor database 202, and uses the read image to determine whether there are multiple users in the booth 400. Hereinafter, the image to be processed is referred to as a target image.

[0139] If it is determined that there are multiple users in the booth 400 ("Yes" in step S203), the process proceeds to step S204; if it is determined that there are not multiple users in the booth 400 ("No" in step S203), the process proceeds to step S205. Figure 3In this method, if it is determined that there are two or more users in the booth 400, it can be determined that there are multiple users in the booth 400. If there is only one user in the booth 400, or if it is determined that there is no user in the booth 400, it can be determined that there are not multiple users in the booth 400.

[0140] In step S204 , the gaze estimation unit 302 applies the above-described technique for estimating the gaze direction to the object image, and estimates the gaze direction of each of the plurality of users.

[0141] In step S205 , the business medium detection unit 303 applies the above-mentioned object detection technology to the object image, and detects a plurality of business media used by each of the plurality of users.

[0142] In step S206, the service medium detection unit 303 determines whether the gaze directions of the multiple users estimated in step S204 are toward the multiple service media detected in step S205. For example, if the multiple users include a first user and a second user, the service medium detection unit 303 determines whether the first user's gaze direction is toward the first service medium used by the first user, and determines whether the second user's gaze direction is toward the second service medium used by the second user.

[0143] If it is determined that multiple business media exist in the line of sight of multiple users ("Yes" in step S207), processing proceeds to step S208. If it is determined that multiple business media do not exist in the line of sight of multiple users ("No" in step S207), processing proceeds to step S209. Determining that multiple business media exist in the line of sight of multiple users may mean that the line of sight of all users handling business within booth 400 is directed toward multiple business media corresponding to each of those users. For example, if the business medium detection unit 303 determines that the line of sight of a first user is directed toward a first business medium used by the first user, and the line of sight of a second user is directed toward a second business medium used by a second user, the business medium detection unit 303 determines that multiple business media exist in the line of sight of each of the multiple users. Alternatively, if the business medium detection unit 303 determines that at least one of the line of sight of the first user and the line of sight of the second user is not directed toward a business medium, the business medium detection unit 303 determines that multiple business media do not exist in the line of sight of each of the multiple users.

[0144] In step S208, content processing unit 304 reads the content image to be displayed from content database 201, applies a softening process to the read content image, and displays the softened content image on screen 403. If booth 400 is equipped with multiple screens corresponding to multiple users, the softened content image can be displayed on each of the multiple screens. Alternatively, in step S208, instead of applying a softening process to the read content image, a process can be applied to change the area corresponding to the moving object included in the read content image to a background image corresponding to that area, and then display the image on screen 403. Alternatively, a process can be applied to reduce the brightness of the read content image, and then display the image on screen 403.

[0145] When it is determined that a business medium exists in the line of sight of one of the multiple users ("Yes" in step S209), the processing proceeds to step S210; when it is determined that a business medium does not exist in the line of sight of any of the multiple users ("No" in step S209), the processing proceeds to step S211. For example, when the business medium detection unit 303 determines that the line of sight of the first user is toward the first business medium used by the first user, or when it is determined that the line of sight of the second user is toward the second business medium used by the second user, it is determined that a business medium exists in the line of sight of one of the multiple users. When the business medium detection unit 303 determines that the line of sight of the first user is not toward the first business medium used by the first user, and when it is determined that the line of sight of the second user is not toward the second business medium used by the second user, it is determined that a business medium does not exist in the line of sight of any of the multiple users.

[0146] In step S210, the content processing unit 304 reads out a content image having motion information indicating a small amount of motion from the content database 201 and displays it on the screen 403. Furthermore, if a content image is displayed on the screen 403 at the time of step S209, the content image displayed in step S210 may have a smaller amount of motion than the content image already displayed.

[0147] In step S211, the content processing unit 304 reads the content image to be displayed from the content database 201 and displays it on the screen 403. The amount of motion of the content image displayed in step S211 may be greater than the amount of motion of the content image displayed in step S210. Furthermore, if the content image is displayed on the screen 403 at the time of step S209, step S211 may maintain the display of the already displayed content image.

[0148] The object image to be processed may include a plurality of object images. The plurality of object images may include a first object image corresponding to a first user and a second object image corresponding to a second user.

[0149] As mentioned above, in Figure 12 In the example of the processing shown, when all the gazes of multiple users handling business in booth 400 are directed toward the business medium, the form of the content image displayed on screen 403 is changed to reduce the attention of all the users, thereby suppressing the decline in the concentration of all the users.

[0150] In addition, when the gaze of one of the multiple users handling business in compartment 400 is directed toward the business medium, while the gazes of other users are not directed toward the business medium, a content image with little motion is displayed, thereby suppressing the decline in concentration of the user who is focused on handling business, while increasing the sense of relaxation of the user who is not focused on business, and improving the work efficiency of the user who is not focused on business.

[0151] Industrial applicability

[0152] According to the present disclosure, a decrease in user concentration on business using business media is suppressed, and thus the present disclosure is useful in the technical field of improving a work environment.

[0153] Description of Reference Numerals

[0154] 1: Information Processing System

[0155] 200: Database

[0156] 201: Content Database

[0157] 202: Sensor Database

[0158] 300: Processor

[0159] 301: User determination unit

[0160] 302: Line of sight estimation unit

[0161] 303: Business Media Detection Department

[0162] 304: Content Processing Department

[0163] 400: Compartment

[0164] 401: User

[0165] 402: Camera

[0166] 403: Screen

[0167] 404: Business medium

[0168] 601a: First display area

[0169] 601b: Second display area

Claims

1. An information processing method, executed by a computer, wherein: Displaying a content image on the display unit, Determine whether the user is looking towards a service medium used by the user to process services, When it is determined that the user is directing his or her gaze toward the business medium, the form of the content image displayed on the display unit is changed so as to reduce the user's attention.

2. The information processing method according to claim 1, wherein: Changing the form of the content image includes softening the content image.

3. The information processing method according to claim 2, wherein: If it is determined that the user is looking toward the business medium, the degree of softening of the content image is increased according to the length of time the user is looking toward the business medium.

4. The information processing method according to claim 1, wherein: The content image is a moving image, Changing the form of the content image means changing a region corresponding to the moving object included in the content image to a background image corresponding to the region.

5. The information processing method according to claim 1, wherein: Changing the form of the content image includes reducing the brightness of the content image. The information processing method according to claim 1 , wherein: If it is determined that the user is not looking toward the business medium after the form of the content image is changed, the form of the content image is restored to the form before the change.

7. The information processing method according to claim 1, wherein: The display unit includes a first display area facing the user and a second display area different from the first display area. Displaying the content image on the display unit includes displaying the content image in the first display area and the second display area, wherein a first content image included in the content image is displayed in the first display area, and a second content image included in the content image is displayed in the second display area. Changing the form of the content image includes changing the form of the first content image and not changing the form of the second content image.

8. The information processing method according to claim 1, wherein: The content image includes a first content motion image and a second content motion image with a smaller motion amount than the first content motion image, When it is determined that the user is looking toward the business medium, the first content moving image displayed on the display unit is changed to the second content moving image.

9. The information processing method according to claim 1, wherein: The content image is an image that forms an atmosphere surrounding the user.

10. The information processing method according to claim 1 or 2, wherein: The determination of whether the user's line of sight is directed toward the business medium includes making the determination using sensing data detected by a sensor that detects a state of at least an eye of the user.

11. An information processing system comprising a processor, wherein: The processor performs the following processing: Displaying a content image on the display unit, Determine whether the user is looking towards a service medium used by the user to process services, When it is determined that the user is directing his or her gaze toward the business medium, the form of the content image displayed on the display unit is changed so as to reduce the user's attention.

12. The information processing method according to claim 1, wherein: The display unit includes a back surface and a display surface for displaying the content image. The display unit is arranged between a front seat of a vehicle and a rear seat of the vehicle, The distance between the display surface and the rear seat is smaller than the distance between the back surface and the rear seat.

Images