Information processing system, information processing device, information processing method, and program

Abstract

An information processing device 2 has: a data acquisition unit 231 that acquires display state data for identifying a virtual space image which a user U is watching and in which a display region and a display orientation are changed by operation of the user U; a close watch identification unit 232 for identifying a close watch time point that is a time point when the user U has closely watched at least a part of the virtual space image for not less than a prescribed time period; a feature identification unit 233 for identifying a feature generation time point which is a time point when a prescribed feature was included in the virtual space image displayed in an information terminal 1 used by the user U; and an output unit 234 that outputs the close watch time point and the feature generation time point in association with each other.

Description

Information processing system, information processing device, information processing method and program

[0001] The present invention relates to an information processing system, an information processing device, an information processing method, and a program.

[0002] 2. Description of the Related Art Conventionally, a system is known that outputs information indicating a user's reaction when viewing an image in a virtual reality space or an augmented reality space (hereinafter referred to as a "virtual space image") (see, for example, Patent Document 1).

[0003] International Publication No. 2020 / 032339

[0004] Conventional systems can output information indicating changes in a user's reaction while viewing a virtual space image. Designers or providers of virtual space images (hereinafter referred to as "designers, etc.") can improve the virtual space image by understanding changes in the user's reaction. However, simply understanding changes in the user's reaction makes it difficult for designers, etc., to recognize how the user will react to certain features of the virtual space image. Therefore, there is a problem in that designers, etc., have difficulty understanding how to change the virtual space image.

[0005] The present invention has been made in consideration of these points, and has as its object to make it easier for designers of virtual space images to improve virtual space images.

[0006] An information processing system of a first aspect of the present invention comprises an information terminal and an information processing device capable of communicating with the information terminal, wherein the information terminal has an operation reception unit that receives user operations, and a display processing unit that displays a virtual space image by changing the display area and display orientation of virtual space image data provided from the information processing device based on the user operation received by the operation reception unit, and the information processing device has a data acquisition unit that acquires display state data for identifying the virtual space image that the user is viewing, a gaze identification unit that identifies, based on the display state data, a gaze time point, which is the time point when the user viewing the virtual space image gazes at at least a portion of the virtual space image for a predetermined period of time or more, a feature identification unit that identifies a feature occurrence time point, which is the time point when a predetermined feature was included in the virtual space image displayed on the information terminal, and an output unit that outputs the gaze time point and the feature occurrence time point in association with each other.

[0007] The data acquisition unit may acquire end data from the information terminal indicating that the user has finished viewing the virtual space image, and the output unit may output information indicating the end point of viewing indicated by the end data, in association with the gaze point and the feature occurrence point.

[0008] The output unit may output state data indicating a result of estimating the biological state of the user based on the gaze time point and the feature occurrence time point.

[0009] The feature specifying unit may specify the feature occurrence time point when a composition of the virtual space image displayed on the information terminal includes a one-point perspective composition.

[0010] The feature identification unit may identify the feature occurrence time point including at least one of a type, a size, and a position of guidance information indicating a route that the user can travel in the virtual space image.

[0011] The gaze specifying unit may specify a time point when the user gazes at the guidance information. When the virtual space image includes a plurality of pieces of guidance information, the gaze specifying unit may specify a piece of guidance information that has been gazed at for a longest period of time among the plurality of pieces of guidance information.

[0012] The feature identification unit may identify the proportion of time that each of the multiple objects is displayed in the virtual space image displayed within a predetermined period of time including the time of gaze, and the output unit may output information indicating the proportion of time in association with the virtual space image.

[0013] The output unit may output information indicating a time period during which the user has continued to view the virtual space image.

[0014] The information processing device may further include a storage unit that stores attributes of the user, and the output unit may output the gaze time point and the feature occurrence time point in association with the attributes of the user.

[0015] The output unit may display, on the same time axis, multiple images showing the gaze points and the feature occurrence points in chronological order when the user continues viewing the virtual space image for different periods of time.

[0016] An information processing device of a second aspect of the present invention has a data acquisition unit that acquires display state data for identifying a virtual space image that a user is viewing and whose display area and display orientation change depending on the user's operation, a gaze identification unit that identifies a gaze point, which is the point at which the user gazes at at least a portion of the virtual space image for a predetermined period of time or more, a feature identification unit that identifies a feature occurrence point, which is the point at which a predetermined feature was included in the virtual space image displayed on an information terminal used by the user, and an output unit that associates the gaze point with the feature occurrence point and outputs them.

[0017] An information processing method of a third aspect of the present invention includes the steps of: executing a computer, acquiring display state data for identifying a virtual space image that a user is looking at and whose display area and display orientation change due to the user's operation; identifying a gaze point, which is the point at which the user gazes at at least a portion of the virtual space image for a predetermined period of time or more; identifying a feature occurrence point, which is the point at which a predetermined feature is included in the virtual space image displayed on an information terminal used by the user; and outputting the gaze point and the feature occurrence point in association with each other.

[0018] A fourth aspect of the program of the present invention is a program for causing a computer to execute the following steps: acquiring display state data for identifying a virtual space image that a user is viewing and whose display area and display orientation change due to the user's operation; identifying a gaze point, which is the point at which the user gazes at at least a portion of the virtual space image for a predetermined period of time or more; identifying a feature occurrence point, which is the point at which a predetermined feature was included in the virtual space image displayed on an information terminal used by the user; and outputting the gaze point and the feature occurrence point in association with each other.

[0019] According to the present invention, it is possible to provide an effect that designers of virtual space images can easily improve the virtual space images.

[0020] 1 is a diagram showing an overview of an information processing system S. FIG. 1 is a diagram schematically showing the entirety of virtual space image data. FIG. 2 is a diagram showing an example of a virtual space image viewed by a user U. FIG. 2 is a diagram showing the configuration of an information terminal 1. FIG. 3 is a diagram showing the configuration of an information processing device 2. FIG. 3 is a diagram showing an example of a screen including information output by an output unit 234. FIG. 4 is a diagram showing an example of a screen on which an estimation result of a biological state is displayed. FIG. 4 is a diagram showing an example of an image showing, in chronological order, gaze points (dashed lines) and points (solid lines) at which a predetermined feature was included when the viewing durations are different. FIG. 5 is a diagram showing an overview of an information processing system S1. FIG. 6 is a diagram showing an example of a screen displayed on an information terminal 1. FIG. 7 is a diagram showing the configuration of an information terminal 1. FIG. 8 is a diagram showing an example of recommended state data. FIG. 9 is a diagram showing an example of recommended state data. FIG. 10 is a diagram showing the configuration of an information processing device 2. FIG. 11 is a sequence diagram showing an example of a processing flow in the information processing system S1. FIG. 12 is a diagram showing an overview of an information processing system S2. FIG. 13 is a diagram showing the configuration of an information processing device 4.

[0021] 1 is a diagram illustrating an overview of the information processing system S. The information processing system S is a system that presents a virtual space image to a user U and enables a designer of virtual space image data or the like to grasp the point in time at which the user U gazed while viewing the virtual space image and the point in time at which a predetermined feature was included in the virtual space image viewed by the user U. The predetermined feature of the virtual space image is at least one of the following: the virtual space image includes a predetermined composition (e.g., a one-point perspective composition or a two-point perspective composition), the virtual space image includes an image of a predetermined object, the virtual space image includes route guidance information, or the proportion of a predetermined color in the virtual space image is equal to or greater than a threshold.

[0022] The virtual space image data is, for example, image data representing a three-dimensional space created by computer graphics, but it may also include image data generated by photographing a real subject, or may include augmented reality image data.

[0023] The virtual space image data is configured so that the displayed area changes when, for example, the user U operates the terminal on which the virtual space image is displayed. That is, the virtual space image data corresponds to an area larger than the image area that can be displayed on the terminal used by the user U. The virtual space image data is image data that virtually represents a space in which the user U can move the viewing position, such as an event venue, a streetscape, or the interior of a building.

[0024] As an example, if the virtual space image data is image data showing the space of an event venue, the event organizer, who is the provider of the virtual space image data, wants the user U to view as many areas as possible. For example, if the event organizer wants to promote a product, they want the user U to proceed to the area where the product is being explained. However, depending on the configuration of the virtual space image data, the user U may become confused about which direction to proceed or lose interest midway, causing them to stop viewing the virtual space image.

[0025] The information processing system S outputs the time when the user U gazed while viewing the virtual space image and the characteristics of the virtual space image that changed while the user U was viewing it, making it easier for designers of the virtual space image data to understand the characteristics of the virtual space image that the user U gazed at. The information processing system S can also output information indicating the time when the user U finished viewing the virtual space image, in association with information indicating the time when the user U gazed at the virtual space image and information indicating the characteristics of the virtual space image.

[0026] By looking at this information, designers and the like can understand how the virtual space image should be configured to make it easier for user U to stop viewing, and how the virtual space image should be configured to make it easier for user U to continue viewing, which makes it easier for designers and the like of virtual space images to improve the virtual space image data.

[0027] An overview of the information processing system S will be described below with reference to Fig. 1. The information processing system S includes an information terminal 1, an information processing device 2, and an output terminal 3. The information terminal 1, the information processing device 2, and the output terminal 3 are connected to a network N, and can transmit and receive data to and from other devices.

[0028] The information terminal 1 is a terminal used by the user U to view virtual space images, and may be, for example, a personal computer, a tablet, a smartphone, or a terminal in the shape of glasses or goggles. The information terminal 1 displays on a display a virtual space image based on virtual space image data received from the information processing device 2 via the network N.

[0029] The information terminal 1 changes the position and orientation of the virtual space image displayed on the display from the virtual space image data based on the operation of the user U. The information terminal 1 transmits display status data indicating the position and orientation in the virtual space image data of the area of ​​the virtual space image displayed on the display to the information processing device 2. The display status data is data including, for example, information indicating the position in the space image data where the information terminal 1 is displaying and the orientation of the user U, and is associated with time. The display status data may be data captured of the screen on which the information terminal 1 is displaying the virtual space image. In this case, the captured data is associated with time. Note that the time is represented, for example, by the elapsed time since the user U started viewing the virtual space image.

[0030] A position in the virtual space image data is represented by three-dimensional coordinates or two-dimensional coordinates based on the origin of the three-dimensional space represented by the virtual space image data. The origin of the three-dimensional space is, for example, the position where the user U begins to view the virtual space image (i.e., the position where the user U begins to move). A position in the virtual space image data is, for example, the coordinates of the virtual space image data corresponding to the center position of the display.

[0031] The virtual space image data may include, as metadata, coordinates indicating the location of predetermined information or predetermined objects included in the virtual space image data. The metadata may include, for example, coordinates of the location where guidance information such as a sign or an advertisement is displayed. The metadata may also include coordinates of the location where the object is displayed in association with the type of the object.

[0032] The orientation of the virtual space image data is expressed by its orientation relative to a reference orientation of the three-dimensional space represented by the virtual space image data. The reference orientation is, for example, the orientation of the virtual space image displayed on the display before the user U performs an operation to change the display position and orientation of the virtual space image.

[0033] The information processing device 2 provides the virtual space image data to the information terminal 1. Furthermore, the information processing device 2 outputs the time when the user U gazed and the features of the virtual space image viewed by the user U in association with each other based on the display state data received from the information terminal 1. The information processing device 2 outputs, for example, information indicating the time when the user U gazed and the time when the virtual space image including the predetermined features was displayed to the output terminal 3 via the network N.

[0034] 2A and 2B are diagrams showing a schematic representation of the entire virtual space image data. The virtual space image data shown in Fig. 2A shows the layout of the event venue as seen from above. The hatched areas in Fig. 2 represent buildings, signs, signage, etc. Fig. 2B shows that the user U has moved within the event venue along the route shown by the dashed lines in accordance with the user U's operations.

[0035] 3A and 3B are diagrams showing examples of virtual space images seen by a user U. Fig. 3A is a virtual space image seen by the user U when the user U is facing the direction of the arrow at point A in Fig. 2B. Fig. 3B is a virtual space image seen by the user U when the user U is facing the direction of the arrow at point B in Fig. 2B. Fig. 3C is a virtual space image seen by the user U when the user U is facing the direction of the arrow at point C in Fig. 2B.

[0036] The object marked with the symbol K in Fig. 3(b) is an example of guidance information that indicates the direction of travel to the user U. The object marked with the symbol K has information written thereon, for example, about the content of an event being held at a location in the direction of the object, or information indicating the direction in which the user U should travel.

[0037] 4 is a diagram showing the configuration of the information terminal 1. The information terminal 1 has a communication unit 11, a display unit 13, an operation unit 14, a storage unit 15, and a control unit 16. The control unit 16 has a data acquisition unit 161, an operation acceptance unit 162, and a display processing unit 163.

[0038] The communication unit 11 has a communication interface for transmitting and receiving data to and from the information processing device 2 via the network N. The communication unit 11 receives, for example, virtual space image data transmitted from the information processing device 2 and inputs the received virtual space image data to the data acquisition unit 161. The communication unit 11 also acquires display state data indicating the position and orientation of the virtual space image displayed on the display unit 13 from the display processing unit 163, associates the display state data with time, and transmits the data to the information processing device 2. The communication unit 11 may also transmit captured data of the virtual space image displayed on the display unit 13 as display state data, associates the data with time, and transmits the data to the information processing device 2.

[0039] The display unit 13 is a display that displays various types of information, such as a virtual space image based on virtual space image data.

[0040] The operation unit 14 is a device that receives operations from the user U, and includes, for example, a keyboard and a mouse. The operation unit 14 inputs data indicating the content of the operation performed by the user U to the operation reception unit 162.

[0041] The storage unit 15 has storage media such as a read-only memory (ROM), a random access memory (RAM), and a solid-state drive (SSD). The storage unit 15 stores programs executed by the control unit 16. The storage unit 15 also temporarily stores virtual space image data, display status data, and the like. For example, the storage unit 15 stores the display status data in association with the elapsed time since the user U started viewing the virtual space image.

[0042] The control unit 16 has a CPU (Central Processing Unit). The control unit 16 executes the programs stored in the storage unit 15, thereby functioning as a data acquisition unit 161, an operation acceptance unit 162, and a display processing unit 163.

[0043] The data acquisition unit 161 acquires virtual space image data transmitted from the information processing device 2. The data acquisition unit 161 stores the acquired virtual space image data in the storage unit 15. The data acquisition unit 161 may input the acquired virtual space image data to the display processing unit 163.

[0044] The operation accepting unit 162 accepts an operation performed by the user U on the operation unit 14. Specifically, the operation accepting unit 162 accepts, as the content of the operation by the user U, coordinates indicating the position where the user U touches the display unit 13 or clicks the mouse while the virtual space image is displayed on the display unit 13, for example.

[0045] The display processing unit 163 displays a virtual space image based on the virtual space image data on the display unit 13. For example, based on an operation of the user U accepted by the operation accepting unit 162, the display processing unit 163 changes the display area and display orientation of the virtual space image data provided from the information processing device 2, thereby displaying a virtual space image corresponding to image data of a portion of the virtual space image data.

[0046] 5 is a diagram showing the configuration of the information processing device 2. The information processing device 2 has a communication unit 21, a storage unit 22, and a control unit 23. The control unit 23 has a data acquisition unit 231, a gaze identification unit 232, a feature identification unit 233, and an output unit 234.

[0047] The communication unit 21 has a communication interface for transmitting and receiving data to and from the information terminal 1 and the output terminal 3 via the network N. The communication unit 21 inputs the received data to the data acquisition unit 231. The communication unit 21 also transmits the data input from the output unit 234.

[0048] The storage unit 22 has storage media such as a ROM, a RAM, and an SSD. The storage unit 22 stores programs executed by the control unit 23. The storage unit 22 also stores virtual space image data. The storage unit 22 stores the display state data acquired by the data acquisition unit 231 via the communication unit 21 in association with the user U.

[0049] The control unit 23 has a CPU, and functions as a data acquisition unit 231 , a gaze identification unit 232 , a feature identification unit 233 , and an output unit 234 by executing the programs stored in the storage unit 22 .

[0050] The data acquisition unit 231 acquires display state data for identifying the virtual space image that the user U is viewing from the information terminal 1 via the communication unit 21. The data acquisition unit 231 stores the acquired display state data in the storage unit 22 in association with the user U. The data acquisition unit 231 may store the display state data in the storage unit 22 in association with the elapsed time since the user U started viewing the virtual space image data. The data acquisition unit 231 may acquire end data from the information terminal 1 indicating that the user U has finished viewing the virtual space image.

[0051] The gaze identification unit 232 identifies a gaze time point, which is a time point when the user U viewing the virtual space image gazes at at least a part of the virtual space image for a predetermined period of time or more, based on the display state data. The predetermined period of time is the minimum time during which the user U is expected not to move the virtual space image when a virtual space image in which the user U is interested is displayed, and is, for example, 3 seconds. The predetermined period of time may be a time that can be set by a designer or the like via an external device such as the output terminal 3, or may be a fixed time.

[0052] The gaze identification unit 232 compares the positions and orientations indicated by the plurality of display state data corresponding to the plurality of times, and identifies a time point at which the user U is gazing when the user U is standing still in the same position and facing the same orientation for a predetermined period of time. When the display state data is image data captured by the information terminal 1, the gaze identification unit 232 identifies a time point at which the user U is gazing when the plurality of captured data corresponding to the plurality of times are identical for a predetermined period of time.

[0053] The gaze identification unit 232 may identify the position in the virtual space image data at which the user U gazes. For example, the gaze identification unit 232 identifies the center position of the virtual space image displayed on the information terminal 1 at the time the gaze time is identified as the position at which the user U gazes. If the information terminal 1 has a function to detect the line of sight of the user U, the gaze identification unit 232 may identify the position of the line of sight of the user U detected by the information terminal 1 at the time of gaze as the position at which the user U gazes.

[0054] The gaze identification unit 232 may identify the point in time when the user U gazes at the guidance information. The gaze identification unit 232, for example, identifies coordinates indicating the position of the guidance information by referring to metadata of the virtual space image data, and identifies the point in time when the guidance information is included within a predetermined range from the center position of the display of the information terminal 1 as the point in time when the user U gazes at the guidance information. This predetermined range is a range in which the user U looking directly at the display of the information terminal 1 can clearly view the guidance information, for example, a range with a radius of 5 cm from the center position.

[0055] When the virtual space image includes a plurality of pieces of guidance information, the gaze identification unit 232 may identify, among the plurality of pieces of guidance information, the guidance information that has been gazed at for a longer period of time. When the virtual space image includes a plurality of pieces of guidance information, the gaze identification unit 232 may identify, as the gazed-at guidance information, one or more pieces of guidance information that have been gazed at for a period of time equal to or greater than a threshold value.

[0056] The feature identification unit 233 identifies a feature occurrence time point, which is a time point at which a predetermined feature was included in the virtual space image displayed on the information terminal 1 used by the user U. For example, the feature identification unit 233 analyzes the shape of an object included in the virtual space image indicated by the display state data, to identify a time point at which the composition of the virtual space image displayed on the information terminal 1 was a one-point perspective composition as the feature occurrence time point. The feature identification unit 233 may also identify a time point at which at least one of the type, size, or position of guidance information indicating a route that the user U can travel in the virtual space image was included as the feature occurrence time point.

[0057] The feature identification unit 233 may identify the proportion of time each of a plurality of objects is displayed in the virtual space image displayed within a predetermined period of time including the gaze point. The plurality of objects may be, for example, an aisle, an intersection, guidance information, a green area (e.g., grass or tree leaves), a brown area (e.g., tree trunks or branches), or a landmark. This predetermined period of time may be, for example, the time during which the user U continuously gazes at the same position. This predetermined period of time may be the time from when the user U starts viewing the virtual space image to when the user U finishes viewing it. This predetermined period of time may be the period from the gaze point immediately before the user U finishes viewing the virtual space image to when the user U finishes viewing it.

[0058] The output unit 234 associates and outputs the gaze time points identified by the gaze identification unit 232 and the feature occurrence time points identified by the feature identification unit 233. The output unit 234 transmits information indicating the gaze time points and information indicating the feature occurrence time points to the output terminal 3, for example, via the communication unit 21. The output unit 234 may output information indicating the viewing end time points indicated by the end data acquired from the information terminal 1 by the data acquisition unit 231, in association with the gaze time points and the feature occurrence time points.

[0059] Fig. 6 is a diagram showing an example of a screen including information output by the output unit 234. Fig. 6 includes regions R1, R2, R3, R4, and R5. When the output unit 234 displays the screen shown in Fig. 6 on the output terminal 3, the output unit 234 can change the information included on the screen in accordance with an operation performed on the output terminal 3 by a designer or the like.

[0060] Area R1 is an area where multiple thumbnail images of multiple virtual space images, each representing a different amount of time that has elapsed since user U began viewing the virtual space images, are displayed. The multiple thumbnail images correspond to the virtual space images that user U was viewing at the time when a multiple of a predetermined time (e.g., 20 seconds) has elapsed since the start of viewing. A designer or the like using output terminal 3 can change the information displayed in other areas by selecting one thumbnail image from the multiple thumbnail images displayed in area R1.

[0061] Region R2 is an area where a virtual space image corresponding to a thumbnail image selected by a designer or the like is displayed. Region R2 may also display a virtual space image corresponding to an elapsed time specified by a designer or the like. Region R2 may also display multiple thumbnail images in chronological order. In this case, the output unit 234 may display thumbnail images having characteristics specified by a designer or the like in a different manner from other thumbnail images. For example, the output unit 234 displays thumbnail images including guidance information on the output terminal 3 in a framed state.

[0062] The output unit 234 may display in the region R2 a thumbnail image of the virtual space image corresponding to the time when the feature occurrence time and the gaze time overlap. By displaying such a thumbnail image by the output unit 234, the designer or the like can easily understand which feature the user U gazed at in the virtual space image.

[0063] Area R3 is an area where images are displayed showing the gaze point and the feature occurrence point within a predetermined time from the elapsed time corresponding to the thumbnail image selected by the designer, etc. The horizontal axis in area R3 corresponds to the elapsed time since user U began viewing the virtual space image corresponding to the selected thumbnail image. The vertical axis in area R3 corresponds to the intensity of gaze by user U and the strength of the predetermined feature shown by the virtual space image. The dashed line E in area R3 indicates the time when user U finished viewing.

[0064] The slower the speed at which the user U changes the position of the virtual space image, the stronger the gaze. Also, the stronger the degree to which a specific feature appears in the virtual space image, the stronger the strength of the specific feature. For example, the longer the contour line indicating the one-point perspective composition, the stronger the strength of the one-point perspective composition characteristic. The larger the size of the guidance information, the stronger the strength of the characteristic that the guidance information is included.

[0065] The peaks indicated by the dashed lines in region R3 in Fig. 6 indicate the time points of gaze. The longer the duration of the peak of the peak, the longer the duration of the gaze state. The peaks indicated by the solid lines in region R3 indicate the time points of feature occurrence. The form of the images indicating the time points of gaze and the time points of feature occurrence is not limited to the form shown in Fig. 6 and can be any form.

[0066] As shown in region R3, the output unit 234 may output information indicating the duration of viewing (e.g., the dashed line indicated by symbol E) in association with the time of gaze or the time of occurrence of a feature. By outputting such information, the output unit 234 can easily allow a designer or the like to grasp the correlation between the time the user U continued viewing, the tendency of the user U to gaze, and the tendency of the feature of the virtual space image.

[0067] The output unit 234 may be configured to change the type of feature for which the feature occurrence time point is to be output, according to settings by the designer, etc. Specifically, when "one-point perspective composition" is selected as the feature, the output unit 234 displays the image shown in region R3 in Fig. 6 on the output terminal 3. When "two-point perspective composition" is selected as the feature, the output unit 234 displays, in region R3, an image indicating the time point at which the two-point perspective composition was identified by the feature identification unit 233.

[0068] The output unit 234 may simultaneously display in the region R3, in different modes, images showing the time points at which each of the multiple features was identified by the feature identification unit 233. That is, the output unit 234 may display in the region R3 multiple types of feature occurrence time points corresponding to multiple different features in different modes for each type of feature. By displaying the time points at which multiple different features occurred on the same time axis as the gaze time point in this manner, the designer or the like can easily grasp the features of the virtual space image that the user U is likely to gaze at.

[0069] Region R4 is an area that displays the number of times that user U gazed at the virtual space image during a predetermined period (e.g., the time corresponding to the image shown in region R3), the number of times a predetermined feature was identified in the virtual space image, and the length of time that user U continued viewing the image. In this way, the output unit 234 may further output information indicating the length of time that user U continued viewing the virtual space image, in association with the number of times that user U gazed at the image or the number of times a predetermined feature was included in the virtual space image.

[0070] Region R5 is an area where information indicating the percentage of time that various objects were included in the virtual space image viewed by user U is displayed. As shown in Fig. 6, the output unit 234 outputs the information indicating the percentage of time associated with the virtual space image. Region R5 displays, for example, the percentage of time that each object was displayed within the time corresponding to the image shown in region R3.

[0071] Region R5 may display the percentage at which each object was displayed within a time range specified by the designer or the like. Region R5 may simultaneously display the percentage at which each object was displayed within a predetermined time range immediately before the user U finished viewing the virtual space image, and the percentage at which each object was displayed within another time range. By displaying such information, the designer or the like can easily understand what kind of virtual space image the user U is viewing is likely to cause the user U to stop viewing.

[0072] The output unit 234 may output status data indicating the result of estimating the user U's biological state based on the gaze time point and the feature occurrence time point. The biological state may be, for example, brain waves, heart rate, or sweat rate. The output unit 234 outputs status data indicating the estimation result that a predetermined biological state is reached during a time period when the number of gazes per unit time is high. The predetermined biological state may be, for example, a state in which theta waves are generated in the user U's brain waves. A state in which theta waves are generated is a state in which brain waves of 4 Hz or more but less than 8 Hz are generated. Theta waves are known to have a correlation with the state of spatial cognition, and it is thought that the user U is more likely to remember an image they are viewing when theta waves are large.

[0073] Fig. 7 is a diagram showing an example of a screen displaying the estimated biological state. Fig. 7 includes an area R6 showing the estimated biological state. Fig. 7 shows that the level of theta waves is high during a time period when the frequency of gaze is above a threshold and when the gaze point overlaps with a time when the virtual space image contains a one-point perspective composition. By outputting such state data from the output unit 234, designers and the like can understand the biological state of user U while user U is viewing the virtual space image, making it possible to improve the virtual space image by taking the biological state into consideration.

[0074] The output unit 234 may output status data indicating an estimated result that theta waves are occurring during a time period when the gaze time overlaps with the time when the virtual space image contains a one-point perspective composition. If theta waves are occurring in the user U, and the virtual space image at the time theta waves are occurring contains an image of guidance information indicating the route, an image of an aisle, an image of a landmark, or the like, the user U is likely to continue to look at the virtual space image for a long period of time and easily memorize it. Therefore, by knowing the state of theta waves, designers and the like can easily improve the virtual space image data so that the user U can continue to look at the virtual space image for a long period of time and easily memorize it.

[0075] When the user U is wearing a device for measuring a biological state, and the data acquisition unit 231 acquires data indicating the measured biological state, for example, via the information terminal 1, the output unit 234 may simultaneously display the estimated biological state and the measured biological state. By operating the output unit 234 in this manner, a designer or the like can grasp the difference between the estimated biological state and the actual biological state. As a result, even when the user U is not wearing a device for measuring a biological state, the designer or the like can easily grasp the actual biological state based on the estimated biological state.

[0076] The output unit 234 may input the data indicating the estimated biological state into a machine learning model that has undergone deep learning using the data indicating the estimated biological state and the data indicating the measured biological state as training data, and output data indicating the biological state after correction in the machine learning model.

[0077] Note that Fig. 7 includes region R6 instead of region R2 in Fig. 6, but region R2 may also be included in Fig. 7. In this case, a virtual space image at a time when a predetermined biological state is reached may be displayed in region R2. For example, the output unit 234 displays a virtual space image at a time when theta waves are at a high level in region R2. By outputting a virtual space image at a time when the biological state is at a predetermined state from the output unit 234, designers and the like can easily understand the relationship between the biological state and the virtual space image, and can easily improve the virtual space image.

[0078] The output unit 234 may input information indicating the gaze time and information indicating the feature occurrence time to output an estimation result of the biological state using a machine learning model that outputs a biological state. The machine learning model is a model created by deep learning using as training data a combination of the gaze time of multiple users U who viewed the virtual space image while measuring their biological state, the time when a predetermined feature was included in the virtual space image, and the level of the biological state value (e.g., theta wave) for each time. The output unit 234 inputs information indicating the gaze time and information indicating the feature occurrence time, as displayed in region R3, to the machine learning model and outputs an estimation result of the biological state output from the machine learning model.

[0079] Incidentally, it is considered that the relationship between the gaze time point and the feature occurrence time point differs depending on the attributes (e.g., gender, age, occupation) of the user U. Therefore, the output unit 234 may output the gaze time point and the feature occurrence time point in association with the attributes of the user U stored in the storage unit 22. By configuring the output unit 234 in this way, it becomes easier for a designer or the like to create virtual space image data suitable for each of the multiple attributes of the user U.

[0080] In order for designers etc. to be able to create virtual space image data that is difficult for user U to stop viewing midway, it is desirable for designers etc. to be able to compare the gaze points when user U stops viewing after a short time with the gaze points when user U continues viewing for a long time. Therefore, the output unit 234 may display, on the same time axis, multiple images that chronologically indicate the gaze points and feature occurrence points when user U continues viewing the virtual space image for different lengths of time.

[0081] Fig. 8 is a diagram showing an example of an image showing, in time series, gaze points (dashed lines) and feature occurrence points (solid lines) for different viewing durations. Fig. 8 shows gaze points, etc., when different virtual space images are viewed by users U with the same attributes. The dashed line H in Fig. 8 indicates the average viewing duration. Fig. 8(a) shows the longest viewing time compared to Fig. 8(b) and Fig. 8(c). Fig. 8(c) shows the shortest viewing time.

[0082] If the predetermined feature is the inclusion of a one-point perspective composition, by comparing Figures 8(a), 8(b), and 8(c), the designer or the like can understand that it is desirable to include many one-point perspective compositions. However, simply comparing these figures may result in the designer or the like overlooking factors that correlate with the viewing duration. Therefore, the output unit 234 may output the results of identifying the occurrence patterns of each of multiple features of virtual space images whose viewing duration is longer than the average viewing duration. As an example, the output unit 234 outputs the intervals between positions where the predetermined feature is included in virtual space image data whose viewing duration is longer than the average viewing duration as the results of identifying the occurrence patterns.

[0083] The output unit 234 may output improvement measures for the virtual space images using a machine learning model created by deep learning using as training data patterns of occurrence of multiple predetermined features in a large number of virtual space images whose viewing duration is longer than the average viewing duration. When data indicating a time point at which the predetermined features were included is input, as shown in FIGS. 8( b) and 8(c), the machine learning model outputs data indicating a time point at which the predetermined features were included that was longer than the average viewing duration and closest to the input data. The output unit 234 outputs information indicating the difference between the data input to the machine learning model and the data output by the machine learning model, making it easier for designers and others to understand how to modify the virtual space image data to potentially extend the viewing duration.

[0084] The output unit 234 may output statistical information on the results of a large number of users U viewing the virtual space image. As an example, the output unit 234 may output the average number of gazes and the average duration of viewing when a single virtual space image is viewed by multiple users U. The output unit 234 may also output a correlation diagram in which the number of gazes and the duration of viewing are plotted on the vertical and horizontal axes. By allowing the designer or the like to understand such statistical information, the designer or the like can more easily improve the virtual space image data so that the duration of viewing by a large number of users U is increased.

[0085] The output unit 234 may also display the user U's movement route, the time spent at each point, the user U's biological condition, and the user U's looking angle (the range of direction changed at the same point) on a layout diagram corresponding to the virtual space image as shown in Fig. 2. Furthermore, the output unit 234 may output the result of classifying the user U's state based on the user U's movement into a selection stage where the user does not know how to proceed, a progress stage where the user knows the direction to proceed, and an arrival stage where the user is gazing at the target object.

[0086] Second Embodiment [Outline of Information Processing System S1] Fig. 9 is a diagram showing an outline of the information processing system S1. The information processing system S1 is a system that displays data indicating the state of the user U by superimposing it on a spatial image viewed by the user U. The spatial image may be an image of the space actually viewed by the user U, or may be an image of the metaverse including a virtual reality image (VR image) created by a computer.

[0087] The data indicating the state of the user U is, for example, biometric information image data indicating changes in the values ​​of the user U's biometric information over time, or state data determined based on the biometric information image data. The biometric information is, for example, information based on the user U's heart rate, brain waves, sweating, skin potential, the user U's line of sight measured by a measuring device M, or the mouse or keyboard operation content that sets the screen range indicating the range of the user U's line of sight, such as graph image information. The state of the user U may be what the user U is feeling while looking at the spatial image, or the user U's mental and physical state, determined based on the user U's biometric information. The state of the user U may be, for example, a relaxed state, a tense state, a concentrated state, or an excited state.

[0088] For example, the information terminal 1 displays the state of the user U identified by the information processing device 2 based on the biometric information image data, superimposed on a spatial image. By displaying the state of the user U superimposed on the spatial image viewed by the user U, the user U viewing the spatial image can understand his or her own state, enabling the user U to determine whether the spatial image viewed is suitable for the user U. If the user U determines that the spatial image viewed is not suitable for the user U, the user U can change it to a spatial image suitable for the user U. Such an information processing system S1 is suitable for displaying an image of the user's own avatar in the metaverse.

[0089] 9, the information processing system S1 includes an information terminal 1, an information processing device 2, and an output terminal 3. The information terminal 1, the information processing device 2, and the output terminal 3 are connected to a network N, and can transmit and receive data to and from other devices.

[0090] The information terminal 1 is a terminal used by the user U to view spatial images, and is, for example, a personal computer, a tablet, a smartphone, or a terminal in the shape of glasses or goggles. The information terminal 1 acquires biometric information from, for example, a measuring device M worn by the user U. The information terminal 1 displays biometric image data indicating changes in the value of the biometric information over time together with spatial image data. The biometric image data is, for example, an augmented reality image (AR image). The information terminal 1 transmits the biometric image data and spatial image data to the information processing device 2.

[0091] The information processing device 2 identifies the state of the user U based on the biometric information image data of the user U displayed on the information terminal 1. For example, when the biometric information image data is input, the information processing device 2 identifies the state of the user U by inputting the biometric information image data into a machine learning model that outputs data indicating the state of the user U. The information processing device 2 transmits state data indicating the identified state to the information terminal 1. Upon receiving the state data, the information terminal 1 displays the state data together with the spatial image data.

[0092] 10 is a diagram showing an example of a screen displayed on the information terminal 1. Spatial image data is displayed in an area R11, biometric information image data is displayed in an area R12, and status data transmitted from the information processing device 2 is displayed in an area R13. The user U can understand his / her own status by looking at the biometric information image data displayed in the area R12, but can also objectively understand his / her own status by looking at the status data displayed in the area R13.

[0093] A "change" icon is displayed in region R11 to receive an operation from user U to change the aspect of the spatial image. When user U selects the "change" icon, the aspect of the spatial image displayed in region R1 is changed. The aspect of the spatial image is, for example, the hue, saturation, brightness, degree of unevenness, shape, or arrangement of floors, walls, ceilings, furniture, etc. In response to the selection of the "change" icon, the information terminal 1 may display multiple candidates for the changed spatial image, or may display a recommended spatial image. In addition to clicking and tapping, the method of selecting an icon may also be gazing, blinking, winking, etc. using eye tracking, or a gesture using hand tracking, etc.

[0094] Furthermore, a "Register" icon is displayed in the region R11 for registering the displayed spatial image as a favorite image of the user U. When the "Register" icon is selected, the information terminal 1 stores the displayed spatial image as a favorite image of the user U and transmits the spatial image to the information processing device 2.

[0095] 10(a), the brightness of the spatial image displayed in region R11 is high, and region R13 shows a state in which the degree of relaxation is low and the degree of tension is high. If the spatial image displayed in region R1 is an image of a living room and the user U feels like relaxing in the living room, when the user U selects the "change" icon, the information terminal 1 changes the spatial image displayed in region R13 to a state in which the state of the user U may change.

[0096] 10(b) shows a screen displaying a spatial image with lower brightness than the spatial image displayed in FIG. 10(a). In this state, the fluctuations in the biological information displayed in region R12 are smaller, and region R13 shows an increased level of relaxation. User U can register the spatial image displayed in FIG. 10(b) by selecting the "Register" icon. Note that the aspect of the spatial image may be indicated by hue and saturation instead of or in addition to brightness.

[0097] The information terminal 1 may transmit the displayed spatial image data to the information processing device 2. The information processing device 2 may then output the status data and spatial image data in association with each other to a device other than the information terminal 1. The information processing device 2 may, for example, transmit the status data and spatial image data in association with each other to an output terminal 3 different from the information terminal 1, or may print the status data and spatial image data in association with each other. The output terminal 3 is a computer used by people who design buildings, spaces, or image data thereof (hereinafter referred to as "designers, etc."), such as designers, researchers, or marketers. By displaying the status data and spatial image data in association with each other on the output terminal 3, designers, etc. can understand what impression the user U has of a certain space and use this information in their design.

[0098] 11 is a diagram showing the configuration of the information terminal 1. The information terminal 1 has a display unit 13, an operation unit 14, a storage unit 15, a control unit 16, a first communication unit 17, and a second communication unit 18. The control unit 16 has an operation reception unit 162, a display processing unit 163, a biometric information acquisition unit 164, a status data acquisition unit 165, and a data transmission unit 166.

[0099] The first communication unit 17 has a communication interface for transmitting and receiving data to and from the measuring device M. The first communication unit 17 has a wireless communication interface such as Bluetooth (registered trademark) or Wi-Fi (registered trademark). The first communication unit 17 inputs the biometric information received from the measuring device M to the biometric information acquisition unit 164.

[0100] The second communication unit 18 has a communication interface for transmitting and receiving data to and from the information processing device 2 via the network N. The second communication unit 18 transmits, for example, biometric information image data and spatial image data input from the data transmission unit 166 to the information processing device 2, and inputs status data received from the information processing device 2 to the status data acquisition unit 165.

[0101] The display unit 13 is a display that displays various types of information. For example, the display unit 13 displays a spatial image based on spatial image data. The display unit 13 also creates biometric information image data based on the biometric information of the user U viewing the spatial image, and displays the biometric information image based on the created biometric information image data. Furthermore, when the first communication unit 17 receives status data from the information processing device 2, the display unit 13 displays text based on the status data.

[0102] The operation unit 14 is a device that receives operations from the user U, and includes, for example, a keyboard and a mouse. The operation unit 14 inputs data indicating the content of the operation performed by the user U to the operation reception unit 162.

[0103] The storage unit 15 has storage media such as a ROM, a RAM, and an SSD. The storage unit 15 stores a program executed by the control unit 16. The storage unit 15 also temporarily stores spatial image data, biometric information, biometric information image data, etc. For example, the storage unit 15 stores the biometric information image data and the spatial image data displayed on the display unit 13 at the time the biometric information was acquired, in association with the time the biometric information was acquired.

[0104] Furthermore, the storage unit 15 stores recommended state data that the display processing unit 163 uses to change the mode of the spatial image displayed on the display unit 13. The recommended state data is data that associates information for identifying the spatial image (for example, the name of the spatial image) with a state that is recommended as the state of the user who has viewed the spatial image.

[0105] 12 and 13 are diagrams showing examples of recommended state data. In the example shown in Fig. 12, a recommended state of "surprise level: high" is associated with a spatial image of an entrance. Also, a recommended state of "relaxation level: high" is associated with a spatial image of a living room. In the recommended state data, multiple recommended states may be associated with one spatial image name.

[0106] The recommended state data may be written in the program of the application software executed by the control unit 16, but the desired state in each space may differ depending on the user U. Therefore, the storage unit 15 may store a recommended state for each space image set by the user U via the operation unit 14. Furthermore, the recommended state data may include data indicating a non-recommended state rather than a recommended state.

[0107] As shown in Fig. 13, the recommended state data may be associated with the name of a spatial image and include a numerical value indicating an acceptable range of scores indicating the state of the user U who viewed the spatial image. In the example shown in Fig. 13, for the entrance hall, the required score for "surprise level" is "8 or more," the required score for "openness level" is "5 or more," and the required scores for the other states are arbitrary. In this way, the recommended state data includes data indicating an acceptable range for each type of state, making it easier to reflect the preferences of the user U for each space.

[0108] Furthermore, the storage unit 15 may store data indicating a tendency of the relationship between the appearance of the spatial image and the state of the user U. As an example, the storage unit 15 stores data indicating a tendency that when the user U views a spatial image with low brightness, the user U is more likely to feel calm than when the user U views a spatial image with high brightness, or that when the user U views a spatial image with a one-point perspective composition, the user U is more likely to feel calm than when the user U views a spatial image with a two-point perspective composition. This data is used when the display processing unit 163 changes the appearance of the spatial image to be displayed.

[0109] The control unit 16 has a CPU. The control unit 16 executes the programs stored in the storage unit 15 to function as an operation reception unit 162, a biometric information acquisition unit 164, a status data acquisition unit 165, a display processing unit 163, and a data transmission unit 166.

[0110] The operation acceptance unit 162 accepts an operation by the user U based on data input from the operation unit 14. The operation acceptance unit 162 notifies an appropriate processing unit in the control unit 16 of the content of the accepted operation. For example, the operation acceptance unit 162 notifies the display processing unit 163 that it has accepted an operation by the user U to select the "change" icon shown in FIG. 10 . The operation acceptance unit 162 also notifies the display processing unit 163 that it has accepted an operation to select one candidate from multiple candidates for the changed spatial image displayed on the display unit 13. The method of selecting an icon may be, in addition to clicking or tapping, gaze, blinking, winking, etc. using eye tracking, or a gesture using hand tracking, etc.

[0111] The biometric information acquisition unit 164 acquires biometric information of the user viewing the spatial image on the information terminal 1 via the first communication unit 17. The biometric information acquisition unit 164 inputs the acquired biometric information to the display processing unit 163.

[0112] The status data acquisition unit 165 acquires the status data output by the information processing device 2 via the second communication unit 18. The status data acquisition unit 165 inputs the acquired status data to the display processing unit 163.

[0113] The display processing unit 163 creates screen data to be displayed on the display unit 13, thereby displaying various types of information on the display unit 13. The display processing unit 163, for example, displays a virtual image based on virtual image data stored in the storage unit 15 on the display unit 13. The display processing unit 163 also displays a biometric information image based on biometric information superimposed on a spatial image, as shown in FIG. 10 . Furthermore, when the status data acquisition unit 165 acquires status data via the second communication unit 18, the display processing unit 163 displays text or an image based on the acquired status data superimposed on the spatial image on the display unit 13. The display processing unit 163 may display a biometric information image of the user U in real time, or may display an image showing biometric information for a predetermined time in the past (for example, a few seconds set by the user U).

[0114] When the status data acquired by the status data acquisition unit 165 indicates a predetermined status corresponding to a case where the spatial image data is changed, the display processing unit 163 displays updated spatial image data corresponding to an updated spatial image different from the currently displayed spatial image. The predetermined status is, for example, a status different from the recommended status indicated by the recommended status data associated with the spatial image name and stored in the storage unit 15.

[0115] The display processing unit 163 changes the state of the spatial image so that the state indicated by the status data approaches the state indicated by the recommended status data, or updates the spatial image to another spatial image. The display processing unit 163 changes the brightness or composition of the spatial image, for example, by referring to data indicating the tendency of the relationship between the state of the spatial image and the state of the user U, which is stored in the storage unit 15. The display processing unit 163 may continue changing the state of the spatial image until the state indicated by the status data falls within the allowable range indicated by the recommended status data.

[0116] The display processing unit 163 may display the status data acquired by the status data acquisition unit 165 on the display unit 13, and then display one or more candidate updated spatial images that may bring the status indicated by the status data closer to the status indicated by the recommended status data. The display processing unit 163 determines the candidate updated spatial images by referring to data indicating the tendency of the relationship between the appearance of the spatial images stored in the storage unit 15 and the status of the user U. As an example, when it is necessary to increase the calmness level, the display processing unit 163 determines a candidate updated spatial image that is lower in brightness than the displayed spatial image.

[0117] The display processing unit 163 may display one or more update space image candidates, and then, in response to the operation of selecting one update space image candidate from the one or more update space image candidates being accepted by the operation accepting unit 162, display an update space image corresponding to the selected update space image candidate. By operating in this manner, the display processing unit 163 can change the state of the user U to the recommended state, and can display a space image in a form preferred by the user U on the display unit 13.

[0118] The display processing unit 163 may superimpose an image of the user U's avatar on a spatial image, which is an image of the metaverse. The display processing unit 163 displays, together with the spatial image, an image of the avatar wearing clothing of a color selected by the user U, and a biometric information image or status data. By displaying the image of the user U's avatar and the biometric information image or status data, the display processing unit 163 can adjust the image of the avatar or the image of the metaverse so that the user U's status is appropriate when the avatar selected by the user is displayed in the metaverse.

[0119] The data transmission unit 166 transmits biometric information image data corresponding to the biometric information image to the information processing device 2. For example, the data transmission unit 166 cuts out the biometric information image data displayed in region R2 of Fig. 10 from the screen displayed on the display unit 13, and transmits the cut-out biometric information image data to the information processing device 2. Since the size of the biometric information image data is smaller than the virtual space image data, the data transmission unit 166 operates in this manner, thereby making it possible to reduce the amount of data transmitted to the information processing device 2.

[0120] The data transmission unit 166 may transmit the biometric information image data and spatial image data corresponding to a spatial image in which a biometric information image corresponding to the biometric information image data is displayed to the information processing device 2. The data transmission unit 166 may normally transmit only the biometric information image data to the information processing device 2, and when receiving a request for spatial image data from the information processing device 2, may read from the storage unit 15 the spatial image data that was displayed on the display unit 13 at the time the measuring instrument M acquired biometric information corresponding to the biometric information image data, and transmit the read spatial image data to the information processing device 2.

[0121] Furthermore, when an avatar image is displayed superimposed on the spatial image, the data transmission unit 166 may transmit spatial image data with the avatar image deleted to the information processing device 2. By transmitting spatial image data that does not include the avatar image to the information processing device 2 by the data transmission unit 166, a designer or the like who views the spatial image data output by the information processing device 2 can evaluate and analyze the spatial image data without being affected by the avatar image.

[0122] 14 is a diagram showing the configuration of the information processing device 2A. The information processing device 2 has a communication unit 21, a storage unit 22, and a control unit 23. The control unit 23 has a data acquisition unit 231, a state identification unit 235, an output unit 234, and a data request unit 236.

[0123] The communication unit 21 has a communication interface for transmitting and receiving data to and from the information terminal 1 and the output terminal 3 via the network N. The communication unit 21 inputs the received data to the data acquisition unit 231. The communication unit 21 also transmits data input from the output unit 234 and the data request unit 236.

[0124] The storage unit 22 has storage media such as a ROM, a RAM, and an SSD. The storage unit 22 stores a program executed by the control unit 23. The storage unit 22 also stores data used to identify the state of the user U based on the biometric information image data. The data is, for example, data that associates the waveform characteristics of the biometric information indicated by the biometric information image data with the state of the user U. The data may be a machine learning model that outputs data indicating the state of the user U when the biometric information image data is input.

[0125] Furthermore, the storage unit 22 may store attributes of the user U in association with user identification information for identifying the user U. The attributes of the user U include, for example, age, sex, family structure, or residential area.

[0126] The control unit 23 has a CPU, and functions as a data acquisition unit 231 , a state identification unit 235 , an output unit 234 , and a data request unit 236 by executing the programs stored in the storage unit 22 .

[0127] The data acquisition unit 231 acquires biometric information image data from the information terminal 1 via the communication unit 21. The data acquisition unit 231 acquires the biometric information image data, for example, in association with the user identification information of the user U. The data acquisition unit 231 inputs the acquired biometric information image data to the state identification unit 235 and the output unit 234.

[0128] The data acquisition unit 231 may acquire the spatial image data displayed on the information terminal 1 via the communication unit 21. The data acquisition unit 231 may acquire the biometric information image data and the spatial image data in association with each other, or may acquire the biometric information image data and then acquire the spatial image data displayed on the information terminal 1 at the time when the measuring instrument M acquires the biometric information corresponding to the biometric information image data.

[0129] The condition identification unit 235 identifies the condition of user U based on the biometric information image data. The condition identification unit 235 identifies, in the storage unit 22, a condition associated with waveform characteristics of biometric information indicated by the biometric information image data as the condition of user U. The condition identification unit 235 may input the biometric information image data to a machine learning model that has been trained in advance using a large number of sets of biometric information image data and the condition of user U as training data, and identify the condition of user U based on data indicating the condition output from the machine learning model. The condition identification unit 235 notifies the output unit 234 of the identified user condition.

[0130] The output unit 234 outputs status data indicating the status of the user U identified by the status identification unit 235. The output unit 234 transmits the status data of the user U to the information terminal 1 used by the user U, for example, via the communication unit 21. The output unit 234 may also transmit the status data of the user U to the output terminal 3 via the communication unit 21.

[0131] The output unit 234 may output status data, associated with the spatial image data, indicating the status of the user identified based on the biometric information image data corresponding to the biometric information acquired at the time the spatial image data was displayed. That is, the output unit 234 may transmit the status data in association with the spatial image data that was displayed on the information terminal 1 at the time the measuring device M acquired the biometric information corresponding to the biometric information image data used to identify the status data. By the output unit 234 outputting the status data in association with the spatial image data, a designer or the like who checks these data can easily understand the status of the user U in each space.

[0132] The user U's perception of a space may differ depending on the attributes of the user U. Therefore, the output unit 234 may refer to the storage unit 22 to identify attributes corresponding to the user identification information associated with the biometric information image data acquired by the data acquisition unit 231, and output the spatial image data corresponding to the spatial image that was displayed at the time the biometric information corresponding to the biometric information image data was acquired, the status data, and the identified attributes in association with each other. The output unit 234 outputs the spatial image data and the status data in association with the attributes of the user U, making it easier for a designer or the like who checks these data to understand how the perception of each space differs depending on the attributes of the user U.

[0133] Meanwhile, designers and the like have a need to understand what kind of space makes it easy for user U to enter a predetermined state. Therefore, the output unit 234 may output one or more pieces of spatial image data transmitted in association with one or more pieces of bioinformation image data corresponding to each type of state of user U, in association with the type of state indicated by the state data. As an example, when a designer and the like performs an operation to understand examples of spaces where people can relax, the output unit 234 selects one or more pieces of spatial image data associated with the bioinformation image data identified by the state identification unit 235 as corresponding to a relaxed state, and outputs the selected spatial image data as a space that makes it easy to relax.

[0134] In order to suppress variations due to the individuality of the user U, the output unit 234 may select a plurality of spatial image data associated with a plurality of biometric information image data that have been identified as corresponding to a relaxed state of the user U, from among various states of the user U identified by the state identification unit 235 using the biometric information image data acquired by the data acquisition unit 231 from the plurality of information terminals 1. By looking at such a plurality of spatial image data, a designer or the like can easily grasp what kind of space is likely to be relaxing for a large number of users U.

[0135] Since there is a possibility that the tendency may differ depending on the attributes of the user U, the output unit 234 may output multiple spatial image data in association with the attributes and state types of the user U. In this case, the output unit 234 classifies multiple pieces of biometric information image data corresponding to the states set by a designer or the like, by the attributes of the user U of the information terminal 1 that transmitted the biometric information image data. Then, the output unit 234 selects multiple pieces of spatial image data corresponding to the multiple pieces of biometric information image data for each classified attribute.

[0136] When the output unit 234 outputs spatial image data corresponding to a predetermined state, if the data acquisition unit 231 continues to acquire the spatial image data together with the biometric information image data, the amount of data flowing through the network N increases, and the processing load on the data acquisition unit 231 also increases. Therefore, when the state identification unit 235 identifies that the state of the user U has reached a predetermined state, the data request unit 236 transmits, to the information terminal 1, request data for requesting spatial image data corresponding to the spatial image that was displayed at the time when the biometric information corresponding to the biometric information image data for which the predetermined state was identified was acquired.

[0137] This predetermined state is, for example, a state set by a designer or the like, or a state after a change occurs in the state of the user U. The request data includes, for example, an ID or a timestamp as information for identifying the biometric information image data.

[0138] In response to receiving the request data, the information terminal 1 reads from the storage unit 15 the spatial image data that was displayed at the time the biometric information corresponding to the biometric information image data indicated by the request data was acquired, and the data transmission unit 166 transmits the read spatial image data. The output unit 234 outputs the spatial image data transmitted by the information terminal 1 in response to the data request unit 236 transmitting the request data to the information terminal 1, in association with status data indicating a predetermined status. By configuring the information terminal 1 and the information processing device 2 in this manner, it is possible to reduce the amount of spatial image data transmitted and received between the information terminal 1 and the information processing device 2, and to reduce the load on the network N, the information terminal 1, and the information processing device 2.

[0139] [Processing flow in information processing system S1] Fig. 15 is a sequence diagram showing an example of the processing flow in the information processing system S1. The sequence diagram shown in Fig. 15 illustrates a case where status data is displayed on the information terminal 1. The sequence diagram shown in Fig. 15 starts from a state where the user U is viewing a spatial image.

[0140] In the information terminal 1, while the display processing unit 163 is displaying a spatial image based on the spatial image data on the display unit 13 (S11), the biometric information acquisition unit 164 acquires biometric information of the user U from the measuring device M (S12). The display processing unit 163 creates biometric information image data based on the biometric information and displays the biometric information image on the display unit 13 (S13). In addition, the data transmission unit 166 transmits the biometric information image data to the information processing device 2 (S14).

[0141] In the information processing device 2, the condition identification unit 235 analyzes the biometric information image data received from the information terminal 1 (S21) and identifies the condition of the user U (S22). The output unit 234 outputs the identified condition data (S23). In the example shown in Fig. 15, the output unit 234 transmits the condition data to the information terminal 1, but the output unit 234 may transmit the condition data to the output terminal 3. When the condition data acquisition unit 165 acquires the condition data, the display processing unit 163 displays the condition data on the display unit 13 (S15).

[0142] While the processes from S11 to S15 are being performed, the data request unit 236 monitors whether the state identified by the state identification unit 235 is a predetermined state (S24). If the data request unit 236 determines that the state identified by the state identification unit 235 is a predetermined state (YES in S24), it transmits request data for requesting spatial image data to the information terminal 1 (S25).

[0143] The data transmission unit 166 refers to the spatial image data stored in the storage unit 15 to identify the spatial image data that was displayed on the display unit 13 at the time identified based on the request data (S16), and transmits the identified spatial image data to the information processing device 2 (S17). The output unit 234 outputs the spatial image data in association with the status data (S26).

[0144] While the spatial image data is being displayed on the display unit 13, the operation accepting unit 162 monitors whether an operation for changing the spatial image has been performed in the information terminal 1 (S18). When the operation accepting unit 162 detects that an operation for changing the aspect of the spatial image has been performed (YES in S18), the display processing unit 163 updates the spatial image data so as to improve the condition of the user U, based on the spatial image data being displayed at that time and the condition indicated by the condition data (S19).

[0145] [Effects of information processing system S1] In conventional systems, people other than the user viewing a virtual space, such as a virtual reality space or a space displaying augmented reality images (for example, designers, researchers, or marketers), could know the user's reactions, but there was a problem in that users (for example, including users who simply view the virtual space, as well as designers, researchers, or marketers involved in the creation of the virtual space) were unable to objectively grasp their own state while viewing the space.

[0146] In response to this, the information terminal 1 in the information processing system S1 transmits biometric image data indicating the waveform of biometric information acquired while the user U is viewing the spatial image to the information processing device 2, and the information processing device 2 identifies the user U's condition based on the biometric image data and creates status data. The information terminal 1 displays the status data received from the information processing device 2 together with the spatial image, allowing the user U to objectively grasp their own condition while viewing the spatial image. As a result, the user U can switch to a spatial image that is more suitable for them. Such an information processing system S1 is suitable for applications in which the user's own avatar is displayed in the metaverse.

[0147] Furthermore, in the information processing system S1, the information processing device 2 identifies the condition of the user U by analyzing the biometric information image data displayed on the information terminal 1. Therefore, the information processing device 2 can identify the condition of the user U even when it is not possible to directly acquire biometric information from the measuring device M.

[0148] The information processing device 2 then acquires spatial image data from the information terminal 1, associates the acquired spatial image data with the state data, and outputs the data. By the information processing device 2 outputting such data, it becomes easier for designers and others to understand what kind of spatial image is desirable. Furthermore, by acquiring spatial image data when a predetermined state is detected, the information processing device 2 can reduce the amount of data received compared to when spatial image data is always transmitted from the information terminal 1 to the information processing device 2, and can also lighten the processing load of the information processing device 2.

[0149] The technology of this embodiment is suitable for enabling users viewing real or virtual spaces (including metaverse spaces) to take an active interest in the spaces they experience without relying on the space's designer, and for users to contribute to improving the spaces themselves.

[0150] [Outline of Information Processing System S2] Fig. 16 is a diagram showing an outline of the information processing system S2. The information processing system S2 has an information processing device 4 instead of the information processing device 2 in the information processing system S1, and the information processing device 4 transmits spatial image data to the information terminal 1. Then, while the user U is viewing the spatial image, the measuring device M transmits biometric information to the information processing device 4 via the network N. The information processing device 4 identifies the condition of the user U based on the received biometric information and transmits condition data to the information terminal 1. The information processing device 4 may also transmit biometric information image data to the information terminal 1. With this configuration, the information terminal 1 can display the condition of the user U while displaying the spatial image, as shown in Fig. 10.

[0151] 17 is a diagram showing the configuration of the information processing device 4. The information processing device 4 has a communication unit 41, a storage unit 42, and a control unit 43.

[0152] 14 , and the storage unit 42 corresponds to the storage unit 22 in the information processing device 2. The storage unit 42 stores spatial image data to be displayed on the information terminal 1. When the output unit 433 described later transmits spatial image data, the storage unit 42 stores information indicating the transmission destination of the spatial image data (for example, identification information of the information terminal 1 or identification information of the user U) and the transmission time in association with the spatial image data.

[0153] The storage unit 42 may store the recommended state data stored in the storage unit 15 of the information terminal 1 in the information processing system S1, and data indicating the tendency of the relationship between the state of the spatial image and the state of the user U.

[0154] The control unit 43 has a CPU corresponding to the control unit 23 in the information processing device 2, and functions as a biometric information acquisition unit 431, a state identification unit 432, and an output unit 433 by executing programs stored in the memory unit 42.

[0155] The biometric information acquisition unit 431 acquires the biometric information transmitted by the measuring device M via the communication unit 41. That is, the biometric information acquisition unit 431 acquires the biometric information of the user U who is looking at the spatial image data displayed on the information terminal 1. The biometric information acquisition unit 431 inputs the acquired biometric information to the condition identification unit 432.

[0156] The condition identification unit 432 has a function equivalent to that of the condition identification unit 235 in the information processing device 2, and identifies the condition of the user U based on the biometric information input from the biometric information acquisition unit 431. The condition identification unit 432 inputs condition data indicating the identified condition to the output unit 433.

[0157] The output unit 433 has a function equivalent to that of the display processing unit 163 in the information processing system S1. Specifically, the output unit 433 transmits spatial image data to display the spatial image data on the information terminal 1. Furthermore, the output unit 433 associates status data indicating the status of the user U identified by the status identification unit 432 with the spatial image data that was displayed on the information terminal 1 at the time the biometric information corresponding to the status data was acquired, and outputs the associated data. For example, by transmitting the status data input from the status identification unit 432 to the information terminal 1, the output unit 433 can display the status of the user U on the information terminal 1 while the user U is viewing the spatial image.

[0158] The output unit 433 may change the spatial image data to be displayed on the information terminal 1 based on the state of the user U identified by the state identification unit 432. Specifically, similar to the display processing unit 163 in the information processing system S1, the output unit 433 may change the state of the spatial image data so as to bring the state of the user U closer to the recommended state, using the recommended state data stored in the storage unit 42 and data indicating the tendency of the relationship between the state of the spatial image and the state of the user U.

[0159] The output unit 433 may associate the spatial image data with the status data and output them to the output terminal 3. In this case, the output unit 433 identifies the spatial image data stored in the storage unit 42 in association with the time when the biometric information acquisition unit 431 acquired the biometric information, and transmits the identified spatial image data and status data in association with each other to the output terminal 3. As in the information processing system S1, the output unit 433 may output the spatial image data and status data in further association with the attributes of the user U.

[0160] The information processing device 4 may display the spatial image data and at least one of the biometric information image data and the status data on an information terminal 1 connected to the information processing device 4 without going through the network N. Alternatively, the information processing device 4 may be a computer used by the user U, and the information processing device 4 may display the spatial image data and at least one of the biometric information image data and the status data on a display.

[0161] Furthermore, the information processing device 4 may acquire biometric information of multiple users U from the information terminals 1 used by the multiple users U, and transmit multiple pieces of biometric image data indicating the biometric information of the multiple users U viewing the same spatial image to the multiple information terminals 1 used by the multiple users U viewing the same spatial image. In this case, each information terminal 1 displays the multiple pieces of biometric image data corresponding to the multiple users U transmitted from the information processing device 4 superimposed on the spatial image data. Displaying such a screen on the information terminal 1 enables the user U to compare his or her own condition with that of other people.

[0162] The information processing device 4 may transmit a plurality of biometric information image data corresponding to a plurality of users U to the output terminal 3. In this case, the output terminal 3 displays the plurality of biometric information image data superimposed on the spatial image data. By displaying such a screen on the output terminal 3, a designer or the like can grasp the state of a plurality of users U who view the spatial image, and therefore it becomes possible to design a spatial image taking many users U into consideration.

[0163] [Effects of Information Processing System S2] In the information processing system S2, as in the information processing system S1, the state of the user U while the user U is viewing the spatial image is displayed on the information terminal 1, allowing the user U to objectively grasp his or her own state while viewing the spatial image. In the information processing system S2, the information processing device 4 provides the information terminal 1 with spatial image data and also provides the information terminal 1 with state data identified based on the acquired biometric information. Therefore, even if the information terminal 1 is a general-purpose computer that does not have the function of acquiring biometric information or pre-storing spatial image data, it is possible to obtain the same effects as those of the information processing system S1.

[0164] [Variations] In the above description, image data of graphs based on heart rate or brain waves has been mainly exemplified as biometric image data, but the biometric image data may also be image data indicating the position of the user U's gaze. The image data indicating the position of the user U's gaze is, for example, data of a heat map image in which areas of a spatial image that the user U looks at for a long time are represented by different colors or patterns depending on whether the area is an area that the user U looks at for a short time. In this case, the measuring device M is, for example, a glasses-type or goggle-type terminal, and the area that the user U is looking at is identified by using a sensor included in the terminal to identify the direction of the user U's gaze. By looking at the heat map image, the user U can understand which areas of the spatial image they tend to look at the most.

[0165] The information processing device 2 or the information processing device 4 may output the spatial image data on which the heat map image data is superimposed to the output terminal 3. This allows the designer or the like to design the spatial image taking into consideration areas that the user U is likely to pay attention to. Furthermore, the information processing device 2 or the information processing device 4 may analyze the heat map image data to analyze areas that the user U tends to look at, or analyze the emotions of the user U based on the tendency of the user U's line of sight movement, and output the analysis results. This makes it easier for the designer or the like to design the spatial image taking emotions into consideration.

[0166] (Additional Note) Another aspect of the information processing system of the present invention includes an information terminal and an information processing device capable of communicating with the information terminal, wherein the information terminal has a biometric information acquisition unit that acquires biometric information of a user viewing a spatial image on the information terminal, a display processing unit that causes the information terminal to display a biometric information image based on the biometric information superimposed on the spatial image, and a data transmission unit that transmits biometric information image data corresponding to the biometric information image to the information processing device, and the information processing device may have a data acquisition unit that acquires the biometric information image data from the information terminal, a status identification unit that identifies the status of the user based on the biometric information image data, and an output unit that outputs status data indicating the status of the user identified by the status identification unit.

[0167] The output unit may transmit the status data to the information terminal, and the information terminal may further have a status data acquisition unit that acquires the status data output by the output unit, and the display processing unit may display the status data superimposed on the spatial image.

[0168] The display processing unit may display an updated spatial image that is different from the spatial image currently being displayed when the status data acquired by the status data acquisition unit indicates a predetermined state corresponding to a case in which the spatial image is changed.

[0169] The information terminal may further have an operation reception unit that receives operations from the user, and the display processing unit may display one or more update space image candidates after displaying the status data acquired by the status data acquisition unit, and may display an update space image corresponding to the selected update space image candidate in response to the operation reception unit receiving an operation to select one update space image candidate from the one or more update space image candidates.

[0170] The information processing device may further have a memory unit that stores attributes of the user in association with user identification information for identifying the user, the data acquisition unit acquires the biometric information image data in association with the user identification information, and the output unit may identify the attributes corresponding to the user identification information associated with the biometric information image data acquired by the data acquisition unit by referring to the memory unit, and output spatial image data corresponding to the spatial image that was displayed at the time the biometric information corresponding to the biometric information image data was acquired, the status data, and the identified attributes in association with each other.

[0171] The data transmission unit may transmit the biometric information image data and spatial image data corresponding to the spatial image in which the biometric information image corresponding to the biometric information image data is displayed to the information processing device, and the output unit may output the status data, associated with the spatial image data, indicating the user's status identified based on the biometric information image data corresponding to the biometric information acquired at the time the spatial image data was displayed.

[0172] The data transmission unit may transmit the biometric information image data and spatial image data corresponding to the spatial image in which the biometric information image corresponding to the biometric information image data is displayed to the information processing device, and the output unit may output the one or more spatial image data transmitted in association with one or more of the biometric information image data corresponding to the user's state of each of the types, in association with the type of state indicated by the status data.

[0173] The information processing device may further have a data request unit that, when the state identification unit identifies that the user's state has become a predetermined state, sends request data to the information terminal to request spatial image data corresponding to the spatial image that was displayed at the time the biometric information corresponding to the biometric information image data for which the predetermined state was identified was acquired, and the output unit may output the spatial image data sent by the information terminal in response to the data request unit sending the request data to the information terminal in association with the state data indicating the predetermined state.

[0174] Another aspect of the information processing device of the present invention includes an output unit that displays spatial image data on an information terminal by transmitting the spatial image data, a biometric information acquisition unit that acquires biometric information of a user who is viewing the spatial image data displayed on the information terminal, and a status identification unit that identifies the status of the user based on the biometric information, and the output unit may associate status data indicating the user's status identified by the status identification unit with the spatial image data that was displayed on the information terminal at the time the biometric information corresponding to the status data was acquired, and output the associated data to the information terminal.

[0175] Another aspect of the information processing method of the present invention may include the steps of: executing, by a computer, transmitting spatial image data to display the spatial image data on an information terminal; acquiring biometric information of a user viewing the spatial image data displayed on the information terminal; identifying the user's condition based on the biometric information; and associating status data indicating the identified user's condition with the spatial image data that was displayed on the information terminal at the time the biometric information corresponding to the status data was acquired, and outputting the associated status data to the information terminal.

[0176] Another aspect of the program of the present invention may cause a computer to execute the steps of transmitting spatial image data to an information terminal to display the spatial image data; acquiring biometric information of a user viewing the spatial image data displayed on the information terminal; identifying the user's condition based on the biometric information; and associating status data indicating the identified user's condition with the spatial image data that was displayed on the information terminal at the time the biometric information corresponding to the status data was acquired, and outputting the associated status data to the information terminal.

[0177] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating in any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination also have the effects of the original embodiments.

[0178] REFERENCE SIGNS LIST 1 Information terminal 2 Information processing device 3 Output terminal 4 Information processing device 11 Communication unit 13 Display unit 14 Operation unit 15 Memory unit 16 Control unit 17 First communication unit 18 Second communication unit 21 Communication unit 22 Memory unit 23 Control unit 41 Communication unit 42 Memory unit 43 Control unit 161 Data acquisition unit 162 Operation acceptance unit 163 Display processing unit 164 Biometric information acquisition unit 165 Status data acquisition unit 166 Data transmission unit 231 Data acquisition unit 232 Gaze identification unit 233 Feature identification unit 234 Output unit 235 Status identification unit 236 Data request unit 431 Biometric information acquisition unit 432 Status identification unit 433 Output unit

Claims

1. The system comprises an information terminal and an information processing device capable of communicating with the information terminal, The aforementioned information terminal is An operation reception unit that receives user input, A display processing unit that displays a virtual space image by changing the display area and display orientation of the virtual space image data provided by the information processing device based on the user's operation received by the operation reception unit, It has, The aforementioned information processing device is A data acquisition unit that acquires display state data for identifying the virtual space image that the user is viewing, A gaze identification unit identifies a gaze point, which is the point in time when the user viewing the virtual space image gazes at at least a portion of the virtual space image for a predetermined period of time or longer, based on the display state data. A feature identification unit identifies a feature occurrence time, which is the point in time when a predetermined feature is found in the virtual space image displayed on the information terminal. An output unit that outputs the aforementioned viewing time and the aforementioned feature occurrence time in association, It has, The output unit displays multiple images on the same time axis, showing the time of gaze and the time of feature occurrence in chronological order, for cases where the user continued to view the virtual space image for different periods of time. Information processing system.

2. The data acquisition unit acquires termination data from the information terminal indicating that the user has finished viewing the virtual space image. The output unit outputs information indicating the end of viewing as shown in the end data, in relation to the viewing time and the feature occurrence time. The information processing system according to claim 1.

3. The output unit outputs state data showing the result of estimating the user's biological state based on the gaze time and the feature occurrence time. The information processing system according to claim 1.

4. The feature identification unit identifies the point in time when the composition of the virtual space image displayed on the information terminal includes a one-point perspective composition. The information processing system according to claim 1.

5. The feature identification unit identifies the feature occurrence time, which includes at least one of the type, size, or position of guidance information indicating a path that the user can move in the virtual space image. The information processing system according to claim 1.

6. The attention-identification unit identifies the point in time when the user was paying attention to the guidance information. The information processing system according to claim 5.

7. If the virtual space image contains multiple pieces of the guidance information, the gaze identification unit identifies the guidance information that has been gazed upon for a longer period of time among the multiple pieces of the guidance information. The information processing system according to claim 6.

8. The feature identification unit identifies the proportion of time during which each of the multiple objects is displayed in the virtual space image displayed within a predetermined time including the time of gaze, The output unit outputs information indicating the proportion of time in relation to the virtual space image. The information processing system according to claim 1.

9. The output unit outputs information indicating the duration for which the user continued to view the virtual space image. The information processing system according to claim 1.

10. The information processing device further includes a storage unit for storing the user's attributes, The output unit outputs the viewing time and the feature occurrence time in association with the user's attributes. The information processing system according to claim 1.

11. A data acquisition unit acquires display state data for identifying a virtual space image that the user is viewing and whose display area and orientation change due to the user's operation. A gaze identification unit that identifies a gaze time, which is the point in time when the user gazes at at least a portion of the virtual space image for a predetermined period of time or longer, A feature identification unit identifies a feature occurrence time, which is the point in time when a predetermined feature was found in the virtual space image displayed on the information terminal used by the user. An output unit that outputs the aforementioned viewing time and the aforementioned feature occurrence time in association, It has, The output unit displays multiple images on the same time axis, showing the time of gaze and the time of feature occurrence in chronological order, for cases where the user continued to view the virtual space image for different periods of time. Information processing device.

12. A computer executes A step of acquiring display state data for identifying a virtual space image that the user is viewing, whose display area and display orientation change due to the user's operation, The steps include identifying a viewing time, which is the point in time when the user has been gazing at at least a portion of the virtual space image for a predetermined period of time or longer, The steps include: identifying the feature occurrence time, which is the point in time when a predetermined feature was found in the virtual space image displayed on the information terminal used by the user; A step of outputting the aforementioned observation time and the aforementioned feature occurrence time in association, It has, The output step involves displaying multiple images on the same time axis, showing the time of gaze and the time of feature occurrence in chronological order, for cases where the user continued to view the virtual space image for different periods of time. Information processing methods.

13. On the computer, A step of acquiring display state data for identifying a virtual space image that the user is viewing, whose display area and display orientation change due to the user's operation, The steps include identifying a viewing time, which is the point in time when the user has been gazing at at least a portion of the virtual space image for a predetermined period of time or longer, The steps include: identifying the feature occurrence time, which is the point in time when a predetermined feature was found in the virtual space image displayed on the information terminal used by the user; A step of outputting the aforementioned observation time and the aforementioned feature occurrence time in association, Make it run, The output step involves displaying multiple images on the same time axis, showing the time of gaze and the time of feature occurrence in chronological order, for cases where the user continued to view the virtual space image for different periods of time. program.