Information processing device, information processing method, and program

Abstract

To provide details about an object which a user has seen and recognized in response to a simple operation by the user.SOLUTION: A control unit transmits information about a target facility identified in video captured by a user device based on a user's state in three-dimensional space or the state in three-dimensional space of the user device moving with the user to the user device or an output device located within a predetermined range from the position of the user device based on a command made by the user's gesture obtained via the user device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] A conventional user interface is disclosed in which a user's gesture is recognized from first and second images, and an interaction command corresponding to the recognized user's gesture is determined (for example, Patent Document 1 below). Furthermore, it is disclosed that an image object displayed on the user interface is operated based on the determined interaction command. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2011-517357 Summary of the Invention [Problem to be solved by the invention]

[0004] However, it is not possible to clearly show details of various objects included in an image simply by manipulating image objects, such as by zooming in and out. An aspect of an embodiment of the present disclosure is to provide details of an object visually recognized by a user in response to a simple operation by the user. [Means for solving the problem]

[0005] In one aspect, the disclosed embodiment is exemplified by an information processing device including a control unit. The control unit transmits information about a target facility identified in an image captured by the user device based on the state of the user in three-dimensional space or the state of the user device moving with the user in three-dimensional space to the user device or an output device located within a predetermined range from the position of the user device based on a command made by a user gesture acquired via the user device. [Effects of the Invention]

[0006] The information processing device can provide details about an object that the user has visually recognized in response to a simple operation by the user. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an information system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating components that constitute a fifth generation mobile communication system. [Figure 3] FIG. 3 is a sequence diagram illustrating a process in the information system. [Figure 4] FIG. 4 is a flowchart illustrating the processing of the server. [Figure 5] FIG. 5 is a flowchart illustrating the sensing and information provision process. DETAILED DESCRIPTION OF THE INVENTION

[0008] An information processing device, an information processing method, and a program according to an embodiment will be described below with reference to the drawings. The information processing device is exemplified by the server 6 in FIG. 1 and includes a control unit 60. The control unit 60 acquires information about a target facility identified in an image captured by the camera 27 of the UE2 based on the state of the user in three-dimensional space or the state of the UE2, which is an example of a user device that moves with the user, in three-dimensional space. The control unit 60 then controls the UE2 or the location of the UE2 based on a command by a user gesture acquired via the UE2. The image data is then sent to a display 40, which is an output device located within a predetermined range.

[0009] (System configuration and application examples) 1 is a diagram illustrating an information system 100 of this embodiment. The information system 100 includes User Equipment (hereinafter referred to as UE2), a server 6, a three-dimensional dynamic map database (hereinafter referred to as 3DDB7), a facility database (hereinafter referred to as facility DB8), an era image database (hereinafter referred to as era image DB9), and a display 40. The UE2, the server 6, the 3DDB7, the facility DB8, the era image DB9, and the display 40 are connected by a network N1.

[0010] The network N1 includes a wireless network and a wired network N2. That is, the network N1 is, for example, a Long Term Evolution (LTE) or fifth generation mobile communication The network N1 includes mobile communication systems such as the 5G system (5G) and the 6th generation mobile communication system (6G), wireless LANs (Local Area Networks), etc. The network N1 also includes public networks such as the Internet. In FIG. 1, the mobile communication systems are 5G Core (5GC) and a radio access network (hereinafter referred to as RAN3) are illustrated.

[0011] The server 6 is a normal computer. However, the server 6 may be what is called a Mobile Edge Computing or Multi-access Edge Computing (MEC) server. In this embodiment, the server 6 cooperates with the UE 2 to provide, for example, an Augmented Reality (AR), a Mixed Reality (MR), a Virtual Reality (VR), or other Extended Reality / Cross Reality (XR) environment.

[0012] The server 6 includes a central processing unit (hereinafter referred to as CPU 61), a main memory device 62, The CPU 61 has external devices and executes information processing and communication processing using a computer program. The CPU 61 is also called a processor. The CPU 61 is not limited to a single processor, and may have a multi-processor configuration. The CPU 61 may also include a graphics processing unit (GPU), a digital signal processor (DSP), etc.

[0013] The CPU 61 executes a computer program that has been deployed in an executable manner in the main memory device 62, and provides processing for the server 6. The main memory device 62 stores the computer program executed by the CPU 61, data processed by the CPU 61, etc. The CPU 61 and the main memory device 62 are collectively referred to as the control unit 60.

[0014] Examples of external devices include an external storage device 63, an output device 64, an operation device 65, and a communication device 66. The external storage device 63 is used, for example, as a storage area that supplements the main storage device 62, and stores computer programs executed by the CPU 61, data processed by the CPU 61, etc.

[0015] The output device 64 is, for example, a display device such as a liquid crystal display or an electroluminescence panel. However, the output device 64 may also include a speaker or other device that outputs sound. The operation device 65 is, for example, a touch panel with a touch sensor superimposed on a display. The communication device 66 accesses the network N1 or the like and communicates with a computer or the like connected to the network N1 or the like.

[0016] However, the server 6 is not limited to a single computer as illustrated in Fig. 1. The server 6 may be a system in a cloud environment, in which multiple computers cooperate with each other via a network N1 or the like to execute processing using virtualized resources.

[0017] 3DDB7 includes dynamic maps and displays information on features, including roads, in 3D space. The hardware configuration of the 3DDB 7 is the same as that of the server 6, and includes a CPU, memory, an external storage device, a communication device, etc. However, the 3DDB 7 may be provided in a cloud environment using virtual resources on a network.

[0018] A dynamic map is defined as high-precision three-dimensional geospatial information (basic map information) that can identify the vehicle's position on roads and their surrounding areas at the lane level, and on top of that, various additional map information necessary to support automated driving, etc. Here, additional map information is defined as traffic regulation information that includes static information such as speed limits, as well as dynamic information such as accident and construction information (Public-Private ITS Initiative Roadmap 2016, Strategic Headquarters for the Promotion of an Advanced Information and Communications Network Society).

[0019] Dynamic maps include static data and dynamic data. Static data is called high-precision 3D map data. High-precision 3D map data is a 3D map that includes detailed information on road surfaces and lanes, as well as the location information of structures. High-precision 3D map data includes road section identification information (ID), marker points, latitude and longitude, etc. as location reference data. High-precision 3D map data also includes data on actual features that are associated with the location reference data. Examples of features include road shoulders, dividing lines, stop lines, pedestrian crossings, traffic lights, railroad crossings, and buildings. High-precision 3D map data also includes data on actual features that are associated with location information from, for example, a global navigation satellite system (GNSS) or a global positioning system (GPS). This includes images taken with a camera or 3D data created by a 3D laser scanner, etc.

[0020] Therefore, the server 6 can identify the geographical position corresponding to the three-dimensional position in the image by comparing the location information of the UE 2 and the image captured by the UE 2 at the location identified by the location information with the information in the 3D DB 7. Note that the dynamic data includes, for example, information on pedestrians, accidents, traffic congestion, etc.

[0021] The facility DB8 provides information about facilities at geographical locations. The hardware configuration of the facility DB8 is the same as that of the server 6 or the 3D DB7. Each piece of data (also called a record) in the facility DB8 includes latitude, longitude, facility name, and facility information. The facility information includes, for example, summary information and detailed information. Therefore, by identifying each position (latitude and longitude) in an image captured by the UE2, the server 6 can obtain the name, summary information, and detailed information of the facility at that latitude and longitude position from the facility DB8.

[0022] The period image DB9 stores, for each facility registered in the facility DB8, images of scenery including each facility taken at a past time point in association with time point information indicating the past time point. There is no limitation on the past time point. In response to a request from UE2 specifying information indicating a location (such as latitude and longitude) and a past time point, the server 6 acquires from the period image DB9 a scenery image of a past time point including a facility located at the location, and transmits the image to UE2.

[0023] Display 40 is an example of an output device, and is one of the large display devices installed in the area where UE 2 moves. Display 40 may be a projector that projects an image onto the wall of a building or the like. The geographical location (latitude, longitude, etc.) where display 40 is installed or the geographical location of the building or the like onto which the image is projected is registered and stored in server 6 or facility DB 8.

[0024] UE2 is, for example, an in-vehicle device called In-Vehicle Infotainment (IVI), a smart The UE2 may be an information processing device equipped with a glasses-like head-mounted display (HMD) called smart glasses or AR glasses that can access the network N1. The UE2 may also be a device called a VR terminal. The UE2 may also be equipped with a head-mounted display (HMD). The display may be combined with a headset including a phone and microphone.

[0025] The UE 2 is carried by the user inside or outside the vehicle to provide information and entertainment to the user. The hardware configuration of the UE 2 is logically similar to that of the server 6, although it may differ in shape, scale, and size. Figure 1 illustrates the camera 27 of the UE 2 and the display 24 fitted into the housing.

[0026] UE2 displays information on display 24 that combines virtual information with images of a real three-dimensional space captured by camera 27. UE2 may output sound from a speaker along with the display on display 24. Camera 27 includes a rear camera that captures images in the user's line of sight (the direction behind display 24) and a front camera that captures images of the user in the facing direction (the direction from the surface of display 24 toward the user). Display 24 in FIG. 1 displays images captured by the rear camera of camera 27. If UE2 is an in-vehicle device, camera 27 includes a front camera that captures images ahead in the direction of travel of the vehicle, a right camera that captures images on the right side, a rear camera that captures images behind, and a left camera that captures images on the left side.

[0027] Furthermore, UE2 places a location pointer 241 and a time pointer 242 in the image displayed on display 24. UE2 obtains information about the facility located at the location indicated by location pointer 241 from server 6 and displays it in the image. In the example of FIG. 1, "VIP Department Store" is displayed as the facility name near location pointer 241, and information such as "2F: Restaurant Floor" is displayed. More specifically, UE2 provides the server 6 with the image captured by camera 27 along with the current location information of UE2, and requests that the server 6 identify the location where location pointer 241 is located and provide information about the facility located at the identified location.

[0028] The server 6 acquires dynamic map data or high-precision 3D map data from the 3DBD 7 based on the location information provided by the UE 2. The server 6 then compares the image provided by the UE 2 with the data from the 3DBD 7 to determine the position and direction of the image provided by the UE 2. The server 6 then determines the latitude and longitude of the position in the image where the point pointer 241 is located.

[0029] Then, the server 6 acquires information about facilities that are located at the position where the point pointer 241 is located from the facility DB 8. The server 6 transmits the facility information thus acquired to the UE 2. The UE 2 adds a virtual image formed based on the facility information transmitted from the server 6 to the video captured by the camera 27, and performs XR display.

[0030] However, together with UE2 or instead of UE2, server 6 may display information about facilities present at the position where location pointer 241 is located in the video displayed by UE2 on external display 40. Server 6 may select displays 40 present within a predetermined range from the position of UE2 and display the information about the facilities. Here, the predetermined range may be set by server 6 or may be specified by parameters received from the user via UE2.

[0031] The time pointer 242 includes a cursor 242A, a slide bar 242B, and an era display field 242C. The time pointer 242 accepts an operation to change the time period of the scenery including the facility identified by the location pointer 241. For example, when the cursor 242A is at the left end of the slide bar 242B, the UE2 displays the current image captured by the camera 27 on the display 24. When the user moves the cursor 242A to a position 242D other than the left end of the slide bar 242B, the UE2 goes back in time and identifies the corresponding year. The year is displayed in the four-digit Gregorian calendar format YYYY in the era display field 242C. Then, the UE2 displays on the display 24 an image of a landscape including the facility specified by the location pointer 241, the image being from an era close to the year specified by the time pointer 242.

[0032] More specifically, UE2 transmits the year specified by the time pointer 242 and the facility identification information to server 6, and requests transmission of a video of a past landscape. Server 6 then references the era image DB9, obtains video containing the facility specified by the identification information and from the era closest to the current year, and transmits the video to UE2. UE2 then displays the video of the era transmitted from server 6. Note that the video in the era image DB9 includes still images and videos.

[0033] In this embodiment, the UE 2 and the server 6 cooperate to identify a user's gesture from an image of the user, and recognize a command issued by the user based on the gesture. The UE 2 and the server 6 then provide the user with information corresponding to the command corresponding to the gesture.

[0034] There is no limitation on the division of the processing of the UE2 and the processing of the server 6. For example, the UE2 may recognize the user's gesture, acquire information from the 3D DB7, facility DB8, and era image DB9, and display the information on the display 24, without involving the processing of the server 6. The UE2 may also function simply as a display device equipped with a program such as a browser. In that case, the server 6 may recognize the user's gesture, acquire information from the 3D DB7, facility DB8, and era image DB9, and display the information on the display 24 of the UE2 via the browser.

[0035] The server 6 acquires information about the status of UE2 or a user moving with UE2 acquired by the 5GC including base stations 31-1 and 31-2 via the 5GC. The 5GC including base stations 31-1 and 31-2 identifies the location of UE2 when exchanging signaling messages with UE2. The base stations 31-1 and 31-2 may, for example, detect the angle of the transmission beam (or reception beam) used during signaling. The 5GC may then extend a line segment connecting the base stations 31-1 and 31-2 and a straight line corresponding to the transmission beam (or reception beam) from the positions of the base stations 31-1 and 31-2, which are both ends of this line segment, and determine the position of the intersection of the straight lines as the location of UE2. The 5GC may then apply the principle of triangulation based on the positional relationship between the base stations 31-1 and 31-2 and UE2 to measure the distance from the base stations 31-1 and 31-2 to UE2, the geographical location (latitude, longitude) of UE2, etc. Furthermore, from the change in the position of UE2 over time, 5GC identifies the moving speed and moving direction of UE2. Furthermore, from the change in the moving speed of UE2 over time, 5GC identifies the acceleration of UE2.

[0036] Furthermore, 5GC may use downlink transmission waves from base station 31-1 or base station 31-2 using a principle similar to that of radar. That is, 5GC measures the distance to UE2, the current geographical position (latitude, longitude), moving speed, moving direction, acceleration, etc. of UE2 based on the reflected waves of the transmitted waves reflected from UE2 or a user moving with UE2. Note that base stations 31-1, 31-2, etc. may use both measurements using signaling messages and measurements using reflected waves. For example, base stations 31-1, 31-2, etc. may roughly identify the location of UE2 using a signaling message and measure the location of UE2 in real time with improved accuracy based on the reflection of the transmitted waves. Server 6 acquires the above-mentioned information on the location or movement of UE2 from 5GC. Base stations 31-1, 31-2, etc. are collectively referred to as base stations 31.

[0037] Furthermore, the server 6 acquires from the UE2, through communication with the UE2, the state of the UE2 detected by the UE2 or the state of the user moving together with the UE2. The state of the UE2 is, for example, the position of the UE2, the moving speed, the moving direction, the acceleration, the direction of the visual axis of the camera 27 of the UE2, etc. The state of the user is, for example, an image of the user, the direction of the user's line of sight obtained from the image, the user's gestures, the user's commands identified from the gestures, etc. Furthermore, In addition, the server 6 acquires the state of UE2 or the state of the user acquired by the 5GC from the 5GC. For example, a network function (NF11) such as NWDAF11k or SENSING11n (see FIG. 2) of the 5GC acquires the state of the UE2 or the state of the user from the UE2. The NWDAF11k, SENSING11n, etc. of the 5GC provide the server 6 with information regarding the acquired state of the UE2 or the state of the user.

[0038] (XR display and gesture examples) Below, examples of user gestures recognized by the server 6 are given. Note that, here, the description is given assuming that the server 6 recognizes the gestures. However, as already mentioned, the UE 2 may recognize a user command from the user's gesture and notify the server 6 of the recognized user command via the NWDAF 11k, SENSING 11n, etc. of the 5GC, or directly.

[0039] (1) Based on the position information of UE2 and the direction of the visual axis of camera 27, server 6 (control unit 60) initially sets a facility near point pointer 241, which specifies a position in the space including the depth in the image captured by camera 27, as the first facility. At this time, UE2 may confirm that the direction of the user's line of sight matches the direction of the visual axis of camera 27 within a predetermined allowable range. Therefore, UE2 and server 6 can confirm that the image matches the user's field of view. Server 6 transmits information about the initially set first facility to UE2. UE2 generates virtual image elements such as graphic objects based on the information transmitted from server 6 and displays them in XR together with the image captured by camera 27.

[0040] (2) If the gesture is a forward or backward movement of the hand, the server 6 moves the point pointer 241 on the display 24 of the UE2 in conjunction with the forward or backward movement of the hand in the near or far direction within the user's field of view. Note that the gesture may be a hand sign that remains stationary for a predetermined period of time to indicate the user's forward or backward movement. The server 6 may move the point pointer 241 in the direction of the sign during that predetermined period of time. The server 6 then compares the 3D DB 7 with the video captured by the camera 27 of the UE2 to identify the latitude and longitude in real space that correspond to the position of the point pointer 241 in the three-dimensional space within the video. The server 6 then identifies other facilities that exist near the moved point pointer 241 from the facility DB 8 and sets the identified other facilities as second facilities. The server 6 then transmits information about the set second facility to the UE2. The UE2 displays, in XR, information about the other facility set as the second facility.

[0041] (3) If the gesture is a gesture of pushing the hand forward of the user, the server 6 moves the point pointer 241 to the far side of the user's field of view. Here, the server 6 may perform the process by, for example, assuming that the direction of the visual axis of the camera 27 is the far side of the user's field of view. Then, the server 6 transmits information about a second facility that is located farther away than the first facility to the UE 2 using the same procedure as in (2) above.

[0042] (4) If the gesture is a gesture of pulling back the user's outstretched hand, the server 6 moves the point pointer 241 to a position closer to the user's field of view. Here, the server 6 may perform processing, for example, assuming that the direction opposite to the direction of the visual axis of the camera 27 (the facing direction toward the user) is the direction closer to the user's field of view. Then, the server 6 transmits information about a second facility that is closer than the first facility to the UE 2 using the same procedure as in (2) above.

[0043] (5) The server 6 sets the initial value of the time pointer 242, which is movable on the time axis and designates the present or a time going back in time from the present, to the present. If the gesture is an up-and-down movement of the hand, the server 6 then sets the initial value of the time pointer 242 to the present. The time pointer 242 is moved in conjunction with the past time. The gesture may be a hand sign that remains stationary for a predetermined time indicating above or below the user. The server 6 may move the time pointer 242 in the direction of the sign at that predetermined time. The server 6 then transmits to the UE 2 a video of a scene including the first facility or the second facility at the present time or a time preceding the present time indicated by the time pointer 242.

[0044] (6) If the gesture is, for example, a downward movement of the hand, the server 6 moves the time pointer 242 a predetermined time into the past and transmits to the UE 2 a video of a scene including the first facility or the second facility at a time going back the predetermined time in the past. If the gesture is, for example, an upward movement of the hand, the server 6 moves the time pointer 242 toward the present by a time corresponding to the gesture. The server 6 then transmits to the UE 2 a video of a scene including the first facility or the second facility at a time later than the time before the gesture by a time corresponding to the gesture.

[0045] (7) When the gesture is a gesture of fisting hands, the server 6 transmits detailed information with an increased amount of information about the first facility or the second facility to the UE 2 and causes it to display. When the gesture is a gesture of spreading hands from a fisted state, the server 6 transmits summary information with a reduced amount of information about the first facility or the second facility to the UE 2 and causes it to display.

[0046] (8) If the gesture is a hand-waving gesture, the server 6 transmits to the UE 2 information about a third facility that is obscured by the first facility or the second facility in the user's line of sight.

[0047] The above processes (1) to (8) are examples of processes based on the user's state in three-dimensional space (position, posture, line of sight, movement, etc.) or the state in three-dimensional space of UE2 moving with the user (position, movement, orientation, posture, line of sight, field of view, visual axis, angle of view, pointer position, etc.). These processes are also examples of processes based on commands made by user gestures acquired via UE2. Then, through these processes, server 6 transmits information about the target facility identified in the video captured by UE2 to UE2 or to display 40, which is an output device located within a predetermined range from the position of UE2.

[0048] (Network example) FIG. 2 illustrates components (elementary elements) constituting a fifth-generation mobile communication system (also referred to as a 5G network or 5GNW) in the network N1. In this embodiment, the elements of the 5GC are collectively referred to as a Network Function (hereinafter referred to as NF11), and individually referred to as NEF11e, etc. In FIG. 1, each element is given a generic symbol and an individual symbol in parentheses. Of the elements in FIG. 2, the configurations other than SENSING11n are specified in, for example, 3GPP (registered trademark) TS23.501, and therefore their description will be omitted. Note that DN5 is a data network (such as the Internet) external to 5GC. For example, a server 6 is connected to DN5. The server 6 may be AF12 of 5GC. Furthermore, RAN (Radio Access Network) 3 is an access network to a 5G core network (5GC). RAN3 is composed of base stations 31 (gNB).

[0049] SENSING11n performs sensing processing, including collecting sensing information from UE2 or other external systems and providing the collected sensing information to UE2, AF12, or other external systems (DN5, etc.). However, NWDAF11k may perform the sensing processing instead of SENSING11n. In the following embodiment, SENSING11n will be described as performing the sensing processing.

[0050] (Processing Procedure) 3 is a sequence diagram illustrating processing in the information system 100. In this processing, first, UE2 requests the server 6 to provide information (S1). Upon receiving the request from UE2, the server 6 requests SENSING11n, which is one of the NFs 11 of 5GC, to start sensing of UE2 (S2).

[0051] Upon receiving the request to start sensing, SENSING11n requests UE2 to sense the user's gestures and state via the base station 31 (S3). At this time, SENSING11n also requests the base station 31 to sense UE2 (S4).

[0052] Upon receiving a request to sense UE2, base station 31 senses the state of UE2. Note that here, base station 31 includes both base stations 31-1 and 31-2. For example, at the time of signaling, base station 31 measures the state of UE2, such as the distance to UE2, the current geographical position (latitude, longitude) of UE2, the moving speed, moving direction, acceleration, etc., using the method described in FIG. 1. In addition, base station 31 measures the state of UE2, for example, from reflected waves of downlink transmitted waves. Then, base station 31 reports the sensing result to SENSING11n (S5).

[0053] On the other hand, when UE2 receives a sensing request, it photographs the user with camera 27 and transmits the photographed image to SENSING11 via base station 31 (S6). However, UE2 may also analyze the user's line of sight and gestures from the photographed image, recognize the user's commands, and transmit the recognized line of sight direction and commands to SENSING11. At this time, UE2 may also transmit the current geographical position (latitude, longitude), speed, moving direction, acceleration, the direction of the visual axis of camera 27, etc. to SENSING11.

[0054] Upon receiving the sensing results from the base station 31 and UE2, SENSING11n transmits the sensing results to the server 6 (S7). Note that, if UE2 transmits an image of the user in S6, SENSING11n analyzes the user's line of sight or gesture from the image received from UE2 and recognizes the line of sight direction or the user's command.

[0055] When the server 6 receives the sensing results from the SENSING 11n, i.e., the user's command by gesture, the user's state, the state of the UE 2, etc., the server 6 acquires information to be provided to the UE 2 corresponding to the acquired user's command, the user's state, the state of the UE 2, etc. (S8). That is, the server 6 executes, for example, the above processes (1) to (8). Then, the server 6 transmits the acquired information to the UE 2 (S9).

[0056] The UE2 performs XR display based on the information transmitted from the server 6. That is, the UE2 moves the point pointer 241 in the image captured by the camera 27 in response to the user's gesture. Then, the UE2 displays information about the facility identified by the point pointer 241 (S10). The UE2 also changes the amount of information displayed about the facility in response to the user's gesture. For example, the UE2 displays detailed information or general information about the facility in response to the user's gesture. Note that in the processing of S9, the server 6 may transmit the acquired information to the display 40 together with the UE2 or instead of the UE2, and display it. In this case, the server 6 may perform XR display on the display 40 based on the image captured by the UE2 and the acquired information.

[0057] Furthermore, the UE 2 changes the time indicated by the time pointer 242 in response to the user's gesture. Then, UE2 displays an image of a landscape including the facility at the current or past time corresponding to the time indicated by time pointer 242. Furthermore, UE2 displays information about other facilities hidden behind the facility specified by point pointer 241 in response to the user's gesture.

[0058] 4 is a flowchart illustrating the processing of the server 6. In this processing, the server 6 acquires the sensing result in 5GC (S31). Next, the server 6 acquires information about the state of the UE 2 directly from the UE 2 (S32). Note that the server 6 may omit either the processing of S31 or the processing of S32.

[0059] Next, the server 6 estimates the user's field of view based on the information obtained in the process of S31 or S32, adds virtual information to the image of the real three-dimensional space on the display 24 of the UE2, and causes the UE2 to perform an XR display (S33). At this time, the XR display also displays a location pointer 241 and a time pointer 242. For example, the server 6 may initially set a facility near the location pointer 241 in the three-dimensional space including depth in the image captured by the camera 27 based on the direction of the visual axis of the camera 27 connected to the UE2 and the position information of the UE2 as a first facility. The facility near the location pointer is subject to processing by the server 6, and therefore can be referred to as a target facility. The process of S33 is also an example of transmitting information about the initially set first target facility to the UE2. Note that in the process of S33, the server 6 may set the initial value of the time pointer 242, which is movable on a time axis and designates the present or a past time going back in time from the present, to the present.

[0060] Then, the server 6 performs sensing and information provision (S34). In the processing of S34, the server 6 acquires the state of the UE 2, the state of the user moving with the UE 2, and the user's gestures via the 5GC or directly from the UE 2. Then, in response to the user's gestures, the server 6 causes the UE 2 to display XR on the image captured by the camera 27 of the UE 2.

[0061] 1, the server 6 may display information about the facility identified by the location pointer 241 in the image displayed by the UE 2 on the external display 40, together with the UE 2 or in place of the UE 2. The processing of S34 is an example of transmitting information about the target facility to the UE 2 or to an output device located within a predetermined range from the position of the UE 2, based on a command by a user gesture acquired via the UE 2.

[0062] 5 is a flowchart illustrating the sensing and information provision process (details of S34 in FIG. 4). In this process, the server 6 acquires the sensing results from SENSING11n of 5GC or UE2 (S341). Note that the server 6 may acquire the sensing results from SENSING11n of 5GC, or may acquire them from UE2 without going through SENSING11n. The server 6 may also acquire the sensing results from both SENSING11n of 5GC and UE2.

[0063] Next, the server 6 determines the user's gesture (S342). Alternatively, the UE 2 may analyze the user's gesture and recognize the user's command, and the server 6 may receive the recognized command via the SENSING 11n or directly from the UE 2. The server 6 then determines the command based on the gesture and executes processing in accordance with the command (S343 to S356).

[0064] That is, if the command is to move the point pointer 241 farther away (YES in S343), the server 6 moves the point pointer 241 in the image from its current position farther away in the direction of the visual axis of the camera 27. Then, the server 6 displays information about the facility identified by the point pointer 241 in the XR. If the command is to move the point pointer 241 closer (YES in S345), the server 6 moves the point pointer 241 in the image from its current position closer (toward the user) in the direction of the visual axis of the camera 27. Then, the server 6 displays information about the facility identified by the point pointer 241 on the UE2 in XR (S346).

[0065] A YES determination in S343 and S345 is an example of a case where the gesture is a hand movement in three-dimensional space corresponding to the depth direction in the image. The hand movement includes a hand movement and a restrained finger sign. If a YES determination is made in S343 and S345, the server 6 causes the hand movement corresponding to the depth direction to correspond to a movement in the distance direction in the image. That is, the server 6 moves the point pointer 241 to the far or near side in the image, sets another facility located near the moved point pointer 241 as a second target facility, and displays the information on UE2 or a display 40 located within a predetermined range from UE2.

[0066] Therefore, the processes of S344 and S346 are also an example of transmitting information about the set second facility to UE2 or an output device located within a predetermined range from the position of UE2.

[0067] If the command is to move the time pointer 242 to the past (YES in S347), the server 6 moves the time pointer 242 back in time and causes the UE 2 to display a scene including facilities from that time (S348).If the command is to move the time pointer 242 closer, i.e., toward the present (YES in S349), the server 6 moves the time pointer 242 toward the present and causes the UE 2 to display a scene including facilities from that time (S350).

[0068] YES in S347 and S349 is an example of the server 6 moving the time pointer 242 to the current or past time in response to the gesture. The processing of S348 and S350 is an example of the server 6 transmitting an image of a scene including the first facility or the second facility at the current or past time indicated by the time pointer 242 to UE2 or to a display 40 located within a predetermined range from the position of UE2.

[0069] Furthermore, if the command is to increase the amount of display information, i.e., to display detailed information (YES in S351), the server 6 increases the amount of information to be displayed (S352). If the command is to reduce the amount of display information, i.e., to display outline information (YES in S353), the server 6 reduces the amount of information to be displayed (S354). Furthermore, if the command is to display an obstructed object, i.e., to display another facility hidden behind the facility currently being displayed in XR (YES in S355), the server 6 displays the facility behind the facility currently being displayed in XR in XR (S356). Then, the server 6 determines whether or not to end the process (S357). For example, if UE2 accepts a command to end the process from the user, the server 6 ends the process. If the server 6 does not want to end the process, it repeats the process from S341.

[0070] (Effects of the embodiment) The server 6 acquires the user's state in three-dimensional space or the state in three-dimensional space of the UE2 moving with the user. Then, the server 6 transmits information about a target facility identified in the video captured by the camera 27 of the UE2 based on these acquired states to the UE2 or to a display 40 located within a predetermined range from the position of the UE2. In this case, the server 6 transmits the information about the target facility based on a command, which is information recognized from a user gesture acquired via the UE2. Therefore, the server 6 can provide the UE2 with details about an object viewed by the user in response to a simple operation by the user.

[0071] The server 6 also calculates the location points by corresponding the user's hand movements in the perspective direction in the image. The user moves the point pointer 241. The server 6 then sets another facility that exists near the moved point pointer 241 as a second target facility. The server 6 then transmits information about the set second facility to the UE2 or to the display 40, which is an output device that exists within a predetermined range from the position of the UE2. Therefore, the server 6 can change the target facility to be displayed in XR in response to a simple operation by the user.

[0072] Furthermore, the server 6 transmits an image of a landscape including the first facility or the second facility at the current or past time indicated by the time pointer 242 to the UE2 or to the display 40 located within a predetermined range from the position of the UE2. Thus, the server 6 can display, in response to a simple operation by the user, changes over time in the landscape including the target facility on the UE2 or the display 40.

[0073] <Other embodiments> The above-described embodiment is merely an example, and the present disclosure may be modified and implemented as appropriate without departing from the spirit thereof. Furthermore, the processes and means described in the present disclosure may be freely combined and implemented as long as no technical contradictions arise. Furthermore, a process described as being performed by one device may be shared and executed by multiple devices. Alternatively, a process described as being performed by different devices may be executed by a single device. In a computer system, the hardware configuration (server configuration) by which each function is realized can be flexibly changed.

[0074] The present disclosure can also be realized by supplying a computer program implementing the functions described in the above embodiments to a computer, and having one or more processors in the computer read and execute the program. Such a computer program may be provided to the computer via a non-transitory computer-readable storage medium connectable to the computer's system bus, or via a network N1. Examples of non-transitory computer-readable storage media include any type of disk, such as a magnetic disk, a hard disk drive (HDD), an optical disk (CD-ROM, DVD, Blu-ray disk, etc.), a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, or an optical card. [Explanation of symbols]

[0075] 2 UE, 6 Server, 7 3D DB, 8 Facility DB, 9 Historical Image DB, 24 Display, 27 Camera, 31, 31-1, 31-2 Base Station, 40 Display

Claims

1. An information processing device comprising a control unit that transmits information about a target facility identified in an image captured by a user device based on the state of the user in three-dimensional space or the state of a user device moving with the user in the three-dimensional space to the user device or an output device located within a predetermined range from the position of the user device based on a command made by a gesture of the user obtained via the user device.

2. The control unit based on the direction of the visual axis of a camera connected to the user device or the direction of the line of sight of the user and position information of the user device, initially set a facility near a point pointer that identifies a position in the three-dimensional space including depth in the video as a first target facility, and transmit information about the initially set first target facility to the user device; When the gesture is a hand movement in the three-dimensional space corresponding to a depth direction in the image, the point pointer is moved in such a way that the hand movement corresponding to the depth direction corresponds to a movement in a perspective direction in the image, and another facility present near the moved point pointer is set as a second target facility; The information processing device according to claim 1 , wherein information about the set second target facility is transmitted to the user device or to the output device located within a predetermined range from the position of the user device.

3. The control unit A time pointer that can be moved on the time axis and that specifies the present or a past time going back in time from the present is set to the present as its initial value, 3. The information processing device according to claim 2, wherein the time pointer is moved to the current or past time in accordance with the gesture, and an image of a landscape including the first target facility or the second target facility at the current or past time indicated by the time pointer is transmitted to the user device or to the output device located within a predetermined range from the position of the user device.

4. The computer An information processing method in which information regarding a target facility identified in an image captured by a user device based on the user's state in three-dimensional space or the state in three-dimensional space of a user device moving with the user is transmitted to the user device or an output device located within a predetermined range from the position of the user device based on a command made by the user's gesture obtained via the user device.

5. On the computer, A program for transmitting information about a target facility identified in an image captured by a user device based on the user's state in three-dimensional space or the state in three-dimensional space of a user device moving with the user, to the user device or an output device located within a predetermined range from the position of the user device, based on a command made by the user's gesture obtained via the user device.

Images