Processing system, processing method, processing program

Abstract

To provide a processing system which balances safety of a moving user and continuity of virtual experience.SOLUTION: A processor of a processing system executes: monitoring a visibility area Av viewed by a user via a wearable terminal; and adjusting an occupancy Rc of a display target range Ad of a content image occupying the visibility area Av, the content image being superimposition-displayed on the visibility area as an XR image corresponding to a desired content of the user, according to a risk level Lr in the visibility area Av. The adjustment of the occupancy Rc includes: acquiring outer sensing information Io from an autonomously running device which leads the user; recognizing the risk level Lr based on the outer sensing information Io; and prohibiting a superimposition-display of the content image on a hazard space part Aoh having the risk level reaching a hazard level of an outer sensing space Ao which is an acquisition range for acquiring outer sensing information Ii in the visibility area Av, thereby adjusting the occupancy Rc.SELECTED DRAWING: Figure 6

Description

Technical Field

[0002]

[0001] This disclosure relates to a processing technology for performing XR display-related processing related to the display of XR images to a user.

Background Art

[0002] The disclosed technology of Patent Document [1] enables a user to have a virtual experience by displaying a VR video as an XR image on an HMD (Head Mounted Display). In particular, in this technology, when the movement of the user exceeds the limit position and the risk of collision in the physical environment is detected by the HMD, the display of the HMD is made transmissive.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the disclosed technology of Patent Document [1] above, since the limit position of the collision risk is set based on the user's position in order to trigger the movement of the user wearing the HMD, the range in which safety can be ensured by display transparency remains within the proximity range of the user. As a result, in the disclosed technology of Patent Document [1] where all VR videos are disappeared by display transparency, the continuity of the virtual experience is particularly inhibited in the application to a moving user.

[0005] The objective of this disclosure is to provide a processing system that performs XR display-related processing while balancing the safety of a moving user with the continuity of the virtual experience. Another objective of this disclosure is to provide a processing method that performs XR display-related processing while balancing the safety of a moving user with the continuity of the virtual experience. Yet another objective of this disclosure is to provide a processing program that performs XR display-related processing while balancing the safety of a moving user with the continuity of the virtual experience. [Means for solving the problem]

[0006] The following describes the technical means of solving the problem described in this disclosure. Note that the claims and the reference numerals in parentheses in this section indicate the correspondence with the specific means described in the embodiments detailed later, and do not limit the technical scope of this disclosure.

[0007] The first aspect of this disclosure is, A processing system having a processor (9b) that performs XR display-related processing related to displaying XR images to a moving user, The processor is, This involves monitoring the field of view (Av) as seen by the user through a wearable device (Wt) worn by the user, and The system is configured to adjust the percentage (Rc) of the display area (Ad) of a content image (20) that is superimposed on the field of view as an XR image corresponding to the user's desired content (Cd), according to the risk level (Lr) in the field of view. Adjusting the occupancy rate is To acquire external sensing information (Io) sensed by an autonomous driving device (Ma) that guides the user moving towards the field of view, Recognizing the risk level based on external sensing information, This includes adjusting the occupancy rate by prohibiting the superimposition of content images in the hazard space portion (Aoh) of the external sensing space (Ao), which is the range within the field of view where external sensing information can be acquired, and where the risk level has reached a hazard level requiring hazard notification to moving users.

[0008] A second aspect of this disclosure is, A processing method performed by a processor (9b) to carry out XR display-related processing related to displaying XR images to a moving user, This involves monitoring the field of view (Av) as seen by the user through a wearable device (Wt) worn by the user, and This includes adjusting the percentage (Rc) of the display area (Ad) of a content image (20) that is superimposed on the field of view as an XR image corresponding to the user's desired content (Cd) within the field of view, according to the risk level (Lr) in the field of view. Adjusting the occupancy rate is To acquire external sensing information (Io) sensed by an autonomous driving device (Ma) that guides the user moving towards the field of view, Recognizing the risk level based on external sensing information, This includes adjusting the occupancy rate by prohibiting the superimposition of content images in the hazard space portion (Aoh) of the external sensing space (Ao), which is the range within the field of view where external sensing information can be acquired, and where the risk level has reached a hazard level requiring hazard notification to moving users.

[0009] A third aspect of this disclosure is: A processing program that includes instructions to be executed by a processor (9b) and is stored in a storage medium (9a) to perform XR display-related processing related to displaying XR images to a moving user, This involves monitoring the field of view (Av) as seen by the user through a wearable device (Wt) worn by the user, and The command includes instructions to adjust the percentage (Rc) of the display area (Ad) of a content image (20) that is superimposed on the field of view as an XR image corresponding to the user's desired content (Cd), according to the risk level (Lr) in the field of view. Adjusting the occupancy rate is To acquire external sensing information (Io) sensed by an autonomous driving device (Ma) that guides the user moving towards the field of view, Recognizing the risk level based on external sensing information, This includes adjusting the occupancy rate by prohibiting the superimposition of content images in the hazard space portion (Aoh) of the external sensing space (Ao), which is the range within the field of view where external sensing information can be acquired, and where the risk level has reached a hazard level requiring hazard notification to moving users.

[0010] In the first to third embodiments described above, the field of view visible to the user through the wearable device worn by the user is monitored. The percentage of the field of view occupied by the content image, which is superimposed on the field of view as an XR image corresponding to the user's desired content, is adjusted according to the risk level in the field of view.

[0011] In particular, according to the first to third embodiments, external sensing information sensed by an autonomous driving device leading the user as they move towards the field of view is acquired, and the risk level based on the external sensing information is recognized. As a result, by effectively utilizing the autonomous driving device that performs the user's leading function, the risk level can be recognized for as wide an area as possible within the field of view where the user is moving, thereby ensuring the user's safety.

[0012] Furthermore, according to the first to third embodiments, in the external sensing space that is within the range of external sensing information acquisition in the field of view, the superimposition of content images is prohibited in the hazard space portion where the risk level has reached a hazard level. As a result, the occupancy rate can be adjusted so that the content image disappears only within the hazard space portion of the hazard level that requires hazard notification to moving users, while the superimposition of content images continues outside the hazard space portion in the field of view. Therefore, it is possible to balance the safety ensured by the above principle for moving users with the continuity of the virtual experience.

[0013] The fourth aspect of this disclosure is: A processing system having a processor (9b) that performs XR display-related processing related to displaying XR images to a moving user, The processor is, This involves monitoring the background area (Ab) that is displayed to the user from their mobile device (Mt), and The system is configured to perform the following actions: adjust the percentage (Rc) of the display area (Ad) of a content image (20) superimposed on a background video (24) showing the background area as an XR image corresponding to the user's desired content (Cd), according to the risk level (Lr) in the background area. Adjusting the occupancy rate is To acquire external sensing information (Io) sensed by an autonomous driving device (Ma) that guides the user moving towards the background area, Recognizing the risk level based on external sensing information, This includes adjusting the occupancy rate by prohibiting the overlaying of content images on the video portion (24h) of the background video showing the external sensing space (Ao), which is the range within which external sensing information is acquired in the background area, specifically the portion of the video showing the hazard space (Aoh) where the risk level has reached a hazard level requiring hazard notification to moving users.

[0014] The fifth aspect of the present disclosure is A processing method executed by a processor (9b) for performing XR display-related processing related to the display of XR images to a moving user, Monitoring a background area (Ab) that serves as a display background to the user from a mobile terminal (Mt) carried by the user, Adjusting the occupancy rate (Rc) that the display target range (Ad) of the content image (20) superimposed on the background video (24) showing the background area occupies in the background video according to the desired content (Cd) desired by the user according to the risk level (Lr) in the background area, Adjusting the occupancy rate includes Obtaining external sensing information (Io) sensed by an autonomous driving device (Ma) that guides a user moving towards the background area, Recognizing a risk level based on the external sensing information, Adjusting the occupancy rate by prohibiting the superimposed display of the content image on the video portion (24h) showing the hazard space portion (Aoh) where the risk level has reached the hazard level that requires a hazard notification to the moving user among the background videos showing the external sensing space (Ao) that is the acquisition range of the external sensing information in the background area.

[0015] The sixth aspect of the present disclosure is A processing program stored in a storage medium (9a) and including instructions for causing a processor (9b) to execute XR display-related processing related to the display of XR images to a moving user, Monitoring a background area (Ab) that serves as a display background to the user from a mobile terminal (Mt) carried by the user, Instructions for causing to execute adjusting the occupancy rate (Rc) that the display target range (Ad) of the content image (20) superimposed on the background video (24) showing the background area occupies in the background video according to the desired content (Cd) desired by the user according to the risk level (Lr) in the background area, Adjusting the occupancy rate includes To acquire external sensing information (Io) sensed by an autonomous driving device (Ma) that guides the user moving towards the background area, Recognizing the risk level based on external sensing information, This includes adjusting the occupancy rate by prohibiting the overlaying of content images on the video portion (24h) of the background video showing the external sensing space (Ao), which is the range within which external sensing information is acquired in the background area, specifically the portion of the video showing the hazard space (Aoh) where the risk level has reached a hazard level requiring hazard notification to moving users.

[0016] In the fourth to sixth embodiments, the background area displayed to the user from the mobile device they carry is monitored. The percentage of the background video in which the content image, which is superimposed on the background video as an XR image corresponding to the user's desired content, occupies is adjusted according to the risk level in the background area.

[0017] In particular, according to the fourth to sixth embodiments, external sensing information sensed by an autonomous driving device leading the user as they move toward the background area is acquired, and the risk level based on the external sensing information is recognized. As a result, by effectively utilizing the autonomous driving device that performs the user's leading function, the risk level can be recognized for as wide an area as possible in the background area where the user is moving, thereby ensuring the user's safety.

[0018] Furthermore, according to the fourth to sixth embodiments, in the background video showing the external sensing space that is within the range of external sensing information acquisition in the background area, the superimposition of content images is prohibited for the video portion showing the hazard space where the risk level has reached the hazard level. As a result, in the background video, the content image disappears only in the portion corresponding to the hazard space at the hazard level requiring hazard notification to moving users, while the superimposition of content images continues in the portion corresponding to the outside of the hazard space in the background area, thus the occupancy rate can be adjusted. Therefore, it is possible to balance the safety ensured for moving users by the principle described above with the continuity of the virtual experience. [Brief explanation of the drawing]

[0019] [Figure 1] This is an overall configuration diagram showing the network connection environment of the processing system according to the first embodiment. [Figure 2] This is a schematic diagram showing a wearable device according to the first embodiment. [Figure 3] This is a schematic diagram illustrating the three-dimensional voxel assumed in the infrastructure system according to the first embodiment. [Figure 4] This is a block diagram showing the processing system according to the first embodiment. [Figure 5] This is a flowchart showing the processing flow according to the first embodiment. [Figure 6] This is a flowchart showing the adjustment subroutine in the processing flow according to the first embodiment. [Figure 7] This is a schematic diagram showing the display state of a wearable device according to the first embodiment. [Figure 8] This is a schematic diagram showing the display state of a wearable device according to the first embodiment. [Figure 9] This is a schematic diagram showing the display state of a wearable device according to the first embodiment. [Figure 10] This is a schematic diagram illustrating the adjustment subroutine according to the first embodiment. [Figure 11]This is a schematic diagram illustrating the adjustment subroutine according to the first embodiment. [Figure 12] This is an overall configuration diagram showing the network connection environment of the processing system according to the second embodiment. [Figure 13] This is a schematic diagram showing a mobile terminal according to the second embodiment. [Figure 14] Block diagram shows the processing system according to the second embodiment. [Figure 15] This is a flowchart showing the processing flow according to the second embodiment. [Figure 16] This is a flowchart showing the adjustment subroutine in the processing flow according to the second embodiment. [Figure 17] This is a schematic diagram showing the display state of a mobile terminal according to the second embodiment. [Figure 18] This is a schematic diagram showing the display state of a mobile terminal according to the second embodiment. [Figure 19] This is a schematic diagram showing the display state of a mobile terminal according to the second embodiment. [Figure 20] This is a schematic diagram illustrating the adjustment subroutine according to the second embodiment. [Figure 21] This is a schematic diagram illustrating the adjustment subroutine according to the second embodiment. [Modes for carrying out the invention]

[0020] Hereinafter, several embodiments of this disclosure will be described with reference to the drawings. In each embodiment, the same reference numerals will be used for corresponding components, and redundant explanations may be omitted. Furthermore, if only a part of the configuration is described in each embodiment, the configuration of other embodiments described earlier may be applied to the other parts of that configuration. Moreover, not only the combinations of configurations explicitly stated in the description of each embodiment, but also the configurations of multiple embodiments can be partially combined even if not explicitly stated, as long as there are no particular problems with the combination.

[0021] (First Embodiment) The processing system 1 of the first embodiment shown in Figure 1 performs XR display-related processing related to the display of XR images to a mobile user Us in conjunction with the provision of user services to the user Us. To this end, the processing system 1 is connected via a communication network Nc to a wearable terminal Wt worn by a pre-registered user Us, an autonomous driving device Ma, and an infrastructure system 6. Here, at least one system each of the wearable terminal Wt, autonomous driving device Ma, and infrastructure system 6 is assumed to be connected to the processing system 1.

[0022] The wearable terminal Wt is configured to be wearable on the user Us's face and operable hands-free, at least when connected to the processing system 1. As shown in Figure 2, the wearable terminal Wt is an optical see-through electronic device equipped with a display unit 2 and a sensor unit 3, such as an HMD or smart glasses. Figure 2 shows a schematic image of how the field of view area Av appears from the user Us's perspective, corresponding to one of the user Us's two eyes, with respect to the wearable terminal Wt.

[0023] As shown in Figure 1, the wearable terminal Wt controls the transmission and reception of data via the communication network Nc by a communication unit. The wearable terminal Wt is operated by a control unit according to control commands from the processing system 1 via the communication network Nc, so as to perform XR display-related processing in cooperation with the processing system 1.

[0024] Under the operational control of the wearable terminal Wt's control unit, the display unit 2 shown in Figure 2, such as a virtual image projection type or retinal projection type, superimposes the display of XR images necessary for XR display-related processing (see Figures 7-9 below) onto the real image of the field of view area Av, which is viewed by the user Us through glasses or lenses, and makes it visible to the user Us. At the same time, under the operational control of the wearable terminal Wt, the sensor unit 3 shown in Figure 2 acquires sensing information through sensing processing corresponding to the XR display-related processing. The sensor unit 3 employs at least one of the following: a camera, inertial sensor, GNSS sensor, electrooculography sensor, gaze sensor, infrared sensor, geomagnetic sensor, motion sensor, touch sensor, and microphone. With this configuration, the sensor unit 3 can acquire input representing the user Us's intentions as sensing information.

[0025] The autonomous driving device Ma shown in Figure 1 is an autonomous vehicle or autonomous robot capable of autonomous driving in any direction (forward, backward, left, or right) by electrically driving the wheels 5 based on sensing information from the sensor unit 4. The autonomous driving device Ma controls the transmission and reception of data via the communication network Nc by the communication unit. The autonomous driving device Ma is driven and controlled by the control unit according to control commands from the processing system 1 via the communication network Nc, so as to perform XR display-related processing in cooperation with the processing system 1. Under this drive control, the sensor unit 4 acquires sensing information through sensing processing corresponding to the XR display-related processing. As the sensor unit 4, at least one of the following types is used: camera, inertial sensor, GNSS sensor, LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging), and sonar.

[0026] Infrastructure system 6 is a foundational system that shares three-dimensional spatial information It stored in the infrastructure database Di, serving as a common base for the distributed architecture. Infrastructure system 6 controls the transmission and reception of data via the communication network Nc using a communication unit. Infrastructure system 6 collects the three-dimensional spatial information It, which is provided to individual distributed systems such as processing system 1 that constitute the distributed architecture, from the control unit as needed, and updates the stored information in the infrastructure database Di with the latest information.

[0027] In particular, the infrastructure system 6 manages information by virtually dividing the three-dimensional space to be stored into multiple three-dimensional voxels Vi (i.e., three-dimensional grids) assumed to be in a three-dimensional array, as shown in Figure 3. The infrastructure system 6 stores a dataset linked by metadata for each spatial ID individually assigned to each voxel Vi as three-dimensional spatial information It in the infrastructure database Di, as shown in Figure 1. Here, the three-dimensional spatial information It may, of course, also include two-dimensional grid information linked only to, for example, the bottom surface of the voxel Vi that constitutes the lowest layer of the two-dimensional array along the ground in the three-dimensional array.

[0028] The data used to construct the three-dimensional spatial information It may be collected from at least one of the following: a wearable terminal Wt that works in conjunction with the processing system 1, other wearable terminals, and mobile terminals such as smartphones or tablet terminals. The data used to construct the three-dimensional spatial information It may also be collected from at least one of the following: an autonomous driving device Ma that works in conjunction with the processing system 1, and other mobile devices.

[0029] The data used to construct the three-dimensional spatial information It may be collected from at least one of the following, such as a communication base station, smart pole, and smart street light, which are equipped with infrastructure sensors such as cameras and / or LiDAR. The data used to construct the three-dimensional spatial information It may also be collected from a servicer that provides at least one of the following, such as a map service, weather service, communication service, traffic management service, feature management service, and aeronautical management service, as well as the user services handled by the processing system 1, as described below.

[0030] The construction data collected in this manner to construct the three-dimensional spatial information It may be at least one type of image data, such as video, still images, and point cloud images. The construction data for the three-dimensional spatial information It may also be at least one type of secondary data, provided that information security is ensured, generated by image processing of such image data regarding humans, including the user Us of the processing system 1, within the target space for information storage, such as position data, motion data, posture data, gaze data, behavior data, and pedestrian flow data, as well as intent data representing the intentions of the user Us, as described later.

[0031] The data used to construct the three-dimensional spatial information It may be at least one type of secondary data generated by image processing of image data relating to human belongings, including the user Us's wearable terminal Wt, within the information storage target space, such as position data, motion data, and posture data. The data used to construct the three-dimensional spatial information It may also be at least one type of secondary data generated by image processing of image data relating to a moving object, including an autonomous driving device Ma, within the information storage target space, such as position data, motion data, and posture data.

[0032] The data used to construct the three-dimensional spatial information It may be voice data collected from human voices, including that of user Us of processing system 1, within the information storage target space. The data used to construct the three-dimensional spatial information It may also be at least one type of secondary data, provided that information security is ensured, which is analyzed with respect to humans, including user Us, through voice recognition processing of such voice data, such as location data, movement data, behavioral data, pedestrian flow data, and conversation data, as well as intent data representing user Us's intentions, as described later.

[0033] The construction data for the three-dimensional spatial information It may be at least one of the following: two-dimensional and / or three-dimensional map data, GIS (Geographic Information System) data, road network data, weather data, communication data, line data, traffic data, feature management data, BIM (Building Information Modelling) data, POI (Point of Interest) data, aviation management data, and time data. The construction data for the three-dimensional spatial information It may also be service data related to at least one of the following user services handled by the processing system 1, such as guidance services, transport services, photography services, and online game services.

[0034] As shown in Figure 1, the processing system 1 is a distributed computer system equipped with a communication system 10 and a control system 9, and includes at least one of the following: a cloud server, an edge server, etc. At least a portion of each system 10 and 9 in the processing system 1 may be composed of the communication unit and control unit of a wearable terminal Wt, respectively. At least a portion of each system 10 and 9 in the processing system 1 may be composed of the communication unit and control unit of an autonomous driving device Ma, respectively. At least a portion of each system 10 and 9 in the processing system 1 may be composed of the communication unit and control unit of an infrastructure system 6, respectively.

[0035] The communication system 10 is mainly composed of communication equipment for constructing the communication network Nc. The control system 9 is connected to the communication system 10 via at least one of the following: a wired communication line and a wireless communication line. The control system 9 is constructed including at least one dedicated computer. The dedicated computer that makes up the control system 9 has at least one memory 9a and at least one processor 9b.

[0036] In this processing system 1, the control system 9 executes multiple instructions of the processing program stored in memory 9a using the processor 9b. As a result, the control system 9 constructs multiple functional blocks for performing XR display-related processing. The constructed functional blocks include a recognition block 200, a drive control block 210, and a display control block 220, as shown in Figure 4.

[0037] The processing method by which the processing system 1 performs XR display-related processing through the combined action of blocks 200, 210, and 220 is executed according to the processing flow shown in Figures 5 and 6. This processing flow is executed in response to the sensor unit 3 of the wearable terminal Wt acquiring a service request input, which is requested by user Us with the intention of providing user services, as sensing information. In this processing flow, each "S" represents a set of steps executed by multiple instructions included in the processing program of the first embodiment.

[0038] In step S10 of the processing flow shown in Figure 5, the recognition block 200 (see Figure 4) monitors the field of view area Av, which is visible to the user Us through the wearable terminal Wt. At this time, the monitoring of the field of view area Av is based on at least one type of sensing information from the sensor unit 3 of the wearable terminal Wt, such as camera information and inertial information. As a result, the field of view area Av is recognized as an area that extends in the direction that the user Us's face or gaze is pointing, or in the direction that the wearable terminal Wt is pointing. Furthermore, during the execution of each step following step S10 of the processing flow, the monitoring process of the field of view area Av may be executed in parallel in response to changes in the field of view area Av due to the movement of the user Us, thereby updating the latest recognized field of view area Av.

[0039] In step S20 of the processing flow shown in Figure 5, the recognition block 200 acquires the desired content Cd that user Us intends to receive. At this time, the desired content Cd is recognized based on sensing information about the face of user Us, which is provided by the wearable terminal Wt worn on the user's face. Therefore, the sensing information for recognizing the desired content Cd is acquired through the wearable terminal Wt at the time that it is input to the sensor unit 3 as user Us's intended data.

[0040] In S20, the input for desired content Cd may be gesture input, gaze input, facial expression input, or voice input by the user Us, which may be sensed by the sensor unit 3. At this time, the display unit 2 of the wearable terminal Wt may superimpose an XR image for receiving the input of desired content Cd from the user Us via the sensor unit 3 onto the field of view area Av through the wearable terminal Wt. Furthermore, the recognition of desired content Cd may be realized based on three-dimensional spatial information It obtained from the infrastructure database Di via the communication system 10.

[0041] The items of the desired content Cd recognized in S20 include at least a display theme, which is given to the XR image (image 20 in Figures 7 and 8 described later) displayed by the display unit 2 of the wearable terminal Wt in connection with the provision of user services to the user Us. The display theme refers to a common concept for XR images that are two-dimensional or three-dimensional displays, such as virtual or real living things like monsters, animals, plants, and insects, as well as inanimate objects such as prehistoric buildings, service vehicles, and imaginary objects.

[0042] The desired content Cd item recognized in S20 may include the types of user services that can be provided to user Us through the use of the autonomous driving device Ma. At least one of the following types of user services may be provided: guidance services, transport services, photography services, and online game services.

[0043] In S20, for any user service, at least one route-related item from among, for example, the route including the destination, travel pace, arrival time, and service provision time (i.e., guidance time or transport time) may be included in the desired content CD. In S20, for any user service, at least one voice-related item from among, for example, whether voice output is required, whether voice recognition function is required, and whether dialogue function is required may be included in the desired content CD.

[0044] In S20, for guidance services, the desired content CD may include at least one tourism-related item concerning the destination, such as acceptable crowd levels, availability of desired facilities, and presence of natural scenery. In S20, for transport services, the desired content CD may include at least one luggage-related item, such as the number, size, weight, type, and whether temperature control is required.

[0045] In the case of a photography service in S20, at least one photography-related item from among, for example, the photography schedule, photography timing, photography location, whether or not images or data need to be sent, and whether or not the photos need to be printed, may be included in the desired content CD. In the case of an online game service in S20, at least one game-related item from among, for example, the initial settings of the game character, online game environment settings, online game difficulty level, whether or not images or data need to be sent, and whether or not or not a match needs to be played, may be included in the desired content CD.

[0046] In step S30 of the processing flow shown in Figure 5, the drive control block 210 (see Figure 4) provides user services by controlling the drive of the autonomous driving device Ma within the field of view area Av. At this time, the user service is provided according to the drive control pattern for the autonomous driving device Ma, which is read from memory 9a according to its type, in order to lead the user Us as it moves toward the field of view area Av.

[0047] The guidance service provided by S30 is delivered according to a drive control pattern for route guidance, which drives the autonomous driving device Ma from a standby position to a position near the user Us and leads the user Us to the destination. In this guidance service, the route to guide the user Us may be presented by superimposing an XR image onto the field of view area Av on the wearable terminal Wt, or by display output from a display unit owned by the autonomous driving device Ma. In the guidance service, the route to guide the user Us may be presented by voice output from a voice unit owned by the wearable terminal Wt or the autonomous driving device Ma.

[0048] The transport service by S30 is provided according to a drive control pattern that drives the autonomous driving device Ma, which has a loading compartment, from a standby position to a position near the user Us, and transports the cargo loaded in the loading compartment to the destination while the user Us is guided. In this transport service, the method of loading cargo into the loading compartment may be presented to the user Us by superimposing an XR image onto the field of view area Av on the wearable terminal Wt, or by display output from a display unit owned by the autonomous driving device Ma. In the transport service, the loading of cargo into the loading compartment may be presented to the user Us by voice output from a voice unit owned by the wearable terminal Wt or the autonomous driving device Ma.

[0049] The imaging service provided by S30 is offered according to a drive control pattern that drives the autonomous driving device Ma, which has an imaging unit, from a standby position to a position near the user Us, and takes images from the surroundings while leading the user Us. In this imaging service, the imaging schedule or timing may be presented to the user Us by superimposing the XR image onto the field of view area Av on the wearable terminal Wt, or by display output from the display unit owned by the autonomous driving device Ma. In the imaging service, the imaging schedule or timing may also be presented to the user Us by voice output from the voice unit owned by the wearable terminal Wt or the autonomous driving device Ma.

[0050] In this shooting service, two-dimensional or three-dimensional still images or videos including the user Us may be captured by an autonomous driving device Ma that drives around the user Us. In the shooting service, the focus during shooting may be adjusted based on the distance to the user Us sensed by a sensor unit 4 such as a LiDAR in the autonomous driving device Ma. In the shooting service, the direction of ambient light incidence during shooting may be adjusted based on three-dimensional spatial information It obtained from an infrastructure database Di, or sensing information from a sensor unit 4 such as a camera in the autonomous driving device Ma. In the shooting service, the image data of the user Us may be stored in the memory 9a of the control system 9 or in a storage medium in the wearable terminal Wt.

[0051] The online game service provided by S30 is delivered according to a drive control pattern that drives the autonomous driving device Ma from a standby position to a position near the user Us, leading the user Us to a point in the online game. In this online game service, necessary instructions in the online game may be presented to the user Us by superimposing an XR image onto the field of view area Av on the wearable terminal Wt, or by display output from a display unit owned by the autonomous driving device Ma. In the online game service, necessary instructions in the online game may also be presented to the user Us by voice output from a voice unit owned by the wearable terminal Wt or the autonomous driving device Ma.

[0052] In the processing flow shown in Figure 5, step S40 is executed in parallel with step S30. In step S40, the display control block 220 (see Figure 4) overlays an XR image corresponding to the desired content Cd onto the field of view area Av, which is occupied by the autonomous driving device Ma providing user services in step S30, using the display unit 2 of the wearable terminal Wt. At this time, the display control block 220 adjusts the occupancy rate Rc of the display target range Ad of the content image 20 that is overlaid according to the desired content Cd, as shown in Figures 7 and 8, within the field of view area Av, according to the risk level Lr in the area Av. Here, the occupancy rate Rc in the first embodiment means, for example, the ratio of the display area of ​​the entire display target range Ad to the two-dimensional projection area of ​​the entire field of view area Av on the glass or lens or retina of the display unit 2.

[0053] In S40, the content image 20 includes at least the device image 20a shown in Figures 7 and 8, as an XR image superimposed on the autonomous driving device Ma visible in the field of view area Av, as shown in Figure 2. The superimposed display position of the device image 20a is adjusted according to the drive control pattern in S30, in accordance with the predicted viewing position of the autonomous driving device Ma. This display position adjustment is based on information obtained from the infrastructure database Di via the communication system 10 as three-dimensional spatial information It, to which position-related information of the autonomous driving device Ma is added to each voxel Vi in the user service provision space. As a result, an overlap display is achieved in which the device image 20a of the display theme corresponding to the desired content Cd is superimposed in such a way that the entire autonomous driving device Ma is opaque or semi-transparent from the front.

[0054] In S40, the content image 20 may include the object image 20b shown in Figures 7 and 8, as an XR image superimposed on another object Mb visible in the field of view Av together with the autonomous driving device Ma, as shown in Figure 2. The superimposed display position of the object image 20b is adjusted to match the predicted viewing position of the other object Mb, resulting in an overlap display similar to that of the device image 20a. The other object Mb on which the object image 20b is superimposed in this way is assumed to be at least one of the following: a person other than the user Us of the processing system 1, a moving object other than the autonomous driving device Ma that cooperates with the processing system 1, an aircraft, a structure, and vegetation.

[0055] In S40, the content image 20 may include the ground image 20c shown in Figures 7 and 8, as an XR image superimposed on the ground Mc visible in the field of view area Av along with the autonomous driving device Ma, as shown in Figure 2. The superimposed display position of the ground image 20c is adjusted to match the predicted viewing position of the ground Mc, resulting in a similar overlap display to the device image 20a.

[0056] In S40, as shown in Figure 7, the display range Ad of the content image 20 is set to cover the entire field of view Av, and the occupancy rate Rc is adjusted to its maximum value (e.g., 1). In S40, as shown in Figure 9, the display range Ad of the content image 20 is set to disappear from the entire field of view Av, and the occupancy rate Rc is adjusted to its minimum value (e.g., 0). In S40, as shown in Figure 8, the display range Ad of the content image 20 is set to cover only a portion of the field of view Av, and the occupancy rate Rc is adjusted to an intermediate value between the minimum and maximum values ​​(e.g., greater than 0 and less than 1).

[0057] In S40, if hazard notification to the user Us becomes necessary according to the risk level Lr, the hazard image 22 as an XR image may be superimposed on the ground Mc and / or sky Md within the field of view area Av shown in Figure 2 by the display unit 2, as shown in Figures 8 and 9. The superimposed display position of the hazard image 22 should be adjusted to match the predicted viewing position of the ground Mc and / or sky Md. This adjustment of the display position should also be based on information obtained from the infrastructure database Di via the communication system 10 as three-dimensional spatial information It, in which position-related information of the ground Mc and / or sky Md is added to each voxel Vi in the user service provision space.

[0058] In step S40, as shown in Figure 6, an adjustment subroutine is executed to switch and adjust the occupancy rate Rc according to the risk level Lr in the field of view area Av. Specifically, in step S400 of the adjustment subroutine, the display control block 220 acquires internal sensing information Ii sensed by the sensor unit 3 of the wearable terminal Wt. At this time, the acquisition range of internal sensing information Ii with an acceptable level of accuracy or reliability by the sensor unit 3 is set to the internal sensing space Ai, which is part of the field of view area Av, as shown in Figure 10. Therefore, the internal sensing information Ii acquired for the internal sensing space Ai is obtained from at least one of the sensor units 3, such as a camera and a motion sensor.

[0059] In the S400, the internal sensing space Ai is defined by multiple voxels Vi, centered on the user Us or wearable terminal Wt, whose two-dimensional or three-dimensional distances are assumed to be within a set distance. Therefore, the set distance that determines the limit range of the internal sensing space Ai is set to a fixed value, such as 5m, according to the specifications of the sensor unit 3 and / or the risk requirements for the processing system 1.

[0060] In step S410 of the adjustment subroutine shown in Figure 6, the display control block 220 acquires the risk level Lr in the internal sensing space Ai within the field of view area Av. At this time, the risk level Lr is defined as an indicator that reaches the hazard level Lh when at least a portion of an obstacle Mbo, such as a person, moving object, or structure, which is predicted to interfere with the movement of the user Us as shown in Figure 10, relatively enters the internal sensing space Ai. In other words, the risk level Lr in the internal sensing space Ai is judged to be at a safe level, having moved away from the hazard level Lh, when the obstacle Mbo disappears from the space Ai.

[0061] In the adjustment subroutine S420 shown in Figure 6, the display control block 220 determines whether the risk level Lr in the internal sensing space Ai has reached the hazard level Lh, which requires hazard notification to the moving user Us. At this time, the comparison of the risk level Lr with the hazard level Lh may be performed for each voxel Vi in the user service provision space, or for groups of multiple voxels Vi.

[0062] If a positive determination is made in S420, in S430 of the adjustment subroutine shown in Figure 6, the display control block 220 adjusts the occupancy rate Rc to its minimum value by prohibiting the superimposition of the content image 20 over the entire field of view area Av, as shown in Figure 9, so as to exclude the entire field of view area Av from the display target range Ad of the content image 20. However, in S430, a hazard image 22 that notifies warning characters and / or warning colors according to the risk level Lr that has reached hazard level Lh may be superimposed on the field of view area Av as shown in Figure 9. In this case, the hazard image 22 may be displayed in such a way as to notify at least one of the following, such as the position, direction, distance, and type, of the obstacle Mbo that caused the hazard level Lh in the internal sensing space Ai.

[0063] In addition to displaying the hazard image 22, in S430, the risk level Lr, which has reached hazard level Lh, may be warned by a display output from the display unit owned by the autonomous driving device Ma or by the illumination output of a warning light. In S430, the risk level Lr, which has reached hazard level Lh, may be warned by an audio output from the display unit owned by the autonomous driving device Ma. Upon completion of the execution of S430, the current execution of the adjustment subroutine and processing flow ends.

[0064] If a negative determination is made in S420, the display control block 220 acquires external sensing information Io sensed by the sensor unit 4 of the autonomous driving device Ma in S440 of the adjustment subroutine shown in Figure 6. At this time, the acquisition range of external sensing information Io with an acceptable level of accuracy or reliability by the sensor unit 4 is set to the external sensing space Ao, which is a part of the field of view area Av and is wider than the internal sensing space Ai, as shown in Figure 11. Therefore, the external sensing information Io acquired for the external sensing space Ao should preferably be obtained from at least one of the sensor units 4, such as a camera, LiDAR, and sonar.

[0065] In S440, the external sensing information Io acquired for the external sensing space Ao may include three-dimensional spatial information It obtained from the infrastructure database Di via the communication system 10, along with the sensing information from the sensor unit 4. That is, the external sensing information Io may be three-dimensional spatial information It collected by the infrastructure system 6 and stored in the infrastructure database Di.

[0066] In S440, the external sensing space Ao is defined by multiple voxels Vi, centered on the autonomous driving device Ma, whose distances in two or three dimensions are assumed to be within a set range. Therefore, the set distance that determines the limit range Aol of the external sensing space Ao shown in Figure 11 may be set to a fixed value, such as 6m to 20m, depending on the specifications of the sensor unit 4 and / or the risk requirements for the processing system 1.

[0067] In step S450 of the adjustment subroutine shown in Figure 6, the display control block 220 acquires the risk level Lr in the external sensing space Ao within the field of view area Av. At this time, the risk level Lr is defined as an index in which it is judged that the hazard level Lh has been reached when the two-dimensional or three-dimensional separation distance δo between the limit range Aol of the external sensing space Ao and the user Us, as shown in Figure 11, falls below or less than a threshold. In other words, the risk level Lr in the external sensing space Ao is judged to be at a safe level, having moved away from the hazard level Lh, when the separation distance δo between the limit range Aol of the space Ao and the user Us exceeds or is greater than a threshold.

[0068] In the adjustment subroutine S460 shown in Figure 6, the display control block 220 determines whether the risk level Lr in the external sensing space Ao has reached the hazard level Lh, which requires hazard notification to the moving user Us. At this time, the comparison of the risk level Lr with the hazard level Lh may be performed for each voxel Vi in the user service provision space, or for groups of multiple voxels Vi.

[0069] If a positive determination is made in S460, the display control block 220 in S470 of the adjustment subroutine shown in Figure 6 identifies the hazard space Aoh in the external sensing space Ao where the risk level Lr has reached the hazard level Lh, as shown in Figure 11. At this time, multiple voxels Vi in which the risk level Lr has reached the hazard level Lh are recognized as the hazard space Aoh. Furthermore, in the field of view area Av, if the display prohibition range in which the superimposed display of the content image 20 described later is prohibited continues to the back of the external sensing space Ao, multiple voxels Vi located on the back side may be recognized as the hazard space Aoh (see Figure 8).

[0070] Therefore, the display control block 220 in S470 adjusts the occupancy rate Rc to an intermediate value by prohibiting the superimposition of the content image 20 specifically on the hazard space Aoh, as shown in Figure 8, so that the hazard space Aoh, which has reached hazard level Lh, is excluded from the display target range Ad of the content image 20. As a result, the remaining safe space As, which is outside the hazard space Aoh in the field of view area Av, is set to the display target range Ad of the content image 20, thereby allowing the superimposition of the content image 20.

[0071] In S470, a hazard image 22 that notifies warning characters and / or warning colors according to the risk level Lr which has reached hazard level Lh may be superimposed on the field of view area Av as shown in Figure 8. In this case, the hazard image 22 may be displayed in a manner that notifies the increased risk due to the proximity of user Us, who caused the hazard level Lh in the external sensing space Ao, to the limit range Aol.

[0072] In addition to displaying the hazard image 22, in S470, the risk level Lr, which has reached hazard level Lh, may be warned by a display output from the display unit owned by the autonomous driving device Ma or by the illumination output of a warning light. In S470, the risk level Lr, which has reached hazard level Lh, may be warned by an audio output from the display unit owned by the autonomous driving device Ma. Upon completion of the execution of S470, the current execution of the adjustment subroutine and processing flow ends.

[0073] If a negative result is obtained in S460, the display control block 220 in S480 of the adjustment subroutine shown in Figure 6 adjusts the occupancy rate Rc to its maximum value by allowing the superimposition of the content image 20 over the entire field of view area Av, as shown in Figure 7, so that the entire field of view area Av is set as the display target range Ad of the content image 20. At this time, a limitation image 20cl is superimposed on the field of view area Av as a ground image 20c representing the limit range Aol on the ground Mc of the external sensing space Ao, as shown in Figure 7. Upon completion of the execution of S480, the current execution of the adjustment subroutine and processing flow ends. Therefore, in subsequent executions of the processing flow, the limitation image 20cl will continue to be displayed until a positive result is obtained in either S420 or S460.

[0074] (Effects and Benefits) The effects and advantages of the first embodiment described above will be explained below.

[0075] In the first embodiment, the field of view area Av, which is visible to the user Us through a wearable terminal Wt worn by the user Us, is monitored. The occupancy rate Rc of the display target range Ad of the content image 20, which is superimposed on the field of view area Av as an XR image corresponding to the desired content Cd requested by the user Us, is adjusted according to the risk level Lr in the field of view area Av.

[0076] In this case, particularly according to the first embodiment, external sensing information Io sensed by the autonomous driving device Ma, which leads the user Us as it moves toward the field of view area Av, is acquired, and the risk level Lr based on the external sensing information Io is recognized. As a result, by effectively utilizing the autonomous driving device Ma, which performs the function of leading the user Us, the risk level Lr can be recognized for as wide an area as possible within the field of view area Av, which is the destination of the user Us, thereby ensuring the safety of the user Us.

[0077] Furthermore, according to the first embodiment, in the external sensing space Ao which is the acquisition range of external sensing information Io in the field of view area Av, the superimposed display of content image 20 is prohibited in the hazard space Aoh where the risk level Lr has reached the hazard level Lh. As a result, the content image 20 disappears only within the hazard space Aoh at hazard level Lh, where hazard notification to the moving user Us is required, while the occupancy rate Rc can be adjusted so that the superimposed display of content image 20 continues outside the hazard space Aoh in the field of view area Av. Therefore, it is possible to balance the safety ensured by the above principle for the moving user Us with the continuity of the virtual experience.

[0078] According to the first embodiment, a limitation image 20cl, which is an XR image representing the limit range Aol on the ground Mc of the external sensing space Ao, is superimposed on the field of view area Av. In this case, when the limit range Aol of the external sensing space Ao, where the risk level Lr is recognized, is presented to the user Us, the user Us's safety is ensured by the recognition of the risk level Lr in the external sensing space Ao, and thus the user Us's sense of security can also be ensured. Therefore, in conjunction with the disappearance of the content image 20 limited to the hazard space Aoh, it is possible to ensure the user Us's safety and sense of security while balancing it with the continuity of the virtual experience.

[0079] According to the first embodiment, when internal sensing information Ii sensed by the wearable terminal Wt is acquired, the risk level Lr based on the internal sensing information Ii is recognized. As a result, the risk level Lr can be recognized even within the user Us's immediate vicinity by the wearable terminal Wt, and the user Us's safety can be ensured. However, if the risk level Lr in the internal sensing space Ai, which is the acquisition range of the internal sensing information Ii within the field of view Av, reaches the hazard level Lh, the superimposition display of the content image 20 is prohibited for the entire field of view Av. As a result, it is possible to adjust the occupancy rate Rc in a way that intentionally skews the balance so that safety takes precedence over the continuity of the virtual experience within the user Us's immediate vicinity.

[0080] According to the first embodiment, external sensing information Io collected by the infrastructure system 6 is also acquired together with the external sensing information Io sensed by the autonomous driving device Ma, and the risk level Lr based on that external sensing information Io may be recognized. In this case, not only the autonomous driving device Ma, which plays a leading role for the user Us, but also the infrastructure system 6 is effectively utilized, so that the risk level Lr for the field of view area Av can be recognized with high accuracy, thereby ensuring the safety of the user Us.

[0081] (Second embodiment) The second embodiment is a modification of the first embodiment.

[0082] As shown in Figure 12, the processing system 2001 of the second embodiment is connected via a communication network Nc to a mobile terminal Mt carried by a pre-registered user Us, an autonomous driving device Ma that drives autonomously, and an infrastructure system 6 that provides infrastructure information. Here, multiple terminals are assumed to be the mobile terminal Mt connected to the processing system 1.

[0083] The mobile terminal Mt is configured to be grasped and operated by the user Us with their fingers, at least when connected to the processing system 2001. As shown in Figure 13, the mobile terminal Mt is a small electronic device equipped with a display unit 2002 and a sensor unit 2003, such as a smartphone or tablet terminal. The mobile terminal Mt controls the transmission and reception of data via the communication network Nc by a communication unit, as shown in Figure 12. The mobile terminal Mt is operated and controlled by a control unit according to control commands from the processing system 2001 via the communication network Nc, so as to perform service-related processing in cooperation with the processing system 2001.

[0084] Under the operational control of the control unit of the mobile terminal Mt, the display unit 2002 shown in Figure 13, such as an LCD panel or an organic EL panel, realizes the screen display necessary for service-related processing (see Figures 17-19 described later). At the same time, under the operational control of the mobile terminal Mt, the sensor unit 2003 shown in Figure 13 acquires sensing information through sensing processing according to the service-related processing. The sensor unit 2003 employs at least one of the following: a camera, an inertial sensor, a GNSS (Global Navigation Satellite System) sensor, a touch sensor, and a microphone. With this configuration, the sensor unit 2003 is able to acquire input representing the user Us's intention as sensing information.

[0085] The autonomous driving device Ma shown in Figure 12 is configured similarly to the processing system 1 of the first embodiment, except that it is driven and controlled by a control unit according to control commands from the processing system 2001 via the communication network Nc, so as to perform service-related processing in cooperation with the processing system 2001. The control system 2009 of the processing system 2001 is configured similarly to the control system 9 of the first embodiment, except that at least a part of it may be composed of a control unit for the mobile terminal Mt, and that it executes a processing flow using the mobile terminal Mt, which will be described later.

[0086] In order to perform service-related processing in the control system 2009, the processing system 2001 performs XR display processing through the joint operation of blocks 200, 210, and 220, which are constructed as shown in Figure 14, according to the processing flow shown in Figures 15 and 16.

[0087] In step S2010 of the processing flow shown in Figure 15, the recognition block 200 (see Figure 14) monitors the background area Ab, which will be the display background for user Us, by pointing the mobile terminal Mt carried by user Us towards it. At this time, the background area Ab is recognized as an area extending in the direction that the mobile terminal Mt is facing, based on at least one type of sensing information from the sensor unit 2003 of the mobile terminal Mt, such as camera information and inertial information. Furthermore, during the execution of each step following S2010 of the processing flow, the monitoring process of the background area Ab may be executed in parallel in response to changes in the background area Ab due to the movement of user Us, thereby updating the latest recognized background area Ab.

[0088] In step S2020 of the processing flow shown in Figure 15, the recognition block 200 acquires the desired content Cd by recognition based on sensing information from the mobile terminal Mt carried by the user Us. The sensing information at this time is acquired via the mobile terminal Mt at the same time that it is input to the sensor unit 2003 as user Us's intent data, and is used for the recognition of the desired content Cd. Details regarding the input and items of the desired content Cd are the same as in step S20 of the first embodiment.

[0089] In step S2030 of the processing flow shown in Figure 15, the drive control block 210 (see Figure 14) provides user services by controlling the drive of the autonomous driving device Ma within the background area Ab. At this time, the user service is provided according to the drive control pattern for the autonomous driving device Ma, which is read from memory 9a according to its type, in order to lead the user Us moving toward the background area Ab. Details regarding the drive of the autonomous driving device Ma and the provision of user services are the same as in step S30 of the first embodiment.

[0090] In the processing flow shown in Figure 15, S2040 is executed in parallel with S2030. In S2040, the display control block 220 (see Figure 14) overlays an XR image corresponding to the desired content Cd onto the background video 24, which shows the background area Ab containing the autonomous driving device Ma that is providing user services, using the display unit 2002 of the mobile terminal Mt. At this time, the display control block 220 adjusts the occupancy rate Rc of the display target area Ad of the content image 20 that is overlaid according to the desired content Cd, as shown in Figures 17 and 18, within the background video 24, according to the risk level Lr in the area Ab. Here, the occupancy rate Rc in the second embodiment means the ratio of the display area of ​​the entire display target area Ad to the display area of ​​the entire background video 24 in the display unit 2002.

[0091] In S2040, the background video 24 to be superimposed on the content image 20 is acquired, for example, based on sensing information from a sensor unit 4 such as a camera, or three-dimensional spatial information It obtained from an infrastructure database Di. The background video 24 may be primary video data of the background area Ab, or it may be secondary video data obtained by processing the primary data.

[0092] In S2040, the content image 20 includes at least the device image 20a shown in Figures 17 and 18, as an XR image superimposed on the video portion of the background video 24, corresponding to the location of the autonomous driving device Ma in the background area Ab, as shown in Figure 13. The superimposed display position of the device image 20a is adjusted according to the drive control pattern by S2030, in accordance with the imaging position of the autonomous driving device Ma on the background video 24. As a result, a replacement display is achieved in which the device image 20a of the display theme corresponding to the desired content Cd is superimposed in such a way that it replaces the entire autonomous driving device Ma shown in the background video 24. Details regarding the adjustment of the display position of the device image 20a are the same as in S40 of the first embodiment.

[0093] In S2040, the content image 20 may include an object image 20b shown in Figures 17 and 18, which is an XR image superimposed on the video portion of the background video 24 at the location of another object Mb present in the background area Ab together with the autonomous driving device Ma, as shown in Figure 13. The superimposed display position of the object image 20b is adjusted to match the imaging position of the other object Mb on the background video 24, resulting in a similar replacement display to the device image 20a. Details of the other object Mb on which the object image 20b is replaced are the same as in S40 of the first embodiment.

[0094] In S2040, the content image 20 may include the ground image 20c shown in Figures 17 and 18, as an XR image superimposed on the video portion of the background video 24, corresponding to the location of the ground Mc that exists in the background area Ab together with the autonomous driving device Ma, as shown in Figure 13. The superimposed display position of the ground image 20c is adjusted to match the imaging position of the ground Mc on the background video 24, resulting in a replacement display similar to that of the device image 20a.

[0095] In S2040, as shown in Figure 17, the display range Ad of the content image 20 is set to cover the entire background video 24, adjusting the occupancy rate Rc to its maximum value (e.g., 1). In S2040, as shown in Figure 19, the display range Ad of the content image 20 is set to disappear from the entire background video 24, adjusting the occupancy rate Rc to its minimum value (e.g., 0). In S2040, as shown in Figure 18, the display range Ad of the content image 20 is set to cover only a portion of the background video 24, adjusting the occupancy rate Rc to an intermediate value between the minimum and maximum values ​​(e.g., greater than 0 and less than 1).

[0096] In S2040, if hazard notification to the user Us becomes necessary according to the risk level Lr, a hazard image 22 as an XR image may be superimposed on the ground Mc and / or sky Md shown in the background video 24 as shown in Figure 13 by the display unit 2002, as shown in Figures 18 and 19. The superimposed display position of the hazard image 22 should be adjusted to match the imaging positions of the ground Mc and / or sky Md on the background video 24. Details regarding the adjustment of the display position of the hazard image 22 are the same as in S40 of the first embodiment.

[0097] In this S2040, an adjustment subroutine is executed as shown in Figure 16 to switch and adjust the occupancy rate Rc according to the risk level Lr in the background area Ab. Specifically, in S2400 of the adjustment subroutine, the display control block 220 acquires internal sensing information Ii sensed by the sensor unit 2003 of the mobile terminal Mt. At this time, the acquisition range of internal sensing information Ii with an acceptable level of accuracy or reliability by the sensor unit 2003 is set to the internal sensing space Ai, which is part of the background area Ab, as shown in Figure 20. Details regarding the acquisition of internal sensing information Ii and the definition and setting of the internal sensing space Ai are the same as in S400 of the first embodiment.

[0098] In S2410 of the adjustment subroutine shown in Figure 16, the display control block 220 acquires the risk level Lr in the internal sensing space Ai within the background area Ab. Details regarding the definition and determination of the risk level Lr are the same as in S410 of the first embodiment. Furthermore, in S2420 of the adjustment subroutine, the display control block 220 determines the risk level Lr in accordance with S410 of the first embodiment.

[0099] If a positive result is obtained in S2420, in S2430 of the adjustment subroutine shown in Figure 16, the display control block 220 adjusts the occupancy rate Rc to its minimum value by prohibiting the superimposition of the content image 20 over the entire background video 24, as shown in Figure 19, so that the entire background video 24 is excluded from the display target range Ad of the content image 20. However, in S2430, a hazard image 22 that notifies warning characters and / or warning colors according to the risk level Lr when the hazard level Lh is reached may be superimposed on the background video 24 as shown in Figure 19. Details regarding the display of the hazard image 22 and other warning methods are the same as in S430 of the first embodiment. Upon completion of the execution of S2430, the current execution of the adjustment subroutine and processing flow ends.

[0100] If a negative determination is made in S2420, the display control block 220 acquires external sensing information Io in S2440 of the adjustment subroutine shown in Figure 16, in accordance with S440 of the first embodiment. However, in this case, the acquisition range of external sensing information Io with an acceptable level of accuracy or reliability by the sensor unit 4 is set to an external sensing space Ao that is part of the background area Ab and is wider than the internal sensing space Ai, as shown in Figure 21.

[0101] In the adjustment subroutine S2450 shown in Figure 16, the display control block 220 acquires the risk level Lr in the external sensing space Ao within the background area Ab. Details regarding the definition and determination of the risk level Lr are the same as in S450 of the first embodiment. Furthermore, in the adjustment subroutine S2460, the display control block 220 determines the risk level Lr in accordance with S460 of the first embodiment.

[0102] If a positive determination is made in S2460, the display control block 220 in S2470 of the adjustment subroutine shown in Figure 16 identifies the hazard space Aoh within the external sensing space Ao in accordance with S470 of the first embodiment, as shown in Figure 21. However, within the background area Ab, if the display prohibition range where the superimposed display of the content image 20 is prohibited, as described later, extends to the back of the external sensing space Ao, then multiple voxels Vi located on the back side may be recognized as the hazard space Aoh (see Figure 18).

[0103] Therefore, in S2470, the display control block 220 adjusts the occupancy rate Rc to an intermediate value by prohibiting the superimposition of the content image 20 by focusing on the video portion 24h of the background video 24 that shows the external sensing space Ao, specifically the video portion 24h that shows the hazard space Aoh, as shown in Figure 18. As a result, the remaining portion 24s of the background video 24, which is not the video portion 24h, is set to the display target range Ad of the content image 20, corresponding to the remaining safe space portion As outside the hazard space Aoh in the background area Ab, and thus the superimposition of the content image 20 is permitted. Details regarding the display of the hazard image 22 and other warning methods are the same as in S470 of the first embodiment. Upon completion of the execution of S2470, the current execution of the adjustment subroutine and processing flow is completed.

[0104] If a negative result is obtained in S2460, the display control block 220 in S2480 of the adjustment subroutine shown in Figure 16 adjusts the occupancy rate Rc to its maximum value by allowing the superimposition of the content image 20 over the entire background video 24, as shown in Figure 17, so that the entire background video 24 is set as the display target range Ad of the content image 20. At this time, a limitation image 20cl is superimposed on the content image 20 as a ground image 20c representing the limit range Aol on the ground Mc of the external sensing space Ao, as shown in Figure 17. Upon completion of the execution of S2480, the current execution of the adjustment subroutine and processing flow ends. Therefore, in subsequent executions of the processing flow, the limitation image 20cl will continue to be displayed until a positive result is obtained in either S2420 or S2460.

[0105] (Effects and Benefits) The effects and advantages of the second embodiment described above will be explained below.

[0106] In the second embodiment, the background area Ab, which is visible to user Us through a mobile terminal Mt carried by user Us, is monitored. The percentage Rc of the display target range Ad of the content image 20, which is superimposed on the background video 24 showing the background area Ab as an XR image corresponding to the desired content Cd of user Us, is adjusted according to the risk level Lr in the background area Ab.

[0107] In this case, particularly according to the second embodiment, external sensing information Io sensed by the autonomous driving device Ma, which leads the user Us as it moves toward the background area Ab, is acquired, and the risk level Lr based on the external sensing information Io is recognized. As a result, by effectively utilizing the autonomous driving device Ma, which performs the leading function of the user Us, the risk level Lr can be recognized for as wide an area as possible within the background area Ab, which is the destination of the user Us, thereby ensuring the safety of the user Us.

[0108] Furthermore, according to the second embodiment, in the background video 24 showing the external sensing space Ao, which is within the acquisition range of external sensing information Io in the background area Ab, the superimposition of the content image 20 is prohibited for the video portion 24h showing the hazard space Aoh where the risk level Lr has reached the hazard level Lh. As a result, in the background video 24, the content image 20 disappears only in the portion 24h corresponding to the hazard space Aoh at hazard level Lh, which requires hazard notification to the moving user Us, while the superimposition of the content image 20 continues in the portion 24s corresponding to the outside of the hazard space Aoh in the background area Ab, thus the occupancy rate Rc can be adjusted. Therefore, it is possible to balance the safety ensured for the moving user Us by the principle described above with the continuity of the virtual experience.

[0109] According to the second embodiment, a limitation image 20cl, which is an XR image representing the limit range Aol on the ground Mc of the external sensing space Ao, is superimposed on a background video 24 showing the background area Ab. In this case, when the limit range Aol of the external sensing space Ao, where the risk level Lr is recognized, is presented to the user Us, the user Us's safety is ensured by the recognition of the risk level Lr in the external sensing space Ao, and thus the user Us's sense of security can also be ensured. Therefore, in conjunction with the disappearance of the content image 20 limited to the corresponding portion 24h within the hazard space Aoh, it is possible to ensure the user Us's safety and sense of security while balancing it with the continuity of the virtual experience.

[0110] According to the second embodiment, when internal sensing information Ii sensed by the mobile terminal Mt is acquired, the risk level Lr based on the internal sensing information Ii is recognized. As a result, the risk level Lr can be recognized even within the user Us's immediate vicinity by the mobile terminal Mt, and the user Us's safety can be ensured. However, if the risk level Lr in the internal sensing space Ai, which is the acquisition range of the internal sensing information Ii within the background area Ab, reaches the hazard level Lh, the superimposition of the content image 20 over the entire background video 24 showing the background area Ab is prohibited. As a result, it is possible to adjust the occupancy rate Rc in a way that intentionally skews the balance so that safety takes precedence over the continuity of the virtual experience within the user Us's immediate vicinity.

[0111] According to the second embodiment, external sensing information Io collected by the infrastructure system 6 is also acquired together with the external sensing information Io sensed by the autonomous driving device Ma, and the risk level Lr based on that external sensing information Io may be recognized. In this case, not only the autonomous driving device Ma, which plays a leading role for the user Us, but also the infrastructure system 6 is effectively utilized, so that the risk level Lr for the background area Ab can be recognized with high accuracy, thereby ensuring the safety of the user Us.

[0112] (Other embodiments) Although several embodiments have been described above, this disclosure is not intended to be limited to those embodiments, and can be applied to various embodiments and combinations without departing from the spirit of this disclosure.

[0113] In the modified embodiments of the first and second embodiments, the dedicated computers constituting the control systems 9 and 2009 of the processing system 1 and 2001 may have at least one of a digital circuit and an analog circuit as a processor. Here, the digital circuit is at least one of the following: ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), SOC (System on a Chip), PGA (Programmable Gate Array), and CPLD (Complex Programmable Logic Device). Furthermore, such a digital circuit may have a memory that stores a program.

[0114] In the modified versions of the first and second embodiments, the processing system 1,2001 may be unconnected to the infrastructure system 6. In these modified versions, sensing information from the connection elements Wt, Ma, and Mt with the processing system 1,2001 may be used instead of the three-dimensional spatial information It obtained from the infrastructure database Di.

[0115] In the modifications of the first and second embodiments, three-dimensional spatial information It, which is not associated with voxel Vi, may be obtained from the infrastructure database Di to the processing system 1,2001. In the modifications of the first and second embodiments, two-dimensional grid information may be obtained from the infrastructure database Di to the processing system 1,2001 instead of three-dimensional spatial information It.

[0116] In a modified version of the first embodiment, if a mobile terminal Mt, as in the second embodiment, is connected to the processing system 1 and functions as part of a wearable terminal Wt, the processing flow may be executed in response to the sensor unit 2003 constituting the part acquiring a service request input as sensing information. In a modified version of the first embodiment, if a mobile terminal Mt, as in the second embodiment, is connected to the processing system 1 and functions as part of a wearable terminal Wt, the desired content Cd in S20 of the processing flow may be acquired based on the sensing information of the sensor unit 2003 constituting the part.

[0117] In a modified version of the second embodiment, a smartwatch or a video see-through or non-transparent wearable terminal may be used as the mobile terminal Mt. In a modified version of the second embodiment, when a wearable terminal Wt, as in the first embodiment, is connected to the processing system 2001 and functions as part of the mobile terminal Mt, the processing flow may be executed in response to the sensor unit 3 constituting the part acquiring a service request input as sensing information. In a modified version of the second embodiment, when a wearable terminal Wt, as in the first embodiment, is connected to the processing system 2001 and functions as part of the mobile terminal Mt, the desired content Cd in S2020 of the processing flow may be acquired based on the sensing information of the sensor unit 3 constituting the part.

[0118] In the modified versions of the first and second embodiments, the display of the limitation image 20cl may be omitted in S480 and S2480. In the modified versions of the first and second embodiments, the limitation image 20cl may also be displayed in S470 and S2470 within the display range Ad of the content image 20. In the modified version of the first embodiment, in S430 when the risk level Lr in the internal sensing space Ai reaches the hazard level Lh, the superimposed display of the content image 20 may be prohibited, limited to the internal sensing space Ai within the field of view area Av. In the modified version of the second embodiment, in S2430 when the risk level Lr in the internal sensing space Ai reaches the hazard level Lh, the superimposed display of the content image 20 may be prohibited, limited to the video portion of the background area Ab corresponding to the internal sensing space Ai.

[0119] In addition to the embodiments described so far, the above-described embodiments and modifications may also be implemented in the form of a semiconductor device (e.g., a semiconductor chip) as a processing system 1,2001 having at least one processor 9b and one memory 9a in the control system 9,2009.

[0120] (Additional note) This specification discloses several technical concepts and several combinations thereof, as listed below.

[0121] (Technical thought 1) A processing system having a processor (9b) that performs XR display-related processing related to displaying XR images to a moving user, The aforementioned processor, The user's field of view (Av) is monitored through a wearable device (Wt) worn by the user. The system is configured to adjust the occupancy rate (Rc) of the display target range (Ad) of the content image (20) superimposed on the field of view as an XR image corresponding to the desired content (Cd) requested by the user, according to the risk level (Lr) in the field of view. Adjusting the aforementioned occupancy rate means To acquire external sensing information (Io) sensed by an autonomous driving device (Ma) that guides the user as they move toward the aforementioned field of view area, Recognizing the risk level based on the aforementioned external sensing information, A processing system that includes adjusting the occupancy rate by prohibiting the superimposition of the content image in the hazard space portion (Aoh) of the external sensing space (Ao) which is the range of acquisition of the external sensing information within the field of view, where the risk level has reached a hazard level that requires hazard notification to the moving user.

[0122] (Technical thought 2) Adjusting the aforementioned occupancy rate means A processing system according to technical idea 1, which includes superimposing a limitation image (20cl) as an XR image representing the limit range (Aol) on the ground (Mc) of the external sensing space onto the field of view area.

[0123] (Technical Thought 3) Adjusting the aforementioned occupancy rate means The internal sensing information (Ii) sensed by the aforementioned wearable device is acquired, Recognizing the risk level based on the aforementioned internal sensing information, A processing system according to technical idea 1 or 2, which includes adjusting the occupancy rate by prohibiting the superimposition of the content image over the entire field of view when the risk level in the internal sensing space (Ai), which is the range for acquiring the internal sensing information, reaches a hazard level that requires hazard notification to the moving user.

[0124] (Technical Thought 4) A processing system having a processor (9b) that performs XR display-related processing related to displaying XR images to a moving user, The aforementioned processor, The background area (Ab) that is displayed to the user from the mobile device (Mt) carried by the user is monitored. The system is configured to perform the following actions: adjust the percentage (Rc) of the display area (Ad) of a content image (20) that is superimposed on a background video (24) showing the background area as an XR image corresponding to the desired content (Cd) requested by the user, in accordance with the risk level (Lr) in the background area. Adjusting the aforementioned occupancy rate means To acquire external sensing information (Io) sensed by an autonomous driving device (Ma) that guides the user moving toward the aforementioned background area, Recognizing the risk level based on the aforementioned external sensing information, A processing system that includes adjusting the occupancy rate by prohibiting the superimposition of the content image on the video portion (24h) of the background video that shows the external sensing space (Ao) which is within the range of acquisition of the external sensing information in the background area, where the risk level has reached a hazard level requiring hazard notification to the moving user.

[0125] (Technical Thought 5) Adjusting the aforementioned occupancy rate means A processing system according to technical idea 4, which includes superimposing a limitation image (20cl) as an XR image representing the limit range (Aol) on the ground (Mc) of the external sensing space onto the background video.

[0126] (Technical Thought 6) Adjusting the aforementioned occupancy rate means The acquisition of internal sensing information (Ii) sensed by the aforementioned mobile terminal, Recognizing the risk level based on the aforementioned internal sensing information, A processing system according to technical idea 4 or 5, which includes adjusting the occupancy rate by prohibiting the superimposition of the content image over the entire background video when the risk level in the internal sensing space (Ai) within the background area, which is the range for acquiring the internal sensing information, reaches a hazard level that requires hazard notification to the moving user.

[0127] (Technical Thought 7) Adjusting the aforementioned occupancy rate means A processing system according to any one of the technical concepts 1 to 6, which includes acquiring the external sensing information collected by the infrastructure system (6).

[0128] Furthermore, the technical concepts 1 to 7 described above may also be understood within the respective technical concepts of the methods and programs. [Explanation of symbols]

[0129] 1: 2001: Processing system, 6: Infrastructure system, 9a: Memory, 9b: Processor, 20: Content image, 20a: Content image, 20cl: Limitation image, 24: Background video, 24h: Video portion, Ab: Background area, Ad: Display target range, Ai: Internal sensing space, Ao: External sensing space, Aoh: Hazard space area, Aol: Limit range, Av: Field of view area, Cd: Desired content, Ii: Internal sensing information, Io: External sensing information, Lr: Risk level, Ma: Autonomous driving device, Mc: Ground, Mt: Mobile terminal, Rc: Occupancy rate, Wt: Wearable terminal

Claims

1. A processing system having a processor (9b) that performs XR display-related processing related to displaying XR images to a moving user, The aforementioned processor, The user's field of view (Av) is monitored through a wearable device (Wt) worn by the user. The system is configured to adjust the occupancy rate (Rc) of the display target range (Ad) of the content image (20) superimposed on the field of view as an XR image corresponding to the desired content (Cd) requested by the user, according to the risk level (Lr) in the field of view. Adjusting the aforementioned occupancy rate means To acquire external sensing information (Io) sensed by an autonomous driving device (Ma) that guides the user as they move toward the aforementioned field of view area, Recognizing the risk level based on the aforementioned external sensing information, A processing system that includes adjusting the occupancy rate by prohibiting the superimposition of the content image in the hazard space portion (Aoh) of the external sensing space (Ao) which is the range of acquisition of the external sensing information within the field of view, where the risk level has reached a hazard level that requires hazard notification to the moving user.

2. Adjusting the aforementioned occupancy rate means The processing system according to claim 1, further comprising superimposing a limitation image (20cl) as an XR image representing the limit range (Aol) on the ground (Mc) of the external sensing space onto the field of view area.

3. Adjusting the aforementioned occupancy rate means The internal sensing information (Ii) sensed by the wearable device is acquired, Recognizing the risk level based on the aforementioned internal sensing information, The processing system according to claim 1, further comprising: adjusting the occupancy rate by prohibiting the superimposition of the content image over the entire field of view when the risk level in the internal sensing space (Ai), which is the range for acquiring the internal sensing information, reaches a hazard level that requires hazard notification to the moving user.

4. A processing system having a processor (9b) that performs XR display-related processing related to displaying XR images to a moving user, The aforementioned processor, The background area (Ab) that serves as the display background for the user, from the mobile terminal (Mt) carried by the user, The system is configured to adjust the percentage (Rc) of the display target area (Ad) of a content image (20) superimposed on a background video (24) showing the background area as an XR image corresponding to the desired content (Cd) requested by the user, in accordance with the risk level (Lr) in the background area. Adjusting the aforementioned occupancy rate means To acquire external sensing information (Io) sensed by an autonomous driving device (Ma) that guides the user moving toward the aforementioned background area, Recognizing the risk level based on the aforementioned external sensing information, A processing system that includes adjusting the occupancy rate by prohibiting the superimposition of the content image on the video portion (24h) of the background video that shows the external sensing space (Ao) which is within the range of acquisition of the external sensing information in the background area, where the risk level has reached a hazard level requiring hazard notification to the moving user.

5. Adjusting the aforementioned occupancy rate means The processing system according to claim 4, further comprising superimposing a limitation image (20cl) as an XR image representing the limit range (Aol) on the ground (Mc) of the external sensing space onto the background video.

6. Adjusting the aforementioned occupancy rate means The internal sensing information (Ii) sensed by the aforementioned mobile terminal is acquired, Recognizing the risk level based on the aforementioned internal sensing information, The processing system according to claim 4, further comprising: adjusting the occupancy rate by prohibiting the superimposition of the content image over the entire background video when the risk level in the internal sensing space (Ai) within the background area, which is the range for acquiring the internal sensing information, reaches a hazard level that requires hazard notification to the moving user.

7. Adjusting the aforementioned occupancy rate means The processing system according to claim 1 or 4, further comprising acquiring the external sensing information collected by the infrastructure system (6).

8. A processing method performed by a processor (9b) to carry out XR display-related processing related to displaying XR images to a moving user, The user's field of view (Av) is monitored through a wearable device (Wt) worn by the user. This includes adjusting the percentage (Rc) of the display target range (Ad) of the content image (20) that is superimposed on the field of view as an XR image corresponding to the desired content (Cd) requested by the user, within the field of view, according to the risk level (Lr) in the field of view. Adjusting the aforementioned occupancy rate means To acquire external sensing information (Io) sensed by an autonomous driving device (Ma) that guides the user as they move toward the aforementioned field of view area, Recognizing the risk level based on the aforementioned external sensing information, A processing method that includes adjusting the occupancy rate by prohibiting the superimposition of the content image in the hazard space portion (Aoh) of the external sensing space (Ao) which is the range of acquisition of the external sensing information within the field of view, where the risk level has reached a hazard level that requires hazard notification to the moving user.

9. A processing method performed by a processor (9b) to carry out XR display-related processing related to displaying XR images to a moving user, The background area (Ab) that serves as the display background for the user, from the mobile terminal (Mt) carried by the user, This includes adjusting the percentage (Rc) of the display target area (Ad) of a content image (20) that is superimposed on a background video (24) showing the background area as an XR image corresponding to the desired content (Cd) requested by the user, in accordance with the risk level (Lr) in the background area. Adjusting the aforementioned occupancy rate means To acquire external sensing information (Io) sensed by an autonomous driving device (Ma) that guides the user moving toward the aforementioned background area, Recognizing the risk level based on the aforementioned external sensing information, A processing method that includes adjusting the occupancy rate by prohibiting the superimposition of the content image on the video portion (24h) of the background video that shows the external sensing space (Ao) which is within the range of acquisition of the external sensing information in the background area, where the risk level has reached a hazard level requiring hazard notification to the moving user.

10. A processing program that includes instructions to be executed by a processor (9b) and is stored in a storage medium (9a) for performing XR display-related processing related to displaying XR images to a moving user, The user's field of view (Av) is monitored through a wearable device (Wt) worn by the user. The command includes causing the user to perform the following actions: adjust the occupancy rate (Rc) of the display target range (Ad) of the content image (20) that is superimposed on the field of view as an XR image corresponding to the desired content (Cd) requested by the user, in accordance with the risk level (Lr) in the field of view. Adjusting the aforementioned occupancy rate means To acquire external sensing information (Io) sensed by an autonomous driving device (Ma) that guides the user as they move toward the aforementioned field of view area, Recognizing the risk level based on the aforementioned external sensing information, A processing program that includes adjusting the occupancy rate by prohibiting the superimposition of the content image in the hazard space portion (Aoh) of the external sensing space (Ao) which is the range of acquisition of the external sensing information within the field of view, where the risk level has reached a hazard level that requires hazard notification to the moving user.

11. A processing program that includes instructions to be executed by a processor (9b) and is stored in a storage medium (9a) for performing XR display-related processing related to displaying XR images to a moving user, The background area (Ab) that serves as the display background for the user, from the mobile terminal (Mt) carried by the user, The command includes the following: adjusting the percentage (Rc) of the display target area (Ad) of a content image (20) that is superimposed on a background video (24) showing the background area as an XR image corresponding to the desired content (Cd) requested by the user, in accordance with the risk level (Lr) in the background area. Adjusting the aforementioned occupancy rate means To acquire external sensing information (Io) sensed by an autonomous driving device (Ma) that guides the user moving toward the aforementioned background area, Recognizing the risk level based on the aforementioned external sensing information, A processing program that includes adjusting the occupancy rate by prohibiting the superimposition of the content image on the video portion (24h) of the background video that shows the external sensing space (Ao) which is within the range of acquisition of the external sensing information in the background area, where the risk level has reached a hazard level requiring hazard notification to the moving user.

Images