Processing system, processing method, and processing program

Abstract

To provide a processing system which balances safety of a moving user and continuity of virtual experience.SOLUTION: A processor of a processing system executes: monitoring a visibility area Av viewed by a user via a wearable terminal; and adjusting an occupancy Rc of a display target range Ad of a content image occupying the visibility area Av, the content image being superimposition-displayed on the visibility area as an XR image corresponding to a desired content of the user, according to a risk level Lr in the visibility area Av. The adjustment of the occupancy Rc includes: acquiring outer sensing information Io from an autonomously running device which leads the user; recognizing the risk level Lr based on the outer sensing information Io; and prohibiting a superimposition-display of the content image on a hazard space part Aoh having the risk level reaching a hazard level of an outer sensing space Ao which is an acquisition range for acquiring outer sensing information Ii in the visibility area Av, thereby adjusting the occupancy Rc.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to a processing technique for performing XR display-related processing related to the display of XR images to a user. [Background technology]

[0002] The technology disclosed in Patent Document 1 enables a user to have a virtual experience by displaying a VR video as an XR image on a head mounted display (HMD). In particular, this technology makes the display of the HMD transparent when the user's movement exceeds a limit position and the HMD detects a collision risk in the physical environment. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 10-504917 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the technology disclosed in Patent Document 1, the movement of the user wearing the HMD is used as a trigger, and the collision risk limit position is set based on the user's position, so the range in which safety can be ensured by display transparency is limited to the user's proximity range. As a result, in the technology disclosed in Patent Document 1, in which the VR image is completely lost by display transparency, the continuity of the virtual experience is hindered, especially when applied to a moving user.

[0005] An object of the present disclosure is to provide a processing system that performs XR display-related processing while balancing the safety of a moving user and the continuity of a virtual experience. Another object of the present disclosure is to provide a processing method that performs XR display-related processing while balancing the safety of a moving user and the continuity of a virtual experience. Yet another object of the present disclosure is to provide a processing program that performs XR display-related processing while balancing the safety of a moving user and the continuity of a virtual experience. [Means for solving the problem]

[0006] The technical means of the present disclosure for solving the problems will be described below. Note that the claims and the reference characters in parentheses in this section indicate the corresponding relationship with the specific means described in the embodiments described later in detail, and do not limit the technical scope of the present disclosure.

[0007] A first aspect of the present disclosure is A processing system having a processor (9b) for performing XR display-related processing related to displaying XR images to a moving user, The processor Monitoring a visual field (Av) viewed by a user through a wearable terminal (Wt) worn by the user; The system is configured to execute the steps of: adjusting the occupancy rate (Rc) of the display target range (Ad) of a content image (20) to be superimposed on the viewing area as an XR image corresponding to a desired content (Cd) desired by the user in accordance with a risk level (Lr) in the viewing area; Adjusting the occupancy rate is Acquiring external sensing information (Io) sensed by an autonomous driving device (Ma) leading a user moving toward a visual field area; Recognizing a risk level based on external sensing information; This includes adjusting the occupancy rate by prohibiting the superimposition of content images in a hazard space portion (Aoh) of the external sensing space (Ao), which is the range within the field of view where external sensing information is acquired, where the risk level has reached a hazard level that requires hazard notification to moving users.

[0008] A second aspect of the present disclosure is A processing method executed by a processor (9b) for performing an XR display-related process related to displaying an XR image to a moving user, the processing method comprising: Monitoring a visual field (Av) viewed by a user through a wearable terminal (Wt) worn by the user; The method includes adjusting the occupancy rate (Rc) of the display target range (Ad) of a content image (20) that is superimposed on the field of view as an XR image corresponding to a desired content (Cd) desired by the user in the field of view area in accordance with a risk level (Lr) in the field of view, Adjusting the occupancy rate is Acquiring external sensing information (Io) sensed by an autonomous driving device (Ma) leading a user moving toward a visual field area; Recognizing a risk level based on external sensing information; This includes adjusting the occupancy rate by prohibiting the superimposition of content images in a hazard space portion (Aoh) of the external sensing space (Ao), which is the range within the field of view where external sensing information is acquired, where the risk level has reached a hazard level that requires hazard notification to moving users.

[0009] A third aspect of the present disclosure is A processing program stored in a storage medium (9a) for performing an XR display-related process related to displaying an XR image to a moving user, the processing program including instructions to be executed by a processor (9b), Monitoring a visual field (Av) viewed by a user through a wearable terminal (Wt) worn by the user; and adjusting the occupancy rate (Rc) of the display target range (Ad) of a content image (20) to be superimposed on the field of view as an XR image corresponding to a desired content (Cd) desired by the user in accordance with a risk level (Lr) in the field of view, Adjusting the occupancy rate is Acquiring external sensing information (Io) sensed by an autonomous driving device (Ma) leading a user moving toward a visual field area; Recognizing a risk level based on external sensing information; This includes adjusting the occupancy rate by prohibiting the superimposition of content images in a hazard space portion (Aoh) of the external sensing space (Ao), which is the range within the field of view where external sensing information is acquired, where the risk level has reached a hazard level that requires hazard notification to moving users.

[0010] In this way, in the first to third aspects, the field of view area viewed by the user through the wearable device worn by the user is monitored. The occupancy rate of the display target range of the content image superimposed on the field of view area as an XR image according to the desired content desired by the user is adjusted according to the risk level in the field of view area.

[0011] In this case, particularly according to the first to third aspects, external sensing information sensed by an autonomous driving device leading a user moving toward a visual field is acquired, and a risk level based on the external sensing information is recognized. In this way, by effectively utilizing the autonomous driving device fulfilling the leading function of the user, a risk level can be recognized for as wide a range as possible of the visual field to which the user is moving, and the safety of the user can be ensured.

[0012] Moreover, according to the first to third aspects, in the external sensing space that is the acquisition range of the external sensing information in the field of view area, the superimposed display of the content image is prohibited in the hazard space portion where the risk level reaches the hazard level. This allows the occupancy rate to be adjusted so that the content image disappears only within the hazard space portion where the hazard level requires a hazard notification to the moving user, while the superimposed display of the content image continues outside the hazard space portion in the field of view area. Therefore, it is possible to balance the safety ensured by the above-mentioned principle for the moving user with the continuity of the virtual experience.

[0013] A fourth aspect of the present disclosure is A processing system having a processor (9b) for performing XR display-related processing related to displaying XR images to a moving user, The processor Monitoring a background area (Ab) that is a display background to the user from a mobile terminal (Mt) carried by the user; The apparatus is configured to execute the steps of: adjusting an occupancy rate (Rc) of a display target range (Ad) of a content image (20) to be superimposed on a background video (24) showing the background area as an XR image according to a desired content (Cd) desired by a user, in accordance with a risk level (Lr) in the background area; Adjusting the occupancy rate is Acquiring external sensing information (Io) sensed by an autonomous mobile device (Ma) leading a user moving toward a background area; Recognizing a risk level based on external sensing information; This includes adjusting the occupancy rate by prohibiting the superimposition of a content image on a video portion (24h) showing a hazard space portion (Aoh) where the risk level has reached a hazard level requiring a hazard notification to moving users, out of a background video showing an external sensing space (Ao) which is the acquisition range of external sensing information in the background area.

[0014] A fifth aspect of the present disclosure is a method for manufacturing a semiconductor device comprising: A processing method executed by a processor (9b) for performing an XR display-related process related to displaying an XR image to a moving user, the processing method comprising: Monitoring a background area (Ab) that is a display background to the user from a mobile terminal (Mt) carried by the user; and adjusting an occupancy rate (Rc) of a display target range (Ad) of a content image (20) to be superimposed on a background video (24) showing the background area as an XR image according to a desired content (Cd) desired by a user in accordance with a risk level (Lr) in the background area; Adjusting the occupancy rate is Acquiring external sensing information (Io) sensed by an autonomous mobile device (Ma) leading a user moving toward a background area; Recognizing a risk level based on external sensing information; This includes adjusting the occupancy rate by prohibiting the superimposition of a content image on a video portion (24h) showing a hazard space portion (Aoh) where the risk level has reached a hazard level requiring a hazard notification to moving users, out of a background video showing an external sensing space (Ao) which is the acquisition range of external sensing information in the background area.

[0015] A sixth aspect of the present disclosure is a method for manufacturing a semiconductor device comprising: A processing program stored in a storage medium (9a) for performing an XR display-related process related to displaying an XR image to a moving user, the processing program including instructions to be executed by a processor (9b), Monitoring a background area (Ab) that is a display background to the user from a mobile terminal (Mt) carried by the user; adjusting an occupancy rate (Rc) of a display target range (Ad) of a content image (20) to be superimposed on a background video (24) showing the background area as an XR image according to a desired content (Cd) desired by a user, in accordance with a risk level (Lr) in the background area; Adjusting the occupancy rate is Acquiring external sensing information (Io) sensed by an autonomous mobile device (Ma) leading a user moving toward a background area; Recognizing a risk level based on external sensing information; This includes adjusting the occupancy rate by prohibiting the superimposition of a content image on a video portion (24h) showing a hazard space portion (Aoh) where the risk level has reached a hazard level requiring a hazard notification to moving users, out of a background video showing an external sensing space (Ao) which is the acquisition range of external sensing information in the background area.

[0016] In this way, in the fourth to sixth aspects, the background area that is displayed to the user from the mobile terminal carried by the user is monitored. The occupancy rate of the display target range of the content image that is superimposed on the background video showing the background area as an XR image according to the desired content desired by the user in the background video is adjusted according to the risk level in the background area.

[0017] In this case, particularly according to the fourth to sixth aspects, external sensing information sensed by an autonomous mobile device leading a user moving toward a background area is acquired, and a risk level based on the external sensing information is recognized. In this way, by effectively utilizing the autonomous mobile device fulfilling the leading function of the user, a risk level can be recognized for as wide a range as possible of the background area to which the user is moving, and the safety of the user can be ensured.

[0018] Moreover, according to the fourth to sixth aspects, in the background video showing the external sensing space that is the acquisition range of the external sensing information in the background area, the content image is prohibited from being superimposed on the video portion showing the hazard space portion where the risk level reaches the hazard level. As a result, the content image disappears only in the portion of the background video that corresponds to the inside of the hazard space portion where the hazard level requires a hazard notification to the moving user, while the occupancy rate can be adjusted so that the content image continues to be superimposed on the portion of the background area that corresponds to the outside of the hazard space portion. Therefore, it is possible to balance the safety ensured by the above-mentioned principle for the moving user with the continuity of the virtual experience. [Brief description of the drawings]

[0019] [Figure 1] 1 is an overall configuration diagram showing a network connection environment of a processing system according to a first embodiment. [Diagram 2] FIG. 1 is a schematic diagram showing a wearable terminal according to a first embodiment. [Diagram 3] 2 is a schematic diagram for explaining a three-dimensional voxel assumed in the infrastructure system according to the first embodiment. FIG. [Figure 4] 1 is a block diagram showing a processing system according to a first embodiment; [Diagram 5] 4 is a flowchart showing a processing flow according to the first embodiment. [Figure 6] 10 is a flowchart showing an adjustment subroutine in the processing flow according to the first embodiment. [Figure 7] FIG. 2 is a schematic diagram showing a display state of the wearable terminal according to the first embodiment. [Figure 8] FIG. 2 is a schematic diagram showing a display state of the wearable terminal according to the first embodiment. [Figure 9] FIG. 2 is a schematic diagram showing a display state of the wearable terminal according to the first embodiment. [Figure 10] FIG. 5 is a schematic diagram for explaining an adjustment subroutine according to the first embodiment. [Figure 11]FIG. 5 is a schematic diagram for explaining an adjustment subroutine according to the first embodiment. [Figure 12] FIG. 11 is an overall configuration diagram showing a network connection environment of a processing system according to a second embodiment. [Figure 13] FIG. 11 is a schematic diagram showing a mobile terminal according to a second embodiment. [Figure 14] FIG. 11 is a block diagram showing a processing system according to a second embodiment. [Figure 15] 10 is a flowchart showing a processing flow according to a second embodiment. [Figure 16] 13 is a flowchart showing an adjustment subroutine in the processing flow according to the second embodiment. [Figure 17] FIG. 11 is a schematic diagram showing a display state of a mobile terminal according to a second embodiment. [Figure 18] FIG. 11 is a schematic diagram showing a display state of a mobile terminal according to a second embodiment. [Figure 19] FIG. 11 is a schematic diagram showing a display state of a mobile terminal according to a second embodiment. [Figure 20] FIG. 10 is a schematic diagram for explaining an adjustment subroutine according to the second embodiment. [Figure 21] FIG. 10 is a schematic diagram for explaining an adjustment subroutine according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, multiple embodiments of the present disclosure will be described with reference to the drawings. In addition, by assigning the same reference numerals to corresponding components in each embodiment, duplicated descriptions may be omitted. In addition, when only a part of the configuration is described in each embodiment, the configuration of the other embodiment described above can be applied to the other parts of the configuration. Furthermore, in addition to the combination of configurations explicitly stated in the description of each embodiment, configurations of multiple embodiments can be partially combined together even if not explicitly stated, as long as there is no particular problem with the combination.

[0021] First embodiment 1 performs XR display-related processing related to displaying XR images to a moving user Us in association with providing a user service to the user Us. To this end, the processing system 1 is connected to a wearable device Wt worn by a pre-registered user Us, an autonomous mobile device Ma, and an infrastructure system 6 via a communication network Nc. It is assumed that at least one system each of the wearable device Wt, the autonomous mobile device Ma, and the infrastructure system 6 is connected to the processing system 1.

[0022] The wearable terminal Wt is configured so that the user Us can wear it on their face and can operate it hands-free at least when connected to the processing system 1. As shown in Fig. 2, the wearable terminal Wt is an optical see-through electronic device equipped with a display unit 2 and a sensor unit 3, such as an HMD or smart glasses. Note that Fig. 2 shows a schematic image of how the visual field area Av of the wearable terminal Wt appears from the user Us's side in a portion corresponding to one of both eyes of the user Us.

[0023] The wearable terminal Wt controls the transmission and reception of data via the communication network Nc by a communication unit as shown in Fig. 1. The wearable terminal Wt is operated and controlled by the control unit according to a control command from the processing system 1 via the communication network Nc so as to perform XR display-related processing in cooperation with the processing system 1.

[0024] Under the operational control of the control unit of the wearable terminal Wt, the display unit 2 shown in FIG. 2, such as a virtual image projection type or a retinal projection type, superimposes an XR image (see FIGS. 7 to 9 described later) required for XR display-related processing on a real image in the field of view Av viewed by the user Us through glasses or lenses, and allows the user Us to view it. At the same time, under the operational control of the wearable terminal Wt, the sensor unit 3 shown in FIG. 2 acquires sensing information by a sensing process according to the XR display-related processing. As the sensor unit 3, at least one of a camera, an inertial sensor, a GNSS sensor, an electrooculography sensor, a gaze sensor, an infrared sensor, a geomagnetic sensor, a motion sensor, a touch sensor, and a microphone is adopted. With this configuration, the sensor unit 3 can acquire an input representing the intention of the user Us as sensing information.

[0025] The autonomous driving device Ma shown in FIG. 1 is an autonomous vehicle or an autonomous driving robot that can autonomously travel in any direction, forward, backward, left or right, by electrically driving wheels 5 based on sensing information from a sensor unit 4. The autonomous driving device Ma controls the transmission and reception of data via a communication network Nc by a communication unit. The autonomous driving device Ma is driven and controlled by a control unit according to a control command from the processing system 1 via the communication network Nc so as to perform XR display-related processing in cooperation with the processing system 1. Under this drive control, the sensor unit 4 acquires sensing information by sensing processing according to the XR display-related processing. As the sensor unit 4, at least one of a camera, an inertial sensor, a GNSS sensor, a LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging), a sonar, etc. is adopted.

[0026] The infrastructure system 6 is a base system that shares the three-dimensional spatial information It stored in the infrastructure database Di as a common foundation for the distributed architecture. The infrastructure system 6 controls the transmission and reception of data via the communication network Nc using a communication unit. The infrastructure system 6 collects the three-dimensional spatial information It, which is provided to individual distributed systems such as the processing system 1 constituting the distributed architecture, as needed using a control unit, and updates the information stored in the infrastructure database Di to the latest information.

[0027] In particular, the infrastructure system 6 manages the information by virtually dividing the three-dimensional space to be stored into a plurality of three-dimensional voxels Vi (i.e., a three-dimensional grid) assumed to be arranged in a three-dimensional array as shown in Fig. 3. The infrastructure system 6 stores a data set linked by metadata for each space ID individually assigned to each voxel Vi in an infrastructure database Di as three-dimensional spatial information It as shown in Fig. 1. Here, the three-dimensional spatial information It may of course include two-dimensional grid information linked only to, for example, the bottom surface of the voxel Vi constituting the lowest layer of the two-dimensional array along the ground in the three-dimensional array.

[0028] The data constituting the three-dimensional space information It may be collected from at least one of the wearable terminal Wt cooperating with the processing system 1, other wearable terminals, and mobile terminals such as smartphones or tablet terminals. The data constituting the three-dimensional space information It may be collected from at least one of the autonomous driving device Ma cooperating with the processing system 1, and other moving bodies.

[0029] The data constituting the three-dimensional spatial information It may be collected from at least one of, for example, a communication base station, a smart pole, and a smart street light, which are equipped with infrastructure sensors such as a camera and / or LiDAR. The data constituting the three-dimensional spatial information It may be collected from, for example, a service provider providing at least one of a map service, a weather service, a communication service, a traffic management service, a feature management service, and an aviation management service, as well as user services handled by the processing system 1, which will be described later.

[0030] The construction data collected in this manner to construct the three-dimensional space information It may be at least one type of image data, for example, video, still images, point cloud images, etc. The construction data of the three-dimensional space information It may be at least one type of secondary data, for which information security is guaranteed, for example, position data, movement data, posture data, gaze data, behavior data, and people flow data generated about humans including the user Us of the processing system 1 in the target space for information accumulation by image processing of such image data, and intention data (described later) that indicates the intention of the user Us.

[0031] The construction data of the three-dimensional space information It may be at least one type of secondary data, such as position data, movement data, and posture data, generated by image processing of image data regarding human belongings including the wearable terminal Wt of the user Us in the information accumulation target space. The construction data of the three-dimensional space information It may be at least one type of secondary data, such as position data, movement data, and posture data, generated by image processing of image data regarding a moving body including the autonomous driving device Ma in the information accumulation target space.

[0032] The construction data of the three-dimensional space information It may be voice data obtained by collecting voices of people including the user Us of the processing system 1 in the information storage target space. The construction data of the three-dimensional space information It may be at least one type of secondary data for which information security is guaranteed, such as position data, movement data, behavior data, people flow data, and conversation data analyzed regarding people including the user Us by voice recognition processing of such voice data, and intention data (described later) that indicates the intention of the user Us.

[0033] The construction data of the three-dimensional space information It may be at least one of, for example, two-dimensional and / or three-dimensional map data, GIS (Geographic Information System) data, road network data, weather data, communication data, line data, traffic data, feature management data, BIM (Building Information Modelling) data, POI (Point of Interest) data, aviation management data, and time data. The construction data of the three-dimensional space information It may be service data related to at least one of, for example, a guidance service, a transportation service, a photography service, and an online game service, which will be described later as user services handled by the processing system 1.

[0034] 1, the processing system 1 is a distributed computer system equipped with a communication system 10 and a control system 9, and constructed to include at least one of, for example, a cloud server and an edge server. At least a portion of each of the systems 10 and 9 in the processing system 1 may be configured by a communication unit and a control unit of the wearable terminal Wt. At least a portion of each of the systems 10 and 9 in the processing system 1 may be configured by a communication unit and a control unit of the autonomous mobile device Ma. At least a portion of each of the systems 10 and 9 in the processing system 1 may be configured by a communication unit and a control unit of the infrastructure system 6.

[0035] The communication system 10 is mainly composed of communication devices for constructing the communication network Nc. The control system 9 is connected to the communication system 10 via at least one of a wired communication line and a wireless communication line. The control system 9 is constructed to include at least one dedicated computer. The dedicated computer that constructs the control system 9 has at least one memory 9a and one processor 9b.

[0036] In the processing system 1, the control system 9 executes multiple commands of a processing program stored in the memory 9a using the processor 9b. This causes the control system 9 to construct multiple functional blocks for performing XR display-related processing. The constructed functional blocks include a recognition block 200, a drive control block 210, and a display control block 220, as shown in FIG.

[0037] The processing method in which the processing system 1 performs XR display-related processing through cooperation between these blocks 200, 210, and 220 is executed according to the processing flow shown in Figures 5 and 6. This processing flow is executed in response to the sensor unit 3 of the wearable terminal Wt acquiring, as sensing information, a service request input that the user Us makes with the intention of requesting the provision of a user service. Note that each "S" in this processing flow refers to multiple steps executed by multiple commands included in the processing program of the first embodiment.

[0038] In S10 of the process flow shown in FIG. 5, the recognition block 200 (see FIG. 4) monitors the field of view area Av visually recognized by the user Us through the wearable terminal Wt. At this time, the monitoring of the field of view area Av is based on at least one type of sensing information, for example, camera information and inertial information, by the sensor unit 3 of the wearable terminal Wt. As a result, the field of view area Av is recognized as an area extending in the direction of the face or gaze of the user Us, or in the direction of the wearable terminal Wt. Note that, during the execution of each step following S10 of the process flow, the latest recognized field of view area Av may be updated by executing a monitoring process of the field of view area Av in parallel as the field of view area Av changes due to the movement of the user Us.

[0039] 5, the recognition block 200 acquires a desired content Cd that the user Us intends and desires. At this time, the desired content Cd is recognized based on sensing information about the face by the wearable terminal Wt worn by the user Us on the face. Therefore, the sensing information for recognizing the desired content Cd is acquired through the wearable terminal Wt at the timing when the sensing information is input to the sensor unit 3 as the intention data of the user Us.

[0040] As the input of the desired content Cd in S20, a gesture input, a gaze input, an expression input, or a voice input by the user Us may be sensed by the sensor unit 3. At this time, in the display unit 2 of the wearable terminal Wt, an XR image for receiving the input of the desired content Cd from the user Us by the sensor unit 3 may be superimposed and displayed on the viewing area Av through the wearable terminal Wt. Note that the recognition of the desired content Cd may be realized based on three-dimensional space information It acquired from the infrastructure database Di via the communication system 10.

[0041] The items of the desired content Cd recognized in S20 include at least a display theme that is given to an XR image (image 20 in Figs. 7 and 8 described later) displayed by the display unit 2 of the wearable terminal Wt in association with the provision of a user service to the user Us. The display theme refers to a concept common to XR images that are displayed in two or three dimensions, such as virtual or real living things, such as monsters, animals, plants, and insects, as well as inanimate objects, such as prehistoric buildings, service vehicles, and imaginary objects.

[0042] The items of the desired content Cd recognized in S20 may include the types of user services that can be provided to the user Us by using the autonomous mobile device Ma. As the types of user services, at least one of a guidance service, a transportation service, a photography service, an online game service, and the like is provided.

[0043] In S20, for any user service, the desired content Cd may include at least one route-related item, for example, a route including a destination, a moving pace, an arrival time, and a service provision time (i.e., a guide time or a transport time), etc. In S20, for any user service, the desired content Cd may include at least one voice-related item, for example, a voice output requirement, a voice recognition function requirement, and a dialogue function requirement.

[0044] In the case of a guidance service in S20, the desired content Cd may include at least one of the following tourist-related items with respect to a destination such as a tourist spot: an acceptable degree of congestion, the presence or absence of a desired facility, the presence or absence of nature, etc. In the case of a transportation service in S20, the desired content Cd may include at least one of the following luggage-related items: the number, size, weight, type, and the need for temperature control, etc.

[0045] In the case of a photography service in S20, the desired content Cd may include at least one photography-related item among, for example, a photography schedule, photography timing, photography location, whether or not to transmit images or data, whether or not to print photos, etc. In the case of an online game service in S20, the desired content Cd may include at least one game-related item among, for example, initial settings of a game character, environmental settings of an online game, difficulty level of an online game, whether or not to transmit images or data, whether or not to play a game, etc.

[0046] In S30 of the process flow shown in Fig. 5, the drive control block 210 (see Fig. 4) provides a user service by controlling the drive of the autonomous mobile device Ma within the visual field area Av. At this time, a drive control pattern for the autonomous mobile device Ma that matches the type of user service is read from the memory 9a, and the user service is provided according to the drive control pattern so as to lead the user Us moving toward the visual field area Av.

[0047] The guidance service by S30 is provided according to a drive control pattern of route guidance that drives the autonomous mobile device Ma from a standby position to a position around the user Us and leads the user Us to the destination. In this guidance service, the route for guiding the user Us may be presented by superimposing an XR image on the viewing area Av of the wearable terminal Wt or by display output from a display unit possessed by the autonomous mobile device Ma. In the guidance service, the route for guiding the user Us may be presented by audio output from an audio unit possessed by the wearable terminal Wt or the autonomous mobile device Ma.

[0048] The transport service by S30 is provided according to a drive control pattern in which the autonomous mobile device Ma having a loading compartment is driven from a waiting position to a position around the user Us, and the loaded luggage in the loading compartment is transported to the destination while leading the user Us. In this case, in the transport service, a method for loading luggage into the loading compartment may be presented to the user Us by superimposing an XR image on the viewing area Av of the wearable terminal Wt, or by display output from a display unit possessed by the autonomous mobile device Ma. In the transport service, the loading of luggage into the loading compartment may be presented to the user Us by audio output from an audio unit possessed by the wearable terminal Wt or the autonomous mobile device Ma.

[0049] The photographing service by S30 is provided according to a drive control pattern in which the autonomous mobile device Ma having an imaging unit is driven from a standby position to a position around the user Us and photographs the surroundings while leading the user Us. In this photographing service, the photographing schedule or photographing timing may be presented to the user Us by superimposing an XR image on the viewing area Av of the wearable terminal Wt or by display output from a display unit possessed by the autonomous mobile device Ma. In the photographing service, the photographing schedule or photographing timing may be presented to the user Us by audio output from an audio unit possessed by the wearable terminal Wt or the autonomous mobile device Ma.

[0050] Here, in the photography service, two-dimensional or three-dimensional still images or videos may be photographed so as to include the user Us by the autonomous mobile device Ma driven around the user Us. In the photography service, the focus during photography may be adjusted based on the distance to the user Us sensed by the sensor unit 4, such as LiDAR, in the autonomous mobile device Ma. In the photography service, the external light incidence direction during photography may be adjusted based on the three-dimensional space information It obtained from the infrastructure database Di, or the sensing information from the sensor unit 4, such as a camera, in the autonomous mobile device Ma. In the photography service, image data photographed of the user Us may be stored in the memory 9a of the control system 9 or a storage medium in the wearable terminal Wt.

[0051] The online game service by S30 is provided according to a drive control pattern that drives the autonomous mobile device Ma from a standby position to a position around the user Us and leads the user Us to a capture point in the online game. In this case, in the online game service, necessary instructions in the online game may be presented to the user Us by superimposing an XR image on the viewing area Av of the wearable device Wt or by display output from a display unit possessed by the autonomous mobile device Ma. In the online game service, necessary instructions in the online game may be presented to the user Us by audio output from an audio unit possessed by the wearable device Wt or the autonomous mobile device Ma.

[0052] S40 in the process flow shown in FIG. 5 is executed in parallel with S30. In S40, the display control block 220 (see FIG. 4) causes the display unit 2 of the wearable terminal Wt to ​​superimpose an XR image corresponding to the desired content Cd on the field of view Av in which the autonomous mobile device Ma providing the user service by S30 is located. At this time, the display control block 220 adjusts the occupancy rate Rc of the display target range Ad of the content image 20 of the XR image, which is superimposed and displayed according to the desired content Cd as shown in FIGS. 7 and 8, in the field of view area Av, in accordance with the risk level Lr in the area Av. Here, the occupancy rate Rc in the first embodiment means the ratio of the display area of ​​the entire display target range Ad to the two-dimensional projection area of ​​the entire field of view area Av onto, for example, glasses or lenses or a retina in the display unit 2.

[0053] The content image 20 in S40 includes at least the device image 20a shown in FIGS. 7 and 8 as an XR image superimposed on the autonomous mobile device Ma viewed in the viewing area Av as shown in FIG. 2. The superimposed display position of the device image 20a is adjusted to match the predicted viewing position of the autonomous mobile device Ma according to the drive control pattern by S30. The display position adjustment at this time is based on information acquired from the infrastructure database Di through the communication system 10 as three-dimensional space information It in which position-related information of the autonomous mobile device Ma is added to each voxel Vi in the space in which the user service is provided. As a result, an overlapping display is realized in which the device image 20a with a display theme corresponding to the desired content Cd is superimposed so as to hide the entire autonomous mobile device Ma opaquely or semi-transparently from in front of it.

[0054] The content image 20 in S40 may include an object image 20b shown in Figs. 7 and 8 as an XR image superimposed on another object Mb viewed in the field of view Av together with the autonomous mobile device Ma as shown in Fig. 2. The superimposed display position of the object image 20b is adjusted according to the predicted viewing position of the other object Mb, resulting in an overlap display similar to that of the device image 20a. The other object Mb on which the object image 20b is thus overlapped and displayed may be, for example, at least one of a human being other than the user Us of the processing system 1, a moving body other than the autonomous mobile device Ma that cooperates with the processing system 1, an aircraft, a structure, and plantings.

[0055] The content image 20 in S40 may include a ground image 20c shown in Figures 7 and 8 as an XR image superimposed on the ground Mc viewed in the visual field Av together with the autonomous mobile device Ma as shown in Figure 2. The superimposed display position of the ground image 20c is adjusted according to the predicted viewing position of the ground Mc, so that the device image 20a is similarly overlapped and displayed.

[0056] In S40, the display target range Ad of the content image 20 is set to cover the entire view area Av as shown in Fig. 7, and the occupancy rate Rc is adjusted to a maximum value (e.g., 1). In S40, the display target range Ad of the content image 20 is set to disappear from the entire view area Av as shown in Fig. 9, and the occupancy rate Rc is adjusted to a minimum value (e.g., 0). In S40, the display target range Ad of the content image 20 is set to a part of the view area Av as shown in Fig. 8, and the occupancy rate Rc is adjusted to an intermediate value between the minimum and maximum values ​​(e.g., greater than 0 and less than 1).

[0057] Furthermore, in S40, when a hazard notification to the user Us becomes necessary according to the risk level Lr, a hazard image 22 as an XR image may be superimposed by the display unit 2 on the ground Mc and / or the sky Md in the field of view area Av shown in Fig. 2, as shown in Figs. 8 and 9. The superimposed display position of the hazard image 22 may be adjusted according to the predicted viewing position of the ground Mc and / or the sky Md. The display position adjustment at this time may also be based on information acquired from the infrastructure database Di via the communication system 10 as three-dimensional space information It to which position-related information of the ground Mc and / or the sky Md is added for each voxel Vi in the space where the user service is provided.

[0058] In such S40, an adjustment subroutine that switches and adjusts the occupancy rate Rc according to the risk level Lr in the field of view Av is executed as shown in Fig. 6. Specifically, in S400 of the adjustment subroutine, the display control block 220 acquires internal sensing information Ii sensed by the sensor unit 3 of the wearable terminal Wt. At this time, the acquisition range of the internal sensing information Ii with an acceptable level of accuracy or reliability by the sensor unit 3 is set to an internal sensing space Ai that is a part of the field of view Av as shown in Fig. 10. Thus, the internal sensing information Ii acquired for the internal sensing space Ai is obtained from at least one type of the sensor unit 3, for example, a camera, a motion sensor, etc.

[0059] The internal sensing space Ai in S400 is defined by a plurality of voxels Vi whose two-dimensional or three-dimensional distance is assumed to be within a set distance, with the user Us or the wearable device Wt at the center. The set distance that determines the limit range of the internal sensing space Ai is set to a fixed value such as 5 m, depending on the specifications of the sensor unit 3 and / or the risk requirements for the processing system 1.

[0060] In S410 of the adjustment subroutine shown in Fig. 6, the display control block 220 acquires a risk level Lr in the internal sensing space Ai from within the field of view Av. At this time, the risk level Lr is defined as an index that reaches a hazard level Lh when at least a part of an obstacle Mbo, which is predicted to interfere with the movement of the user Us among other objects Mb such as humans, moving objects, or structures, relatively enters the internal sensing space Ai as shown in Fig. 10. In other words, the risk level Lr in the internal sensing space Ai is determined to be at a safe level that has escaped from the hazard level Lh when the obstacle Mbo disappears from the space Ai.

[0061] 6, the display control block 220 determines whether the risk level Lr in the internal sensing space Ai has reached a hazard level Lh that requires a hazard notification to the moving user Us. At this time, the comparison of the risk level Lr with the hazard level Lh may be performed for each voxel Vi in the user service provision space, or for each voxel group consisting of multiple voxels Vi.

[0062] If a positive determination is made in S420, in S430 of the adjustment subroutine shown in Fig. 6, the display control block 220 adjusts the occupancy rate Rc to a minimum value by prohibiting the superimposed display of the content image 20 in the entire field of view area Av as shown in Fig. 9 so as to exclude the entire field of view area Av from the display target range Ad of the content image 20. However, in S430, a hazard image 22 notifying a warning character and / or a warning color according to the risk level Lr that has reached the hazard level Lh may be superimposed and displayed on the field of view area Av as shown in Fig. 9. In this case, the hazard image 22 may be displayed so as to notify at least one of, for example, the position, direction, distance, and type of the obstacle Mbo that caused the hazard level Lh in the internal sensing space Ai.

[0063] In addition to displaying the hazard image 22, in S430, a warning of the risk level Lr that has reached the hazard level Lh may be issued by a display output from a display unit possessed by the autonomous mobile device Ma or an illumination output from a warning light. In S430, a warning of the risk level Lr that has reached the hazard level Lh may be issued by an audio output from a display unit possessed by the autonomous mobile device Ma. Completion of execution of S430 marks the end of the current execution of the adjustment subroutine and the processing flow.

[0064] If a negative determination is made in S420, in S440 of the adjustment subroutine shown in Fig. 6, the display control block 220 acquires external sensing information Io sensed by the sensor unit 4 of the autonomous mobile device Ma. At this time, the acquisition range of the external sensing information Io sensed by the sensor unit 4 with an acceptable level of accuracy or reliability is set to an external sensing space Ao that is a part of the visual field Av and has a wider range than the internal sensing space Ai, as shown in Fig. 11. Therefore, the external sensing information Io acquired for the external sensing space Ao may be obtained from at least one of the sensor units 4, such as a camera, LiDAR, and sonar.

[0065] In S440, as the external sensing information Io acquired for the external sensing space Ao, three-dimensional space information It acquired from the infrastructure database Di through the communication system 10 may be obtained together with the sensing information by the sensor unit 4. That is, the external sensing information Io may be the three-dimensional space information It collected by the infrastructure system 6 and accumulated in the infrastructure database Di.

[0066] The external sensing space Ao in S440 is defined by a plurality of voxels Vi whose two-dimensional or three-dimensional distances are assumed to be within a set range, with the autonomous mobile device Ma at the center. The set distance that determines the limit range Aol of the external sensing space Ao shown in FIG. 11 may be set to a fixed value, for example, between 6 m and 20 m, depending on the specifications of the sensor unit 4 and / or the risk requirements for the processing system 1.

[0067] In S450 of the adjustment subroutine shown in Fig. 6, the display control block 220 acquires a risk level Lr in the external sensing space Ao from within the field of view Av. At this time, the risk level Lr is defined as an index that is determined to have reached the hazard level Lh when the two-dimensional or three-dimensional separation distance δo between the limit range Aol of the external sensing space Ao and the user Us becomes equal to or less than a threshold value as shown in Fig. 11. In other words, the risk level Lr in the external sensing space Ao is determined to be at a safe level that has escaped from the hazard level Lh when the separation distance δo between the limit range Aol of the space Ao and the user Us exceeds or is equal to or greater than a threshold value.

[0068] 6, the display control block 220 determines whether the risk level Lr in the external sensing space Ao has reached a hazard level Lh that requires a hazard notification to the moving user Us. At this time, the comparison of the risk level Lr with the hazard level Lh may be performed for each voxel Vi in the user service provision space, or for each voxel group consisting of multiple voxels Vi.

[0069] If a positive determination is made in S460, in S470 of the adjustment subroutine shown in Fig. 6, the display control block 220 identifies a hazard space portion Aoh in the external sensing space Ao where the risk level Lr has reached the hazard level Lh as shown in Fig. 11. At this time, a plurality of voxels Vi in which the risk level Lr has reached the hazard level Lh are recognized as the hazard space portion Aoh. Furthermore, a display prohibited range in the field of view Av in which the superimposed display of a content image 20 described later is prohibited may be recognized as a plurality of voxels Vi present on the far side of the external sensing space Ao as the hazard space portion Aoh (see Fig. 8) so that the range continues to the far side of the external sensing space Ao.

[0070] Therefore, the display control block 220 in S470 adjusts the occupancy rate Rc to an intermediate value by restricting the hazard space portion Aoh that has reached the hazard level Lh to be removed from the display target range Ad of the content image 20, as shown in Fig. 8, and prohibiting the superimposed display of the content image 20. As a result, the remaining space portion As of the safety level that is outside the hazard space portion Aoh in the visibility area Av is set to the display target range Ad of the content image 20, and the superimposed display of the content image 20 is permitted.

[0071] In S470, a hazard image 22 notifying a user of a risk level Lr that has reached the hazard level Lh with warning text and / or a warning color may be superimposed on the field of view area Av as shown in Fig. 8. In this case, the hazard image 22 may be displayed to notify a user of an increased risk due to the user Us, who caused the hazard level Lh in the external sensing space Ao, approaching the limit range Aol.

[0072] In addition to displaying the hazard image 22, in S470, a warning of the risk level Lr that has reached the hazard level Lh may be issued by a display output from a display unit possessed by the autonomous mobile device Ma or an illumination output from a warning light. In S470, a warning of the risk level Lr that has reached the hazard level Lh may be issued by an audio output from a display unit possessed by the autonomous mobile device Ma. Completion of execution of S470 marks the end of the current execution of the adjustment subroutine and the processing flow.

[0073] If a negative determination is made in S460, in S480 of the adjustment subroutine shown in Fig. 6, the display control block 220 adjusts the occupancy rate Rc to the maximum value by permitting the content image 20 to be superimposed on the entire viewing area Av as shown in Fig. 7 so as to set the entire viewing area Av as the display target range Ad of the content image 20. At this time, the content image 20 is superimposed on the viewing area Av as a ground image 20c representing the limit range Aol on the ground Mc of the external sensing space Ao, as shown in Fig. 7. Completion of execution of S480 ends the current execution of the adjustment subroutine and the processing flow. Therefore, in the next and subsequent executions of the processing flow, the limitation image 20cl will continue to be displayed until a positive determination is made in either S420 or S460.

[0074] (Action and effect) The effects of the first embodiment described above will be described below.

[0075] In the first embodiment, a visual field area Av viewed by a user Us through a wearable device Wt worn by the user Us is monitored. The occupancy rate Rc of a display target range Ad of a content image 20 superimposed on the visual field area Av as an XR image corresponding to a desired content Cd desired by the user Us is adjusted according to a risk level Lr in the visual field area Av.

[0076] At this time, particularly according to the first embodiment, external sensing information Io sensed by the autonomous mobile device Ma leading the user Us moving toward the visual field area Av is acquired, and the risk level Lr based on the external sensing information Io is recognized. As a result, by effectively utilizing the autonomous mobile device Ma fulfilling the leading function for the user Us, the risk level Lr can be recognized for as wide a range as possible of the visual field area Av to which the user Us is moving, and the safety of the user Us can be ensured.

[0077] Moreover, according to the first embodiment, in the external sensing space Ao that is the acquisition range of the external sensing information Io in the visual field area Av, the content image 20 is prohibited from being superimposed on the hazard space portion Aoh where the risk level Lr has reached the hazard level Lh. This allows the content image 20 to disappear only within the hazard space portion Aoh of the hazard level Lh that requires a hazard notification to the moving user Us, while the occupancy rate Rc can be adjusted so that the superimposed display of the content image 20 continues outside the hazard space portion Aoh in the visual field area Av. Therefore, it is possible to balance the safety ensured by the above-mentioned principle for the moving user Us and the continuity of the virtual experience.

[0078] According to the first embodiment, a limitation image 20cl as an XR image representing a limit range Aol on the ground Mc of the external sensing space Ao is superimposed on the field of view Av. According to this, in a state in which the limit range Aol of the external sensing space Ao in which the risk level Lr is recognized is presented to the user Us, the safety of the user Us is ensured by the recognition of the risk level Lr in the external sensing space Ao, and the sense of security of the user Us can also be ensured. Therefore, coupled with the disappearance of the content image 20 limited to the hazard space portion Aoh, it is possible to ensure the safety and sense of security of the user Us in a balanced manner with the continuity of the virtual experience.

[0079] According to the first embodiment, the risk level Lr based on the internal sensing information Ii sensed by the wearable terminal Wt is recognized by acquiring the internal sensing information Ii. As a result, the risk level Lr is also recognized for the proximity range of the user Us by the wearable terminal Wt, and the safety of the user Us can be ensured. However, when the risk level Lr in the internal sensing space Ai, which is the acquisition range of the internal sensing information Ii, of the visual field area Av, reaches the hazard level Lh, the superimposed display of the content image 20 is prohibited for the entire visual field area Av. As a result, it is possible to realize an intentionally unbalanced adjustment of the occupancy rate Rc for the proximity range of the user Us so as to prioritize safety over the continuity of the virtual experience.

[0080] According to the first embodiment, the external sensing information Io collected by the infrastructure system 6 may also be acquired together with the external sensing information Io sensed by the autonomous mobile device Ma, and the risk level Lr based on the external sensing information Io may be recognized. In this way, by effectively utilizing not only the autonomous mobile device Ma that performs a leading function for the user Us, but also the infrastructure system 6, the risk level Lr for the visual field area Av may be recognized with high accuracy, making it possible to ensure the safety of the user Us.

[0081] Second Embodiment The second embodiment is a modification of the first embodiment.

[0082] 12, a processing system 2001 of the second embodiment is connected to a mobile terminal Mt carried by a pre-registered user Us, an autonomous mobile device Ma that performs autonomous traveling, and an infrastructure system 6 that provides infrastructure information, via a communication network Nc. Here, it is assumed that a plurality of mobile terminals Mt are connected to the processing system 1.

[0083] The mobile terminal Mt is configured to be held and operated by a user Us with his / her fingers at least when connected to the processing system 2001. As shown in Fig. 13, the mobile terminal Mt is a small electronic device including a display unit 2002 and a sensor unit 2003, such as a smartphone or a tablet terminal. The mobile terminal Mt controls transmission and reception of data via the communication network Nc by a communication unit as shown in Fig. 12. The mobile terminal Mt is operated and controlled by the control unit according to a control command from the processing system 2001 via the communication network Nc so as to perform service-related processing in cooperation with the processing system 2001.

[0084] Under the operational control of the control unit of the mobile terminal Mt, a display unit 2002 shown in FIG. 13, such as a liquid crystal panel or an organic EL panel, realizes a screen display (see FIGS. 17 to 19 described later) required for service-related processing. At the same time, under the operational control of the mobile terminal Mt, a sensor unit 2003 shown in FIG. 13 acquires sensing information by a sensing process corresponding to the service-related processing. As the sensor unit 2003, at least one of a camera, an inertial sensor, a GNSS (Global Navigation Satellite System) sensor, a touch sensor, a microphone, and the like is adopted. With this configuration, the sensor unit 2003 can acquire an input indicating the intention of the user Us as sensing information.

[0085] 12 is configured similarly to the processing system 1 of the first embodiment, except that the autonomous mobile device Ma shown in Fig. 12 is driven and controlled by a control unit in accordance with a control command from the processing system 2001 via a communication network Nc so as to perform service-related processing in cooperation with the processing system 2001. A control system 2009 of the processing system 2001 is configured similarly to the control system 9 of the first embodiment, except that at least a part of it may be configured by the control unit of the mobile terminal Mt, and that it executes a processing flow using the mobile terminal Mt, which will be described later.

[0086] In order to perform service-related processing in the control system 2009, the processing system 2001 performs XR display processing in cooperation with blocks 200, 210, and 220 constructed as shown in FIG. 14, according to the processing flows shown in FIGS.

[0087] In S2010 of the process flow shown in Fig. 15, the recognition block 200 (see Fig. 14) monitors a background area Ab that is displayed to the user Us from the display unit 2002 by pointing the mobile terminal Mt carried by the user Us. At this time, the background area Ab is recognized as an area spreading in the direction in which the mobile terminal Mt is facing based on at least one type of sensing information, for example, camera information and inertial information, obtained by the sensor unit 2003 of the mobile terminal Mt. Note that during the execution of each step subsequent to S2010 of the process flow, the latest recognized background area Ab may be updated by executing a monitoring process of the background area Ab in parallel as the background area Ab changes due to the movement of the user Us.

[0088] 15, the recognition block 200 acquires the desired content Cd by recognition based on sensing information at the mobile terminal Mt carried by the user Us. The sensing information at this time is acquired through the mobile terminal Mt at the timing when the sensing information is input to the sensor unit 2003 as the intention data of the user Us, and is used for recognizing the desired content Cd. Details regarding the input and items of the desired content Cd conform to S20 of the first embodiment.

[0089] In S2030 of the process flow shown in Fig. 15, the drive control block 210 (see Fig. 14) provides a user service by controlling the drive of the autonomous mobile device Ma within the background area Ab. At this time, a drive control pattern for the autonomous mobile device Ma that matches the type of user service is read from the memory 9a, and the user service is provided according to the drive control pattern so as to lead the user Us moving toward the background area Ab. Details regarding the drive of the autonomous mobile device Ma and the provision of the user service are similar to S30 in the first embodiment.

[0090] S2040 of the process flow shown in Fig. 15 is executed in parallel with S2030. In S2040, the display control block 220 (see Fig. 14) causes the display unit 2002 of the mobile terminal Mt to display an XR image corresponding to the desired content Cd superimposed on the background video 24 showing the background area Ab in which the autonomous mobile device Ma that is providing the user service by S2030 is located. At this time, the display control block 220 adjusts the occupancy rate Rc of the display target range Ad of the content image 20 of the XR image that is superimposed and displayed according to the desired content Cd as shown in Figs. 17 and 18 in the background video 24 in accordance with the risk level Lr in the area Ab. Here, the occupancy rate Rc in the second embodiment means the ratio of the display area of ​​the entire display target range Ad to the display area of ​​the entire background video 24 on the display unit 2002.

[0091] In S2040, the background video 24 to be superimposed on the content image 20 is acquired based on, for example, sensing information obtained by a sensor unit 4 such as a camera, or three-dimensional space information It obtained from an infrastructure database Di. For the background video 24, primary video data obtained by capturing a background area Ab may be used, or secondary video data obtained by processing the primary data may be used.

[0092] The content image 20 in S2040 includes at least a device image 20a shown in FIGS. 17 and 18 as an XR image displayed superimposed on a video portion corresponding to the position of the autonomous mobile device Ma in the background area Ab on the background video 24 as shown in FIG. 13. The superimposed display position of the device image 20a is adjusted to match the imaging position of the autonomous mobile device Ma on the background video 24 according to the drive control pattern by S2030. As a result, a replacement display is realized in which the device image 20a with a display theme corresponding to the desired content Cd is superimposed so as to replace the entire autonomous mobile device Ma shown in the background video 24. Details regarding the adjustment of the display position of the device image 20a are similar to S40 of the first embodiment.

[0093] The content image 20 in S2040 may include an object image 20b shown in Figs. 17 and 18 as an XR image displayed superimposed on a video portion corresponding to the position of another object Mb that exists in the background area Ab together with the autonomous driving device Ma as shown in Fig. 13 on the background video 24. The superimposed display position of the object image 20b is adjusted according to the imaging position of the other object Mb on the background video 24, resulting in a replacement display similar to the device image 20a. Details of the other object Mb that is replaced by the object image 20b are similar to those of S40 in the first embodiment.

[0094] The content image 20 in S2040 may include a ground image 20c shown in Figures 17 and 18 as an XR image superimposed on the video portion corresponding to the position of the ground Mc that exists in the background area Ab together with the autonomous mobile device Ma on the background video 24 as shown in Figure 13. The superimposed display position of the ground image 20c is adjusted according to the imaging position of the ground Mc on the background video 24, resulting in a replacement display similar to that of the device image 20a.

[0095] In S2040, the display target range Ad of the content image 20 is set to cover the entire background video 24 as shown in Fig. 17, and the occupancy rate Rc is adjusted to a maximum value (e.g., 1). In S2040, the display target range Ad of the content image 20 is set to disappear from the entire background video 24 as shown in Fig. 19, and the occupancy rate Rc is adjusted to a minimum value (e.g., 0). In S2040, the display target range Ad of the content image 20 is set to a part of the background video 24 as shown in Fig. 18, and the occupancy rate Rc is adjusted to an intermediate value between the minimum and maximum values ​​(e.g., more than 0 and less than 1).

[0096] Furthermore, in S2040, when a hazard notification to the user Us becomes necessary according to the risk level Lr, a hazard image 22 as an XR image may be superimposed by the display unit 2002 as shown in Figs. 18 and 19 on the ground Mc and / or the sky Md shown in the background video 24 as shown in Fig. 13. The superimposed display position of the hazard image 22 may be adjusted according to the imaging position of the ground Mc and / or the sky Md on the background video 24. At this time, details regarding the adjustment of the display position of the hazard image 22 are similar to S40 of the first embodiment.

[0097] In such S2040, an adjustment subroutine for switching and adjusting the occupancy rate Rc according to the risk level Lr in the background area Ab is executed as shown in Fig. 16. Specifically, in S2400 of the adjustment subroutine, the display control block 220 acquires internal sensing information Ii sensed by the sensor unit 2003 of the mobile terminal Mt. At this time, the acquisition range of the internal sensing information Ii with an acceptable level of accuracy or reliability by the sensor unit 2003 is set to an internal sensing space Ai that is a part of the background area Ab as shown in Fig. 20. Details regarding the acquisition of the internal sensing information Ii and the definition and setting of the internal sensing space Ai are similar to S400 of the first embodiment.

[0098] 16, the display control block 220 acquires a risk level Lr in the internal sensing space Ai of the background area Ab. Details regarding the definition and determination of the risk level Lr at this time conform to S410 of the first embodiment. Furthermore, at S2420 of the adjustment subroutine, the display control block 220 determines the risk level Lr in accordance with S410 of the first embodiment.

[0099] If a positive determination is made in S2420, in S2430 of the adjustment subroutine shown in Fig. 16, the display control block 220 adjusts the occupancy rate Rc to the minimum value by prohibiting the superimposed display of the content image 20 from the entire area of ​​the background video 24 as shown in Fig. 19 so as to remove the entire area of ​​the background video 24 from the display target range Ad of the content image 20. However, in S2430, a hazard image 22 notifying a user of warning characters and / or a warning color in accordance with a risk level Lr that has reached the hazard level Lh may be superimposed on the background video 24 as shown in Fig. 19. At this time, details of the display of the hazard image 22 and other warning methods conform to those of S430 in the first embodiment. Completion of execution of S2430 terminates the current execution of the adjustment subroutine and the processing flow.

[0100] If a negative determination is made in S2420, the display control block 220 acquires the external sensing information Io in S2440 of the adjustment subroutine shown in Fig. 16 in accordance with S440 of the first embodiment. However, at this time, the acquisition range of the external sensing information Io with an acceptable level of accuracy or reliability by the sensor unit 4 is set to an external sensing space Ao that is a part of the background area Ab as shown in Fig. 21 and is wider than the internal sensing space Ai.

[0101] 16, the display control block 220 acquires a risk level Lr in the external sensing space Ao of the background area Ab. Details regarding the definition and determination of the risk level Lr are similar to those of S450 in the first embodiment. Furthermore, in S2460 of the adjustment subroutine, the display control block 220 determines the risk level Lr in accordance with S460 in the first embodiment.

[0102] If an affirmative determination is made in S2460, in S2470 of the adjustment subroutine shown in Fig. 16, the display control block 220 identifies the hazard space portion Aoh of the external sensing space Ao in accordance with S470 of the first embodiment as shown in Fig. 21. However, in the background area Ab, a display prohibited range in which the superimposed display of the content image 20 is prohibited as described below may be recognized as the hazard space portion Aoh (see Fig. 18) so that the display prohibited range continues to the far side of the external sensing space Ao.

[0103] Therefore, the display control block 220 in S2470 adjusts the occupancy rate Rc to an intermediate value by restricting the content image 20 to the video portion 24h showing the hazard space portion Aoh as shown in FIG. 18 so as to remove the video portion 24h showing the hazard space portion Aoh from the display target range Ad of the content image 20. As a result, the remaining portion 24s of the background video 24 other than the video portion 24h is set to the display target range Ad of the content image 20 in response to the remaining space portion As of the safety level outside the hazard space portion Aoh in the background area Ab, and the content image 20 is permitted to be displayed in a superimposed manner. At this time, the details of the display of the hazard image 22 and other warning methods are similar to those of S470 in the first embodiment. When the execution of S2470 is completed, the current execution of the adjustment subroutine and the processing flow is terminated.

[0104] When a negative determination is made in S2460, in S2480 of the adjustment subroutine shown in Fig. 16, the display control block 220 adjusts the occupancy rate Rc to the maximum value by permitting the content image 20 to be superimposed on the entire background video 24 as shown in Fig. 17 so as to set the entire background video 24 as the display target range Ad of the content image 20. At this time, a limitation image 20cl is displayed as a ground image 20c representing the limit range Aol on the ground Mc of the external sensing space Ao, superimposed on the background video 24 as shown in Fig. 17. Completion of execution of S2480 ends the current execution of the adjustment subroutine and the processing flow. Therefore, in the next and subsequent executions of the processing flow, the limitation image 20cl will continue to be displayed until a positive determination is made in either S2420 or S2460.

[0105] (Action and effect) The effects of the second embodiment described above will now be described.

[0106] In the second embodiment, a background area Ab viewed by a user Us through a mobile terminal Mt carried by the user Us is monitored. The content image 20 is superimposed on a background video 24 showing the background area Ab as an XR image corresponding to a desired content Cd desired by the user Us. The occupancy rate Rc of the display target range Ad in the background video 24 is adjusted according to a risk level Lr in the background area Ab.

[0107] At this time, particularly according to the second embodiment, external sensing information Io sensed by the autonomous mobile device Ma leading the user Us moving toward the background area Ab is acquired, and the risk level Lr based on the external sensing information Io is recognized. As a result, by effectively utilizing the autonomous mobile device Ma fulfilling the leading function for the user Us, the risk level Lr can be recognized for as wide a range as possible of the background area Ab to which the user Us is moving, and the safety of the user Us can be ensured.

[0108] Moreover, according to the second embodiment, in the background video 24 showing the external sensing space Ao that is the acquisition range of the external sensing information Io in the background area Ab, the content image 20 is prohibited from being superimposed on the video portion 24h showing the hazard space portion Aoh where the risk level Lr has reached the hazard level Lh. As a result, in the background video 24, the content image 20 disappears only in the portion 24h corresponding to the inside of the hazard space portion Aoh of the hazard level Lh that requires a hazard notification to the moving user Us, while the occupancy rate Rc can be adjusted so that the content image 20 continues to be superimposed on the portion 24s corresponding to the outside of the hazard space portion Aoh in the background area Ab. Therefore, it is possible to balance the safety ensured by the above-mentioned principle for the moving user Us and the continuity of the virtual experience.

[0109] According to the second embodiment, a limitation image 20cl as an XR image representing a limit range Aol on the ground Mc of the external sensing space Ao is superimposed on a background video 24 showing the background area Ab. According to this, in a state in which the limit range Aol of the external sensing space Ao in which the risk level Lr is recognized is presented to the user Us, the safety of the user Us is ensured by the recognition of the risk level Lr in the external sensing space Ao, and the sense of security of the user Us can also be ensured. Therefore, coupled with the disappearance of the content image 20 limited to the portion 24h corresponding to the hazard space portion Aoh, it is possible to ensure the safety and sense of security of the user Us in a balanced manner with the continuity of the virtual experience.

[0110] According to the second embodiment, the internal sensing information Ii sensed by the mobile terminal Mt is acquired, and the risk level Lr based on the internal sensing information Ii is recognized. As a result, the risk level Lr is also recognized for the proximity range of the user Us by the mobile terminal Mt, and the safety of the user Us can be ensured. However, when the risk level Lr in the internal sensing space Ai, which is the acquisition range of the internal sensing information Ii in the background area Ab, reaches the hazard level Lh, the content image 20 is prohibited from being superimposed on the entire background video 24 in which the background area Ab is reflected. As a result, it is possible to realize an intentionally unbalanced adjustment of the occupancy rate Rc for the proximity range of the user Us so as to prioritize safety over the continuity of the virtual experience.

[0111] According to the second embodiment, the external sensing information Io collected by the infrastructure system 6 may also be acquired together with the external sensing information Io sensed by the autonomous mobile device Ma, and the risk level Lr based on the external sensing information Io may be recognized. In this way, by effectively utilizing not only the autonomous mobile device Ma that fulfills a leading function for the user Us, but also the infrastructure system 6, the risk level Lr for the background area Ab may be recognized with high accuracy, making it possible to ensure the safety of the user Us.

[0112] (Other embodiments) Although several embodiments have been described above, the present disclosure should not be construed as being limited to those embodiments, and can be applied to various embodiments and combinations within the scope not departing from the gist of the present disclosure.

[0113] In the modified examples of the first and second embodiments, the dedicated computer constituting the control system 9, 2009 of the processing system 1, 2001 may have at least one of a digital circuit and an analog circuit as a processor. Here, the digital circuit is at least one of ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), SOC (System on a Chip), PGA (Programmable Gate Array), CPLD (Complex Programmable Logic Device), etc. Furthermore, such a digital circuit may have a memory that stores a program.

[0114] In a modification of the first and second embodiments, the processing system 1, 2001 may be disconnected from the infrastructure system 6. In this modification, sensing information by connection elements Wt, Ma, Mt with the processing system 1, 2001 may be used instead of the three-dimensional space information It obtained from the infrastructure database Di.

[0115] In the modified examples of the first and second embodiments, three-dimensional spatial information It that is not linked to the voxel Vi may be acquired from the infrastructure database Di by the processing system 1, 2001. In the modified examples of the first and second embodiments, two-dimensional grid information may be acquired from the infrastructure database Di by the processing system 1, 2001 instead of the three-dimensional spatial information It.

[0116] In a modification of the first embodiment, when a mobile terminal Mt as in the second embodiment is connected to the processing system 1 and functions as a part of the wearable terminal Wt, the processing flow may be executed in response to the sensor unit 2003 constituting the part acquiring a service request input as sensing information. In a modification of the first embodiment, when a mobile terminal Mt as in the second embodiment is connected to the processing system 1 and functions as a part of the wearable terminal Wt, the desired content Cd in S20 of the processing flow may be acquired based on the sensing information of the sensor unit 2003 constituting the part.

[0117] In a modification of the second embodiment, a smart watch or a video see-through or non-transparent wearable terminal may be used as the mobile terminal Mt. In the modification of the second embodiment, when the wearable terminal Wt as in the first embodiment is connected to the processing system 2001 and functions as a part of the mobile terminal Mt, the processing flow may be executed in response to the sensor unit 3 constituting the part acquiring a service request input as sensing information. In the modification of the second embodiment, when the wearable terminal Wt as in the first embodiment is connected to the processing system 2001 and functions as a part of the mobile terminal Mt, the desired content Cd may be acquired in S2020 of the processing flow based on the sensing information of the sensor unit 3 constituting the part.

[0118] In the modified examples of the first and second embodiments, the display of the limitation image 20cl may be omitted in S480 and S2480. In the modified examples of the first and second embodiments, the limitation image 20cl may be displayed in the display target range Ad of the content image 20 also in S470 and S2470. In the modified example of the first embodiment, in S430 when the risk level Lr in the internal sensing space Ai has reached the hazard level Lh, the superimposed display of the content image 20 may be prohibited only in the internal sensing space Ai of the visual field area Av. In the modified example of the second embodiment, in S2430 when the risk level Lr in the internal sensing space Ai has reached the hazard level Lh, the superimposed display of the content image 20 may be prohibited only in the moving image portion of the background area Ab corresponding to the internal sensing space Ai.

[0119] In addition to the above-described embodiments and modifications, the processing system 1, 2001 having at least one processor 9b and one memory 9a in the control system 9, 2009 may be implemented in the form of a semiconductor device (such as a semiconductor chip).

[0120] (Additional remarks) This specification discloses the following technical ideas and combinations thereof.

[0121] (Technical thought 1) A processing system having a processor (9b) for performing XR display-related processing related to displaying XR images to a moving user, The processor, Monitoring a visual field (Av) viewed by the user through a wearable terminal (Wt) worn by the user; The display device is configured to adjust the occupancy rate (Rc) of a display target range (Ad) of a content image (20) to be superimposed on the viewing area as the XR image corresponding to the desired content (Cd) desired by the user in accordance with a risk level (Lr) in the viewing area; Adjusting the occupancy rate includes: Acquiring external sensing information (Io) sensed by an autonomous driving device (Ma) leading the user moving toward the viewing area; recognizing the risk level based on the external sensing information; and and adjusting the occupancy rate by prohibiting the superimposed display of the content image for a hazard space portion (Aoh) of the external sensing space (Ao) within the field of view area, which is the range within which the external sensing information is acquired, where the risk level has reached a hazard level requiring a hazard notification to the moving user.

[0122] (Technical thought 2) Adjusting the occupancy rate includes: The processing system described in technical idea 1 includes superimposing a limitation image (20cl) as the XR image representing a limit range (Aol) on the ground (Mc) of the external sensing space on the field of view area.

[0123] (Technical Thought 3) Adjusting the occupancy rate includes: Acquiring internal sensing information (Ii) sensed by the wearable device; and recognizing the risk level based on the internal sensing information; and The processing system described in technical idea 1 or 2 includes adjusting the occupancy rate by prohibiting the superimposed display of the content image over the entire field of view when the risk level in an internal sensing space (Ai) within the field of view, which is the range within which the internal sensing information is acquired, reaches a hazard level that requires a hazard notification to the moving user.

[0124] (Technical Thought 4) A processing system having a processor (9b) for performing XR display-related processing related to displaying XR images to a moving user, The processor, Monitoring a background area (Ab) that is a display background to the user from a mobile terminal (Mt) carried by the user; and adjusting an occupancy rate (Rc) of a display target range (Ad) of a content image (20) to be superimposed on a background video (24) showing the background area as the XR image according to a desired content (Cd) desired by the user, in accordance with a risk level (Lr) in the background area; Adjusting the occupancy rate includes: Acquiring external sensing information (Io) sensed by an autonomous mobile device (Ma) leading the user moving toward the background area; recognizing the risk level based on the external sensing information; and and adjusting the occupancy rate by prohibiting the superimposition of the content image on a video portion (24h) of the background video showing an external sensing space (Ao) in the background area which is the acquisition range of the external sensing information, the video portion showing a hazard space portion (Aoh) where the risk level has reached a hazard level requiring a hazard notification to the moving user.

[0125] (Technical Thought 5) Adjusting the occupancy rate includes: The processing system described in technical idea 4 includes superimposing a limitation image (20cl) as the XR image representing the limit range (Aol) on the ground (Mc) of the external sensing space on the background video.

[0126] (Technical Thought 6) Adjusting the occupancy rate includes: Acquiring internal sensing information (Ii) sensed by the mobile terminal; recognizing the risk level based on the internal sensing information; and The processing system described in Technical Idea 4 or 5 includes adjusting the occupancy rate by prohibiting the overlay display of the content image over the entire background video when the risk level in an internal sensing space (Ai) within the background area, which is the range within which the internal sensing information is acquired, reaches a hazard level that requires a hazard notification to the moving user.

[0127] (Technical Thought 7) Adjusting the occupancy rate includes: The processing system according to any one of Technical Ideas 1 to 6, further comprising acquiring the external sensing information collected by an infrastructure system (6).

[0128] The above-mentioned technical concepts 1 to 7 may be understood as the respective technical concepts of a method and a program. [Explanation of symbols]

[0129] 1, 2001: processing system, 6: infrastructure system, 9a: memory, 9b: processor, 20: content image, 20a: content image, 20cl: limitation image, 24: background video, 24h: video portion, Ab: background area, Ad: display target range, Ai: internal sensing space, Ao: external sensing space, Aoh: hazard space portion, Aol: limit range, Av: field of view area, Cd: desired content, Ii: internal sensing information, Io: external sensing information, Lr: risk level, Ma: autonomous driving device, Mc: ground, Mt: mobile terminal, Rc: occupancy rate, Wt: wearable terminal

Claims

1. A processing system having a processor (9b) for performing XR display related processes related to displaying XR images to a mobile user, The processor, Monitoring a visual field (Av) viewed by the user through a wearable terminal (Wt) worn by the user; The display device is configured to adjust the occupancy rate (Rc) of the display target range (Ad) of a content image (20) to be superimposed on the field of view area as the XR image corresponding to the desired content (Cd) desired by the user in accordance with a risk level (Lr) in the field of view area; Adjusting the occupancy rate includes: Acquiring external sensing information (Io) sensed by an autonomous driving device (Ma) leading the user moving toward the viewing area; Recognizing the risk level based on the external sensing information; and and adjusting the occupancy rate by prohibiting the superimposed display of the content image for a hazard space portion (Aoh) of the external sensing space (Ao) which is the acquisition range of the external sensing information in the field of view area and where the risk level has reached a hazard level requiring a hazard notification to the moving user.

2. Adjusting the occupancy rate includes: The processing system according to claim 1 , further comprising: superimposing a limitation image (20cl) as the XR image representing a limit range (Aol) on a ground surface (Mc) of the external sensing space on the field of view area.

3. Adjusting the occupancy rate includes: Acquiring internal sensing information (Ii) sensed by the wearable device; and recognizing the risk level based on the internal sensing information; and The processing system of claim 1 further comprising: when the risk level in an internal sensing space (Ai) within the field of view that is the acquisition range of the internal sensing information reaches a hazard level that requires a hazard notification to the moving user, adjusting the occupancy rate by prohibiting the superimposed display of the content image over the entire field of view.

4. A processing system having a processor (9b) for performing XR display related processes related to displaying XR images to a mobile user, The processor, Monitoring a background area (Ab) that is a display background to the user from a mobile terminal (Mt) carried by the user; and adjusting an occupancy rate (Rc) of a display target range (Ad) of a content image (20) to be superimposed on a background video (24) showing the background area as the XR image according to a desired content (Cd) desired by the user in accordance with a risk level (Lr) in the background area; Adjusting the occupancy rate includes: Acquiring external sensing information (Io) sensed by an autonomous driving device (Ma) leading the user moving toward the background area; Recognizing the risk level based on the external sensing information; and and adjusting the occupancy rate by prohibiting the superimposition of the content image on a video portion (24h) of the background video showing an external sensing space (Ao) in the background area that is the acquisition range of the external sensing information, the video portion showing a hazard space portion (Aoh) where the risk level has reached a hazard level requiring a hazard notification to the moving user.

5. Adjusting the occupancy rate includes: The processing system according to claim 4 , further comprising: superimposing and displaying a limitation image (20cl) as the XR image representing a limit range (Aol) on the ground (Mc) of the external sensing space on the background video.

6. Adjusting the occupancy rate includes: Acquiring internal sensing information (Ii) sensed by the mobile terminal; recognizing the risk level based on the internal sensing information; and The processing system according to claim 4, further comprising: when the risk level in an internal sensing space (Ai) within the background area, which is the range within which the internal sensing information is acquired, reaches a hazard level that requires a hazard notification to the moving user, adjusting the occupancy rate by prohibiting the superimposed display of the content image over the entire background video.

7. Adjusting the occupancy rate includes: The processing system according to claim 1 or 4, further comprising acquiring the external sensing information collected by an infrastructure system (6).

8. A processing method executed by a processor (9b) for performing XR display related processing related to displaying XR images to a mobile user, the processing method comprising: Monitoring a visual field (Av) viewed by the user through a wearable terminal (Wt) worn by the user; The method includes adjusting an occupancy rate (Rc) of a display target range (Ad) of a content image (20) to be superimposed on the field of view area as the XR image corresponding to a desired content (Cd) desired by the user in accordance with a risk level (Lr) in the field of view area, Adjusting the occupancy rate includes: Acquiring external sensing information (Io) sensed by an autonomous driving device (Ma) leading the user moving toward the viewing area; Recognizing the risk level based on the external sensing information; and and adjusting the occupancy rate by prohibiting the superimposed display of the content image for a hazard space portion (Aoh) of the external sensing space (Ao) which is the acquisition range of the external sensing information in the field of view area and where the risk level has reached a hazard level requiring a hazard notification to the moving user.

9. A processing method executed by a processor (9b) for performing XR display related processing related to displaying XR images to a mobile user, the processing method comprising: Monitoring a background area (Ab) that is a display background to the user from a mobile terminal (Mt) carried by the user; adjusting an occupancy rate (Rc) of a display target range (Ad) of a content image (20) to be superimposed on a background video (24) showing the background area as the XR image corresponding to a desired content (Cd) desired by the user in accordance with a risk level (Lr) in the background area; Adjusting the occupancy rate includes: Acquiring external sensing information (Io) sensed by an autonomous driving device (Ma) leading the user moving toward the background area; Recognizing the risk level based on the external sensing information; and and adjusting the occupancy rate by prohibiting the superimposition of the content image on a video portion (24h) of the background video showing an external sensing space (Ao) in the background area which is the acquisition range of the external sensing information, the video portion showing a hazard space portion (Aoh) where the risk level has reached a hazard level requiring a hazard notification to the moving user.

10. A processing program stored in a storage medium (9 a) for performing an XR display-related process related to displaying an XR image to a moving user, the processing program including instructions to be executed by a processor (9 b), Monitoring a visual field (Av) viewed by the user through a wearable terminal (Wt) worn by the user; The instruction includes: adjusting the occupancy rate (Rc) of a display target range (Ad) of a content image (20) to be superimposed on the field of view area as the XR image corresponding to the desired content (Cd) desired by the user in accordance with a risk level (Lr) in the field of view area; Adjusting the occupancy rate includes: Acquiring external sensing information (Io) sensed by an autonomous driving device (Ma) leading the user moving toward the viewing area; Recognizing the risk level based on the external sensing information; and a processing program including: adjusting the occupancy rate by prohibiting the superimposed display of the content image for a hazard space portion (Aoh) of the external sensing space (Ao) which is the acquisition range of the external sensing information in the field of view area and where the risk level has reached a hazard level requiring a hazard notification to the moving user.

11. A processing program stored in a storage medium (9 a) for performing an XR display-related process related to displaying an XR image to a moving user, the processing program including instructions to be executed by a processor (9 b), Monitoring a background area (Ab) that is a display background to the user from a mobile terminal (Mt) carried by the user; adjusting an occupancy rate (Rc) of a display target range (Ad) of a content image (20) to be superimposed on a background video (24) showing the background area as the XR image according to a desired content (Cd) desired by the user in accordance with a risk level (Lr) in the background area, Adjusting the occupancy rate includes: Acquiring external sensing information (Io) sensed by an autonomous driving device (Ma) leading the user moving toward the background area; Recognizing the risk level based on the external sensing information; and and adjusting the occupancy rate by prohibiting the superimposition of the content image on a video portion (24h) of the background video showing an external sensing space (Ao) in the background area which is the acquisition range of the external sensing information, the video portion showing a hazard space portion (Aoh) where the risk level has reached a hazard level requiring a hazard notification to the moving user.

Images