Processing system, processing method, and processing program

Abstract

A processor of this processing system executes: the monitoring a visual field area (Av) visually recognized, through a wearable terminal, by a user; and the adjusting, in accordance with a risk level (Lr) for the visual field area (Av), the occupancy ratio (Rc) of a display target range (Ad) occupying the visual field area, the display target range (Ad) being of a content image superimposed and displayed over the visual field area (Av) as an XR image corresponding to content desired by the user. The adjustment of the occupancy ratio (Rc) includes: acquiring external sensing information (Io) by an autonomous travel device that leads the user; recognizing the risk level (Lr) based on the external sensing information (Io); and adjusting the occupancy ratio (Rc) by prohibiting the superimposition display of the content image with respect to a hazardous space part (Aoh) in which the risk level has reached a hazard level and which is within an external sensing space (Ao) serving as the acquisition range of the external sensing information (Ii) in the visual field area (Av).

Description

Processing system, processing method, and processing program CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based on Patent Application No. 2023-106120 filed in Japan on June 28, 2023, and the contents of the original application are incorporated by reference in their entirety.

[0002] The present disclosure relates to processing techniques for performing XR display-related processing related to the display of XR images to a user.

[0003] The technology disclosed in Patent Literature 1 enables a user to have a virtual experience by displaying VR video as an XR image on a head-mounted display (HMD). In particular, this technology makes the display of the HMD transparent when the HMD detects a collision risk in the physical environment due to the user's movement exceeding a limit position.

[0004] Special Publication No. 10-504917

[0005] However, in the technology disclosed in Patent Document 1, the movement of the user wearing the HMD is used as a trigger, and the collision risk limit is set based on the user's position, so the range in which safety can be ensured by display transparency is limited to the user's proximity. As a result, the technology disclosed in Patent Document 1, in which the VR image is completely erased by display transparency, disrupts the continuity of the virtual experience, especially when applied to a moving user.

[0006] An object of the present disclosure is to provide a processing system that performs XR display-related processing while balancing the safety of a mobile user and the continuity of a virtual experience. Another object of the present disclosure is to provide a processing method that performs XR display-related processing while balancing the safety of a mobile user and the continuity of a virtual experience. Yet another object of the present disclosure is to provide a processing program that performs XR display-related processing while balancing the safety of a mobile user and the continuity of a virtual experience.

[0007] The technical means of the present disclosure for solving the problems will be described below.

[0008] A first aspect of the present disclosure is a processing system having a processor that performs XR display-related processing related to the display of XR images to a moving user, wherein the processor is configured to monitor a field of view viewed by the user through a wearable terminal worn by the user, and adjust the occupancy rate of the field of view that is occupied by the display target range of a content image that is superimposed on the field of view area as an XR image corresponding to a desired content desired by the user, in accordance with a risk level in the field of view area, wherein adjusting the occupancy rate includes acquiring external sensing information sensed by an autonomous driving device leading the user moving toward the field of view area, recognizing the risk level based on the external sensing information, and adjusting the occupancy rate by prohibiting the superimposition of content images in hazard space portions of the external sensing space that are the acquisition range of the external sensing information in the field of view area, where the risk level has reached a hazard level that requires hazard notification to the moving user.

[0009] A second aspect of the present disclosure is a processing method executed by a processor to perform XR display-related processing related to the display of XR images to a moving user, the method including: monitoring a field of view viewed by the user through a wearable terminal worn by the user; and adjusting the occupancy rate of the field of view area occupied by the display target range of a content image superimposed on the field of view area as an XR image corresponding to a desired content desired by the user, in accordance with the risk level in the field of view area, wherein adjusting the occupancy rate includes: acquiring external sensing information sensed by an autonomous driving device leading the user moving toward the field of view area; recognizing the risk level based on the external sensing information; and adjusting the occupancy rate by prohibiting the superimposition of content images in hazard space portions of the external sensing space within the field of view area that are the acquisition range of the external sensing information, where the risk level has reached a hazard level that requires hazard notification to the moving user.

[0010] A third aspect of the present disclosure is a processing program stored in a storage medium for performing XR display-related processing related to the display of XR images to a moving user, and including instructions for causing a processor to execute the XR display-related processing, the processing including instructions for monitoring a field of view viewed by the user through a wearable terminal worn by the user, and adjusting, in accordance with a risk level in the field of view, the occupancy rate of a display target range of a content image that is superimposed on the field of view area as an XR image corresponding to a desired content desired by the user, wherein adjusting the occupancy rate includes acquiring external sensing information sensed by an autonomous driving device leading the user moving toward the field of view area, recognizing a risk level based on the external sensing information, and adjusting the occupancy rate by prohibiting the superimposition of content images in hazard space portions of the external sensing space that is the acquisition range of the external sensing information in the field of view area, where the risk level has reached a hazard level that requires hazard notification to the moving user.

[0011] In the first to third aspects, the field of view of the user is monitored through the wearable device worn by the user. The occupancy rate of the display target range of the content image superimposed on the field of view as an XR image corresponding to the desired content of the user is adjusted according to the risk level in the field of view.

[0012] In particular, according to the first to third aspects, external sensing information sensed by an autonomous mobile device leading a user moving toward a field of view is acquired, and a risk level based on the external sensing information is recognized. By effectively utilizing the autonomous mobile device that performs the user's leading function, the risk level can be recognized for as wide a range as possible of the field of view to which the user is moving, thereby ensuring the safety of the user.

[0013] Furthermore, according to the first to third aspects, in the external sensing space within the field of view, which is the acquisition range of external sensing information, the superimposed display of the content image is prohibited in the hazard space portion where the risk level reaches the hazard level. This allows the content image to disappear only within the hazard space portion where the hazard level requires hazard notification to the moving user, while the occupancy rate can be adjusted so that the superimposed display of the content image continues outside the hazard space portion in the field of view. Therefore, it is possible to balance the safety ensured by the above-mentioned principle for the moving user with the continuity of the virtual experience.

[0014] A fourth aspect of the present disclosure is a processing system having a processor that performs XR display-related processing related to the display of XR images to a moving user, wherein the processor is configured to monitor a background area that serves as a display background for the user from a mobile terminal carried by the user, and adjust, in accordance with a risk level in the background area, an occupancy rate of a display target range of a content image that is superimposed on a background video showing the background area as an XR image corresponding to a desired content desired by the user, wherein the adjusting of the occupancy rate includes: acquiring external sensing information sensed by an autonomous driving device leading the user moving toward the background area; recognizing the risk level based on the external sensing information; and adjusting the occupancy rate by prohibiting the superimposition of the content image on a video portion showing a hazard space portion where the risk level has reached a hazard level requiring a hazard notification to the moving user, among the background video showing an external sensing space that is the acquisition range of the external sensing information in the background area.

[0015] A fifth aspect of the present disclosure is a processing method executed by a processor to perform XR display-related processing related to the display of XR images to a moving user, the method including: monitoring a background area that serves as a display background for the user from a mobile terminal carried by the user; and adjusting, in accordance with a risk level in the background area, an occupancy rate of a display target range of a content image that is superimposed on a background video showing the background area as an XR image corresponding to a desired content desired by the user, in the background video; wherein adjusting the occupancy rate includes: acquiring external sensing information sensed by an autonomous driving device leading the user moving toward the background area; recognizing the risk level based on the external sensing information; and adjusting the occupancy rate by prohibiting the superimposition of the content image on a video portion showing a hazard space portion where the risk level has reached a hazard level requiring a hazard notification to the moving user, among the background video showing an external sensing space that is the acquisition range of the external sensing information in the background area.

[0016] A sixth aspect of the present disclosure is a processing program stored in a storage medium for performing XR display-related processing related to the display of XR images to a moving user, and including instructions for causing a processor to execute the XR display-related processing, the processing including instructions for executing the following: monitoring a background area that serves as a display background for the user from a mobile terminal carried by the user; and adjusting, in accordance with a risk level in the background area, an occupancy rate of a display target range of a content image that is superimposed on a background video showing the background area as an XR image corresponding to a desired content desired by the user. The adjusting of the occupancy rate includes: acquiring external sensing information sensed by an autonomous driving device leading the user moving toward the background area; recognizing the risk level based on the external sensing information; and adjusting the occupancy rate by prohibiting the superimposition of the content image on a video portion showing a hazard space portion where the risk level has reached a hazard level requiring a hazard notification to the moving user, among the background video showing an external sensing space that is the acquisition range of the external sensing information in the background area.

[0017] In this manner, in the fourth to sixth aspects, a background area that serves as a display background for a user from a mobile device carried by the user is monitored. The content image, which is superimposed on a background video showing the background area as an XR image according to the user's desired content, occupies a display target range of the content image in the background video, and the occupancy rate of the content image is adjusted according to the risk level in the background area.

[0018] In particular, according to the fourth to sixth aspects, external sensing information sensed by an autonomous mobile device leading a user moving toward a background area is acquired, and a risk level based on the external sensing information is recognized. In this way, by effectively utilizing the autonomous mobile device fulfilling the user's leading function, the risk level can be recognized for as wide a range as possible of the background area to which the user is moving, thereby ensuring the safety of the user.

[0019] Furthermore, according to the fourth to sixth aspects, in the background video showing the external sensing space within the background area where external sensing information is acquired, the content image is prohibited from being superimposed on the video portion showing the hazard space where the risk level reaches the hazard level. This allows the content image to disappear only in the background video portion corresponding to the inside of the hazard space where the hazard level requires hazard notification to the moving user, while the occupancy rate can be adjusted so that the content image continues to be superimposed on the portion of the background area corresponding to the outside of the hazard space. This makes it possible to balance the safety ensured by the above-mentioned principles for the moving user with the continuity of the virtual experience.

[0020] FIG. 1 is an overall configuration diagram showing a network connection environment of a processing system according to a first embodiment. FIG. 2 is a schematic diagram showing a wearable terminal according to the first embodiment. FIG. 3 is a schematic diagram for explaining three-dimensional voxels assumed in the infrastructure system according to the first embodiment. FIG. 4 is a block diagram showing a processing system according to the first embodiment. FIG. 5 is a flowchart showing a processing flow according to the first embodiment. FIG. 6 is a flowchart showing an adjustment subroutine of the processing flow according to the first embodiment. FIG. 7 is a schematic diagram showing a display state of a wearable terminal according to the first embodiment. FIG. 8 is a schematic diagram showing a display state of a wearable terminal according to the first embodiment. FIG. 9 is a schematic diagram showing a display state of a wearable terminal according to the first embodiment. FIG. 10 is a schematic diagram for explaining the adjustment subroutine according to the first embodiment. FIG. 11 is an overall configuration diagram showing a network connection environment of a processing system according to a second embodiment. FIG. 12 is a schematic diagram showing a mobile terminal according to the second embodiment. FIG. 13 is a block diagram showing a processing system according to the second embodiment. FIG. 14 is a flowchart showing a processing flow according to the second embodiment. FIG. 15 is a flowchart showing an adjustment subroutine of the processing flow according to the second embodiment. FIG. 16 is a schematic diagram showing a display state of a mobile terminal according to the second embodiment. 10 is a schematic diagram for explaining an adjustment subroutine according to a second embodiment.FIG. 11 is a schematic diagram for explaining an adjustment subroutine according to a second embodiment.FIG.

[0021] Hereinafter, multiple embodiments of the present disclosure will be described with reference to the drawings. Note that corresponding components in each embodiment are designated by the same reference numerals, and redundant description may be omitted. Furthermore, when only a portion of the configuration is described in each embodiment, the configuration of another previously described embodiment may be applied to the remaining portions of the configuration. Furthermore, in addition to the combinations of configurations explicitly stated in the description of each embodiment, configurations of multiple embodiments may be partially combined together even if not explicitly stated, provided that there is no particular problem with the combination.

[0022] 1 performs XR display-related processing related to displaying XR images to a moving user Us in conjunction with providing a user service to the user Us. To this end, the processing system 1 is connected to a wearable device Wt worn by a pre-registered user Us, an autonomous mobile device Ma, and an infrastructure system 6 via a communication network Nc. It is assumed that at least one system each of the wearable device Wt, the autonomous mobile device Ma, and the infrastructure system 6 is connected to the processing system 1.

[0023] The wearable terminal Wt is configured so that the user Us can wear it on their face and operate it hands-free, at least when connected to the processing system 1. As shown in Fig. 2, the wearable terminal Wt is an optical see-through electronic device, such as an HMD or smart glasses, that includes a display unit 2 and a sensor unit 3. Note that Fig. 2 of the wearable terminal Wt is a schematic image diagram showing how the visual field area Av of the user Us looks from the user Us's side in a portion corresponding to one of their eyes.

[0024] The wearable terminal Wt uses a communication unit to control the transmission and reception of data via the communication network Nc as shown in Fig. 1. The wearable terminal Wt is operated and controlled by the control unit in accordance with control commands from the processing system 1 via the communication network Nc so as to perform XR display-related processing in cooperation with the processing system 1.

[0025] Under operational control by the control unit of the wearable device Wt, the display unit 2 (shown in FIG. 2 ), such as a virtual image projection or retinal projection display unit, superimposes an XR image (see FIGS. 7 to 9 , described below) required for XR display-related processing on a real image of the field of view Av viewed by the user Us through glasses or lenses, allowing the user Us to view it. At the same time, under operational control of the wearable device Wt, the sensor unit 3 (shown in FIG. 2 ) acquires sensing information through sensing processing in accordance with the XR display-related processing. The sensor unit 3 may be, for example, at least one of a camera, inertial sensor, GNSS sensor, electrooculography sensor, gaze sensor, infrared sensor, geomagnetic sensor, motion sensor, touch sensor, and microphone. This configuration enables the sensor unit 3 to acquire input representing the user Us's intention as sensing information.

[0026] The autonomous mobile device Ma shown in FIG. 1 is an autonomous mobile vehicle or autonomous mobile robot that can autonomously travel in any direction, forward, backward, leftward, or rightward, by electrically driving wheels 5 based on sensing information from a sensor unit 4. The autonomous mobile device Ma uses a communication unit to control the transmission and reception of data via a communication network Nc. The autonomous mobile device Ma is driven and controlled by a control unit in accordance with control commands from the processing system 1 via the communication network Nc so as to perform XR display-related processing in cooperation with the processing system 1. Under this drive control, the sensor unit 4 acquires sensing information through sensing processing in accordance with the XR display-related processing. The sensor unit 4 may be at least one of a camera, an inertial sensor, a GNSS sensor, LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging), a sonar, or the like.

[0027] The infrastructure system 6 is a base system that shares three-dimensional spatial information It stored in the infrastructure database Di as a common foundation for the distributed architecture. The infrastructure system 6 controls the transmission and reception of data via the communication network Nc using a communication unit. The infrastructure system 6 collects the three-dimensional spatial information It, which is provided to individual distributed systems such as the processing system 1 that make up the distributed architecture, using a control unit as needed, and updates the information stored in the infrastructure database Di to the latest information.

[0028] In particular, the infrastructure system 6 manages information by virtually dividing the three-dimensional space to be stored into a plurality of three-dimensional voxels Vi (i.e., a three-dimensional grid) that are assumed to be arranged in a three-dimensional array as shown in Fig. 3. The infrastructure system 6 stores a data set linked by metadata for each space ID individually assigned to each voxel Vi in the infrastructure database Di as three-dimensional space information It as shown in Fig. 1. Of course, the three-dimensional space information It may include two-dimensional grid information that is linked only to, for example, the bottom surface of the voxels Vi that constitute the lowest layer of the two-dimensional array along the ground in the three-dimensional array.

[0029] The data constituting the three-dimensional space information It may be collected from at least one of a wearable device Wt that cooperates with the processing system 1, other wearable devices, and mobile devices such as smartphones or tablet devices. The data constituting the three-dimensional space information It may be collected from at least one of an autonomous mobile device Ma that cooperates with the processing system 1, and other moving bodies.

[0030] The data constituting the three-dimensional spatial information It may be collected from at least one of a communication base station, a smart pole, a smart street light, etc., equipped with infrastructure sensors such as a camera and / or LiDAR. The data constituting the three-dimensional spatial information It may also be collected from a service provider that provides at least one of a map service, a weather service, a communication service, a traffic management service, a feature management service, an aviation management service, and a user service (described later) handled by the processing system 1.

[0031] The construction data collected in this manner to construct the three-dimensional space information It may be at least one type of image data, for example, video, still images, point cloud images, etc. The construction data of the three-dimensional space information It may be at least one type of secondary data, for which information security is guaranteed, generated by image processing of such image data regarding people, including the user Us of the processing system 1, within the target space for information accumulation, such as position data, movement data, posture data, gaze data, behavior data, and people flow data, as well as intention data (described below) that represents the intention of the user Us.

[0032] The construction data of the three-dimensional space information It may be at least one type of secondary data, such as position data, movement data, and posture data, generated by image processing of image data regarding human belongings, including the wearable terminal Wt of the user Us, within the information accumulation target space.The construction data of the three-dimensional space information It may be at least one type of secondary data, such as position data, movement data, and posture data, generated by image processing of image data regarding moving objects, including the autonomous mobile device Ma, within the information accumulation target space.

[0033] The construction data of the three-dimensional space information It may be voice data obtained by collecting voices of people, including the user Us of the processing system 1, in the information storage target space. The construction data of the three-dimensional space information It may be at least one type of secondary data, for which information security is guaranteed, analyzed regarding people, including the user Us, by voice recognition processing of such voice data, such as position data, movement data, behavior data, people flow data, and conversation data, as well as intention data (described below) that represents the intention of the user Us.

[0034] The construction data of the three-dimensional space information It may be at least one of, for example, two-dimensional and / or three-dimensional map data, GIS (Geographic Information System) data, road network data, weather data, communication data, line data, traffic data, feature management data, BIM (Building Information Modeling) data, POI (Point of Interest) data, aviation management data, and time data. The construction data of the three-dimensional space information It may also be service data related to at least one of, for example, a guidance service, a transportation service, a photography service, and an online game service, which will be described later as user services handled by the processing system 1.

[0035] 1 , the processing system 1 is a distributed computer system equipped with a communication system 10 and a control system 9, and constructed to include at least one of, for example, a cloud server and an edge server. At least a portion of each of the systems 10 and 9 in the processing system 1 may be configured by a communication unit and a control unit of a wearable terminal Wt. At least a portion of each of the systems 10 and 9 in the processing system 1 may be configured by a communication unit and a control unit of an autonomous mobile device Ma. At least a portion of each of the systems 10 and 9 in the processing system 1 may be configured by a communication unit and a control unit of an infrastructure system 6.

[0036] The communication system 10 is mainly composed of communication devices for constructing the communication network Nc. The control system 9 is connected to the communication system 10 via at least one of a wired communication line and a wireless communication line. The control system 9 is constructed to include at least one dedicated computer. The dedicated computer that constructs the control system 9 has at least one memory 9a and one processor 9b.

[0037] In the processing system 1, the control system 9 executes multiple instructions of a processing program stored in the memory 9a using the processor 9b. As a result, multiple function blocks for performing XR display-related processing are constructed in the control system 9. The constructed function blocks include a recognition block 200, a drive control block 210, and a display control block 220, as shown in FIG.

[0038] The processing method in which the processing system 1 performs XR display-related processing through the cooperation of these blocks 200, 210, and 220 is executed according to the processing flows shown in Figures 5 and 6. This processing flow is executed in response to the sensor unit 3 of the wearable device Wt acquiring, as sensing information, a service request input by the user Us intending to request the provision of a user service. Note that each "S" in this processing flow represents multiple steps executed by multiple commands included in the processing program of the first embodiment.

[0039] In S10 of the processing flow shown in Figure 5, the recognition block 200 (see Figure 4) monitors the field of view area Av visually recognized by the user Us through the wearable device Wt. At this time, the monitoring of the field of view area Av is based on at least one type of sensing information, such as camera information and inertial information, obtained by the sensor unit 3 of the wearable device Wt. As a result, the field of view area Av is recognized as an area extending in the direction of the user Us's face or gaze, or in the direction of the wearable device Wt. Note that during the execution of each step following S10 of the processing flow, the latest recognized field of view area Av may be updated by concurrently executing a monitoring process for the field of view area Av as the field of view area Av changes due to the movement of the user Us.

[0040] 5 , the recognition block 200 acquires the desired content Cd that the user Us intends and desires. At this time, the desired content Cd is recognized based on sensing information about the face of the user Us obtained by the wearable device Wt worn on the user Us's face. Therefore, the sensing information for recognizing the desired content Cd is acquired through the wearable device Wt at the timing when the sensing information is input to the sensor unit 3 as intention data intended by the user Us.

[0041] The input of the desired content Cd in S20 may be a gesture input, a gaze input, a facial expression input, or a voice input by the user Us, which may be sensed by the sensor unit 3. At this time, on the display unit 2 of the wearable device Wt, an XR image for receiving the input of the desired content Cd from the user Us by the sensor unit 3 may be superimposed on the field of view Av through the wearable device Wt. Note that the desired content Cd may be recognized based on three-dimensional space information It acquired from the infrastructure database Di via the communication system 10.

[0042] The items of the desired content Cd recognized in S20 include at least a display theme that is given to the XR image (image 20 in FIGS. 7 and 8 described later) that the display unit 2 of the wearable device Wt displays in conjunction with the provision of a user service to the user Us. The display theme refers to a concept common to XR images that are displayed in two or three dimensions, such as virtual or real living things, such as monsters, animals, plants, and insects, as well as inanimate objects, such as prehistoric buildings, service vehicles, and imaginary objects.

[0043] The items of the desired content Cd recognized in S20 may include the types of user services that can be provided to the user Us by using the autonomous mobile device Ma. The types of user services may include, for example, at least one of a guidance service, a transportation service, a photography service, an online game service, and the like.

[0044] In S20, for any user service, the desired content Cd may include at least one route-related item, for example, a route including a destination, a moving pace, an arrival time, and a service provision time (i.e., a guidance time or a transport time).In S20, for any user service, the desired content Cd may include at least one voice-related item, for example, whether or not voice output is required, whether or not a voice recognition function is required, and whether or not a dialogue function is required.

[0045] In the case of a guidance service in S20, the desired content Cd may include at least one sightseeing-related item regarding a destination such as a tourist spot, such as an acceptable level of congestion, the presence or absence of facilities desired to be used, the presence or absence of nature, etc. In the case of a transportation service in S20, the desired content Cd may include at least one luggage-related item, such as the number, size, weight, type, and whether temperature control is required, etc.

[0046] In the case of a photography service in S20, the desired content Cd may include at least one photography-related item from among, for example, a photography schedule, photography timing, photography location, whether or not images or data need to be sent, whether or not photos need to be printed, etc. In the case of an online game service in S20, the desired content Cd may include at least one game-related item from, for example, initial settings for game characters, environmental settings for the online game, difficulty level of the online game, whether or not images or data need to be sent, whether or not a match is needed, etc.

[0047] 5, the drive control block 210 (see FIG. 4) provides a user service by controlling the drive of the autonomous mobile device Ma within the field of view Av. At this time, a drive control pattern for the autonomous mobile device Ma that matches the type of user service is read from the memory 9a, and the user service is provided in accordance with the drive control pattern so as to lead the user Us moving toward the field of view Av.

[0048] The guidance service by S30 is provided according to a drive control pattern for route guidance in which the autonomous mobile device Ma is driven from a standby position to a position near the user Us and leads the user Us to the destination. In this case, the guidance service may present the route for guiding the user Us by superimposing an XR image on the field of view Av of the wearable device Wt or by display output from a display unit possessed by the autonomous mobile device Ma. In the guidance service, the route for guiding the user Us may be presented by audio output from an audio unit possessed by the wearable device Wt or the autonomous mobile device Ma.

[0049] The transportation service by S30 is provided according to a drive control pattern in which the autonomous mobile device Ma having a loading compartment is driven from a standby position to a position near the user Us, and the loaded luggage in the loading compartment is transported to the destination while the user Us leads the way. In this transportation service, a method for loading luggage into the loading compartment may be presented to the user Us by superimposing an XR image on the field of view Av of the wearable device Wt or by display output from a display unit possessed by the autonomous mobile device Ma. In the transportation service, the loading of luggage into the loading compartment may be presented to the user Us by audio output from an audio unit possessed by the wearable device Wt or the autonomous mobile device Ma.

[0050] The photography service by S30 is provided according to a drive control pattern in which the autonomous mobile device Ma having an imaging unit is driven from a standby position to a position near the user Us and photographs the surrounding area while the user Us leads the way. In this photography service, the photography schedule or photography timing may be presented to the user Us by superimposing an XR image on the field of view Av of the wearable device Wt or by display output from a display unit possessed by the autonomous mobile device Ma. In the photography service, the photography schedule or photography timing may be presented to the user Us by audio output from an audio unit possessed by the wearable device Wt or the autonomous mobile device Ma.

[0051] In the photography service, the autonomous mobile device Ma driven around the user Us may capture two-dimensional or three-dimensional still images or videos that include the user Us. In the photography service, the focus during photography may be adjusted based on the distance to the user Us sensed by a sensor unit 4, such as a LiDAR, in the autonomous mobile device Ma. In the photography service, the incident direction of external light during photography may be adjusted based on three-dimensional spatial information It obtained from the infrastructure database Di or sensing information from a sensor unit 4, such as a camera, in the autonomous mobile device Ma. In the photography service, image data captured of the user Us may be stored in the memory 9a of the control system 9 or in a storage medium in the wearable device Wt.

[0052] The online game service according to S30 is provided according to a drive control pattern that drives the autonomous mobile device Ma from a standby position to a position near the user Us and leads the user Us to a capture point in the online game. In this case, in the online game service, necessary instructions for the online game may be presented to the user Us by superimposing an XR image on the field of view Av of the wearable device Wt or by display output from a display unit possessed by the autonomous mobile device Ma. In the online game service, necessary instructions for the online game may be presented to the user Us by audio output from an audio unit possessed by the wearable device Wt or the autonomous mobile device Ma.

[0053] S40 in the processing flow shown in FIG. 5 is executed in parallel with S30. In S40, the display control block 220 (see FIG. 4) causes the display unit 2 of the wearable device Wt to ​​superimpose an XR image corresponding to the desired content Cd on the field of view Av of the autonomous mobile device Ma currently providing user services in S30. At this time, the display control block 220 adjusts the occupancy rate Rc of the display target range Ad of the content image 20 of the XR image superimposed on the field of view Av according to the desired content Cd, as shown in FIGS. 7 and 8 , in accordance with the risk level Lr for the field of view Av. Here, the occupancy rate Rc in the first embodiment refers to the ratio of the display area of ​​the entire display target range Ad to the two-dimensional projection area of ​​the entire field of view Av onto, for example, glasses, lenses, or the retina of the display unit 2.

[0054] The content image 20 in S40 includes at least the device image 20a shown in FIGS. 7 and 8 as an XR image superimposed on the autonomous mobile device Ma visible in the field of view Av as shown in FIG. 2. The superimposed display position of the device image 20a is adjusted to match the predicted viewing position of the autonomous mobile device Ma according to the drive control pattern in S30. This display position adjustment is based on information acquired from the infrastructure database Di via the communication system 10 as three-dimensional space information It in which position-related information of the autonomous mobile device Ma is added to each voxel Vi in the user service provision space. As a result, an overlapping display is realized in which the device image 20a with a display theme corresponding to the desired content Cd is superimposed on the autonomous mobile device Ma so as to obscure the entire autonomous mobile device Ma opaquely or semi-transparently from in front of it.

[0055] The content image 20 in S40 may include an object image 20b shown in Figures 7 and 8 as an XR image superimposed on another object Mb that is viewed in the field of view Av together with the autonomous mobile device Ma as shown in Figure 2. The superimposed display position of the object image 20b is adjusted to match the predicted viewing position of the other object Mb, resulting in an overlapping display similar to that of the device image 20a. The other object Mb that is displayed with the object image 20b overlapping in this way may be, for example, at least one of a person other than the user Us of the processing system 1, a moving object other than the autonomous mobile device Ma that cooperates with the processing system 1, an aircraft, a structure, and plantings.

[0056] The content image 20 in S40 may include a ground image 20c shown in Figures 7 and 8 as an XR image superimposed on the ground Mc visible in the field of view Av together with the autonomous mobile device Ma as shown in Figure 2. The superimposed display position of the ground image 20c is adjusted to match the predicted viewing position of the ground Mc, thereby resulting in a similar overlapping display of the device image 20a.

[0057] In S40, the display target range Ad of the content image 20 is set to cover the entire view area Av as shown in Fig. 7, and the occupancy rate Rc is adjusted to a maximum value (e.g., 1). In S40, the display target range Ad of the content image 20 is set to disappear from the entire view area Av as shown in Fig. 9, and the occupancy rate Rc is adjusted to a minimum value (e.g., 0). In S40, the display target range Ad of the content image 20 is set to cover a portion of the view area Av as shown in Fig. 8, and the occupancy rate Rc is adjusted to an intermediate value between the minimum and maximum values ​​(e.g., greater than 0 and less than 1).

[0058] Furthermore, in S40, when a hazard notification to the user Us becomes necessary in accordance with the risk level Lr, a hazard image 22 as an XR image may be superimposed by the display unit 2 on the ground Mc and / or the sky Md within the field of view Av shown in FIG. 2, as shown in FIGS. 8 and 9. The superimposed display position of the hazard image 22 may be adjusted according to the predicted visual position of the ground Mc and / or the sky Md. This display position adjustment may also be based on information acquired from the infrastructure database Di via the communication system 10 as three-dimensional space information It, in which position-related information of the ground Mc and / or the sky Md is added to each voxel Vi in the space where the user service is provided.

[0059] In S40, an adjustment subroutine is executed to switch and adjust the occupancy rate Rc according to the risk level Lr in the field of view Av, as shown in FIG. 6 . Specifically, in S400 of the adjustment subroutine, the display control block 220 acquires internal sensing information Ii sensed by the sensor unit 3 of the wearable device Wt. At this time, the acquisition range of the internal sensing information Ii sensed by the sensor unit 3 with an acceptable level of accuracy or reliability is set to an internal sensing space Ai, which is part of the field of view Av, as shown in FIG. 10 . Therefore, the internal sensing information Ii acquired for the internal sensing space Ai is obtained from at least one type of sensor unit 3, such as a camera or a motion sensor.

[0060] The internal sensing space Ai in S400 is defined by a plurality of voxels Vi whose two-dimensional or three-dimensional distance is assumed to be within a set distance, with the user Us or the wearable device Wt at the center. The set distance that determines the limit range of the internal sensing space Ai is set to a fixed value, such as 5 m, depending on the specifications of the sensor unit 3 and / or the risk requirements for the processing system 1.

[0061] In S410 of the adjustment subroutine shown in Figure 6, the display control block 220 acquires a risk level Lr in the internal sensing space Ai from the field of view Av. At this time, the risk level Lr is defined as an index that will reach the hazard level Lh when at least a portion of an obstacle Mbo, such as a human, a moving object, or a structure, which is predicted to interfere with the movement of the user Us, relatively enters the internal sensing space Ai as shown in Figure 10. In other words, when the obstacle Mbo disappears from the internal sensing space Ai, the risk level Lr in the internal sensing space Ai is determined to be at a safe level that has escaped the hazard level Lh.

[0062] 6, the display control block 220 determines whether the risk level Lr in the internal sensing space Ai has reached a hazard level Lh that requires a hazard notification to the moving user Us. The comparison of the risk level Lr with the hazard level Lh may be performed for each voxel Vi in the user service provision space or for each voxel group consisting of multiple voxels Vi.

[0063] If a positive determination is made in S420, in S430 of the adjustment subroutine shown in Fig. 6, the display control block 220 adjusts the occupancy rate Rc to a minimum value by prohibiting superimposed display of the content image 20 over the entire field of view Av so as to exclude the entire field of view Av from the display target range Ad of the content image 20, as shown in Fig. 9. However, in S430, a hazard image 22 notifying warning text and / or a warning color in accordance with the risk level Lr that has reached the hazard level Lh may be superimposed on the field of view Av as shown in Fig. 9. In this case, the hazard image 22 may be displayed to notify at least one of, for example, the position, direction, distance, and type of the obstacle Mbo that caused the hazard level Lh in the internal sensing space Ai.

[0064] In addition to displaying the hazard image 22, in S430, a warning of the risk level Lr that has reached the hazard level Lh may be issued by a display output from a display unit possessed by the autonomous mobile device Ma or by an illumination output from a warning light. In S430, a warning of the risk level Lr that has reached the hazard level Lh may be issued by an audio output from a display unit possessed by the autonomous mobile device Ma. Completion of execution of S430 marks the end of the current execution of the adjustment subroutine and the processing flow.

[0065] If a negative determination is made in S420, in S440 of the adjustment subroutine shown in FIG. 6 , the display control block 220 acquires external sensing information Io sensed by the sensor unit 4 of the autonomous mobile device Ma. At this time, the acquisition range of the external sensing information Io sensed by the sensor unit 4 with an acceptable level of accuracy or reliability is set to an external sensing space Ao, which is part of the field of view Av and is wider than the internal sensing space Ai, as shown in FIG. 11 . Therefore, the external sensing information Io acquired for the external sensing space Ao may be obtained from at least one type of sensor unit 4, such as a camera, LiDAR, or sonar.

[0066] In S440, as the external sensing information Io acquired for the external sensing space Ao, three-dimensional space information It acquired from the infrastructure database Di via the communication system 10 may be obtained together with the sensing information by the sensor unit 4. In other words, the external sensing information Io may be three-dimensional space information It collected by the infrastructure system 6 and stored in the infrastructure database Di.

[0067] The external sensing space Ao in S440 is defined by a plurality of voxels Vi whose two-dimensional or three-dimensional distances are assumed to fall within a set range, with the autonomous mobile device Ma at the center. The set distance that determines the limit range Aol of the external sensing space Ao shown in FIG. 11 may be set to a fixed value, such as 6 m to 20 m, depending on the specifications of the sensor unit 4 and / or the risk requirements for the processing system 1.

[0068] At S450 of the adjustment subroutine shown in FIG. 6 , the display control block 220 acquires a risk level Lr in the external sensing space Ao from the field of view Av. The risk level Lr is defined as an index indicating that the hazard level Lh has been reached when the two-dimensional or three-dimensional distance δo between the limit range Ao1 of the external sensing space Ao and the user Us becomes equal to or less than a threshold, as shown in FIG. 11 . In other words, the risk level Lr in the external sensing space Ao is determined to be at a safe level that has escaped from the hazard level Lh when the distance δo between the limit range Ao1 of the space Ao and the user Us exceeds or exceeds a threshold.

[0069] 6, the display control block 220 determines whether the risk level Lr in the external sensing space Ao has reached a hazard level Lh that requires a hazard notification to the moving user Us. The comparison of the risk level Lr with the hazard level Lh may be performed for each voxel Vi in the user service provision space or for each voxel group consisting of multiple voxels Vi.

[0070] If a positive determination is made in S460, the display control block 220 in S470 of the adjustment subroutine shown in Fig. 6 identifies a hazard space portion Aoh in the external sensing space Ao where the risk level Lr has reached the hazard level Lh, as shown in Fig. 11. At this time, a plurality of voxels Vi where the risk level Lr has reached the hazard level Lh are recognized as the hazard space portion Aoh. Furthermore, a display prohibited range in the field of view Av, in which the superimposed display of a content image 20 described below is prohibited, may be recognized as the hazard space portion Aoh (see Fig. 8) so that the range continues to the far side of the external sensing space Ao.

[0071] Therefore, the display control block 220 in S470 adjusts the occupancy rate Rc to an intermediate value by restricting the hazard space portion Aoh that has reached the hazard level Lh to be excluded from the display target range Ad of the content image 20, as shown in Fig. 8, and prohibiting the superimposed display of the content image 20. As a result, the remaining space portion As of the safety level that is outside the hazard space portion Aoh in the view area Av is set to the display target range Ad of the content image 20, and the superimposed display of the content image 20 is permitted.

[0072] In S470, a hazard image 22 notifying a user of a risk level Lr that has reached the hazard level Lh with warning text and / or a warning color may be superimposed on the field of view Av as shown in Fig. 8. In this case, the hazard image 22 may be displayed to notify the user of an increased risk due to the user Us, who caused the hazard level Lh in the external sensing space Ao, approaching the limit range Ao1.

[0073] In addition to displaying the hazard image 22, in S470, a warning that the risk level Lr has reached the hazard level Lh may be issued by a display output from a display unit possessed by the autonomous mobile device Ma or by an illumination output from a warning light. In S470, a warning that the risk level Lr has reached the hazard level Lh may be issued by an audio output from a display unit possessed by the autonomous mobile device Ma. Completion of execution of S470 marks the end of the current execution of the adjustment subroutine and the processing flow.

[0074] If a negative determination is made in S460, the display control block 220 adjusts the occupancy rate Rc to the maximum value in S480 of the adjustment subroutine shown in FIG. 6 by permitting the content image 20 to be superimposed on the entire field of view Av as the display target range Ad of the content image 20, as shown in FIG. 7. At this time, a limitation image 20cl is displayed superimposed on the field of view Av as a ground image 20c representing the limit range Aol on the ground Mc of the external sensing space Ao, as shown in FIG. 7. Completion of S480 marks the end of the adjustment subroutine and the current execution of the processing flow. Therefore, in subsequent executions of the processing flow, the limitation image 20cl will continue to be displayed until a positive determination is made in either S420 or S460.

[0075] (Operations and Effects) Operations and effects of the first embodiment described above will be described below.

[0076] In the first embodiment, a field of view Av visually recognized by a user Us through a wearable device Wt worn by the user Us is monitored. The occupancy rate Rc of a display target range Ad of a content image 20 superimposed on the field of view Av as an XR image corresponding to a desired content Cd desired by the user Us is adjusted according to a risk level Lr in the field of view area Av.

[0077] At this time, particularly according to the first embodiment, external sensing information Io sensed by the autonomous mobile device Ma leading the user Us moving toward the field of view Av is acquired, and the risk level Lr based on the external sensing information Io is recognized. In this way, by effectively utilizing the autonomous mobile device Ma fulfilling the leading function for the user Us, the risk level Lr can be recognized for as wide a range as possible of the field of view Av to which the user Us is moving, and the safety of the user Us can be ensured.

[0078] Moreover, according to the first embodiment, in the external sensing space Ao, which is the acquisition range of the external sensing information Io in the field of view Av, the superimposed display of the content image 20 is prohibited in the hazard space portion Aoh where the risk level Lr has reached the hazard level Lh. This allows the content image 20 to disappear only within the hazard space portion Aoh at the hazard level Lh, which requires a hazard notification to the moving user Us, while the occupancy rate Rc can be adjusted so that the superimposed display of the content image 20 continues outside the hazard space portion Aoh in the field of view Av. Therefore, it is possible to balance the safety ensured by the above-described principle for the moving user Us with the continuity of the virtual experience.

[0079] According to the first embodiment, a limitation image 20cl is an XR image representing a limit range Aol on the ground Mc of the external sensing space Ao, and is superimposed on the field of view Av. This ensures the safety of the user Us by recognizing the risk level Lr in the external sensing space Ao, thereby ensuring a sense of security for the user Us. Therefore, coupled with the disappearance of the content image 20 limited to the hazard space Aoh, it is possible to ensure the safety and sense of security of the user Us while maintaining a balance with the continuity of the virtual experience.

[0080] According to the first embodiment, internal sensing information Ii sensed by the wearable device Wt is acquired, and a risk level Lr based on the internal sensing information Ii is recognized. This allows the wearable device Wt to ​​recognize the risk level Lr for the user Us's proximity range as well, ensuring the safety of the user Us. However, if the risk level Lr in the internal sensing space Ai within the field of view Av, which is the acquisition range of the internal sensing information Ii, reaches the hazard level Lh, superimposed display of the content image 20 is prohibited for the entire field of view Av. This makes it possible to intentionally adjust the occupancy rate Rc for the user Us's proximity range so as to prioritize safety over the continuity of the virtual experience.

[0081] According to the first embodiment, the external sensing information Io collected by the infrastructure system 6 may also be acquired together with the external sensing information Io sensed by the autonomous mobile device Ma, and the risk level Lr may be recognized based on the external sensing information Io. In this way, by effectively utilizing not only the autonomous mobile device Ma, which performs a leading function for the user Us, but also the infrastructure system 6, the risk level Lr for the field of view Av can be recognized with high accuracy, making it possible to ensure the safety of the user Us.

[0082] Second Embodiment The second embodiment is a modification of the first embodiment.

[0083] 12 , a processing system 2001 according to the second embodiment is connected to a mobile terminal Mt carried by a pre-registered user Us, an autonomous mobile device Ma that performs autonomous driving, and an infrastructure system 6 that provides infrastructure information, via a communication network Nc. Here, it is assumed that a plurality of mobile terminals Mt are connected to the processing system 1.

[0084] The mobile terminal Mt is configured so that it can be held and operated by the user Us with his or her fingers, at least when connected to the processing system 2001. As shown in Fig. 13 , the mobile terminal Mt is a small electronic device equipped with a display unit 2002 and a sensor unit 2003, such as a smartphone or a tablet terminal. The mobile terminal Mt controls the transmission and reception of data via the communication network Nc using a communication unit, as shown in Fig. 12 . The mobile terminal Mt is operated and controlled by the control unit in accordance with control commands from the processing system 2001 via the communication network Nc, so as to perform service-related processing in cooperation with the processing system 2001.

[0085] Under the operational control of the control unit of the mobile terminal Mt, a display unit 2002 shown in FIG. 13 , such as a liquid crystal panel or organic EL panel, realizes the screen display required for service-related processing (see FIGS. 17 to 19 described below). At the same time, under the operational control of the mobile terminal Mt, a sensor unit 2003 shown in FIG. 13 acquires sensing information through sensing processing corresponding to the service-related processing. The sensor unit 2003 may be, for example, at least one of a camera, an inertial sensor, a GNSS (Global Navigation Satellite System) sensor, a touch sensor, a microphone, or the like. This configuration enables the sensor unit 2003 to acquire input representing the user Us's intention as sensing information.

[0086] 12 is configured in the same manner as the processing system 1 of the first embodiment, except that the autonomous mobile device Ma is driven and controlled by a control unit in accordance with control commands from the processing system 2001 via a communication network Nc so as to perform service-related processing in cooperation with the processing system 2001. A control system 2009 of the processing system 2001 is configured in the same manner as the control system 9 of the first embodiment, except that at least a part of the control system 2009 may be configured by the control unit of the mobile terminal Mt, and that it executes a processing flow using the mobile terminal Mt, which will be described later.

[0087] In order to perform service-related processing in the control system 2009, a processing method in which the processing system 2001 performs XR display processing through cooperation of blocks 200, 210, and 220 constructed as shown in FIG. 14 is executed in accordance with the processing flows shown in FIGS. 15 and 16.

[0088] 15 , the recognition block 200 (see FIG. 14 ) monitors a background area Ab, which serves as a display background for the user Us from the display unit 2002, by pointing the mobile terminal Mt carried by the user Us. At this time, the background area Ab is recognized as an area extending in the direction in which the mobile terminal Mt is facing, based on at least one type of sensing information obtained by the sensor unit 2003 of the mobile terminal Mt, such as camera information and inertial information. Note that during the execution of each step subsequent to S2010 of the processing flow, the latest recognized background area Ab may be updated by concurrently executing a monitoring process for the background area Ab as the background area Ab changes due to the movement of the user Us.

[0089] 15 , the recognition block 200 acquires the desired content Cd through recognition based on sensing information from the mobile terminal Mt carried by the user Us. The sensing information at this time is acquired via the mobile terminal Mt at the timing when the sensing information is input to the sensor unit 2003 as the user Us's intention data, and is used to recognize the desired content Cd. Details regarding the input and items of the desired content Cd conform to S20 of the first embodiment.

[0090] In S2030 of the processing flow shown in Figure 15, the drive control block 210 (see Figure 14) provides a user service by controlling the drive of the autonomous mobile device Ma within the background area Ab. At this time, a drive control pattern for the autonomous mobile device Ma that matches the type of user service is read from the memory 9a, and the user service is provided in accordance with the drive control pattern so as to lead the user Us moving toward the background area Ab. Details regarding the drive of the autonomous mobile device Ma and the provision of the user service are similar to S30 of the first embodiment.

[0091] S2040 in the processing flow shown in FIG. 15 is executed in parallel with S2030. In S2040, the display control block 220 (see FIG. 14 ) causes the display unit 2002 of the mobile terminal Mt to superimpose an XR image corresponding to the desired content Cd on the background video 24, which shows the background area Ab in which the autonomous mobile device Ma, which is providing the user service in S2030, is located. At this time, the display control block 220 adjusts the occupancy Rc of the display target range Ad of the content image 20 of the XR image, which is superimposed and displayed according to the desired content Cd as shown in FIGS. 17 and 18 , in the background video 24, in accordance with the risk level Lr of the area Ab. Here, the occupancy Rc in the second embodiment refers to the ratio of the display area of ​​the entire display target range Ad to the display area of ​​the entire background video 24 on the display unit 2002.

[0092] In S2040, the background video 24 to be superimposed on the content image 20 is acquired based on, for example, sensing information from a sensor unit 4 such as a camera, or three-dimensional space information It obtained from an infrastructure database Di. The background video 24 may be primary video data obtained by capturing a background area Ab, or secondary video data obtained by processing the primary data.

[0093] The content image 20 in S2040 includes at least a device image 20a shown in FIGS. 17 and 18 as an XR image superimposed on a video portion corresponding to the position of the autonomous mobile device Ma in the background video 24, as shown in FIG. 13 . The superimposed display position of the device image 20a is adjusted to match the imaging position of the autonomous mobile device Ma on the background video 24, in accordance with the drive control pattern in S2030. As a result, a replacement display is realized in which the device image 20a with a display theme corresponding to the desired content Cd is superimposed so as to replace the entire autonomous mobile device Ma shown in the background video 24. Details regarding the adjustment of the display position of the device image 20a are similar to S40 of the first embodiment.

[0094] The content image 20 in S2040 may include an object image 20b shown in FIGS. 17 and 18 as an XR image superimposed on a video portion corresponding to the position of another object Mb that exists in the background area Ab together with the autonomous mobile device Ma as shown in FIG. 13 . The superimposed display position of the object image 20b is adjusted to match the imaging position of the other object Mb on the background video 24, resulting in a replacement display similar to the device image 20a. Details of the other object Mb that is replaced by the object image 20b are similar to S40 of the first embodiment.

[0095] The content image 20 in S2040 may include a ground image 20c shown in Figures 17 and 18 as an XR image superimposed on the video portion, corresponding to the position of the ground Mc that exists in the background area Ab together with the autonomous mobile device Ma on the background video 24 as shown in Figure 13. The superimposed display position of the ground image 20c is adjusted to match the imaging position of the ground Mc on the background video 24, resulting in a replacement display similar to that of the device image 20a.

[0096] In S2040, the display target range Ad of the content image 20 is set to cover the entire background video 24 as shown in Fig. 17, and the occupancy rate Rc is adjusted to a maximum value (for example, 1). In S2040, the display target range Ad of the content image 20 is set to disappear from the entire background video 24 as shown in Fig. 19, and the occupancy rate Rc is adjusted to a minimum value (for example, 0). In S2040, the display target range Ad of the content image 20 is set to cover a portion of the background video 24 as shown in Fig. 18, and the occupancy rate Rc is adjusted to an intermediate value between the minimum and maximum values ​​(for example, greater than 0 and less than 1).

[0097] Furthermore, in S2040, when a hazard notification to the user Us becomes necessary in accordance with the risk level Lr, the display unit 2002 may superimpose a hazard image 22 as an XR image as shown in Figures 18 and 19 on the ground Mc and / or the sky Md shown in the background video 24 as shown in Figure 13. The superimposed display position of the hazard image 22 may be adjusted to match the imaging position of the ground Mc and / or the sky Md on the background video 24. In this case, details regarding the adjustment of the display position of the hazard image 22 are similar to S40 of the first embodiment.

[0098] In S2040, an adjustment subroutine is executed to switch and adjust the occupancy rate Rc in accordance with the risk level Lr in the background area Ab, as shown in FIG. 16 . Specifically, in S2400 of the adjustment subroutine, the display control block 220 acquires internal sensing information Ii sensed by the sensor unit 2003 of the mobile terminal Mt. At this time, the acquisition range of the internal sensing information Ii sensed by the sensor unit 2003 with an acceptable level of accuracy or reliability is set to an internal sensing space Ai that is part of the background area Ab, as shown in FIG. 20 . Details regarding the acquisition of the internal sensing information Ii and the definition and setting of the internal sensing space Ai are similar to those of S400 in the first embodiment.

[0099] 16, the display control block 220 acquires the risk level Lr in the internal sensing space Ai of the background area Ab. The details of the definition and determination of the risk level Lr conform to S410 of the first embodiment. Furthermore, in S2420 of the adjustment subroutine, the display control block 220 determines the risk level Lr in accordance with S410 of the first embodiment.

[0100] If a positive determination is made in S2420, in S2430 of the adjustment subroutine shown in FIG. 16 , the display control block 220 adjusts the occupancy rate Rc to the minimum value by prohibiting superimposed display of the content image 20 over the entire background video 24, as shown in FIG. 19 , so as to exclude the entire background video 24 from the display target range Ad of the content image 20. However, in S2430, a hazard image 22 notifying a user of warning text and / or a warning color in accordance with a risk level Lr that has reached the hazard level Lh may be superimposed on the background video 24, as shown in FIG. 19 . In this case, details regarding the display of the hazard image 22 and other warning methods conform to S430 in the first embodiment. Completion of execution of S2430 marks the end of the adjustment subroutine and the current execution of the processing flow.

[0101] If a negative determination is made in S2420, the display control block 220 acquires the external sensing information Io in S2440 of the adjustment subroutine shown in Fig. 16 in accordance with S440 of the first embodiment. However, at this time, the acquisition range of the external sensing information Io, for which the accuracy or reliability of the sensor unit 4 is at an acceptable level, is set to an external sensing space Ao, which is part of the background area Ab as shown in Fig. 21 and which is wider than the internal sensing space Ai.

[0102] 16, the display control block 220 acquires a risk level Lr in the external sensing space Ao of the background area Ab. Details regarding the definition and determination of the risk level Lr conform to S450 of the first embodiment. Furthermore, in S2460 of the adjustment subroutine, the display control block 220 determines the risk level Lr in accordance with S460 of the first embodiment.

[0103] If a positive determination is made in S2460, in S2470 of the adjustment subroutine shown in Fig. 16, the display control block 220 identifies a hazard space portion Aoh within the external sensing space Ao in accordance with S470 of the first embodiment, as shown in Fig. 21. However, in the background area Ab, a display prohibited range in which the superimposed display of the content image 20 is prohibited as described below may be recognized as the hazard space portion Aoh (see Fig. 18) so that the range continues to the far side of the external sensing space Ao.

[0104] Therefore, in S2470, the display control block 220 adjusts the occupancy rate Rc to an intermediate value by restricting the video portion 24h showing the hazard space portion Aoh of the background video 24 showing the external sensing space Ao to be excluded from the display target range Ad of the content image 20, as shown in FIG. 18 , thereby prohibiting the superimposed display of the content image 20. As a result, the remaining portion 24s of the background video 24 other than the video portion 24h is set in the display target range Ad of the content image 20 in correspondence with the safety level remaining space portion As outside the hazard space portion Aoh in the background area Ab, thereby permitting the superimposed display of the content image 20. Details of the display of the hazard image 22 and other warning methods at this time conform to S470 in the first embodiment. Completion of execution of S2470 marks the end of the current execution of the adjustment subroutine and processing flow.

[0105] If a negative determination is made in S2460, the display control block 220 adjusts the occupancy rate Rc to the maximum value in S2480 of the adjustment subroutine shown in FIG. 16 by permitting the content image 20 to be superimposed on the entire background video 24, as shown in FIG. 17, so as to set the entire background video 24 as the display target range Ad of the content image 20. At this time, a limitation image 20cl is displayed superimposed on the background video 24 as a ground image 20c representing the limit range Aol on the ground Mc of the external sensing space Ao, as shown in FIG. 17. Completion of S2480 marks the end of the adjustment subroutine and the current execution of the processing flow. Therefore, in subsequent executions of the processing flow, the limitation image 20cl will continue to be displayed until a positive determination is made in either S2420 or S2460.

[0106] (Operations and Effects) Operations and effects of the second embodiment described above will be described below.

[0107] In the second embodiment, a background area Ab viewed by the user Us through a mobile terminal Mt carried by the user Us is monitored. The content image 20 is superimposed on a background video 24 showing the background area Ab as an XR image corresponding to a desired content Cd desired by the user Us. The occupancy rate Rc of the display target range Ad of the content image 20 in the background video 24 is adjusted according to a risk level Lr in the background area Ab.

[0108] In this case, particularly according to the second embodiment, external sensing information Io sensed by the autonomous mobile device Ma leading the user Us moving toward the background area Ab is acquired, and the risk level Lr based on the external sensing information Io is recognized. In this way, by effectively utilizing the autonomous mobile device Ma fulfilling the leading function for the user Us, the risk level Lr can be recognized for as wide a range as possible of the background area Ab to which the user Us is moving, and the safety of the user Us can be ensured.

[0109] Moreover, according to the second embodiment, in the background video 24 showing the external sensing space Ao, which is the acquisition range of the external sensing information Io in the background area Ab, the content image 20 is prohibited from being superimposed on the video portion 24h showing the hazard space portion Aoh where the risk level Lr has reached the hazard level Lh. As a result, in the background video 24, the content image 20 disappears only in the portion 24h corresponding to the inside of the hazard space portion Aoh at the hazard level Lh, which requires a hazard notification to the moving user Us, while the occupancy rate Rc can be adjusted so that the content image 20 continues to be superimposed on the portion 24s corresponding to the outside of the hazard space portion Aoh in the background area Ab. Therefore, it is possible to balance the safety ensured by the above-mentioned principle for the moving user Us with the continuity of the virtual experience.

[0110] According to the second embodiment, a limitation image 20cl, which is an XR image representing a limit range Aol on the ground Mc of the external sensing space Ao, is superimposed on a background video 24 showing the background area Ab. Accordingly, when the limit range Aol of the external sensing space Ao, in which the risk level Lr is recognized, is presented to the user Us, the user Us's safety is ensured by recognizing the risk level Lr in the external sensing space Ao, thereby ensuring the user Us's sense of security. Therefore, coupled with the disappearance of the content image 20 limited to the portion 24h corresponding to the hazard space Aoh, the safety and sense of security of the user Us can be ensured in a balance with the continuity of the virtual experience.

[0111] According to the second embodiment, internal sensing information Ii sensed by the mobile terminal Mt is acquired, and a risk level Lr based on the internal sensing information Ii is recognized. This allows the mobile terminal Mt to recognize the risk level Lr for the user Us's proximity range as well, ensuring the safety of the user Us. However, if the risk level Lr in the internal sensing space Ai within the background area Ab, which is the acquisition range of the internal sensing information Ii, reaches the hazard level Lh, superimposition of the content image 20 is prohibited over the entire background video 24 that displays the background area Ab. This makes it possible to intentionally adjust the occupancy rate Rc within the user Us's proximity range so as to prioritize safety over the continuity of the virtual experience.

[0112] According to the second embodiment, external sensing information Io collected by the infrastructure system 6 may also be acquired together with external sensing information Io sensed by the autonomous mobile device Ma, and a risk level Lr may be recognized based on the external sensing information Io. In this way, by effectively utilizing not only the autonomous mobile device Ma, which performs a leading function for the user Us, but also the infrastructure system 6, the risk level Lr for the background area Ab can be recognized with high accuracy, thereby ensuring the safety of the user Us.

[0113] (Other Embodiments) Although multiple embodiments have been described above, the present disclosure should not be construed as being limited to those embodiments, and can be applied to various embodiments and combinations within the scope that does not deviate from the gist of the present disclosure.

[0114] In the modified examples of the first and second embodiments, the dedicated computer constituting the control system 9, 2009 of the processing system 1, 2001 may have at least one of a digital circuit and an analog circuit as a processor. Here, the digital circuit is at least one of an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a system on a chip (SOC), a programmable gate array (PGA), and a complex programmable logic device (CPLD). Furthermore, such a digital circuit may have a memory that stores a program.

[0115] In a modification of the first and second embodiments, the processing system 1, 2001 may be disconnected from the infrastructure system 6. In this modification, sensing information from connection elements Wt, Ma, and Mt with the processing system 1, 2001 may be used instead of the three-dimensional spatial information It obtained from the infrastructure database Di.

[0116] In the modified examples of the first and second embodiments, three-dimensional spatial information It that is not linked to the voxel Vi may be acquired from the infrastructure database Di by the processing system 1, 2001. In the modified examples of the first and second embodiments, two-dimensional grid information may be acquired from the infrastructure database Di by the processing system 1, 2001 instead of the three-dimensional spatial information It.

[0117] In a modification of the first embodiment, when a mobile terminal Mt as in the second embodiment is connected to the processing system 1 and functions as part of a wearable terminal Wt, the processing flow may be executed in response to the sensor unit 2003 constituting the part acquiring a service request input as sensing information. In a modification of the first embodiment, when a mobile terminal Mt as in the second embodiment is connected to the processing system 1 and functions as part of a wearable terminal Wt, the desired content Cd in S20 of the processing flow may be acquired based on sensing information of the sensor unit 2003 constituting the part.

[0118] In a modification of the second embodiment, a smart watch or a video see-through or non-transparent wearable terminal may be used as the mobile terminal Mt. In the modification of the second embodiment, when a wearable terminal Wt as in the first embodiment is connected to the processing system 2001 and functions as part of the mobile terminal Mt, the processing flow may be executed in response to the sensor unit 3 constituting the part acquiring a service request input as sensing information. In the modification of the second embodiment, when a wearable terminal Wt as in the first embodiment is connected to the processing system 2001 and functions as part of the mobile terminal Mt, the desired content Cd in S2020 of the processing flow may be acquired based on sensing information from the sensor unit 3 constituting the part.

[0119] In modified examples of the first and second embodiments, display of the limitation image 20cl may be omitted in S480 and S2480. In modified examples of the first and second embodiments, the limitation image 20cl may also be displayed in the display target range Ad of the content image 20 in S470 and S2470. In modified example of the first embodiment, in S430 when the risk level Lr in the internal sensing space Ai has reached the hazard level Lh, superimposed display of the content image 20 may be prohibited only within the internal sensing space Ai in the field of view Av. In modified example of the second embodiment, in S2430 when the risk level Lr in the internal sensing space Ai has reached the hazard level Lh, superimposed display of the content image 20 may be prohibited only within the moving image portion of the background area Ab corresponding to the internal sensing space Ai.

[0120] In addition to the forms described so far, the above-mentioned embodiments and variations may be implemented in the form of a semiconductor device (e.g., a semiconductor chip) as a processing system 1, 2001 having at least one processor 9b and one memory 9a in a control system 9, 2009.

[0121] (Additional Remarks) This specification discloses the following technical ideas and combinations thereof.

[0122] (Technical Idea 1) A processing system having a processor (9b) and performing XR display-related processing related to displaying XR images to a moving user, wherein the processor is configured to execute the following: monitor a field of view (Av) viewed by the user through a wearable terminal (Wt) worn by the user; and adjust an occupancy rate (Rc) of a display target range (Ad) of a content image (20) to be superimposed on the field of view as the XR image corresponding to a desired content (Cd) desired by the user in accordance with a risk level (Lr) in the field of view, wherein adjusting the occupancy rate includes: acquiring external sensing information (Io) sensed by an autonomous driving device (Ma) leading the user moving toward the field of view; and recognizing the risk level based on the external sensing information. A processing system including: adjusting the occupancy rate by prohibiting the superimposed display of the content image for a hazard space portion (Aoh) of the external sensing space (Ao) in the field of view area that is the acquisition range of the external sensing information, where the risk level has reached a hazard level that requires hazard notification to the moving user.

[0123] (Technical Idea 2) The processing system described in Technical Idea 1, wherein adjusting the occupancy rate includes superimposing a limitation image (20cl) as the XR image representing the limit range (Aol) on the ground (Mc) of the external sensing space on the field of view.

[0124] (Technical Idea 3) The processing system described in Technical Idea 1 or 2, wherein adjusting the occupancy rate includes: acquiring internal sensing information (Ii) sensed by the wearable terminal; recognizing the risk level based on the internal sensing information; and adjusting the occupancy rate by prohibiting the superimposed display of the content image over the entire field of view when the risk level in an internal sensing space (Ai) within the field of view that is the acquisition range of the internal sensing information reaches a hazard level that requires a hazard notification to the moving user.

[0125] (Technical Idea 4) A processing system having a processor (9b) and performing XR display-related processing related to displaying an XR image to a moving user, wherein the processor is configured to execute the following: monitor a background area (Ab) that serves as a display background for the user from a mobile terminal (Mt) carried by the user; and adjust an occupancy rate (Rc) of a display target range (Ad) of a content image (20) that is superimposed on a background video (24) showing the background area as the XR image according to a desired content (Cd) desired by the user in the background video according to a risk level (Lr) in the background area, wherein adjusting the occupancy rate includes: acquiring external sensing information (Io) sensed by an autonomous mobile device (Ma) leading the user moving toward the background area; and recognizing the risk level based on the external sensing information. and adjusting the occupancy rate by prohibiting the superimposition of the content image on a video portion (24h) of the background video that shows an external sensing space (Ao) in the background area that is the acquisition range of the external sensing information, the hazard space portion (Aoh) where the risk level has reached a hazard level that requires hazard notification to the moving user.

[0126] (Technical Idea 5) The processing system described in Technical Idea 4, wherein adjusting the occupancy rate includes superimposing a limitation image (20cl) as the XR image representing the limit range (Aol) on the ground (Mc) of the external sensing space on the background video.

[0127] (Technical Idea 6) The processing system described in Technical Idea 4 or 5, wherein adjusting the occupancy rate includes: acquiring internal sensing information (Ii) sensed by the mobile terminal; recognizing the risk level based on the internal sensing information; and adjusting the occupancy rate by prohibiting the superimposed display of the content image over the entire area of ​​the background video when the risk level in an internal sensing space (Ai) within the background area that is the acquisition range of the internal sensing information reaches a hazard level that requires a hazard notification to the moving user.

[0128] (Technical Idea 7) The processing system according to any one of Technical Ideas 1 to 6, wherein adjusting the occupancy rate includes acquiring the external sensing information collected by an infrastructure system (6).

[0129] The above-mentioned technical concepts 1 to 7 may be understood as the respective technical concepts of the method and the program.

Claims

1. A processing system having a processor (9b) and performing XR display-related processing related to displaying XR images to a moving user, wherein the processor is configured to execute the following: monitoring a field of view (Av) viewed by the user through a wearable terminal (Wt) worn by the user; and adjusting an occupancy rate (Rc) of a display target range (Ad) of a content image (20) to be superimposed on the field of view as the XR image corresponding to a desired content (Cd) desired by the user in accordance with a risk level (Lr) in the field of view, wherein adjusting the occupancy rate includes: acquiring external sensing information (Io) sensed by an autonomous driving device (Ma) leading the user moving toward the field of view; and recognizing the risk level based on the external sensing information. and adjusting the occupancy rate by prohibiting the superimposed display of the content image for a hazard space portion (Aoh) of the external sensing space (Ao) which is the acquisition range of the external sensing information in the field of view area and where the risk level has reached a hazard level requiring a hazard notification to the moving user.

2. The processing system of claim 1, wherein adjusting the occupancy rate includes superimposing a limitation image (20cl) as the XR image representing a limit range (Aol) on the ground (Mc) of the external sensing space on the field of view area.

3. The processing system of claim 1, wherein adjusting the occupancy rate includes: acquiring internal sensing information (Ii) sensed by the wearable device; recognizing the risk level based on the internal sensing information; and, when the risk level in an internal sensing space (Ai) within the field of view that is the acquisition range of the internal sensing information reaches a hazard level that requires a hazard notification to the moving user, adjusting the occupancy rate by prohibiting the superimposed display of the content image over the entire field of view.

4. A processing system having a processor (9b) and performing XR display-related processing related to displaying an XR image to a moving user, wherein the processor is configured to execute the following: monitoring a background area (Ab) that serves as a display background for the user from a mobile terminal (Mt) carried by the user; and adjusting an occupancy rate (Rc) of a display target range (Ad) of a content image (20) that is superimposed on a background video (24) showing the background area as the XR image according to a desired content (Cd) desired by the user, in accordance with a risk level (Lr) in the background area, the adjusting of the occupancy rate comprising: acquiring external sensing information (Io) sensed by an autonomous driving device (Ma) leading the user moving toward the background area; and recognizing the risk level based on the external sensing information; and adjusting the occupancy rate by prohibiting the superimposition of the content image on a video portion (24h) of the background video showing an external sensing space (Ao) in the background area that is the acquisition range of the external sensing information, the video portion showing a hazard space portion (Aoh) where the risk level has reached a hazard level requiring a hazard notification to the moving user.

5. The processing system of claim 4, wherein adjusting the occupancy rate includes superimposing a limitation image (20cl) as the XR image representing a limit range (Aol) on the ground (Mc) of the external sensing space on the background video.

6. The processing system of claim 4, wherein adjusting the occupancy rate includes: acquiring internal sensing information (Ii) sensed by the mobile terminal; recognizing the risk level based on the internal sensing information; and, when the risk level in an internal sensing space (Ai) of the background area that is the acquisition range of the internal sensing information reaches a hazard level that requires a hazard notification to the moving user, adjusting the occupancy rate by prohibiting the superimposition of the content image over the entire area of ​​the background video.

7. The processing system according to claim 1 or 4, wherein adjusting the occupancy rate includes obtaining the external sensing information collected by an infrastructure system (6).

8. A processing method executed by a processor (9b) to perform XR display-related processing related to displaying XR images to a moving user, comprising: monitoring a field of view (Av) viewed by the user through a wearable terminal (Wt) worn by the user; and adjusting an occupancy rate (Rc) of a display target range (Ad) of a content image (20) to be superimposed on the field of view as the XR image corresponding to a desired content (Cd) desired by the user in accordance with a risk level (Lr) in the field of view, wherein adjusting the occupancy rate includes: acquiring external sensing information (Io) sensed by an autonomous driving device (Ma) leading the user moving toward the field of view; and recognizing the risk level based on the external sensing information. and adjusting the occupancy rate by prohibiting the superimposed display of the content image for a hazard space portion (Aoh) of the external sensing space (Ao) which is the acquisition range of the external sensing information in the field of view area and where the risk level has reached a hazard level requiring a hazard notification to the moving user.

9. A processing method executed by a processor (9b) to perform an XR display-related process related to displaying an XR image to a moving user, comprising: monitoring a background area (Ab) that is a display background for the user from a mobile terminal (Mt) carried by the user; and adjusting an occupancy rate (Rc) of a display target range (Ad) of a content image (20) that is superimposed on a background video (24) showing the background area as the XR image according to a desired content (Cd) desired by the user in the background video according to a risk level (Lr) in the background area, wherein adjusting the occupancy rate includes: acquiring external sensing information (Io) sensed by an autonomous driving device (Ma) leading the user moving toward the background area; and recognizing the risk level based on the external sensing information; and adjusting the occupancy rate by prohibiting the superimposition of the content image on a video portion (24h) of the background video showing an external sensing space (Ao) in the background area which is the acquisition range of the external sensing information, the video portion showing a hazard space portion (Aoh) where the risk level has reached a hazard level requiring a hazard notification to the moving user.

10. A processing program stored in a storage medium (9a) for performing XR display-related processing related to displaying XR images to a moving user, the processing program including instructions for causing a processor (9b) to execute the XR display-related processing, the processing program including instructions for executing the following: monitoring a field of view (Av) viewed by the user through a wearable terminal (Wt) worn by the user; and adjusting an occupancy rate (Rc) of a display target range (Ad) of a content image (20) to be superimposed on the field of view as the XR image corresponding to a desired content (Cd) desired by the user, in the field of view, in accordance with a risk level (Lr) in the field of view, the adjusting of the occupancy rate comprising: acquiring external sensing information (Io) sensed by an autonomous driving device (Ma) leading the user moving toward the field of view; and recognizing the risk level based on the external sensing information. a processing program including: adjusting the occupancy rate by prohibiting the superimposed display of the content image for a hazard space portion (Aoh) of the external sensing space (Ao) which is the acquisition range of the external sensing information in the field of view area and where the risk level has reached a hazard level requiring a hazard notification to the moving user.

11. A processing program stored in a storage medium (9a) for performing an XR display-related process related to the display of an XR image to a moving user, the processing program including instructions for causing a processor (9b) to execute the XR display-related process, the processing program including instructions for executing the following: monitoring a background area (Ab) that serves as a display background to the user from a mobile terminal (Mt) carried by the user; and adjusting an occupancy rate (Rc) of a display target range (Ad) of a content image (20) that is superimposed on a background video (24) showing the background area as the XR image according to a desired content (Cd) desired by the user, in accordance with a risk level (Lr) in the background area, the adjusting of the occupancy rate including: acquiring external sensing information (Io) sensed by an autonomous driving device (Ma) leading the user moving toward the background area; and recognizing the risk level based on the external sensing information; and adjusting the occupancy rate by prohibiting the superimposition of the content image on a video portion (24h) of the background video showing an external sensing space (Ao) in the background area which is the acquisition range of the external sensing information, the video portion showing a hazard space portion (Aoh) where the risk level has reached a hazard level requiring a hazard notification to the moving user.

Images