Processing system, processing method, and processing program

Abstract

A processor of a processing system performs the following: monitoring a field of view (Av) visually recognized by a user through a wearable terminal; and adjusting, in accordance with the risk level (Lr) in the field-of-view region (Av), the occupancy (Rc) of a display target range (Ad) of a content image displayed on the field-of-view region (Av) as an XR image corresponding to the content desired by the user in an overlapping manner, in the field-of-view region, on the basis of the risk level (Lr) in the field-of-view region (Av). The adjustment of the occupancy (Rc) includes: acquiring external sensing information (Io) of an autonomous travel device that guides a user; identifying a risk level (Lr) based on the external sensing information (Io); and a step for adjusting the occupancy (Rc) by prohibiting the superimposed display of the content image in a danger space section (Aoh) in which the risk level has reached the danger level in an external sensing space (Ao) that is the acquisition range of the external sensing information (Ii) in the field of view (Av).

Description

[0001] Cross-reference of related applications

[0002] This application is based on Japanese Patent Application No. 2023-106120 filed on June 28, 2023, and incorporates the contents of the base application in its entirety by reference. Technical Field

[0003] This disclosure relates to a processing technique for performing XR display-related processing associated with displaying XR images to a user. Background Technology

[0004] The technology disclosed in Patent Document 1 enables users to experience virtual reality by displaying VR images as XR images in an HMD (Head Mounted Display). In this technology, the HMD displays a perspective view, particularly when the user's movement crosses a boundary and the HMD detects a collision risk in the physical environment.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Publication No. 10-504917.

[0008] However, in the technology disclosed in Patent Document 1, the boundary position of the collision risk is set based on the user's position and triggered by the movement of the user wearing the HMD. Therefore, the range within which safety can be ensured by displaying a see-through display is limited to the user's approach range. As a result, in the technology disclosed in Patent Document 1, where the VR image disappears entirely due to the see-through display, the continuity of the virtual experience is particularly hindered in applications involving moving users. Summary of the Invention

[0009] The present disclosure addresses the challenge of providing a processing system that balances mobile user security with the continuity of the virtual experience while performing XR display-related processing. Another challenge of the present disclosure is to provide a processing method that balances mobile user security with the continuity of the virtual experience while performing XR display-related processing. Yet another challenge of the present disclosure is to provide a processing program that balances mobile user security with the continuity of the virtual experience while performing XR display-related processing.

[0010] The technical means of this disclosure used to solve the problem will now be described.

[0011] The first method of this disclosure is,

[0012] A processing system having a processor that performs XR display-related processing associated with displaying XR images to a moving user.

[0013] The processor is configured to execute the following:

[0014] Monitoring the area of ​​the user's field of vision as perceived by the user through a wearable device; and

[0015] The occupancy of the displayed object within the field of view is adjusted based on the risk level within the field of view. This adjustment is made by overlaying the content image (which corresponds to the user's desired content) onto the field of view.

[0016] Adjusting market share includes the following:

[0017] Acquire external sensing information sensed by an autonomous driving device that guides a user moving toward the field of vision;

[0018] Identify risk levels based on externally sensed information; and

[0019] Overlapping of content images is prohibited in dangerous spaces to adjust occupancy. A dangerous space is an external sensing space within the field of view that is within the range of external sensing information acquisition and whose risk level has reached the point where a hazard notification to a moving user is required.

[0020] The second method disclosed herein is,

[0021] A processing method, executed by a processor, to perform XR display-related processing associated with displaying an XR image to a moving user.

[0022] This processing method includes the following:

[0023] Monitoring the area of ​​the user's field of vision as perceived by the user through a wearable device; and

[0024] The occupancy of the displayed object within the field of view is adjusted based on the risk level within the field of view. This adjustment is made by overlaying the content image (which corresponds to the user's desired content) onto the field of view.

[0025] Adjusting market share includes the following:

[0026] Acquire external sensing information sensed by an autonomous driving device that guides a user moving toward the field of vision;

[0027] Identify risk levels based on externally sensed information; and

[0028] Overlapping of content images is prohibited in dangerous spaces to adjust occupancy. A dangerous space is an external sensing space within the field of view that is within the range of external sensing information acquisition and whose risk level has reached the point where a hazard notification to a moving user is required.

[0029] The third method disclosed herein is,

[0030] A processing program, stored in a storage medium for performing XR display-related processing associated with displaying an XR image to a moving user, and comprising commands for causing a processor to perform the XR display-related processing.

[0031] This handler contains commands for performing the following:

[0032] Monitoring the area of ​​the user's field of vision as perceived by the user through a wearable device; and

[0033] The occupancy of the displayed object within the field of view is adjusted based on the risk level within the field of view. This adjustment is made by overlaying the content image (which corresponds to the user's desired content) onto the field of view.

[0034] Adjusting market share includes the following:

[0035] Acquire external sensing information sensed by an autonomous driving device that guides a user moving toward the field of vision;

[0036] Identify risk levels based on externally sensed information; and

[0037] Overlapping of content images is prohibited in dangerous spaces to adjust occupancy. A dangerous space is an external sensing space within the field of view that is within the range of external sensing information acquisition and whose risk level has reached the point where a hazard notification to a moving user is required.

[0038] In this way, in the first to third methods, the field of view area, which is visually recognized by the user through a wearable terminal worn by the user, is monitored. Therefore, the occupancy of the displayed object area in the field of view area, which is an XR image corresponding to the user's desired content, is adjusted according to the risk level in the field of view area.

[0039] At this point, specifically according to the first to third methods, risk levels based on external sensing information are identified by acquiring external sensing information sensed by the autonomous driving device that guides the user moving toward the field of vision. Accordingly, by effectively utilizing the autonomous driving device that guides the user, risk levels can be identified over the widest possible range of the field of vision that becomes the user's destination, thereby ensuring the user's safety.

[0040] Furthermore, according to the first to third methods, for dangerous areas within the external sensing space that fall within the field of view and are considered the acquisition range of external sensing information, where the risk level reaches a dangerous level, the overlapping display of content images is prohibited. Accordingly, the occupancy rate can be adjusted such that: content images disappear only within dangerous areas where a dangerous level notification to a moving user is required; on the other hand, overlapping display of content images continues outside dangerous areas within the field of view. Therefore, for moving users, a balance can be achieved between security ensured according to the principles described above and the continuity of the virtual experience.

[0041] The fourth method disclosed herein is,

[0042] A processing system having a processor that performs XR display-related processing associated with displaying XR images to a moving user.

[0043] The processor is configured to execute the following:

[0044] Monitor the background area, which becomes the display background from the user's mobile terminal; and

[0045] The proportion of the display object's area within the background video is adjusted based on the risk level in the background area, using an XR image that corresponds to the user's desired content and is overlaid on the background video.

[0046] Adjusting market share includes the following:

[0047] Acquire external sensing information sensed by an autonomous driving device that guides a user moving toward a background area;

[0048] Identify risk levels based on externally sensed information; and

[0049] The overlapping display of content images in the background video of the existing external sensing space and the video portion of the existing dangerous space is prohibited to adjust the occupancy. The external sensing space becomes the range for obtaining external sensing information in the background area. The dangerous space is a space where the risk level has reached the level that requires a hazard notification to the moving user.

[0050] The fifth method disclosed herein is,

[0051] A processing method, executed by a processor, to perform XR display-related processing associated with displaying an XR image to a moving user.

[0052] This processing method includes the following:

[0053] Monitor the background area, which becomes the display background from the user's mobile terminal; and

[0054] The proportion of the display object's area within the background video is adjusted based on the risk level in the background area, using an XR image that corresponds to the user's desired content and is overlaid on the background video.

[0055] Adjusting market share includes the following:

[0056] Acquire external sensing information sensed by an autonomous driving device that guides a user moving toward a background area;

[0057] Identify risk levels based on externally sensed information; and

[0058] The overlapping display of content images in the background video of the existing external sensing space and the video portion of the existing dangerous space is prohibited to adjust the occupancy. The external sensing space becomes the range for obtaining external sensing information in the background area. The dangerous space is a space where the risk level has reached the level that requires a hazard notification to the moving user.

[0059] The sixth method disclosed herein is,

[0060] A processing program, stored in a storage medium for performing XR display-related processing associated with displaying an XR image to a moving user, and comprising commands for causing a processor to perform the XR display-related processing.

[0061] This handler contains commands for performing the following:

[0062] Monitor the background area, which becomes the display background from the user's mobile terminal; and

[0063] The proportion of the display object's area within the background video is adjusted based on the risk level in the background area, using an XR image that corresponds to the user's desired content and is overlaid on the background video.

[0064] Adjusting market share includes the following:

[0065] Acquire external sensing information sensed by an autonomous driving device that guides a user moving toward a background area;

[0066] Identify risk levels based on externally sensed information; and

[0067] The overlapping display of content images in the background video of the existing external sensing space and the video portion of the existing dangerous space is prohibited to adjust the occupancy. The external sensing space becomes the range for obtaining external sensing information in the background area. The dangerous space is a space where the risk level has reached the level that requires a hazard notification to the moving user.

[0068] In methods four through six, a background area is monitored, which becomes the display background from the user's mobile terminal to the user. Therefore, the occupancy of the display object range in the background video, which is overlaid on the background video as an XR image corresponding to the user's desired content, is adjusted according to the risk level in the background area.

[0069] At this point, specifically according to methods four through six, risk levels based on external sensing information are identified by acquiring external sensing information sensed by the autonomous driving device guiding the user moving toward the background area. Accordingly, by effectively utilizing the autonomous driving device that guides the user, risk levels can be identified over the widest possible range of the background area that becomes the user's destination, thereby ensuring user safety.

[0070] Furthermore, according to methods four through six, the overlapping display of content images in the video portion reflecting the existing dangerous space within the background video corresponding to the existing external sensing space is prohibited. This external sensing space is the range for acquiring external sensing information within the background area, and the dangerous space is a space where the risk level has reached a dangerous level. Therefore, in the background video, the occupancy can be adjusted such that: only within the portion corresponding to the dangerous space where a danger level notification to a moving user is required, the content image disappears; on the other hand, the overlapping display of content images continues in the portion of the background area corresponding to the outside of the dangerous space. Thus, for moving users, a balance can be achieved between security ensured according to the principles described above and the continuity of the virtual experience. Attached Figure Description

[0071] Figure 1 This is an overall structural diagram showing the network connection environment of the processing system based on the first embodiment.

[0072] Figure 2 This is a schematic diagram illustrating a wearable terminal based on the first embodiment.

[0073] Figure 3This is a schematic diagram used to illustrate a hypothetical three-dimensional voxel in an infrastructure system based on the first embodiment.

[0074] Figure 4 This is a block diagram representing the processing system based on the first embodiment.

[0075] Figure 5 This is a flowchart illustrating the processing flow based on the first embodiment.

[0076] Figure 6 This is a flowchart illustrating the adjustment subroutine in the processing flow based on the first embodiment.

[0077] Figure 7 This is a schematic diagram showing the display state of a wearable terminal based on the first embodiment.

[0078] Figure 8 This is a schematic diagram showing the display state of a wearable terminal based on the first embodiment.

[0079] Figure 9 This is a schematic diagram showing the display state of a wearable terminal based on the first embodiment.

[0080] Figure 10 This is a schematic diagram illustrating the adjustment subroutine based on the first embodiment.

[0081] Figure 11 This is a schematic diagram illustrating the adjustment subroutine based on the first embodiment.

[0082] Figure 12 This is an overall structural diagram showing the network connection environment of the processing system based on the second embodiment.

[0083] Figure 13 This is a schematic diagram illustrating a mobile terminal based on the second embodiment.

[0084] Figure 14 This is a block diagram representing a processing system based on the second embodiment.

[0085] Figure 15 This is a flowchart illustrating the processing flow based on the second embodiment.

[0086] Figure 16 This is a flowchart illustrating the adjustment subroutine in the processing flow based on the second embodiment.

[0087] Figure 17 This is a schematic diagram showing the display state of a mobile terminal based on the second embodiment.

[0088] Figure 18 This is a schematic diagram showing the display state of a mobile terminal based on the second embodiment.

[0089] Figure 19 This is a schematic diagram showing the display state of a mobile terminal based on the second embodiment.

[0090] Figure 20 This is a schematic diagram illustrating the adjustment subroutine based on the second embodiment.

[0091] Figure 21 This is a schematic diagram illustrating the adjustment subroutine based on the second embodiment. Detailed Implementation

[0092] Hereinafter, several embodiments of the present disclosure will be described with reference to the accompanying drawings. Furthermore, in each embodiment, corresponding constituent elements are sometimes labeled with the same symbols, and repeated descriptions are omitted. Additionally, where only a portion of the structure is described in each embodiment, the structures of other previously described embodiments can be applied to the other parts of that structure. Moreover, not only can combinations of structures specified in the descriptions of each embodiment be made, but structures of multiple embodiments can be partially combined with each other even without specification, provided that such combinations do not particularly hinder them.

[0093] (First Implementation)

[0094] Figure 1 The processing system 1 of the first embodiment shown performs XR display-related processing associated with providing user services to a mobile user Us and displaying XR images to that user Us. Therefore, the processing system 1 is connected via a communication network Nc to a wearable terminal Wt worn by the pre-registered user Us, an autonomous driving device Ma, and an infrastructure system 6. Here, at least one system is envisioned as the wearable terminal Wt, the autonomous driving device Ma, and the infrastructure system 6 connected to the processing system 1.

[0095] The wearable terminal Wt is configured such that, at least when connected to the processing system 1, the user Us can be mounted on the face and perform handless operations. For example... Figure 2 As shown, the wearable terminal Wt is an optically transparent electronic device with a display unit 2 and a sensor unit 3, such as an HMD or smart glasses. Furthermore, regarding the wearable terminal Wt, Figure 2 The diagram illustrates the observation of the visual field area Av as seen from the user's side, corresponding to one of the user's two eyes.

[0096] Wearable terminal Wt controls, for example, through a communication unit. Figure 1 Such data transmission and reception via the communication network Nc. The wearable terminal Wt is controlled by the control unit according to the control commands from the processing system 1 via the communication network Nc, so as to jointly perform XR display-related processing with the processing system 1.

[0097] Under the operational control of the wearable terminal Wt, such as virtual image projection type or retinal projection type, etc. Figure 2 The display unit 2 shown displays the XR images required for XR display-related processing (see below). Figures 7-9 The image is superimposed on the real image of the visual field Av, which is visually recognized by the user Us through glasses or lenses, and is thus recognized by the user Us. Simultaneously, under the operational control of the wearable terminal Wt, Figure 2 The sensor unit 3 shown acquires sensing information through sensing processing corresponding to XR display-related processing. As the sensor unit 3, at least one of the following is employed: a camera, an inertial sensor, a GNSS sensor, an eye potential sensor, a gaze sensor, an infrared sensor, a geomagnetic sensor, a motion sensor, a touch sensor, and a microphone. With this structure, the sensor unit 3 can acquire input representing the user Us's intention as sensing information.

[0098] Figure 1 The autonomous driving device Ma shown is an automated vehicle or robot capable of autonomously moving in any direction—forward, backward, left, or right—by electrically driving the wheels 5 based on sensing information from sensor unit 4. The autonomous driving device Ma controls the transmission and reception of data via communication network Nc through a communication unit. The autonomous driving device Ma is driven by a control unit according to control commands from processing system 1 via communication network Nc, so as to jointly execute XR display-related processing with processing system 1. Under this drive control, sensor unit 4 acquires sensing information through sensing processing corresponding to XR display-related processing. As sensor unit 4, at least one of the following is employed: a camera, an inertial sensor, a GNSS sensor, LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging), and sonar.

[0099] Infrastructure System 6 is a foundational system that shares the accumulated three-dimensional spatial information It in the infrastructure database Di as a general foundation for the distributed architecture. Infrastructure System 6 controls the transmission and reception of data via the communication network Nc through a communication unit. Infrastructure System 6 continuously collects the three-dimensional spatial information It provided to individual distributed systems such as Processing System 1, which constitute the distributed architecture, through a control unit, and updates the accumulated information in the infrastructure database Di to the latest information.

[0100] Here, in particular, the infrastructure system 6 will be used as an object for information accumulation in three-dimensional space, such as Figure 3As shown, the system is virtually divided into multiple three-dimensional voxels Vi (i.e., a three-dimensional mesh) arranged in a hypothetical three-dimensional pattern and its information is managed. Infrastructure system 6 uses a dataset associated with each voxel Vi individually assigned a spatial ID via metadata as a basis for... Figure 1 Such three-dimensional spatial information It is accumulated in the infrastructure database Di. Here, the three-dimensional spatial information It can of course also include, for example, two-dimensional mesh information associated only with the lower surface, of the voxels Vi that constitute the two-dimensional arrangement along the ground in the three-dimensional arrangement.

[0101] The data used to construct the 3D spatial information It can be collected from at least one of a wearable terminal Wt that works in conjunction with the processing system 1, and other wearable terminals, as well as mobile terminals such as smartphones or tablets. The data used to construct the 3D spatial information It can also be collected from at least one of an autonomous driving device Ma that works in conjunction with the processing system 1, and other mobile bodies.

[0102] The data used to construct the 3D spatial information It can also be collected from at least one of the following: communication base stations, smart poles, and smart streetlights equipped with infrastructure sensors such as cameras and / or LiDAR. The data used to construct the 3D spatial information It can also be collected from servers that provide at least one of the following: map services, weather services, communication services, traffic management services, feature management services, and aviation management services, as well as user services (described later) processed by processing system 1.

[0103] The data used to construct the three-dimensional spatial information It can also be at least one type of image data, such as video, still images, and point cloud images. The data used to construct the three-dimensional spatial information It can also be at least one type of secondary data generated within the information accumulation object space for a person including the user Us of the processing system 1 through image processing of such image data, such as location data, motion data, posture data, gaze data, action data, and pedestrian flow data, as well as intent data (described later) representing the user Us's intent, while ensuring information security.

[0104] The data used to construct 3D spatial information It can also be secondary data generated within the information accumulation object space for a wearable terminal Wt containing a user Us, or a person's carry-on, through image data processing, such as position data, motion data, and posture data, and is at least one of these. The data used to construct 3D spatial information It can also be secondary data generated within the information accumulation object space for a moving body containing an autonomous driving device Ma, through image data processing, such as position data, motion data, and posture data, and is at least one of these.

[0105] The data used to construct the three-dimensional spatial information It can also be sound data containing the voices of users Us in the processing system 1, collected within the information accumulation object space. The data used to construct the three-dimensional spatial information It can also be at least one type of secondary data that has been analyzed for users Us through sound recognition processing of such sound data, including, for example, location data, motion data, action data, crowd flow data, conversation data, and intent data (described later) representing the intent of users Us, while ensuring information security.

[0106] The construction data for the three-dimensional spatial information It can also be at least one of the following: two-dimensional and / or three-dimensional map data, GIS (Geographic Information System) data, road network data, meteorological data, communication data, route data, traffic data, feature management data, BIM (Building Information Modelling) data, POI (Point of Interest) data, aviation management data, and time data. The construction data for the three-dimensional spatial information It can also be service data associated with at least one of the user services processed by processing system 1, such as guidance services, transport services, photography services, and online game services.

[0107] like Figure 1 As shown, the processing system 1 is a distributed computer system comprising a communication system 10 and a control system 9, and for example, includes at least one of cloud servers and edge servers. At least a portion of each system 10, 9 in the processing system 1 may also be composed of a communication unit and a control unit of a wearable terminal Wt. At least a portion of each system 10, 9 in the processing system 1 may also be composed of a communication unit and a control unit of an autonomous driving device Ma. At least a portion of each system 10, 9 in the processing system 1 may also be composed of a communication unit and a control unit of the infrastructure system 6.

[0108] The communication system 10 is primarily composed of communication equipment used to construct the communication network Nc. The control system 9 is connected to the communication system 10 via at least one of wired and wireless communication lines. The control system 9 is constructed by including at least one dedicated computer. The dedicated computer constructing the control system 9 has at least one memory 9a and at least one processor 9b.

[0109] In such a processing system 1, the control system 9 executes multiple commands of a processing program stored in the memory 9a via the processor 9b. Thus, the control system 9 constructs multiple functional blocks for performing XR display-related processing. For example... Figure 4As shown, the constructed functional blocks include an identification block 200, a drive control block 210, and a display control block 220.

[0110] Processing system 1 performs XR display-related processing through the coordination of these blocks 200, 210, and 220 according to the processing method. Figure 5 , 6 The processing flow shown is executed. This processing flow is executed in response to the sensor unit 3 of the wearable terminal Wt acquiring service request input representing the user Us's intention to request user services as sensing information. Furthermore, each "S" in this processing flow represents a multiple step executed by multiple commands included in the processing procedure of the first embodiment.

[0111] exist Figure 5 In S10 of the processing flow shown, the identification block 200 (refer to...) Figure 4 The visual field region Av, visually identified by the user Us via the wearable terminal Wt, is monitored. The monitoring of the visual field region Av is based on at least one sensing information generated by the sensor unit 3 of the wearable terminal Wt, such as camera information and inertial information. As a result, the visual field region Av is identified as an area extending in the orientation of the user Us's face or gaze, or in the orientation of the wearable terminal Wt. Furthermore, in subsequent steps of the processing flow S10, monitoring processing of the visual field region Av can be performed in parallel in response to changes in the visual field region Av due to movement of the user U, thereby updating the latest identified visual field region Av.

[0112] exist Figure 5 In S20 of the processing flow shown, the identification block 200 acquires the user Us's intent and desired content Cd. At this time, the desired content Cd is identified by sensing information about the face generated by the wearable terminal Wt worn by the user Us on their face. Therefore, the sensing information used to identify the desired content Cd is acquired by the wearable terminal Wt at the moment it is input to the sensor unit 3 as intent data of the user Us's intent.

[0113] As input to the desired content Cd in S20, user Us's gesture input, gaze input, facial expression input, or voice input can be sensed by sensor unit 3. At this time, in the display unit 2 of the wearable terminal Wt, the XR image used to receive the input of the desired content Cd from user Us via sensor unit 3 can also be overlaid and displayed on the field of view Av through the wearable terminal Wt. Furthermore, the recognition of the desired content Cd can also be achieved based on the three-dimensional spatial information It obtained from the infrastructure database Di via communication system 10.

[0114] Among the items of desired content Cd identified in S20, at least one XR image (described later) is displayed to the display unit 2 of the wearable terminal Wt as user services are provided to user Us. Figure 7 , Figure 8 Image 20) is assigned a display theme. A display theme refers to a common concept in XR images that are displayed in two dimensions or three dimensions, such as imaginary or real creatures like monsters, animals, plants, and insects, as well as non-living things like pre-historic buildings, service vehicles, and imaginary objects.

[0115] The items in the desired content Cd identified in S20 may include categories of user services that can be provided to user Us by utilizing the autonomous driving device Ma. As categories of user services, at least one of the following is prepared: guidance service, transport service, photography service, and online gaming service.

[0116] In S20, regardless of which user is being served, at least one path-related item, such as the path to the destination, walking speed, arrival time, and service provision time (i.e., guidance time or delivery time), can be included in the desired content Cd. In S20, regardless of which user is being served, at least one voice-related item, such as whether voice output is required, whether voice recognition function is required, and whether dialogue function is required, can be included in the desired content Cd.

[0117] In the case of a guidance service in S20, at least one tourism-related item regarding the destination, such as the permissible level of congestion, the availability of facilities to be used, and the presence of natural landscapes, may also be included in the desired content Cd. In the case of a transport service in S20, at least one cargo-related item, such as the number, size, weight, type, and whether temperature management is required, may also be included in the desired content Cd.

[0118] In the case of a shooting service in S20, at least one shooting-related item, such as shooting schedule, shooting time, shooting location, whether images or data need to be sent, and whether photo printing is required, can be included in the desired content Cd. In the case of an online game service in S20, at least one game-related item, such as initial settings for game characters, online game environment settings, online game difficulty, whether images or data need to be sent, and whether battles are required, can be included in the desired content Cd.

[0119] exist Figure 5 In the processing flow shown in S30, the drive control block 210 (refer to...) Figure 4User services are provided by controlling the drive of the autonomous driving device Ma within the field of vision area Av. At this time, user services are provided according to the drive control mode of the autonomous driving device Ma that matches the category of user services, by reading the drive control mode from the memory 9a, so as to guide the user Us moving toward the field of vision area Av.

[0120] The guidance service based on S30 is provided according to a path-guided drive control mode, which is as follows: the autonomous driving device Ma is driven from a standby position to a location near the user Us, and the user Us is guided to the destination. In this guidance service, the path to guide the user Us can be indicated by overlaying an XR image onto the field of view Av in the wearable terminal Wt, or by display output from the display unit of the autonomous driving device Ma. The path to guide the user Us can also be indicated by audio output from the audio unit of the wearable terminal Wt or the autonomous driving device Ma.

[0121] The transport service based on S30 is provided according to a drive control mode, which is as follows: the autonomous driving device Ma, equipped with a loading chamber, is driven from a standby position to a location near the user Us, and while guiding the user Us, the goods loaded in the loading chamber are transported to the destination. During this transport service, the loading method for the goods into the loading chamber can be indicated to the user Us by overlaying an XR image onto the field of view Av of the wearable terminal Wt, or through display output from the display unit of the autonomous driving device Ma. The user Us can also be prompted to load goods into the loading chamber through audio output from the audio unit of the wearable terminal Wt or the autonomous driving device Ma.

[0122] The shooting service based on S30 is provided according to a drive control mode, which is as follows: the autonomous driving device Ma, equipped with a camera unit, is driven from its standby position to a location near the user Us, and it guides the user Us while taking pictures from the surroundings. During this shooting service, the user Us can be notified of the shooting schedule or shooting time by overlaying an XR image onto the field of view Av of the wearable terminal Wt, or through display output from the display unit of the autonomous driving device Ma. The user Us can also be notified of the shooting schedule or shooting time through sound output from the sound unit of the wearable terminal Wt or the autonomous driving device Ma.

[0123] Here, in the shooting service, two-dimensional or three-dimensional still images or videos can be captured in a manner that includes the user Us by an autonomous driving device Ma driven around the user Us. In the shooting service, the focus during shooting can also be adjusted based on the distance to the user Us sensed by a sensor unit 4, such as LiDAR, in the autonomous driving device Ma. In the shooting service, the orientation of external light incidence during shooting can also be adjusted based on three-dimensional spatial information It obtained from the infrastructure database Di, or sensing information from sensor units 4, such as cameras, in the autonomous driving device Ma. The image data captured of the user Us during the shooting service can also be stored in the memory 9a of the control system 9 or in the storage medium of the wearable terminal Wt.

[0124] The online game service based on S30 is provided according to a drive control mode, which is as follows: the autonomous driving device Ma is driven from a standby position to a location near the user Us, and the user Us is guided to a strategic location in the online game. At this time, in this online game service, necessary instructions for the online game can be provided to the user Us by overlaying XR images onto the field of view Av of the wearable terminal Wt, or through display output from the display unit of the autonomous driving device Ma. In this online game service, necessary instructions for the online game can also be provided to the user Us through sound output from the sound unit of the wearable terminal Wt or the autonomous driving device Ma.

[0125] Figure 5 The processing flow shown, S40 and S30, are executed in parallel. In S40, display control block 220 (see reference...) Figure 4 The XR image corresponding to the desired content Cd is overlaid and displayed on the field of view Av entered by the autonomous driving device Ma, which is providing user services via S30, through the display unit 2 of the wearable terminal Wt. At this time, the display control block 220 adjusts the XR image according to the risk level Lr in the area Av. Figure 7 , 8 As shown, the occupancy Rc of the display object range Ad in the field of view Av is the content image 20 that is to be overlaid according to the desired content Cd. Here, the occupancy Rc in the first embodiment refers to the ratio of the display area of ​​the entire display object range Ad to the two-dimensional projection area of ​​the entire field of view Av on, for example, the glasses or lens in the display unit 2, or the retina.

[0126] The content image 20 in S40 contains at least... Figure 7 , Figure 8 The device image 20a shown is displayed as an overlay on, as shown in the image. Figure 2The XR image of the autonomous driving device Ma is visually recognized within the field of view Av, as shown. The overlapping display position of the device image 20a is adjusted according to the drive control mode based on S30 and the visually predicted position of the autonomous driving device Ma. The display position adjustment at this time is based on the three-dimensional spatial information It, which is the location association information of the autonomous driving device Ma attached to each voxel Vi in the space for providing services to the user, and is obtained from the infrastructure database Di through the communication system 10. As a result, the device image 20a, which is the display subject corresponding to the desired content Cd, is overlaid on the entire autonomous driving device Ma in a way that hides it opaquely or semi-transparently.

[0127] The content image 20 in S40 may also include Figure 7 , Figure 8 The object image 20b shown is displayed as an overlay on, as shown in... Figure 2 The XR image of another object Mb, which is visually recognized in the field of view Av along with the autonomous driving device Ma, is shown. The overlapping display position of the object image 20b is adjusted to match the visual recognition prediction position of the other object Mb, thereby achieving the same overlapping display as the device image 20a. Other objects Mb that are thus overlapping the object image 20b include, hypothetically, at least one of the following: a person other than the user Us of the processing system 1, a moving body other than the autonomous driving device Ma cooperating with the processing system 1, a flying body, a structure, and plants.

[0128] The content image 20 in S40 may also include Figure 7 , 8 The ground image 20c shown is displayed as an overlay on... Figure 2 The XR image of the ground Mc, which is visually recognized in the field of view Av along with the autonomous driving device Ma, is shown. The overlapping display position of the ground image 20c is adjusted to match the visual recognition prediction position of the ground Mc, thereby becoming the same overlapping display as the device image 20a.

[0129] In S40, via, as Figure 7 As shown, the display object range Ad of the content image 20 is set to the entire area within the field of view Av, thereby adjusting the occupancy Rc to its maximum value (e.g., 1). In S40, by means of... Figure 9 As shown, the display object range Ad of the content image 20 is set to disappear from the entire area within the field of view Av, thereby adjusting the occupancy Rc to a minimum value (e.g., 0). In S40, by means of... Figure 8 The display object range Ad of the content image 20 is set to a portion of the field of view Av, thereby adjusting the occupancy Rc to an intermediate value between the minimum and maximum values ​​(e.g., greater than 0 and less than 1).

[0130] In S40, if a hazard notification needs to be sent to user Us based on the risk level Lr, it can also be displayed via display unit 2. Figure 8 , 9 As shown, the hazardous image 22, which is an XR image, is overlaid on... Figure 2 The field of view Av shows the ground Mc and / or overhead Md. The overlapping display position of the hazard image 22 can be adjusted to match the visually recognized predicted position of the ground Mc and / or overhead Md. This display position adjustment can also be based on three-dimensional spatial information It, which is the voxel Vi in the space providing services to the user, supplemented with positional information related to the ground Mc and / or overhead Md, and obtained from the infrastructure database Di via the communication system 10.

[0131] In this S40, the adjustment subroutine that switches and adjusts the occupancy rate Rc according to the risk level Lr in the field of view Av is as follows: Figure 6 It is executed as shown. Specifically, in adjustment subroutine S400, display control block 220 acquires internal sensing information Ii sensed by sensor unit 3 of wearable terminal Wt. At this time, as shown... Figure 10 As shown, the range for acquiring internal sensing information Ii, where the accuracy or reliability of sensor unit 3 is at an allowable level, is set to an internal sensing space Ai that is part of the field of view region Av. Therefore, the internal sensing information Ii acquired from the internal sensing space Ai is obtained from at least one of the sensor unit 3, such as a camera and a motion sensor.

[0132] The internal sensing space Ai in S400 is defined by multiple voxels Vi centered on the user Us or wearable terminal Wt, with their distances in two or three dimensions assumed to be within a set distance. Therefore, the set distance that determines the boundary range of the internal sensing space Ai is set to a fixed value, such as 5m, based on the specifications of the sensor unit 3 and / or the risk requirements of the processing system 1.

[0133] exist Figure 6 In the adjustment subroutine S410 shown, the display control block 220 acquires the risk level Lr in the internal sensing space Ai within the field of view Av. At this time, the risk level Lr is defined as an index such as: when the risk level Lr is within the range of other objects Mb, such as people, moving objects, or structures, it is considered to be low. Figure 10 When at least a portion of an obstacle Mbo, which is predicted to interfere with the movement of user Us, enters the internal sensing space Ai, the indicator reaches the danger level Lh. In other words, the risk level Lr within the internal sensing space Ai is determined to be at a safe level, having moved away from danger level Lh, when obstacle Mbo disappears from the space Ai.

[0134] exist Figure 6 In the adjustment subroutine S420 shown, the display control block 220 determines whether the risk level Lr in the internal sensing space Ai has reached the danger level Lh that requires a danger notification to the moving user Us. At this time, the comparison between the risk level Lr and the danger level Lh can be performed according to each voxel Vi in the user service provision space, or according to the voxel group of each of the multiple voxels Vi.

[0135] In the case where a positive determination is made in S420, Figure 6 In the adjustment subroutine S430 shown, the display control block 220 is as follows: Figure 9 In this way, overlapping display of the content image 20 is prohibited in the entire area within the field of view Av, so that the entire area is excluded from the display object range Ad of the content image 20, thereby adjusting the occupancy to a minimum value. However, in S430, the danger image 22 that notifies warning text and / or warning color according to the risk level Lr that has reached the danger level Lh can be as follows: Figure 9 As shown, it is overlaid in the field of view area Av. At this time, the hazard image 22 can be displayed to inform of at least one of the following: location, orientation, distance, and category of the obstacle Mbo that is the main cause of the hazard level Lh in the internal sensing space Ai.

[0136] In addition to displaying the hazard image 22, in S430, a warning of risk level Lr reaching hazard level Lh can also be issued via display output from the display unit of the autonomous driving device Ma or by illuminating a warning light. In S430, a warning of risk level Lr reaching hazard level Lh can also be issued via audible output from the display unit of the autonomous driving device Ma. Upon completion of S430, the execution of this subroutine and processing flow ends.

[0137] In the case where a negative determination is made in S420, Figure 6 In the S440 adjustment subroutine shown, the display control block 220 acquires external sensing information Io sensed by the sensor unit 4 of the autonomous driving device Ma. At this time, the accuracy or reliability of the sensor unit 4 is within the allowable range for acquiring external sensing information Io, such as... Figure 11 As shown, the external sensing space Ao is set as a part of the field of view region Av and is a larger area than the internal sensing space Ai. Therefore, the external sensing information Io acquired for the external sensing space Ao can be obtained from at least one of the sensor units 4, such as a camera, LiDAR, and sonar.

[0138] In S440, the external sensing information Io acquired for the external sensing space Ao, and the three-dimensional spatial information It obtained from the infrastructure database Di by the communication system 10, can be obtained together with the sensing information of the sensor unit 4. That is, the external sensing information Io can also be the three-dimensional spatial information It collected and accumulated in the infrastructure database Di by the infrastructure system 6.

[0139] In S440, the external sensing space Ao is defined by multiple voxels Vi, centered on the autonomous driving device Ma, whose distances in two or three dimensions are hypothetically within a set range. Therefore, the external sensing space Ao is determined by... Figure 11 The set distance of the limit range Aol shown can also be set to a fixed value, such as 6m to 20m, corresponding to the specifications of the sensor unit 4 and / or the risk requirements of the processing system 1.

[0140] exist Figure 6 In the adjustment subroutine S450 shown, the display control block 220 acquires the risk level Lr in the external sensing space Ao within the field of view Av. At this time, the risk level Lr is defined as follows: when the boundary range Aol of the external sensing space Ao is such that... Figure 11 As shown, when the separation distance δo in two or three dimensions is below or less than the threshold, the indicator is judged to have reached the danger level Lh. In other words, when the separation distance δo between the boundary range Aol of the external sensing space Ao and the user Us exceeds or is above the threshold, the risk level Lr within the space Ao is judged to be at a safe level that has moved away from the danger level Lh.

[0141] exist Figure 6 In the adjustment subroutine S460 shown, the display control block 220 determines whether the risk level Lr in the external sensing space Ao has reached the danger level Lh that requires a danger notification to the moving user Us. At this time, the comparison between the risk level Lr and the danger level Lh can be performed according to each voxel Vi in the user service provision space, or according to the voxel groups of multiple voxels Vi.

[0142] In the case where a positive determination is made in S460, Figure 6 In the adjustment subroutine S470 shown, the display control block 220 identifies the external sensing space Ao as shown in the image. Figure 11 As shown, the danger zone Aoh has a risk level Lr that reaches a danger level Lh. At this time, multiple voxels Vi that have reached a risk level Lh are identified as the danger zone Aoh. Furthermore, multiple voxels Vi existing inside the external sensing space Ao can also be identified as the danger zone Aoh (see reference). Figure 8), so that the display prohibition range for the overlapping display of the content image 20 described later in the field of view Av is also extended to the inner side.

[0143] Therefore, the display control block 220 in S470 is as follows: Figure 8 By narrowing the dangerous space Aoh to the level of danger Lh and prohibiting the overlapping display of the content image 20, the dangerous space Aoh is excluded from the display target range Ad of the content image 20, thereby adjusting the occupancy Rc to an intermediate value. As a result, the remaining space As outside the dangerous space Aoh in the field of view Av is set as the display target range Ad of the content image 20, thereby allowing the overlapping display of the content image 20.

[0144] In S470, the hazard image 22, which provides warning text and / or warning color based on the risk level Lr that has reached hazard level Lh, can also be as follows: Figure 8 As shown, it is overlaid in the field of view area Av. At this time, the hazard image 22 can also be displayed to notify the user Us, who is the main cause of the hazard level Lh in the external sensing space Ao, of the increased risk caused by approaching the boundary range Aol.

[0145] In addition to displaying the hazard image 22, in S470, a warning of risk level Lr reaching hazard level Lh can also be issued via display output from the display unit of the autonomous driving device Ma or by illuminating a warning light. In S470, a warning of risk level Lr reaching hazard level Lh can also be issued via audible output from the display unit of the autonomous driving device Ma. Upon completion of S470, the execution of this subroutine and processing flow ends.

[0146] In the case where a negative decision is made in S460, Figure 6 In the adjustment subroutine S480 shown, the display control block 220 allows, for example... Figure 7 The content image 20 is displayed in an overlapping manner relative to the entire area within the field of view Av, so that the entire area is set as the display object range Ad of the content image 20, thereby adjusting the occupancy Rc to the maximum value. At this time, in the content image 20, the ground image 20c, which represents the boundary range Aol on the ground Mc of the external sensing space Ao, and the restricted image 20cl are as follows: Figure 7 As shown, it is overlaid in the field of view area Av. By completing the execution of S480, the current execution of the adjustment subroutine and processing flow ends. Therefore, in the next execution of the processing flow, the restricted image 20cl continues to be displayed until a positive determination is made in either S420 or S460.

[0147] (Effects)

[0148] The effects of the first embodiment described above will be explained below.

[0149] In the first embodiment, the field of view Av, which is visually recognized by the user Us through a wearable terminal Wt worn by the user Us, is monitored. Therefore, the occupancy Rc of the display object range Ad of the content image 20, which is overlaid on the field of view Av as an XR image corresponding to the desired content Cd of the user Us, is adjusted according to the risk level Lr in the field of view Av.

[0150] At this point, particularly according to the first embodiment, risk level Lr is identified based on external sensing information Io by acquiring external sensing information sensed by the autonomous driving device Ma, which guides the user Us moving toward the field of vision Av. Accordingly, by effectively utilizing the autonomous driving device Ma, which performs the function of guiding the user, risk level Lr can be identified over the widest possible range of the field of vision Av, which becomes the user Us's destination, thereby ensuring the user's safety.

[0151] Furthermore, according to the first embodiment, for the dangerous space Aoh within the external sensing space Ao, which is the acquisition range of external sensing information Io in the field of view Av, where the risk level Lr reaches the danger level Lh, the overlapping display of the content image 20 is prohibited. Accordingly, the occupancy rate Rc can be adjusted such that the content image 20 disappears only within the dangerous space Aoh at the danger level Lh where a danger notification to the moving user Us is required; on the other hand, the overlapping display of the content image 20 continues outside the dangerous space Aoh in the field of view Av. Therefore, for the moving user Us, a balance can be achieved between security ensured according to the principles described above and the continuity of the virtual experience.

[0152] According to the first embodiment, a restricted image 20cl, which is an XR image representing the boundary range Aol on the ground Mc of the external sensing space Ao, is overlaid on the field of view Av. Accordingly, while the user Us is informed of the boundary range Aol of the external sensing space Ao where the risk level Lr is identified, the user Us's safety is ensured by identifying the risk level Lr within the external sensing space Ao, thereby also ensuring the user Us's sense of security. Therefore, combined with the disappearance of the content image 20 confined within the dangerous space Aoh, the user Us's safety and sense of security can be ensured while maintaining the continuity of the virtual experience.

[0153] According to the first embodiment, risk level Lr is identified based on internal sensing information Ii sensed by the wearable terminal Wt. Accordingly, risk level Lr can also be identified within the proximity range of the user Us via the wearable terminal Wt, thereby ensuring the safety of the user Us. However, if the risk level Lr in the internal sensing space Ai, which is the acquisition range of the internal sensing information Ii within the field of view Av, reaches a danger level Lh, overlapping display of the content image 20 with the entire area of ​​the field of view Av is prohibited. Accordingly, for the proximity range of the user Us, the occence Rc, which intentionally disrupts the balance, can be adjusted in a manner that prioritizes safety over the continuity of the virtual experience.

[0154] According to the first embodiment, the risk level Lr based on the external sensing information Io collected by the infrastructure system 6 can also be identified by acquiring the external sensing information Io sensed by the autonomous driving device Ma. Accordingly, not only is the autonomous driving device Ma, which guides the user Us, effectively utilized, but the infrastructure system 6 is also effectively utilized, thereby enabling high-precision identification of the risk level Lr for the field of view Av, thus ensuring the safety of the user Us.

[0155] (Second Implementation)

[0156] The second embodiment is a variation of the first embodiment.

[0157] like Figure 12 As shown, the processing system 2001 of the second embodiment is connected via a communication network Nc to a mobile terminal Mt carried by a pre-registered user Us, an autonomous driving device Ma, and an infrastructure system 6 that provides infrastructure information. Here, multiple terminals are envisioned as mobile terminals Mt connected to the processing system 1.

[0158] The mobile terminal Mt is configured such that, at least when connected to the processing system 2001, the user Us can hold and operate it with their finger. For example... Figure 13 As shown, the mobile terminal Mt is a small electronic device equipped with a display unit 2002 and a sensor unit 2003, such as a smartphone or tablet terminal. The mobile terminal Mt controls devices such as… Figure 12 Such data transmission and reception via communication network Nc. The mobile terminal Mt is controlled by the control unit according to control commands from the processing system 2001 via communication network Nc, so as to jointly perform service-related processing with the processing system 2001.

[0159] Under the operational control of the control unit of the mobile terminal Mt, such as an LCD panel or an OLED panel, etc. Figure 13The display unit 2002 shown implements the screen display required for service-related processing (see below). Figures 17-19 Meanwhile, under the operational control of the mobile terminal Mt, Figure 13 The sensor unit 2003 shown acquires sensing information through sensing processing corresponding to service-related processing. As the sensor unit 2003, at least one of the following is employed: a camera, an inertial sensor, a GNSS (Global Navigation Satellite System) sensor, a touch sensor, and a microphone. With this structure, the sensor unit 2003 can acquire input representing the user Us's intention as sensing information.

[0160] Figure 12 The autonomous driving device Ma shown is configured similarly to the processing system 1 of the first embodiment, except that its drive control is performed by the control unit according to control commands from the processing system 2001 via the communication network Nc to jointly perform service-related processing with the processing system 2001. The control system 2009 of the processing system 2001 is configured similarly to the control system 9 of the first embodiment, except that at least a portion of it can be composed of the control unit of the mobile terminal Mt, and that it executes the processing flow using the mobile terminal Mt described later.

[0161] In order to perform service-related processing in control system 2009, processing system 2001 performs the following steps: Figure 14 The XR display processing method is performed by the coordinated operation of blocks 200, 210, and 220 as shown. Figure 15 , 16 The processing flow shown will be executed.

[0162] exist Figure 15 In S2010 of the processing flow shown, the identification block 200 (refer to...) Figure 14 By using the orientation of the mobile terminal Mt carried by the user Us, the background area Ab, which becomes the display background from the display unit 2002 to the user Us, is monitored. At this time, the background area Ab is identified as an area extending in the orientation of the mobile terminal Mt based on at least one sensor information generated by the sensor unit 2003 of the mobile terminal Mt, such as camera information and inertial information. Furthermore, in the subsequent steps of the processing flow S2010, in response to changes in the background area Ab due to the movement of the user U, the monitoring processing of the background area Ab can be performed in parallel, thereby updating the latest identified background area Ab.

[0163] exist Figure 15In S2020 of the processing flow shown, the identification block 200 obtains the desired content Cd by identifying sensing information in the mobile terminal Mt carried by the user Us. This sensing information is obtained through the mobile terminal Mt at the moment the user Us's intent data is input to the sensor unit 2003, and is thus used for the identification of the desired content Cd. Details related to the input and items of the desired content Cd follow S20 of the first embodiment.

[0164] exist Figure 15 In the processing flow shown in S2030, the drive control block 210 (refer to...) Figure 14 User services are provided by controlling the drive of the autonomous driving device Ma within the background area Ab. At this time, user services are provided according to the drive control mode of the autonomous driving device Ma, which matches its category, read from the memory 9a, to guide the user Us moving towards the background area Ab. Details related to the drive of the autonomous driving device Ma and the provision of user services are as per S30 of the first embodiment.

[0165] Figure 15 The processing flow shown, S2040 and S2030, are executed in parallel. In S2040, display control block 220 (see reference) Figure 14 The display unit 2002 of the mobile terminal Mt overlays the XR image corresponding to the desired content Cd onto the background video 24 of the background area Ab that is currently being entered by the autonomous driving device Ma providing user services via S2030. At this time, the display control block 220 adjusts the XR image according to the risk level Lr in the background area Ab. Figure 17 , 18 As shown, the occupancy Rc of the display object range Ad in the background video 24 is the ratio of the total display area of ​​the display object range Ad to the total display area of ​​the background video 24 in the display unit 2002.

[0166] In S2040, the background video 24, which becomes the overlay display object of the content image 20, is acquired based on sensing information from sensor unit 4, such as a camera, or three-dimensional spatial information It obtained from infrastructure database Di. The background video 24 can use either primary video data captured of the background region Ab or secondary video data that has been processed from that primary data.

[0167] The content image 20 in S2040 contains at least […]. Figure 17 , 18 The device image 20a shown is as follows Figure 13The location of the autonomous driving device Ma, which exists in the background area Ab as shown, is superimposed on the XR image of the background video 24, corresponding to the location of the autonomous driving device Ma. The superimposed display position of the device image 20a is adjusted according to the drive control mode of S2030 and the camera position of the autonomous driving device Ma on the background video 24. As a result, the device image 20a, which corresponds to the display theme of the desired content Cd, is superimposed on the autonomous driving device Ma reflected in the background video 24 in a manner that completely replaces it. Details related to the adjustment of the display position of the device image 20a are as described in S40 of the first embodiment.

[0168] The content image 20 in S2040 may also include Figure 17 , 18 The object image 20b shown is, as... Figure 13 The XR image of the other object Mb, which exists in the background area Ab along with the autonomous driving device Ma, is overlaid on the video portion of the background video 24. The overlaid display position of the object image 20b is adjusted to match the camera position of the other object Mb on the background video 24, thereby achieving the same displacement display as the device image 20a. Details of the other object Mb that are displacement-displayed in the object image 20b are as described in S40 of the first embodiment.

[0169] The content image 20 in S2040 may also include Figure 17 , 18 The ground image 20c shown is as follows Figure 13 The XR image 20c, which is an XR image superimposed on the video portion of the background video 24, corresponds to the location of the ground Mc in the background area Ab, which exists together with the autonomous driving device Ma. The superimposed display position of the ground image 20c is adjusted to match the camera position of the ground Mc on the background video 24, thereby achieving the same displacement display as the device image 20a.

[0170] In S2040, via, as Figure 17 As shown, the display object range Ad of the content image 20 is set to the entire area of ​​the background video 24, thereby adjusting the occupancy Rc to its maximum value (e.g., 1). In S2040, by means of... Figure 19 As shown, the display object range Ad of the content image 20 is set to disappear from the entire area of ​​the background video 24, thereby adjusting the occupancy Rc to a minimum value (e.g., 0). In S2040, by means of... Figure 18 The display object range Ad of the content image 20 is set to be part of the background video 24, thereby adjusting the occupancy Rc to an intermediate value between the minimum and maximum values ​​(e.g., more than 0 and less than 1).

[0171] In S2040, if a hazard notification needs to be sent to user Us based on the risk level Lr, it can also be displayed via the display unit 2002. Figure 18 , Figure 19 As shown, the hazardous image 22, which is an XR image, is overlaid on the image as shown. Figure 13 The ground Mc and / or overhead Md are thus projected onto the background video 24. The overlapping display position of the hazard image 22 can be adjusted to match the camera position of the ground Mc and / or overhead Md on the background video 24. At this time, the details related to the adjustment of the display position of the hazard image 22 follow S40 of the first embodiment.

[0172] In such an S2040, such as Figure 16 As shown, an adjustment subroutine is executed to switch and adjust the occupancy rate Rc according to the risk level Lr in the background area Ab. Specifically, in S2400 of the adjustment subroutine, the display control block 220 acquires the internal sensing information Ii sensed by the sensor unit 2003 of the mobile terminal Mt. At this time, the accuracy or reliability of the sensor unit 2003 is within the allowable range of acquiring the internal sensing information Ii, such as... Figure 20 The internal sensing space Ai is set as part of the background region Ab, as shown. Details related to the acquisition of internal sensing information Ii, and the definition and setting of the internal sensing space Ai, are as described in S400 of the first embodiment.

[0173] exist Figure 16 In S2410 of the adjustment subroutine shown, the display control block 220 acquires the risk level Lr in the internal sensing space Ai within the background region Ab. At this time, details related to the definition and determination of the risk level Lr are presented according to S410 of the first embodiment. Furthermore, in S2420 of the adjustment subroutine, the display control block 220 determines the risk level Lr according to S410 of the first embodiment.

[0174] In the case where a positive determination was made in S2420, Figure 16 In the adjustment subroutine S2430 shown, the display control block 220 is as follows: Figure 19 In this way, overlapping display of the content image 20 is prohibited over the entire area of ​​the background video 24, so that the entire area is excluded from the display object range Ad of the content image 20, thereby adjusting the occupancy Rc to a minimum value. However, in S2430, the danger image 22 that provides warning text and / or warning color according to the risk level Lr that has reached the danger level Lh is also possible. Figure 19 As shown, it is overlaid on the background video 24. At this time, details related to the display of the danger image 22 and other warning methods are displayed according to S430 of the first embodiment. By completing the execution of S2430, this execution of the adjustment subroutine and processing flow ends.

[0175] In the case where a negative determination was made in S2420, Figure 16 In the adjustment subroutine S2440 shown, the display control block 220 acquires external sensing information Io according to S440 of the first embodiment. However, at this time, the accuracy or reliability of the sensor unit 4 is within the allowable range of acquiring external sensing information Io, such as... Figure 21 As shown, it is set as the outer sensing space Ao, which is part of the background region Ab and becomes a larger range than the inner sensing space Ai.

[0176] exist Figure 16 In S2450 of the adjustment subroutine shown, the control block 220 acquires the risk level Lr in the external sensing space Ao within the background region Ab. At this time, details related to the definition and determination of the risk level Lr are presented according to S450 of the first embodiment. Furthermore, in S2460 of the adjustment subroutine, the control block 220 determines the risk level Lr according to S460 of the first embodiment.

[0177] In the case where a positive determination was made in S2460, Figure 16 In the adjustment subroutine S2470 shown, the display control block 220 identifies, according to S470 of the first embodiment, as follows: Figure 21 The dangerous space Aoh is shown in the external sensing space Ao. However, multiple voxels Vi existing inside the external sensing space Ao can also be identified as the dangerous space Aoh (see reference). Figure 18 This is such that the display prohibition range for the overlapping display of content images 20 in the background area Ab, as described later, also extends to the inner side.

[0178] Therefore, the display control block 220 in S2470 is as follows: Figure 18 The video portion 24h of the background video 24, which reflects the dangerous space Aoh, is reduced to reflect the external sensing space Ao, and the overlapping display of the content image 20 is prohibited. This excludes the video portion 24h from the display target range Ad of the content image 20, thereby adjusting the occupancy Rc to an intermediate value. As a result, corresponding to the remaining space As in the background area Ab that is outside the safety level of the dangerous space Aoh, the remaining portion 24s in the background video 24 that is outside the video portion 24h is set as the display target range Ad of the content image 20, thereby allowing the overlapping display of the content image 20. At this time, the details related to the display of the dangerous image 22 and other warning methods are as per S470 of the first embodiment. By completing the execution of S2470, the current execution of the adjustment subroutine and processing flow ends.

[0179] In the case where a negative determination is made in S2460, Figure 16 In the adjustment subroutine S2480 shown, the display control block 220 allows, for example... Figure 17 The content image 20 is displayed overlapping the entire area of ​​the background video 24, so that the entire area is set as the display object range Ad of the content image 20, thereby adjusting the occupancy Rc to the maximum value. At this time, in the content image 20, the ground image 20c, which represents the boundary range Aol on the ground Mc of the external sensing space Ao, and the restricted image 20cl are as follows: Figure 17 As shown, it is overlaid on the background video 24. By completing the execution of S2480, the current execution of the adjustment subroutine and processing flow ends. Therefore, in the next subsequent execution of the processing flow, the restricted image 20c1 continues to be displayed until a positive determination is made in either S2420 or S2460.

[0180] (Effects)

[0181] The effects of the second embodiment described above will be explained below.

[0182] In the second embodiment, the background area Ab, visually identified by the user Us via the mobile terminal Mt carried by the user Us, is monitored. Therefore, the occanceling percentage Rc of the display object range Ad in the background video 24, which is overlaid on the content image 20 that corresponds to the desired content Cd of the user Us and reflects the background area Ab, is adjusted according to the risk level Lr in the background area Ab.

[0183] At this point, particularly according to the second embodiment, the risk level Lr based on the external sensing information Io is identified by acquiring the external sensing information Io sensed by the autonomous driving device Ma, which guides the user Us moving toward the background area Ab. Accordingly, by effectively utilizing the autonomous driving device Ma, which performs the function of guiding the user Us, the risk level Lr can be identified over the widest possible range of the background area Ab, which becomes the user Us's destination, thereby ensuring the safety of the user Us.

[0184] Furthermore, according to the second embodiment, the overlapping display of the content image 20 is prohibited in the video portion 24h of the background video 24 corresponding to the existing external sensing space Ao, which corresponds to the dangerous space portion Aoh of the existing risk level Lr reaching the danger level Lh. This external sensing space Ao becomes the acquisition range of external sensing information Io in the background area Ab. Therefore, in the background video 24, the occupancy Rc can be adjusted such that the content image 20 disappears only in the portion 24h corresponding to the dangerous space portion Aoh of the danger level Lh that requires a danger notification to the moving user Us, while the overlapping display of the content image 20 continues in the portion 24s of the background area Ab corresponding to the outside of the dangerous space portion Aoh. Therefore, for the moving user Us, a balance between security ensured according to the principle described above and the continuity of the virtual experience can be achieved.

[0185] According to the second embodiment, a restricted image 20cl, representing the boundary range Aol on the ground Mc of the external sensing space Ao, is overlaid on a background video 24 reflecting the existing background area Ab. Accordingly, while informing the user Us of the boundary range Aol of the external sensing space Ao where the risk level Lr is identified, the user Us's safety is ensured by identifying the risk level Lr within the external sensing space Ao, thereby also ensuring the user Us's sense of security. Therefore, combined with the disappearance of the content image 20 confined to the portion 24h corresponding to the dangerous space Aoh, the user Us's safety and sense of security can be ensured while maintaining the continuity of the virtual experience.

[0186] According to the second embodiment, risk level Lr is identified based on internal sensing information Ii sensed by the mobile terminal Mt. Accordingly, risk level Lr can also be identified within the proximity range of the user Us via the mobile terminal Mt, thereby ensuring the safety of the user Us. However, if the risk level Lr in the internal sensing space Ai, which is the acquisition range of the internal sensing information Ii in the background area Ab, reaches danger level Lh, overlapping display of content image 20 is prohibited across the entire area of ​​the background video 24 corresponding to the existing background area Ab. Accordingly, for the proximity range of the user Us, the occence Rc, which intentionally disrupts the balance, can be adjusted in a manner that prioritizes safety over the continuity of the virtual experience.

[0187] According to the second embodiment, the risk level Lr based on the external sensing information Io collected by the infrastructure system 6 can also be identified by acquiring the external sensing information Io sensed by the autonomous driving device Ma. Accordingly, not only is the autonomous driving device Ma, which guides the user Us, effectively utilized, but the infrastructure system 6 is also effectively utilized, thereby enabling high-precision identification of the risk level Lr for the background area Ab, thus ensuring the safety of the user Us.

[0188] (Other implementation methods)

[0189] The above describes several embodiments, but this disclosure is not limited to these embodiments and can be applied to various embodiments and combinations without departing from the spirit of this disclosure.

[0190] In variations of the first and second embodiments, the dedicated computer constituting the control system 9 and 2009 of the processing system 1 and 2001 may have at least one of digital circuitry and analog circuitry as a processor. Here, the digital circuitry may be, for example, at least one of ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), SOC (System on a Chip), PGA (Programmable Gate Array), and CPLD (Complex Programmable Logic Device). Furthermore, such a digital circuitry may also have a memory storing a program.

[0191] In variations of the first and second embodiments, the processing systems 1 and 2001 may also be non-connected to the infrastructure system 6. In this variation, the sensing information of the connection elements Wt, Ma, and Mt with the processing systems 1 and 2001 can be used instead of the three-dimensional spatial information It obtained from the infrastructure database Di.

[0192] In variations of the first and second embodiments, processing systems 1 and 2001 may also obtain three-dimensional spatial information Ti, which is not associated with voxel Vi, from the infrastructure database Di. In variations of the first and second embodiments, processing systems 1 and 2001 may also obtain two-dimensional mesh information from the infrastructure database Di instead of three-dimensional spatial information It.

[0193] In a variation of the first embodiment, where the mobile terminal Mt is connected to the processing system 1 and functions as part of the wearable terminal Wt, as in the second embodiment, the processing flow can also be executed in response to the sensor unit 2003 constituting that part acquiring a service request input as sensing information. In another variation of the first embodiment, where the mobile terminal Mt is connected to the processing system 1 and functions as part of the wearable terminal Wt, in step S20 of the processing flow, it is desired that the content Cd can also be acquired based on the sensing information from the sensor unit 2003 constituting that part.

[0194] In a variation of the second embodiment, a smartwatch or a video-transparent or non-transparent wearable terminal may also be used as the mobile terminal Mt. In another variation of the second embodiment, where the wearable terminal Wt is connected to the processing system 2001 and functions as part of the mobile terminal Mt as in the first embodiment, the processing flow can be executed in response to the sensor unit 3 constituting that part acquiring a service request input as sensing information. In another variation of the second embodiment, where the wearable terminal Wt is connected to the processing system 2001 and functions as part of the mobile terminal Mt as in the first embodiment, in S2020 of the processing flow, it is desired that the content Cd can also be acquired based on the sensing information from the sensor unit 3 constituting that part.

[0195] In variations of the first and second embodiments, the display of the restricted image 20cl may be omitted in S480 and S2480. In variations of the first and second embodiments, the restricted image 20cl may be displayed within the display object range Ad of the content image 20 in S470 and S2470. In a variation of the first embodiment, in S430 when the risk level Lr in the internal sensing space Ai reaches the danger level Lh, overlapping display of the content image 20 may be prohibited only within the internal sensing space Ai in the field of view area Av. In a variation of the second embodiment, in S2430 when the risk level Lr in the internal sensing space Ai reaches the danger level Lh, overlapping display of the content image 20 may be prohibited only within the video portion corresponding to the internal sensing space Ai in the background area Ab.

[0196] In addition to the above description, the above-described embodiments and variations can also be implemented as processing systems 1 and 2001, each having at least one processor 9b and one memory 9a in control systems 9 and 2009, in the form of semiconductor devices (e.g., semiconductor chips).

[0197] (Postscript)

[0198] This specification discloses several technical ideas and combinations thereof, as listed below.

[0199] (Technical Idea 1)

[0200] A processing system having a processor (9b) that performs XR display-related processing associated with displaying an XR image to a moving user.

[0201] The processor is configured to execute the following:

[0202] Monitoring the field of view (Av) perceived by the user through the wearable terminal (Wt) worn by the user; and

[0203] The occupancy (Rc) of the display object range (Ad) in the field of view is adjusted based on the risk level (Lr) in the field of view, and the content image (20) that is overlaid on the field of view as the XR image corresponding to the user's desired content (Cd) is adjusted.

[0204] Adjusting the market share includes the following:

[0205] Acquire external sensing information (Io) sensed by the autonomous driving device (Ma) that guides the user moving toward the field of vision;

[0206] Identify the risk level based on the externally sensed information; and

[0207] Overlapping of the content image is prohibited in the dangerous space (Aoh) to adjust the occupancy. The dangerous space is the space in the external sensing space (Ao) that is within the field of view and is within the range of external sensing information acquisition, where the risk level has reached a level that requires a hazard notification to the moving user.

[0208] (Technical Idea 2)

[0209] In the processing system described in Technical Idea 1,

[0210] Adjusting the market share includes the following:

[0211] A limited image (20cl) of the XR image, which represents the boundary range (Aol) on the ground (Mc) of the external sensing space, is overlaid on the field of view.

[0212] (Technical Idea 3)

[0213] In the processing system described in technical concept 1 or 2,

[0214] Adjusting the market share includes the following:

[0215] Acquire internal sensing information (Ii) sensed by the wearable terminal;

[0216] Identify the risk level based on the internally sensed information; and

[0217] If the risk level in the internal sensing space (Ai) that is within the field of view area, which is the acquisition range of the internal sensing information, reaches a level that requires a danger notification to the moving user, the overlapping display of the content image is prohibited in the entire area of ​​the field of view area, thereby adjusting the occupancy.

[0218] (Technical Idea 4)

[0219] A processing system having a processor (9b) that performs XR display-related processing associated with displaying an XR image to a moving user.

[0220] The processor is configured to execute the following:

[0221] The monitoring background area (Ab) becomes the display background from the mobile terminal (Mt) carried by the user to the user; and

[0222] The content image (20) of the background video (24) that is overlaid on the background video (24) reflecting the background area is adjusted according to the risk level (Lr) in the background area, and the occupancy (Rc) of the display object range (Ad) in the background video is adjusted as the XR image corresponding to the user's desired content (Cd).

[0223] Adjusting the market share includes the following:

[0224] Acquire external sensing information (Io) sensed by the autonomous driving device (Ma) that guides the user moving toward the background area;

[0225] Identify the risk level based on the externally sensed information; and

[0226] The overlapping display of the content image in the background video of the existing external sensing space (Ao) and the video portion of the existing dangerous space (Aoh) is prohibited in 24h, thereby adjusting the occupancy rate. The external sensing space becomes the acquisition range of the external sensing information in the background area, and the dangerous space is the space where the risk level has reached the level that requires a danger notification to the moving user.

[0227] (Technical Idea 5)

[0228] In the processing system described in Technical Idea 4,

[0229] Adjusting the market share includes the following:

[0230] A limited image (20cl) of the XR image, which represents the boundary range (Aol) on the ground (Mc) of the external sensing space, is overlaid on the background video.

[0231] (Technical Idea 6)

[0232] In the processing system described in technical concept 4 or 5,

[0233] Adjusting the market share includes the following:

[0234] Acquire internal sensing information (Ii) sensed by the mobile terminal;

[0235] Identify the risk level based on the internally sensed information; and

[0236] If the risk level in the internal sensing space (Ai) that is within the acquisition range of the internal sensing information in the background area reaches a danger level that requires a danger notification to the moving user, the overlapping display of the content image is prohibited in the entire area of ​​the background video, thereby adjusting the occupancy.

[0237] (Technical Idea 7)

[0238] In any of the processing systems described in technical ideas 1 to 6,

[0239] Adjusting the market share includes the following:

[0240] Acquire the external sensing information collected by the infrastructure system (6).

[0241] Furthermore, the aforementioned technical ideas 1 to 7 can also be grasped through the various technical ideas of methods and procedures.

Claims

1. A processing system having a processor (9b) for performing XR display-related processing associated with displaying an XR image to a moving user, characterized in that, The processor is configured to execute the following: Monitoring the field of view (Av) perceived by the user through the wearable terminal (Wt) worn by the user; and The occupancy (Rc) of the display object range (Ad) in the field of view is adjusted based on the risk level (Lr) in the field of view, and the content image (20) that is overlaid on the field of view as the XR image corresponding to the user's desired content (Cd) is adjusted. Adjusting the market share includes the following: Acquire external sensing information (Io) sensed by the autonomous driving device (Ma) that guides the user moving toward the field of vision; Identify the risk level based on the externally sensed information; as well as Overlapping of the content image is prohibited in the dangerous space (Aoh) to adjust the occupancy. The dangerous space is the space in the external sensing space (Ao) that is within the field of view and is within the range of external sensing information acquisition, where the risk level has reached a level that requires a hazard notification to the moving user.

2. The processing system according to claim 1, characterized in that, Adjusting the market share includes the following: A limited image (20cl) of the XR image, which represents the boundary range (Aol) on the ground (Mc) of the external sensing space, is overlaid on the field of view.

3. The processing system according to claim 1, characterized in that, Adjusting the market share includes the following: Acquire internal sensing information (Ii) sensed by the wearable terminal; Identify the risk level based on the internal sensing information; as well as If the risk level in the internal sensing space (Ai) that is within the field of view area, which is the acquisition range of the internal sensing information, reaches a level that requires a danger notification to the moving user, the overlapping display of the content image is prohibited in the entire area of ​​the field of view area, thereby adjusting the occupancy.

4. A processing system having a processor (9b) that performs XR display-related processing associated with displaying an XR image to a moving user, characterized in that, The processor is configured to execute the following: The monitoring background area (Ab) becomes the display background from the mobile terminal (Mt) carried by the user to the user; and The content image (20) of the background video (24) that is overlaid on the background video (24) reflecting the background area is adjusted according to the risk level (Lr) in the background area, and the occupancy (Rc) of the display object range (Ad) in the background video is adjusted as the XR image corresponding to the user's desired content (Cd). Adjusting the market share includes the following: Acquire external sensing information (Io) sensed by the autonomous driving device (Ma) that guides the user moving toward the background area; Identify the risk level based on the externally sensed information; as well as The overlapping display of the content image in the background video of the existing external sensing space (Ao) and the existing dangerous space (Aoh) is prohibited in 24h, thereby adjusting the occupancy. The external sensing space becomes the acquisition range of the external sensing information in the background area, and the dangerous space is the space where the risk level has reached the level that requires a danger notification to the moving user.

5. The processing system according to claim 4, characterized in that, Adjusting the market share includes the following: A limited image (20cl) of the XR image, which represents the boundary range (Aol) on the ground (Mc) of the external sensing space, is overlaid on the background video.

6. The processing system according to claim 4, characterized in that, Adjusting the market share includes the following: Acquire internal sensing information (Ii) sensed by the mobile terminal; Identify the risk level based on the internal sensing information; as well as If the risk level in the internal sensing space (Ai) that is within the acquisition range of the internal sensing information in the background area reaches a level that requires a danger notification to the moving user, the overlapping display of the content image is prohibited in the entire area of ​​the background video, thereby adjusting the occupancy.

7. The processing system according to claim 1 or 4, characterized in that, Adjusting the market share includes the following: Acquire the external sensing information collected by the infrastructure system (6).

8. A processing method, executed by a processor (9b), to perform XR display-related processing associated with displaying an XR image to a moving user, characterized in that, This processing method includes the following: Monitoring the field of view (Av) perceived by the user through the wearable terminal (Wt) worn by the user; and The occupancy (Rc) of the display object range (Ad) in the field of view is adjusted based on the risk level (Lr) in the field of view, and the content image (20) that is overlaid on the field of view as the XR image corresponding to the user's desired content (Cd) is adjusted. Adjusting the market share includes the following: Acquire external sensing information (Io) sensed by the autonomous driving device (Ma) that guides the user moving toward the field of vision; Identify the risk level based on the externally sensed information; as well as Overlapping of the content image is prohibited in the dangerous space (Aoh) to adjust the occupancy. The dangerous space is the space in the external sensing space (Ao) that is within the field of view and is within the range of external sensing information acquisition, where the risk level has reached a level that requires a hazard notification to the moving user.

9. A processing method, executed by a processor (9b), to perform XR display-related processing associated with displaying an XR image to a moving user, characterized in that, This processing method includes the following: The monitoring background area (Ab) becomes the display background from the mobile terminal (Mt) carried by the user to the user; and The content image (20) of the background video (24) that is overlaid on the background video (24) reflecting the background area is adjusted according to the risk level (Lr) in the background area, and the occupancy (Rc) of the display object range (Ad) in the background video is adjusted as the XR image corresponding to the user's desired content (Cd). Adjusting the market share includes the following: Acquire external sensing information (Io) sensed by the autonomous driving device (Ma) that guides the user moving toward the background area; Identify the risk level based on the externally sensed information; as well as The overlapping display of the content image in the background video of the existing external sensing space (Ao) and the existing dangerous space (Aoh) is prohibited in 24h, thereby adjusting the occupancy. The external sensing space becomes the acquisition range of the external sensing information in the background area, and the dangerous space is the space where the risk level has reached the level that requires a danger notification to the moving user.

10. A processing program stored in a storage medium (9a) for performing XR display-related processing associated with displaying an XR image to a moving user, and comprising commands for causing a processor (9b) to perform the XR display-related processing, characterized in that, The handler contains commands for performing the following: Monitoring the field of view (Av) perceived by the user through the wearable terminal (Wt) worn by the user; and The occupancy (Rc) of the display object range (Ad) in the field of view is adjusted based on the risk level (Lr) in the field of view, and the content image (20) that is overlaid on the field of view as the XR image corresponding to the user's desired content (Cd) is adjusted. Adjusting the market share includes the following: Acquire external sensing information (Io) sensed by the autonomous driving device (Ma) that guides the user moving toward the field of vision; Identify the risk level based on the externally sensed information; as well as Overlapping of the content image is prohibited in the dangerous space (Aoh) to adjust the occupancy. The dangerous space is the space in the external sensing space (Ao) that is within the field of view and is within the range of external sensing information acquisition, where the risk level has reached a level that requires a hazard notification to the moving user.

11. A processing program stored in a storage medium (9a) for performing XR display-related processing associated with displaying an XR image to a moving user, and comprising commands for causing a processor (9b) to perform the XR display-related processing, characterized in that, The handler contains commands for performing the following: The monitoring background area (Ab) becomes the display background from the mobile terminal (Mt) carried by the user to the user; and The content image (20) of the background video (24) that is overlaid on the background video (24) reflecting the background area is adjusted according to the risk level (Lr) in the background area, and the occupancy (Rc) of the display object range (Ad) in the background video is adjusted as the XR image corresponding to the user's desired content (Cd). Adjusting the market share includes the following: Acquire external sensing information (Io) sensed by the autonomous driving device (Ma) that guides the user moving toward the background area; Identify the risk level based on the externally sensed information; as well as The overlapping display of the content image in the background video of the existing external sensing space (Ao) and the existing dangerous space (Aoh) is prohibited in 24h, thereby adjusting the occupancy. The external sensing space becomes the acquisition range of the external sensing information in the background area, and the dangerous space is the space where the risk level has reached the level that requires a danger notification to the moving user.

Images