Processing system, processing method, and processing program

Abstract

A processor of a processing system for executing service-related processing is configured to execute the following contents: acquiring content information (Ci) identifying service content (Cs) and display content (Cd) of a user's intention based on sensing information of a mobile terminal (Mt) carried by the user; monitoring a background region (Ab) as a display background by a user facing the mobile terminal (Mt); providing, in the current background area (Ab), a user service of a category corresponding to the service content (Cs) by driving control of the autonomous travel device (Ma); and displaying an XR image corresponding to the display content (Cd) on the presence position (Pm) of the autonomous travel device (Mt) in a mobile terminal (Mt) that displays a background video indicating that the autonomous travel device (Ma) that provides the user service enters the background region (Ab), wherein the XR image is superimposed on the presence position (Pm) of the autonomous travel device (Mt).

Description

[0001] Cross - reference to related applications

[0002] This application is based on Japanese Patent Application No. 2022 - 199677 filed in Japan on December 14, 2022, and the entire contents of the basic application are incorporated by reference. Technical field

[0003] The present disclosure relates to a processing technology for performing service - related processing associated with user services provided to users. Background art

[0004] In the technology disclosed in Patent Document 1, on a display device such as a monitor, a touch panel, or an HMD (Head Mounted Display), a content image in which a frame image such as a video is aligned with the size of an imaginary object as display content and overlapped is displayed.

[0005] Prior art documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent No. 6609078 Gazette

[0008] In the technology disclosed in the above - mentioned Patent Document 1, in a distribution service of a video such as a live - broadcast content, a technology of using a virtual character for user identification as an imaginary object is adopted. On the contrary, in recent years, the service contents that are the contents of user services are not limited to such distribution services, but are diversified. For example, in smart cities planned to be built in various countries around the world, service contents coordinated with autonomous driving devices are expected. At the same time, as display contents, not only virtual characters for user identification are expected, but images matching diversified service contents are expected to satisfy user preferences. Summary of the invention

[0009] The technical problem of the present invention is to be able to provide user services that satisfy user preferences through coordination with autonomous driving devices.

[0010] Hereinafter, the technical means of the present disclosure for solving the technical problem will be described.

[0011] A first aspect of the present disclosure is

[0012] A processing system having a processor that performs service - related processing associated with user services provided to users

[0013] The processor is configured to perform the following:

[0014] Obtain content information of service content and display content that identifies a user's intention based on sensing information of a mobile terminal carried by the user;

[0015] Monitor the background area that serves as the display background by the user facing the mobile terminal;

[0016] In the current background area, provide user services of the category corresponding to the service content in the content information through the driving control of the autonomous driving device; and

[0017] In the mobile terminal that displays the background video of the background area entered by the autonomous driving device providing the user service, overlap and display the XR image corresponding to the display content in the content information at the position where the autonomous driving device exists.

[0018] The second mode of the present disclosure is

[0019] A processing method, executed by a processor to perform service association processing associated with the user service provided to the user, the processing method includes the following:

[0020] Obtain content information that identifies the service content and display content of the user's intention based on the sensing information of the mobile terminal carried by the user;

[0021] Monitor the background area that serves as the display background by the user facing the mobile terminal;

[0022] In the current background area, provide user services of the category corresponding to the service content in the content information through the driving control of the autonomous driving device; and

[0023] In the mobile terminal that displays the background video of the background area entered by the autonomous driving device providing the user service, overlap and display the XR image corresponding to the display content in the content information at the position where the autonomous driving device exists.

[0024] The third mode of the present disclosure is

[0025] A processing program, including commands stored in a storage medium and executed by a processor to perform user association services associated with the user service provided to the user,

[0026] The commands include the following:

[0027] Obtain content information that identifies the service content and display content of the user's intention based on the sensing information of the mobile terminal carried by the user;

[0028] Monitor the background area that serves as the display background by the user facing the mobile terminal;

[0029] In the current background area, provide user services of the category corresponding to the service content in the content information through the driving control of the autonomous driving device; and

[0030] In a mobile terminal that displays a background video of a background area where an autonomous driving device providing user services enters, an XR image corresponding to the displayed content in the content information is overlapped and displayed at the position where the autonomous driving device exists.

[0031] According to these first to third methods, based on the sensing information of the mobile terminal carried by the user, content information of service content that identifies the user's intention is obtained. At the same time, by the user facing the mobile terminal, the background area serving as the display background is monitored. Therefore, in the current background area, user services of a category corresponding to the service content are provided through the driving control of the autonomous driving device. This means that among diverse user services, user services that meet the user's preferences can also be provided through coordination with the autonomous driving device.

[0032] Moreover, according to the first to third methods, content information of the displayed content that identifies the user's intention is also obtained based on the sensing information of the mobile terminal. Therefore, as described above, in a mobile terminal that displays a background video of a background area where an autonomous driving device providing user services enters, an XR image corresponding to the displayed content is overlapped and displayed at the position where the autonomous driving device exists. This means that by overlapping and displaying an XR image corresponding to the displayed content that matches the service content provided by the autonomous driving device, the satisfaction of the user's preferences can be improved.

[0033] The fourth method of the present disclosure is

[0034] A processing system having a processor that executes service-related processing associated with user services provided to a user

[0035] The processor is configured to execute the following:

[0036] Obtain content information of service content and displayed content that identify the user's intention based on the sensing information of a wearable terminal worn by the user;

[0037] Monitor the visual field area visually confirmed by the user through the wearable terminal;

[0038] In the current visual field area, provide user services of a category corresponding to the service content in the content information through the driving control of the autonomous driving device; and

[0039] In a wearable terminal where an autonomous driving device providing user services enters the visual field area, overlap and display an XR image corresponding to the displayed content in the content information at the position where the autonomous driving device exists.

[0040] The fifth method of the present disclosure is

[0041] A processing method, executed by a processor, for performing service - related processing associated with a user service provided to a user, the processing method comprising the following:

[0042] Obtaining content information that identifies the service content and display content of the user's intention based on the sensing information of the wearable terminal worn by the user;

[0043] Monitoring the visual field area visually confirmed by the user through the wearable terminal;

[0044] In the current visual field area, providing a user service of a category corresponding to the service content in the content information through the driving control of the autonomous driving device; and

[0045] In the wearable terminal where the autonomous driving device providing the user service enters the visual field area, overlapping and displaying the XR image corresponding to the display content in the content information at the position where the autonomous driving device exists.

[0046] The sixth aspect of the present disclosure is

[0047] A processing program, comprising commands stored in a storage medium and executed by a processor, for performing user - related services associated with a user service provided to a user,

[0048] The commands comprise the following:

[0049] Obtaining content information that identifies the service content and display content of the user's intention based on the sensing information of the wearable terminal worn by the user;

[0050] Monitoring the visual field area visually confirmed by the user through the wearable terminal;

[0051] In the current visual field area, providing a user service of a category corresponding to the service content in the content information through the driving control of the autonomous driving device; and

[0052] In the wearable terminal where the autonomous driving device providing the user service enters the visual field area, overlapping and displaying the XR image corresponding to the display content in the content information at the position where the autonomous driving device exists.

[0053] According to the above - mentioned fourth to sixth aspects, based on the sensing information of the wearable terminal worn by the user, content information that identifies the service content of the user's intention is obtained. At the same time, the visual field area visually confirmed by the user through the wearable terminal is monitored. Therefore, in the current visual field area, a user service of a category corresponding to the service content is provided through the driving control of the autonomous driving device. This means that among diverse user services, user services that meet the user's preferences can also be provided through coordination with the autonomous driving device.

[0054] Moreover, according to the fourth to sixth aspects, content information of the display content that identifies the user's intention is also obtained based on the sensing information of the mobile terminal. Therefore, as described above, in the wearable terminal where the autonomous driving device providing the user service enters the visual field area, the XR image corresponding to the display content is overlapped and displayed at the position where the autonomous driving device exists. This means that by overlapping and displaying the XR image corresponding to the display content that matches the service content provided by the autonomous driving device, the satisfaction of user preference can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is an overall structural diagram showing the network connection environment of the processing system according to the first embodiment.

[0056] Figure 2 is a structural diagram showing the display state of the mobile terminal according to the first embodiment.

[0057] Figure 3 is a schematic diagram for explaining the three-dimensional voxels assumed in the infrastructure system according to the first embodiment.

[0058] Figure 4 is a block diagram showing the processing system according to the first embodiment.

[0059] Figure 5 is a flowchart showing the processing flow according to the first embodiment.

[0060] Figure 6 is a structural diagram showing the display state of the mobile terminal according to the first embodiment.

[0061] Figure 7 is a structural diagram showing the display state of the mobile terminal according to the first embodiment.

[0062] Figure 8 is a perspective view showing the background area of the mobile terminal according to the first embodiment.

[0063] Figure 9 is a perspective view for explaining the drive control mode of the autonomous driving device according to the first embodiment.

[0064] Figure 10 is a structural diagram showing the display state of the mobile terminal according to the first embodiment.

[0065] Figure 11 is an overall structural diagram showing the network connection environment of the processing system according to the second embodiment.

[0066] Figure 12 is a structural diagram showing the display state of the wearable terminal according to the second embodiment.

[0067] Figure 13It is a block diagram showing the processing system of the second embodiment.

[0068] Figure 14 It is a flowchart showing the processing flow of the second embodiment. Detailed implementation manners

[0069] Hereinafter, a plurality of embodiments of the present disclosure will be described with reference to the drawings. In addition, in each embodiment, sometimes the same reference numerals are assigned to corresponding components and repeated descriptions are omitted. Further, when only a part of the structure is described in each embodiment, the structure of the other part of the structure can be applied to the structure of the other previously described embodiment. Furthermore, not only can the combinations of the structures specified in the description of each embodiment be made, but also the structures of a plurality of embodiments can be partially combined with each other without being particularly obstructive to the combination, even if not specified.

[0070] (First Embodiment)

[0071] Figure 1 The processing system 1 of the first embodiment shown performs service-related processing associated with the user service provided to the user Us. Therefore, the processing system 1 is connected to the mobile terminal Mt carried by the previously registered user Us, the autonomous driving device Ma that autonomously travels, and the infrastructure system Is that provides infrastructure information via the communication network Nc. At least one is assumed as the connection elements Mt, Ma, Is to the processing system 1.

[0072] As Figure 1 , 2 shown, the mobile terminal Mt is configured such that at least when connected to the processing system 1, the user Us of the system 1 can hold it with a finger and operate it. The mobile terminal Mt is a small electronic device having a display unit 2 and a sensor unit 3, and is, for example, a smart phone or a tablet terminal. The mobile terminal Mt controls the transmission and reception of data via the communication network Nc through the communication unit. The mobile terminal Mt performs operation control by the control unit according to the control instruction from the system 1 via the communication network Nc to jointly execute service-related processing with the processing system 1.

[0073] Under the operation control of the control unit of the mobile terminal Mt, for example, in the display unit 2 such as a liquid crystal panel or an organic EL panel, as Figure 2The display required for implementing service association processing is shown. Meanwhile, under the action control of the mobile terminal Mt, the sensor unit 3 obtains sensing information through sensing processing corresponding to the service association processing. As the sensor unit 3, at least one of, for example, a camera, an inertial sensor, a GNSS (Global Navigation Satellite System) sensor, a touch sensor, and a microphone is adopted. With such a structure, the sensor unit 3 can obtain an input from the same user Us representing the intention of the user Us as sensing information.

[0074] Figure 1 The autonomous driving device Ma shown is an autonomous driving vehicle or an autonomous driving robot that can autonomously drive in any direction of front, rear, left, and right by electrically driving the wheels 5 based on the sensing information of the sensor unit 4. The autonomous driving device Ma controls the transmission and reception of data via the communication network Nc through the communication unit. The autonomous driving device Ma is driven and controlled by the control unit according to the control instruction from the system 1 via the communication network Nc to jointly execute service association processing with the processing system 1. Under this driving control, the sensor unit 4 obtains sensing information through sensing processing corresponding to the service association processing. As the sensor unit 4, at least one of, for example, a camera, an inertial sensor, a GNSS sensor, a LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging), and a sonar is adopted.

[0075] The infrastructure system Is, as a common basis for a decentralized architecture, is a basic system that shares the three-dimensional space information Ti accumulated in the infrastructure database Di. The infrastructure system Is controls the transmission and reception of data via the communication network Nc through the communication unit. The infrastructure system Is collects at any time the three-dimensional space information Ti provided to individual decentralized systems such as the processing system 1 that constitutes the decentralized architecture by the control unit, and updates the accumulated information in the infrastructure database Di to the latest information.

[0076] Here, in particular, the infrastructure system Is virtually divides the three-dimensional space to be accumulated with information, as Figure 3 shown, into a plurality of three-dimensional voxels Vi (i.e., a three-dimensional grid) imagined in a three-dimensional arrangement and manages the information. The infrastructure system Is uses a data set in which each space ID assigned to each voxel Vi is associated with metadata as the three-dimensional space information Ti, as Figure 1Accumulated in the infrastructure database Di as shown. Here, the three-dimensional space information Ti may of course also include two-dimensional grid information related only to the lower surface, for example, of the voxel Vi that forms the lowermost layer of the two-dimensional arrangement along the ground in the three-dimensional arrangement.

[0077] The data for constructing the three-dimensional space information Ti can also be collected from at least one of the mobile terminal Mt that is common with the processing system 1, other mobile terminals, and wearable terminals such as HMDs or smart glasses. The data for constructing the three-dimensional space information Ti can also be collected from at least one of the autonomous driving device Ma that is common with the processing system 1 and other moving bodies. The data for constructing the three-dimensional space information Ti can also be collected from at least one of infrastructure sensors such as cameras and / or LiDARs, such as communication base stations, smart poles, and smart streetlights. The data for constructing the three-dimensional space information Ti can also be collected from a server that provides at least one of, for example, map services, meteorological services, communication services, traffic management services, ground object management services, and air traffic management services, as well as the user services described later processed by the processing system 1.

[0078] The construction data collected and used to construct the three-dimensional space information Ti can, for example, be at least one type of image data such as videos, still images, and point cloud images. The construction data of the three-dimensional space information Ti can also be at least one type of secondary data that ensures information security, such as position data, motion data, pose data, gaze data, movement data, and crowd flow data, as well as intention data indicating the intention of the user Us described later, generated for the person including the user Us of the processing system 1 in the object space where information is accumulated through image processing of such image data. The construction data of the three-dimensional space information Ti can also be at least one type of secondary data such as position data, motion data, and pose data generated for the mobile terminal Mt including the user Us in the information storage object space through image processing of the image data of a person's carried items. The construction data of the three-dimensional space information Ti can also be at least one type of secondary data such as position data, motion data, and pose data generated for the moving body including the autonomous driving device Ma in the information accumulation object space through image processing of the image data.

[0079] The construction data of the three-dimensional space information Ti can also be voice data that collects the voice of a person, i.e., the user Us of the processing system 1, within the information accumulation object space. The construction data of the three-dimensional space information Ti can also be at least one type of secondary data that ensures information security, such as position data, motion data, action data, pedestrian flow data, conversation data, and intention data representing the intention of the user Us described later, etc., which are analyzed for the person including the user Us through voice recognition processing of such voice data. The construction data of the three-dimensional space information Ti can also be at least one type of data such as two-dimensional and / or three-dimensional map data, GIS (Geographic Information System) data, road network data, meteorological data, communication data, line data, traffic data, ground object management data, BIM (Building Information Modelling) data, POI (Point of Interest) data, aviation management data, and time data, etc. The construction data of the three-dimensional space information Ti can also be service data associated with at least one type of user service processed by the processing system 1, such as a photography service, a trash can guiding service, a delivery service, and a guiding service, etc., which will be described later.

[0080] As Figure 1 shown, the processing system 1 is a distributed computer system constructed by including at least one type of system such as a cloud server and an edge server, etc., and having a communication system 10 and a control system 20. At least a part of each of the systems 10 and 20 in the processing system 1 can also be respectively composed of a communication unit and a control unit of a mobile terminal Mt. At least a part of each of the systems 10 and 20 in the processing system 1 can also be respectively composed of a communication unit and a control unit of an autonomous driving device Ma. At least a part of each of the systems 10 and 20 in the processing system 1 can also be respectively composed of a communication unit and a control unit of an infrastructure system Is.

[0081] The communication system 10 is mainly composed of communication devices for constructing a communication network Nc. The control system 20 is connected to the communication system 10 via at least one of a wired communication line and a wireless communication line. The control system 20 is constructed by including at least one dedicated computer. The dedicated computer constructing the control system 20 has at least one memory 22 and one processor 24.

[0082] In such a processing system 1, the control system 20 executes a plurality of commands of a processing program stored in the memory 22 through the processor 24. As a result, the control system 20 constructs a plurality of functional blocks for performing service-related processing associated with the user service provided to the user Us. As Figure 4As shown, the constructed functional blocks include an identification block 200, a monitoring block 210, a drive control block 220, and a display control block 230.

[0083] The processing system 1 executes the processing method of service association processing through these blocks 200, 210, 220, and 230 together, and executes according to Figure 5 the processing flow shown. This processing flow is executed in response to a service request input, which is sensed information representing the intention of the user US to request a user service, obtained by the sensor unit 3 of the mobile terminal Mt. In addition, each "S" in this processing flow respectively means a plurality of steps executed by a plurality of commands included in the processing program of the first embodiment.

[0084] In S200 of the processing flow, the identification block 200 (refer to Figure 4 ) obtains content information Ci that identifies the service content Cs and the display content Cd that match the intention of the user Us based on the sensed information of the mobile terminal Mt carried by the user Us. The service content Cs identified in this S200 refers to the category of user services that can be provided to the user Us of the processing system 1. Here, as the category of user services corresponding to the service content Cs, in the first embodiment, at least two of a photography service, a trash can guiding service, a delivery service, and a guiding service are prepared.

[0085] The display content Cd identified in S200 refers to the category of XR images 2a, 2d, 2e that are displayed on the display unit 2 of the mobile terminal Mt as shown in Figure 2 , 6 when providing the user service corresponding to the service content Cs, especially the category of the XR image 2a. Here, in the XR image 2a in which the category intended by the user Us is identified as the display content Cd, a plurality of categories of character images representing imaginary or real creatures such as monsters, animals, plants, and insects in two or three dimensions are prepared. In contrast, in the XR images 2d, 2e given a common theme corresponding to the category of the XR image 2a, a plurality of categories of character images representing imaginary or real creatures such as monsters, animals, plants, and insects, as well as inanimate objects such as prehistoric buildings, service vehicles, and imaginary objects in two or three dimensions are prepared.

[0086] In Figure 5 S200, the sensed information used to respectively identify the service content Cs and the display content Cd is input as the intention of the user Us to the sensor unit 3, and thus obtained by the mobile terminal Mt. At this time, in the display unit 2 of the mobile terminal Mt, it can also be as shown in Figure 7As shown, as the selection input of the user Us from the service content Cs and the display content Cd, an input reception image 2b for receiving a touch input through the sensor unit 3 is displayed. As the selection input of the user Us from the service content Cs and the display content Cd, the gesture input, line-of-sight input, expression input, or voice input of the user Us can also be sensed by the sensor unit 3. These gesture inputs, line-of-sight inputs, expression inputs, or voice inputs can also be recognized based on the three-dimensional space information Ti obtained from the infrastructure database Di.

[0087] In Figure 5 In S210 of the processing flow shown, the monitoring block 210 (refer to Figure 4 ) monitors the background area Ab that becomes the display background of the XR image 2a in the display unit 2 as the user Us faces the mobile terminal Mt, as Figure 8 shown. At this time, in the information storage object space where a plurality of voxels Vi are assumed as Figure 3 shown through the infrastructure system Is, in the service space As where user services of the category corresponding to the service content Cs are provided as Figure 8 shown, the background area Ab is monitored. Therefore, by based on at least one of, for example, camera information and inertial information in the sensing information of the sensor unit 3 of the mobile terminal Mt, the monitoring of the background area Ab extended in the orientation of the mobile terminal Mt is performed.

[0088] In Figure 5 In S220 of the processing flow shown, the drive control block 220 (refer to Figure 4 ) provides user services of the category corresponding to the service content Cs in the content information Ci by driving and controlling the autonomous driving device Ma in the current background area Ab. At this time, by reading the drive control mode of the autonomous driving device Ma that matches the service content Cs from the memory 22, the user service is provided according to the drive control mode.

[0089] Such a photography service in S220 is provided by a drive control mode that is as follows: driving the autonomous driving device Ma having a camera unit from a standby position to the current position of the user Us or a landmark around the user Us, taking a picture of the user Us from the surroundings, and then returning to the standby position. At this time, in the photography service, the photography timing or the photography period can also be prompted to the user Us through the sound output from the sound unit of the autonomous driving device Ma. In the photography service, the photography timing or the photography period can also be prompted to the user Us through the display output from the display unit of the autonomous driving device Ma.

[0090] In the photography service, as Figure 9As shown by the white hollow arrow in the figure, it is also possible to take two-dimensional or three-dimensional static images or videos in a way that includes the user Us by means of an autonomous driving device Ma driven around the user Us. In the photography service, it is also possible to adjust the focus during photography based on the distance from the user Us sensed by a sensor unit 4 such as LiDAR in the autonomous driving device Ma. In the photography service, it is also possible to adjust the incident azimuth of external light during photography based on the three-dimensional space information Ti obtained from the infrastructure database Di or the sensing information from a sensor unit 4 such as a camera in the autonomous driving device Ma.

[0091] The trash can guiding service of S220 is provided through a driving control mode, which is a mode of driving an autonomous driving device Ma with a trash can from a standby position to the current position of the user Us, and after receiving the user Us's intention to discard trash into the trash can, returning to the standby position. At this time, in the trash can guiding service, it is also possible to indicate to the user Us the discarding of trash into the trash can through the sound output from the sound unit of the autonomous driving device Ma. In the trash can guiding service, it is also possible to indicate to the user Us the discarding of trash into the trash can through the display output from the display unit of the autonomous driving device Ma.

[0092] The delivery service of S220 is provided through a driving control mode, which is a mode of driving an autonomous driving device Ma with a delivery box from a standby position to the current position of the user Us, and after receiving the user Us's intention to load goods into the delivery box, returning to the standby position. At this time, in the delivery service, it is also possible to indicate to the user Us the loading of goods into the delivery box through the sound output from the sound unit of the autonomous driving device Ma. In the delivery service, it is also possible to indicate to the user Us the loading of goods into the delivery box through the display output from the display unit of the autonomous driving device Ma.

[0093] The guidance service of S220 is provided through a drive control mode, which is a mode of driving the autonomous driving device Ma from a standby position to the current position of a guidance target such as the user Us or a registered family member, and returning to the standby position after performing path guidance for the guidance target. At this time, in the guidance service, the path for guiding the guidance target can also be indicated through sound output from the sound unit of the autonomous driving device Ma. In the guidance service, the path for guiding the guidance target can also be indicated through display output from the display unit of the autonomous driving device Ma. Further, before such path indication, in the mobile terminal Mt, for example, an input acceptance image for accepting input of path-related information such as a destination and / or a transportation means desired to be used from the user Us through the sensor unit 3 can also be displayed on the display unit 2. In addition, a drive control mode for accommodating and transporting the carry-on luggage of the guidance target in the delivery box of the autonomous driving device Ma can also be given along with the path indication.

[0094] Figure 5 S230 shown in the processing flow is executed in parallel with S220. In S230, the display control block 230 (refer to Figure 4 ) causes the background video 2c of the background area Ab into which the autonomous driving device Ma that provides user services through S220 enters, such as Figure 8 、 9 , to be displayed on the display unit 2 of the mobile terminal Mt as Figure 2 、 6 . Here, the background video 2c is obtained, for example, based on the sensing information of the sensor unit 4 such as a camera or the three-dimensional space information Ti obtained from the infrastructure database Di. At this time, the background video 2c can use the primary video data of the background area Ab or the secondary video data obtained by processing a part of the primary data such as the ground or the sky.

[0095] Therefore, Figure 5 in S230, the display control block 230 overlays and displays the XR image 2a corresponding to the display content Cd in the content information Ci on the presence position Pm of the autonomous driving device Ma displayed on the background video 2c on the display unit 2 as Figure 2 、 6 . At this time, the overlay display position of the XR image 2a is adjusted in accordance with the drive control mode of the autonomous driving device Ma in S220 to match the presence position Pm on the background video 2c. Thus, the XR image 2a is replaced and displayed at the presence position Pm of the autonomous driving device Ma on the background video 2c.

[0096] The display control block 230 in S230 is in the background area Ab together with the autonomous driving device Ma asFigure 8 The position Po where other object Oo enters as such is also as Figure 2 , 6 such that the XR image 2d is overlapped and displayed. At this time, the overlapping display position of the XR image 2d is adjusted to match the position Po where the other object Oo exists. In this way, the other object Oo that replaces and displays the XR image 2d at the position Po on the background video 2c is, for example, at least one of a person other than the user Us of the processing system 1, a moving body other than the autonomous driving device Ma sharing with the processing system 1, and a ground object.

[0097] In S230, for the plurality of voxels Vi obtained by hypothetically dividing the service space As as Figure 3 such, the display control block 230 acquires the three-dimensional space information Ti in which the position-related information associated with the respective existence positions Pm, Po of the autonomous driving device Ma and the other object Oo is attached from the infrastructure database Di. Therefore, the display control block 230 in S230 overlaps and displays the XR images 2a, 2d at the respective existence positions Pm, Po based on the acquired three-dimensional space information Ti. Thus, particularly in the first embodiment, the role image associated with the display content Cd in the content information Ci is overlapped and displayed as the XR images 2a, 2d at the respective existence positions Pm, Po.

[0098] The display control block 230 in S230 may also overlap and display a scenery other than the respective existence positions Pm, Po of the autonomous driving device Ma and the other object Oo in the background area Ab Figure 2 as shown in the XR image 2e. In this case, in the first embodiment, as the XR image 2e, the role image associated with the display content Cd may also be overlapped and displayed. At the same time, the overlapping display position of the XR image 2e can be adjusted based on the three-dimensional space information Ti acquired from the infrastructure database Di so as to deviate from the respective overlapping positions of the XR images 2a, 2d.

[0099] Through S230 like this, in the photography service, the background video 2c in which at least the XR images 2a, 2d are overlapped with the existence position Pm of the autonomous driving device Ma driven to photograph the user Us and the existence position Po of the surrounding object Oo is displayed and output to the user Us. At this time, by switching the display from Figure 10 as shown in Figure 2 the background video 2c to Figure 9As a still image or video of the captured photographic image 2f of the user Us and their background area, the XR image 2a can also be overlapped and displayed at the position Ps of at least the former of the user Us and their companion in the photographic image 2f. In this way, the image data of the photographic image 2f of the user Us and the background area with the XR image 2a overlapped can also be stored in the internal or external storage medium of the mobile terminal Mt.

[0100] In S230, in the trash bin guiding service, at least the XR images 2a, 2d are overlapped on the background video 2c of the position Pm of the autonomous driving device Ma driven to receive the disposal of trash from the user Us and the position Po of the surrounding objects Oo, and are displayed and output to the user Us. At this time, the XR image 2a corresponding to the autonomous driving device Ma can also be a character image imitating the trash disposal opening of the trash bin as the mouth of a living being. The XR image 2a corresponding to the autonomous driving device Ma can also include an indication image indicating the trash disposal opening corresponding to the category of the trash to be disposed of among the multiple trash disposal openings of the trash bin.

[0101] In S230, in the delivery service, at least the XR images 2a, 2d are overlapped on the background video 2c of the position Pm of the autonomous driving device Ma driven to receive the loading of goods from the user Us and the position Po of the surrounding objects Oo, and are displayed and output to the user Us. At this time, the XR image 2a corresponding to the autonomous driving device Ma can also be a character image imitating the loading opening of the delivery box as the mouth of a living being.

[0102] In S230, in the guiding service, at least the XR images 2a, 2d are overlapped on the background video 2c of the position Pm of the autonomous driving device Ma driven to guide the guided person and the position Po of the surrounding objects Oo, and are displayed and output to the guided person. At this time, it is also possible to switch and display a still image or video of the photographic image of the background area where the guiding path is captured from the background video 2c. In this way, the image data of the photographic image of the background area where the guiding path is captured can also be stored in the internal or external storage medium of the mobile terminal Mt.

[0103] When the provision of the user service corresponding to the service content Cs in the content information Ci ends through S220, S230, the above Figure 5 execution of the processing flow of the first embodiment shown is completed. Subsequently, when the next service request input is obtained through the sensor unit 3 of the mobile terminal Mt, the next execution of the processing flow starts.

[0104] (Function and effect)

[0105] Hereinafter, the function and effect of the first embodiment described above will be described.

[0106] According to the first embodiment, based on the sensing information of the mobile terminal Mt carried by the user Us, content information Ci of service content Cs that identifies the intention of the user Us is obtained. At the same time, by the user Us facing the mobile terminal Mt, the background area Ab serving as the display background is monitored. Therefore, in the current background area Ab, through the drive control of the autonomous driving device Ma, a user service corresponding to the category of the service content Cs is provided. This means that even among diverse user services, through coordination with the autonomous driving device Ma, a user service that meets the user's preferences can be provided.

[0107] Moreover, according to the first embodiment, based on the sensing information of the mobile terminal Mt, content information Ci of display content Cd that identifies the intention of the user Us is also obtained. Therefore, in the mobile terminal Mt that displays the background video 2c of the background area Ab where the autonomous driving device Ma that provides the user service as described above enters, the XR image 2a corresponding to the display content Cd is overlapped and displayed at the presence position Pm of the autonomous driving device Ma. This means that by the overlapping display of the XR image 2a corresponding to the display content Cd that matches the service content Cs provided by the autonomous driving device Ma, the satisfaction of the user's preferences can be improved.

[0108] According to the first embodiment, the XR image 2d is also overlapped and displayed at the presence position Po of other objects Oo that enter the background area Ab together with the autonomous driving device Ma. Thereby, the overlapping display of the XR images 2a, 2d corresponding to the display content Cd that matches the service content Cs provided by the autonomous driving device Ma is achieved within a wide range in the background video 2c of the background area Ab, and the satisfaction of the user's preferences can be improved.

[0109] According to the first embodiment, three-dimensional space information Ti is obtained from the infrastructure database Di. The three-dimensional space information Ti is information in which position-related information associated with the respective presence positions Pm, Po of the autonomous driving device Ma and other objects Oo is attached to a plurality of voxels Vi obtained by hypothetically dividing the service space As that provides the user service. Therefore, at the respective presence positions Pm, Po of the autonomous driving device Ma and other objects Oo based on the obtained three-dimensional space information Ti, the XR images 2a, 2d that can satisfy the user's preferences in the background video 2c of the background area Ab can be replaced to appropriate positions and overlapped and displayed. Therefore, it is possible to suppress the user's discomfort caused by the position deviation of the overlapping display and at the same time improve the satisfaction of the user's preferences.

[0110] According to the first embodiment, a user service is provided by a drive control mode of the autonomous driving device Ma that matches the service content Cs in the content information Ci. Thus, by driving the autonomous driving device Ma in coordination with the user service, the satisfaction of user preference can be improved. At the same time, according to the first embodiment, the overlapping display position of the XR image 2a with respect to the background video 2c is adjusted in accordance with the position Pm of the autonomous driving device Ma according to the drive control mode. Thus, the XR image 2a corresponding to the display content Cd that matches the service content Cs can be overlapped and displayed within the background video 2c following the drive of the autonomous driving device Ma, and thus the satisfaction of user preference can be further improved. Here, particularly in the first embodiment, by using the three-dimensional space information Ti from the infrastructure database Di, the followability of the XR image 2a with respect to the drive of the autonomous driving device Ma can be improved, and thus the suppression effect of user discomfort caused by the position deviation of the overlapping display can also be improved.

[0111] According to the first embodiment, the character images associated with the display content Cd in the content information Ci are overlapped and displayed as the XR images 2a, 2d at the positions Pm, Po of the autonomous driving device Ma and other objects Oo. Thus, the character images that can satisfy user preference in the background video 2c of the background area Ab can be replaced from the display of the autonomous driving device Ma and other objects Oo, and visually confirmed by the user Us. Therefore, in particular, the satisfaction of user preference with respect to the display content Cd can be improved.

[0112] (Second Embodiment)

[0113] The second embodiment is a modified example of the first embodiment.

[0114] As Figure 11 shown, the processing system 2001 of the second embodiment is connected to a wearable terminal Wt worn by a pre-registered user Us, an autonomous driving device Ma, and an infrastructure system Is via a communication network Nc. At least one is assumed for each of the connection elements Wt, Ma, Is to the processing system 1. In addition, as described in the first embodiment, the mobile terminal Mt carried by the user Us may or may not be connected to the processing system 2001 via the communication network Nc.

[0115] The wearable terminal Wt is configured such that when connected to the processing system 2001 at least, the user Us of the system 2001 can wear it on the face and perform a hands-free operation. As Figure 12As shown, the wearable terminal Wt is an optically transparent electronic device equipped with a display unit 2002 and a sensor unit 2003, such as an HMD or smart glass. The wearable terminal Wt controls the reception and transmission of data via the communication network Nc through the communication unit. The wearable terminal Wt performs operation control by the control unit according to the control instruction from the system 2001 via the communication network Nc to jointly execute service-related processing with the processing system 2001. In addition, regarding the wearable terminal Wt, Figure 12 It schematically shows the observation situation of the field of view area Av from the user Us side, which corresponds to the part corresponding to one eye in the two eyes of the user Us.

[0116] Under the operation control of the control unit of the wearable terminal Wt, the virtual image projection type or retinal projection type display unit 2002 overlaps the display required for service-related processing on the real image of the field of view area Av visually confirmed by the user Us, such as Figure 12 shown, and enables the user Us to visually confirm it. At the same time, under the operation control of the wearable terminal Wt, the sensor unit 2003 acquires sensing information through sensing processing corresponding to service-related processing. As the sensor unit 2003, for example, at least one of a camera, an inertial sensor, a GNSS sensor, an electrooculogram sensor, a gaze sensor, an infrared sensor, a geomagnetic sensor, a motion sensor, a touch sensor, and a microphone is adopted. With such a structure, the sensor unit 2003 can acquire the input from the user Us representing the intention of the user Us as sensing information.

[0117] Figure 11 The autonomous driving device Ma shown is configured in the same way as the processing system 1 of the first embodiment, except that it is driven and controlled by the control unit according to the control instruction from the system 2001 via the communication network Nc to jointly execute service-related processing with the processing system 2001. The control system 2020 of the processing system 2001 is configured in the same way as the control system 20 of the first embodiment, except that at least a part of it can be constituted by the control unit of the wearable terminal Wt, and it executes the processing flow in the form of the wearable terminal Wt based on the following functional block structure.

[0118] As Figure 13 shown, the functional blocks constructed to execute service-related processing in the control system 2020 include an identification block 2200, a monitoring block 2210, a drive control block 2220, and a display control block 2230. The processing system 2001 jointly executes the processing method of service-related processing through these blocks 2200, 2210, 2220, 2230, according to Figure 14The processing flow shown is executed. In response to the sensor unit 2003 of the wearable terminal Wt obtaining a service request input indicating the intention of the user Us to request a user service as sensing information, this processing flow is executed. In addition, each "S" in this processing flow respectively means a plurality of steps executed by a plurality of commands included in the processing program of the second embodiment.

[0119] In S2200 of the processing flow, the recognition block 2200 (refer to Figure 13 ) obtains the content information Ci of the service content Cs and the display content Cd that recognize the intention of the user Us based on the sensing information related to the face of the wearable terminal Wt worn by the user Us on the face. Therefore, by inputting the sensing information as the user intention into the sensor unit 2003, the sensing information is obtained through the wearable terminal Wt. At this time, in the display unit 2002 of the wearable terminal Wt, in order to accept a line-of-sight input corresponding to a change in the direction of the line of sight or an expression input corresponding to a change in the facial expression as a selection input of the content Cs and Cd from the user Us to the sensor unit 2003, the input acceptance image 2b according to the first embodiment may also be overlapped and displayed on the actual image of the visual field area Av. Except for the above points, S2200 is executed according to S200 of the first embodiment.

[0120] In Figure 14 In S2210 of the processing flow shown, the monitoring block 2210 (refer to Figure 13 ) monitors the visual field area Av visually confirmed by the user Us through the wearable terminal Wt. At this time, in the information storage object space assuming a plurality of voxels Vi by the infrastructure system Is, in the service space As where user services of the category corresponding to the service content Cs are provided as Figure 12 such, the visual field area Av is monitored. Therefore, based on at least one of, for example, camera information and inertial information in the sensing information of the sensor unit 2003 of the wearable terminal Wt, the monitoring of the visual field area Av extended by, for example, glass or a lens is performed in the orientation from the face of the user Us to the line of sight or the orientation of the wearable terminal Wt.

[0121] In Figure 14 In S2220 of the processing flow shown, the drive control block 2220 (refer to Figure 13 ) provides the user service of the category corresponding to the service content Cs in the content information Ci through the drive control of the autonomous driving device Ma in the current visual field area Av. Except for this point, S2220 is executed according to S220 of the first embodiment.

[0122] Figure 14 S2230 of the processing flow shown is executed in parallel with S2220. In S2230, the display control block 2230 (refer toFigure 13 )In the wearable terminal Wt where the autonomous driving device Ma that provides user services through S2220 enters the visual field area Av, the XR image 2a corresponding to the display content Cd in the content information Ci is overlapped and displayed as shown in Figure 12 such a way on the presence position Pm of the autonomous driving device Ma. At this time, the overlapping display position of the XR image 2a is adjusted to match the presence position Pm in the visual field area Av according to the driving control mode of the autonomous driving device Ma of S2220. Thus, in the visual field area Av, the XR image 2a becomes a three-dimensional overlapping display that overlaps with the whole in a way that is opaque or semi-transparent and hidden from the front compared to the whole of the autonomous driving device Ma at the presence position Pm.

[0123] The display control block 2230 in S2230 also overlaps and displays the XR image 2d on the presence position Po of other objects Oo that enter the visual field area Av together with the autonomous driving device Ma as shown in Figure 12 such a way. At this time, the overlapping display position of the XR image 2d is adjusted to match the presence position Po of the other object Oo. In this way, the other object Oo with the XR image 2d overlapping at the presence position Po in the visual field area Av becomes at least one of, for example, a person other than the user Us of the processing system 2001, a moving body other than the autonomous driving device Ma shared with the processing system 2001, and a ground object.

[0124] Except for these points, S2230 is executed according to S230 of the first embodiment. As a result, in various user services according to S2230 of the first embodiment, the XR images 2a and 2d that are visually confirmed by overlapping at least on the presence position Pm of the autonomous driving device Ma and the presence position Po of the surrounding object Oo are displayed and output to the user Us.

[0125] When the provision of the user service corresponding to the service content Cs in the content information Ci ends through S2220 and S2230, the execution of the processing flow of the second embodiment shown above is completed. After that, when the next service request input is obtained through the sensor unit 2003 of the wearable terminal Wt, the next execution of the processing flow starts. Figure 14

[0126] (Function and effect)

[0127] Hereinafter, the function and effect of the second embodiment described above will be described.

[0128] ​According to the second embodiment, based on the sensing information of the wearable terminal Wt worn by the user Us, the content information Ci of the service content Cs that identifies the intention of the user Us is obtained. At the same time, the visual field area Av visually confirmed by the user Us through the wearable terminal Wt is monitored. Therefore, in the current visual field area Av, through the drive control of the autonomous driving device Ma, a user service corresponding to the category of the service content Cs is provided. This means that even among diverse user services, through coordination with the autonomous driving device Ma, a user service that meets the user's preference can be provided.

[0129] Moreover, according to the second embodiment, based on the sensing information of the mobile terminal Mt, the content information Ci of the display content Cd that identifies the intention of the user Us is also obtained. Therefore, as described above, in the wearable terminal Wt where the autonomous driving device Ma that provides the user service as described above enters the visual field area Av, the XR image 2a corresponding to the display content Cd is overlapped and displayed at the presence position Pm of the autonomous driving device Ma. This means that by overlapping and displaying the XR image 2a corresponding to the display content Cd that matches the service content Cs provided by the autonomous driving device Ma, the satisfaction of the user's preference can be improved.

[0130] According to the second embodiment, the XR image 2d is also overlapped and displayed at the presence position Po of other objects Oo that enter the visual field area Av together with the autonomous driving device Ma. Thereby, in a wide range of the visual field area Av, the XR images 2a and 2d corresponding to the display content Cd that matches the service content Cs provided by the autonomous driving device Ma are overlapped and displayed, and the satisfaction of the user's preference can be improved.

[0131] According to the second embodiment, the content information Ci is obtained based on the sensing information related to the face of the user Us wearing the wearable terminal Wt. Thus, the service content Cs and the display content Cd of the intention of the user Us as the content information Ci can be automatically recognized from the face wearing the wearable terminal Wt. Therefore, the convenience in terms of improving the satisfaction of the user's preference can also be improved.

[0132] According to the second embodiment, three-dimensional space information Ti is obtained from the infrastructure database Di. The three-dimensional space information Ti is information in which position-related information associated with the respective existence positions Pm and Po of the autonomous driving device Ma and other objects Oo is attached to a plurality of three-dimensional voxels Vi obtained by hypothetically dividing the service space As that provides user services. Therefore, on the respective existence positions Pm and Po of the autonomous driving device Ma and other objects Oo based on the obtained three-dimensional space information Ti, the XR images 2a and 2d that can satisfy the user's preference in the visual field area Av can be appropriately overlapped and displayed from the front position. Therefore, it is possible to suppress the user's discomfort caused by the positional deviation of the overlapping display and improve the satisfaction of the user's preference at the same time.

[0133] According to the second embodiment, a user service is provided by a driving control mode of the autonomous driving device Ma that matches the service content Cs in the content information Ci. Thus, by driving the autonomous driving device Ma in coordination with the user service, the satisfaction of the user's preference can be improved. At the same time, according to the second embodiment, the overlapping display position of the XR image 2a with respect to the visual field area Av is adjusted in accordance with the existence position Pm of the autonomous driving device Ma according to the driving control mode. As a result, the XR image 2a corresponding to the display content Cd that matches the service content Cs can be overlapped and displayed within the visual field area Av following the driving of the autonomous driving device Ma, and thus the satisfaction of the user's preference can be further improved. Here, particularly in the second embodiment, by using the three-dimensional space information Ti from the infrastructure database Di, the followability of the XR image 2a with respect to the driving of the autonomous driving device Ma can be improved, and thus the suppression effect of the user's discomfort caused by the positional deviation of the overlapping display can also be improved.

[0134] According to the second embodiment, the character images associated with the display content Cd in the content information Ci are overlapped and displayed as the XR images 2a and 2d at the existence positions Pm and Po of the autonomous driving device Ma and other objects Oo. Thus, the character images that can satisfy the user's preference in the visual field area Av can be overlapped with the front positions of the autonomous driving device Ma and other objects Oo and visually confirmed by the user Us. Therefore, in particular, the satisfaction of the user's preference for the display content Cd can be improved.

[0135] (Other embodiments)

[0136] As described above, a plurality of embodiments have been described, but the present disclosure is not construed as being limited to these embodiments, and can be applied to various embodiments and combinations without departing from the gist of the present disclosure.

[0137] In the modification examples of the first and second embodiments, the dedicated computers that constitute the control systems 20 and 2020 of the processing systems 1 and 2001 may also have at least one of a digital circuit and an analog circuit as a processor. Here, the digital circuit is, for example, at least one of an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), an SOC (System on a Chip), a PGA (Programmable Gate Array), a CPLD (Complex Programmable Logic Device), etc. In addition, such a digital circuit may also have a memory that stores a program.

[0138] In the modification examples of the first and second embodiments, the processing systems 1 and 2001 may also be non-connected to the infrastructure system Is. In this case, the sensing information of the connection elements Mt, Ma, and Wt of the processing systems 1 and 2001 may be used instead of the three-dimensional space information Ti obtained from the infrastructure database Di.

[0139] In the modification examples of the first and second embodiments, the processing systems 1 and 2001 may also obtain three-dimensional space information Ti that is not associated with the voxel Vi from the infrastructure database Di. In the modification examples of the first and second embodiments, the processing systems 1 and 2001 may also obtain two-dimensional grid information instead of the three-dimensional space information Ti from the infrastructure database Di.

[0140] In the modification example of the first embodiment, a smartwatch, a video see-through type or non-transmissive type wearable terminal may also be used as the mobile terminal Mt. In the modification example of the second embodiment, when the mobile terminal Mt is connected to the processing system 2001 and functions as a part of the wearable terminal Wt, a service request input may also be obtained as sensing information according to the sensor unit 3 that constitutes this part, and the processing flow may be executed. In the modification example of the second embodiment, when the mobile terminal Mt is connected to the processing system 2001 and functions as a part of the wearable terminal Wt, content information Ci that identifies the service content Cs and the display content Cd in S2200 of the processing flow may also be obtained according to the sensing information of the sensor unit 3 that constitutes this part.

[0141] In a modification of S230 of the first embodiment and S2230 of the second embodiment, the XR image 2d may not be overlapped and displayed at the position Po of the presence of another object Oo that enters the areas Ab and Av together with the autonomous driving device Ma. In a modification of S2200 of the second embodiment, according to S200 of the first embodiment, the sensor unit 2003 may sense the gesture input or voice input of the user Us. In a modification of S2200 of the second embodiment, according to S200 of the first embodiment, the gesture input, gaze input, expression input, or voice input of the user Us may be recognized based on the three-dimensional space information Ti obtained from the infrastructure database Di.

[0142] In addition to the above description methods, the above-described embodiments and modifications may also be implemented in the form of a processing system 1, 2001 that each has at least one processor 24 and one memory 22 in the control systems 20, 2020, in the form of a semiconductor device (such as a semiconductor chip, etc.).

[0143] (Postscript)

[0144] This specification discloses a plurality of technical ideas listed below and their multiple combinations.

[0145] (Technical idea 1)

[0146] A processing system having a processor (24) that performs service-related processing associated with a user service provided to a user,

[0147] The processor is configured to perform the following:

[0148] Based on the sensing information of the mobile terminal (Mt) carried by the user, obtain the content information (Ci) of the service content (Cs) and display content (Cd) that identify the user's intention;

[0149] By the user facing the mobile terminal, monitor the background area (Ab) as the display background;

[0150] In the current background area, provide a user service of the category corresponding to the service content in the content information through the driving control of the autonomous driving device (Ma); and

[0151] In the mobile terminal that displays the background video (2c) of the background area entered by the autonomous driving device that provides the user service, overlap and display the XR image (2a) corresponding to the display content in the content information at the presence position (Pm) of the autonomous driving device.

[0152] (Technical idea 2)

[0153] In the processing system described in Technical idea 1,

[0154] The overlapping display of the XR image includes the following:

[0155] The XR image (2d) is overlapped and displayed at the position of existence (Po) of other objects (Oo) in the background area that enters together with the autonomous driving device.

[0156] (Technical idea 3)

[0157] A processing system having a processor (24) that executes service-related processing associated with user services provided to a user.

[0158] The processor is configured to execute the following:

[0159] Based on the sensing information of the wearable terminal (Wt) worn by the user, content information (Ci) of the service content (Cs) and the display content (Cd) that identifies the user's intention is obtained;

[0160] The visual field area (Av) visually confirmed by the user through the wearable terminal is monitored;

[0161] In the current visual field area, a user service of the category corresponding to the service content in the content information is provided through the drive control of the autonomous driving device (Ma); and

[0162] In the wearable terminal where the autonomous driving device that provides the user service enters the visual field area, the XR image (2a) corresponding to the display content in the content information is overlapped and displayed at the position of existence (Pm) of the autonomous driving device.

[0163] (Technical idea 4)

[0164] In the processing system described in Technical idea 3,

[0165] The overlapping display of the XR image includes the following:

[0166] The XR image (2d) is overlapped and displayed at the position of existence of other objects (Oo) that enter the visual field area together with the autonomous driving device.

[0167] (Technical idea 5)

[0168] In the processing system described in Technical idea 3 or 4,

[0169] Detection is performed, including:

[0170] Based on the sensing information related to the face of the user wearing the wearable terminal, content information is obtained.

[0171] (Technical idea 6)

[0172] In the processing system described in any one of Technical ideas 1 to 5,

[0173] The processor is configured to further execute the following:

[0174] Obtain three-dimensional space information (Ti) from the infrastructure database (Di), where the three-dimensional space information (Ti) is information in which position association information associated with the existence position is attached to a plurality of three-dimensional voxels (Vi) obtained by hypothetically dividing the service space (As) that provides user services.

[0175] The overlapping display of the XR image includes the following:

[0176] Overlap and display the XR image at the existence position based on the three-dimensional space information.

[0177] (Technical idea 7)

[0178] In the processing system according to any one of Technical ideas 1 to 5,

[0179] Providing user services includes the following:

[0180] Provide user services according to the drive control mode of the autonomous driving device that matches the service content in the content information.

[0181] (Technical idea 8)

[0182] In the processing system according to Technical idea 7,

[0183] The overlapping display of the XR image includes the following:

[0184] Adjust the overlapping display position of the XR image in accordance with the existence position according to the drive control mode.

[0185] (Technical idea 9)

[0186] In the processing system according to Technical idea 6,

[0187] Providing user services includes the following:

[0188] Provide user services according to the drive control mode of the autonomous driving device that matches the service content in the content information,

[0189] The overlapping display of the XR image includes the following:

[0190] Adjust the overlapping display position of the XR image in accordance with the existence position according to the drive control mode.

[0191] (Technical idea 10)

[0192] In the processing system according to any one of Technical ideas 1 to 9,

[0193] The overlapping display of the XR image includes the following:

[0194] The character image associated with the display content in the content information is overlapped and displayed as an XR image at the existing position. In addition, the above technical ideas 1 to 10 can also be grasped by the respective technical ideas of the method and the program.

Claims

1. A processing system having a processor (24) that executes service - related processing associated with a user service provided to a user, characterized in that, the processor is configured to execute the following: Based on sensing information of a mobile terminal (Mt) carried by the user, obtain content information (Ci) that identifies service content (Cs) and display content (Cd) that reflects the user's intention; By the user facing the mobile terminal, monitor a background area (Ab) that serves as a display background; In the current background area, provide the user service of the category corresponding to the service content in the content information through drive control of an autonomous driving device (Ma); and In the mobile terminal that displays a background video (2c) of the autonomous driving device entering the background area while providing the user service, superimpose and display an XR image (2a) corresponding to the display content in the content information at the presence position (Pm) of the autonomous driving device.

2. The processing system according to claim 1, characterized in that, Superimposing and displaying the XR image includes the following: Superimpose and display the XR image (2d) at the presence position (Po) of other objects (Oo) that enter the background area together with the autonomous driving device.

3. A processing system having a processor (24) that executes service - related processing associated with a user service provided to a user, characterized in that, the processor is configured to execute the following: Based on sensing information of a wearable terminal (Wt) worn by the user, obtain content information (Ci) that identifies service content (Cs) and display content (Cd) that reflects the user's intention; Monitor the visual field area (Av) visually confirmed by the user through the wearable terminal; In the current visual field area, provide the user service of the category corresponding to the service content in the content information through drive control of an autonomous driving device (Ma); and In the wearable terminal where the autonomous driving device providing the user service enters the visual field area, superimpose and display an XR image (2a) corresponding to the display content in the content information at the presence position (Pm) of the autonomous driving device.

4. The processing system according to claim 3, characterized in that, Superimposing and displaying the XR image includes the following: Superimpose and display the XR image (2d) at the presence position (Po) of other objects (Oo) that enter the visual field area together with the autonomous driving device.

5. The processing system according to claim 3, characterized in that, Obtaining the content information includes the following: Obtain the content information based on the sensing information related to the face of the user wearing the wearable terminal.

6. The processing system according to any one of claims 1 to 5, characterized in that, the processor is configured to further execute the following: Obtain three-dimensional space information (Ti) from the infrastructure database (Di), where the three-dimensional space information (Ti) is information in which position association information associated with the presence position is attached to a plurality of three-dimensional voxels (Vi) obtained by hypothetically dividing the service space (As) that provides the user service. The overlapping display of the XR image includes the following: Overlap and display the XR image at the presence position based on the three-dimensional space information.

7. The processing system according to any one of claims 1 to 5, characterized in that Providing the user service includes the following: Provide the user service according to the driving control mode of the autonomous driving device that matches the service content in the content information.

8. The processing system according to claim 7, characterized in that The overlapping display of the XR image includes the following: Adjust the overlapping display position of the XR image to match the presence position according to the driving control mode.

9. The processing system according to claim 6, characterized in that Providing the user service includes the following: Provide the user service according to the driving control mode of the autonomous driving device that matches the service content in the content information. The overlapping display of the XR image includes the following: Adjust the overlapping display position of the XR image to match the presence position according to the driving control mode.

10. The processing system according to any one of claims 1 to 5, characterized in that The overlapping display of the XR image includes the following: Overlap and display the role image associated with the display content in the content information as the XR image at the presence position.

11. A processing method, executed by a processor (24), to perform service-related processing associated with a user service provided to a user, characterized in that, Includes the following: Obtain content information (Ci) that identifies the service content (Cs) and display content (Cd) of the user's intention based on the sensing information of the mobile terminal (Mt) carried by the user; Monitor the background area (Ab) that serves as the display background by the user facing the mobile terminal; In the current background area, provide the user service of the category corresponding to the service content in the content information through the driving control of the autonomous driving device (Ma); And In the mobile terminal that displays the background video (2c) of the autonomous driving device providing the user service entering the background area, overlap and display the XR image (2a) corresponding to the display content in the content information at the presence position (Pm) of the autonomous driving device.

12. A processing method, executed by a processor (24), to perform service - associated processing related to a user service provided to a user, characterized in that, Includes the following: Obtain content information (Ci) that identifies the service content (Cs) and display content (Cd) of the user's intention based on the sensing information of the wearable terminal (Wt) worn by the user; Monitor the visual field area (Av) visually confirmed by the user through the wearable terminal; In the current visual field area, provide the user service of the category corresponding to the service content in the content information through the driving control of the autonomous driving device (Ma); And In the wearable terminal in which the autonomous driving device providing the user service enters the visual field area, an XR image (2a) corresponding to the display content in the content information is superimposed and displayed at the position (Pm) where the autonomous driving device exists.

13. A processing program, comprising commands stored in a storage medium (22) and executed by a processor (24) to execute a user-related service associated with a user service provided to a user, characterized in that the commands include the following: Based on the sensing information of the mobile terminal (Mt) carried by the user, content information (Ci) that identifies the service content (Cs) and the display content (Cd) of the user's intention is obtained; By the user facing the mobile terminal, the background area (Ab) as the display background is monitored; In the current background area, the user service of the category corresponding to the service content in the content information is provided by driving control of the autonomous driving device (Ma); and In the mobile terminal that displays the background video (2c) of the background area where the autonomous driving device providing the user service enters, an XR image (2a) corresponding to the display content in the content information is superimposed and displayed at the position (Pm) where the autonomous driving device exists.

14. A processing program, comprising commands stored in a storage medium (22) and executed by a processor (24) to execute a user-related service associated with a user service provided to a user, characterized in that the commands include the following: Based on the sensing information of the wearable terminal (Wt) worn by the user, content information (Ci) that identifies the service content (Cs) and the display content (Cd) of the user's intention is obtained; The visual field area (Av) visually confirmed by the user through the wearable terminal is monitored; In the current visual field area, the user service of the category corresponding to the service content in the content information is provided by driving control of the autonomous driving device (Ma); and In the wearable terminal in which the autonomous driving device providing the user service enters the visual field area, an XR image (2a) corresponding to the display content in the content information is superimposed and displayed at the position (Pm) where the autonomous driving device exists.

Images