Processing system, processing method, processing program

Abstract

To provide a user service that satisfies the preference of a user by cooperation with an autonomous traveling device.SOLUTION: A processor in a processing system for performing service-related processing is configured to: obtain, based on sensing information from a mobile terminal Mt carried by a user, content information Ci formed by recognizing service content Cs and display content Cd intended by the user; monitor a background area Ab which serves as a display background toward which the user directs the mobile terminal Mt; provide user services of a type corresponding to the service content Cs by means of drive control of an autonomous traveling device Ma in the current background area Ab; and superimpose an XR image corresponding to the display content Cd on an existing location Pm of the autonomous traveling device Ma on the mobile terminal Mt which displays a background video of the background area Ab that the autonomous traveling device Ma providing the user services enters.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a processing technology for performing service-related processing related to user services provided to users.

Background Art

[0002] The technology disclosed in Patent Document 1 displays a content image in which a virtual object as display content is sized and superimposed on a frame image such as a video on a display device such as a monitor, a touch panel, or an HMD (Head Mounted Display).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The technology disclosed in the above Patent Document 1 remains at a technology that employs an avatar for identifying a user as a virtual object in a video distribution service such as a live content. In contrast, in recent years, service content that is the content of user services has diversified not only in such distribution services, but also, for example, in smart cities planned to be built in various countries around the world, service content that cooperates with autonomous driving devices is expected. Along with that, as display content, it is expected to satisfy user preferences with an image that matches not only the avatar for user identification but also diversified service content.

[0005] An object of the present disclosure is to enable the provision of a user service that satisfies user preferences through cooperation with an autonomous driving device.

Means for Solving the Problems

[0006] Hereinafter, the technical means of the present disclosure for solving the problems will be described. Note that the reference numerals in parentheses described in the claims and this column indicate the correspondence with the specific means described in the embodiments to be described in detail later, and do not limit the technical scope of the present disclosure.

[0007] The first aspect of the present disclosure is a processing system having a processor (24) and performing service-related processing related to user services provided to a user, wherein the processor acquires content information (Ci) recognizing service content (Cs) and display content (Cd) intended by the user based on sensing information of a mobile terminal (Mt) carried by the user, monitors a background area (Ab) that becomes a display background when the user turns the mobile terminal, provides a user service of a type corresponding to the service content among the content information by driving control of an autonomous driving device (Ma) in the current background area, and is configured to execute superimposing and displaying an XR image (2a) corresponding to the display content among the content information on the presence position (Pm) of the autonomous driving device in a mobile terminal that displays a background video (2c) of the background area where the autonomous driving device providing the user service enters.

[0008] The second aspect of the present disclosure is a processing method executed by a processor (24) to perform service-related processing related to user services provided to a user, acquiring content information (Ci) recognizing service content (Cs) and display content (Cd) intended by the user based on sensing information of a mobile terminal (Mt) carried by the user, monitoring a background area (Ab) that becomes a display background when the user turns the mobile terminal, Among the content information, providing a user service of a type corresponding to service content in the current background area by driving control of the autonomous driving device (Ma); In a mobile terminal that displays a background video (2c) of a background area where an autonomous driving device that provides a user service enters, superimposing and displaying an XR image (2a) corresponding to display content among the content information at the position (Pm) where the autonomous driving device exists.

[0009] The third aspect of the present disclosure is A processing program including instructions stored in a storage medium (22) and executed by a processor (24) to perform service-related processing related to user services provided to users, The instructions are Obtaining content information (Ci) that recognizes service content (Cs) and display content (Cd) intended by the user based on sensing information of a mobile terminal (Mt) carried by the user; Monitoring a background area (Ab) that becomes a display background when the user turns the mobile terminal; Among the content information, providing a user service of a type corresponding to service content in the current background area by driving control of the autonomous driving device (Ma); In a mobile terminal that displays a background video (2c) of a background area where an autonomous driving device that provides a user service enters, causing an XR image (2a) corresponding to display content among the content information to be superimposed and displayed at the position (Pm) where the autonomous driving device exists.

[0010] According to these first to third aspects, content information recognizing the service content intended by the user is acquired based on the sensing information of the mobile terminal carried by the user. At the same time, the background area that becomes the display background when the user turns the mobile terminal is monitored. Therefore, a user service of a type corresponding to the service content is provided by driving control of the autonomous driving device in the current background area. This means that among various user services, it is possible to provide a user service that satisfies user preferences through cooperation with the autonomous driving device.

[0011] Moreover, according to the first to third aspects, content information recognizing the display content intended by the user is also acquired based on the sensing information of the mobile terminal. Therefore, in a mobile terminal that displays the background video of the background area where the autonomous driving device that provides the user service as described above enters, an XR image corresponding to the display content is superimposed and displayed at the position where the autonomous driving device exists. This means that it is possible to enhance the satisfaction of user preferences by superimposing and displaying an XR image corresponding to the display content that matches the service content provided by the autonomous driving device.

[0012] The fourth aspect of the present disclosure is a processing system having a processor (24) and performing service-related processing related to user services provided to the user, wherein the processor acquires content information (Ci) recognizing the service content (Cs) and display content (Cd) intended by the user based on the sensing information of the wearable terminal (Wt) worn by the user, monitors the visual field area (Av) visually recognized by the user through the wearable terminal, provides a user service of a type corresponding to the service content among the content information by driving control of the autonomous driving device (Ma) in the current visual field area, In a wearable terminal in which an autonomous driving device that provides a user service enters a visual field area, an XR image (2a) corresponding to display content among content information is superimposed and displayed on the presence position (Pm) of the autonomous driving device.

[0013] The fifth aspect of the present disclosure is A processing method executed by a processor (24) for performing service-related processing related to a user service provided to a user, acquiring content information (Ci) that recognizes service content (Cs) and display content (Cd) intended by the user based on sensing information of a wearable terminal (Wt) worn by the user; monitoring a visual field area (Av) visually recognized by the user through the wearable terminal; providing a user service of a type corresponding to the service content among the content information by driving control of an autonomous driving device (Ma) in the current visual field area; In a wearable terminal in which an autonomous driving device that provides a user service enters a visual field area, an XR image (2a) corresponding to display content among the content information is superimposed and displayed on the presence position (Pm) of the autonomous driving device.

[0014] The sixth aspect of the present disclosure is A processing program stored in a storage medium (22) and including instructions executed by a processor (24) for performing service-related processing related to a user service provided to a user, The instructions are to cause the content information (Ci) that recognizes the service content (Cs) and the display content (Cd) intended by the user to be acquired based on the sensing information of the wearable terminal (Wt) worn by the user; to cause the visual field area (Av) visually recognized by the user through the wearable terminal to be monitored; Among the content information, a user service of a type corresponding to service content is provided by driving control of the autonomous driving device (Ma) in the current visual field area. In a wearable terminal where an autonomous driving device that provides a user service enters the visual field area, an XR image (2a) corresponding to display content among the content information is superimposed and displayed at the existence position (Pm) of the autonomous driving device. This includes.

[0015] According to these fourth to sixth aspects, content information that recognizes the service content intended by the user is acquired based on the sensing information of the wearable terminal worn by the user. At the same time, the visual field area visually recognized by the user through the wearable terminal is monitored. Therefore, a user service of a type corresponding to the service content is provided by driving control of the autonomous driving device in the current visual field area. This means that among various user services, it is possible to provide a user service that satisfies user preference through cooperation with the autonomous driving device.

[0016] Moreover, according to the fourth to sixth aspects, content information that recognizes the display content intended by the user is also acquired based on the sensing information of the mobile terminal. Therefore, in a wearable terminal where an autonomous driving device that provides the user service as described above enters the visual field area, an XR image corresponding to the display content is superimposed and displayed at the existence position of the autonomous driving device. This means that it is possible to enhance the satisfaction of user preference by superimposing and displaying an XR image corresponding to the display content that matches the service content provided by the autonomous driving device.

Brief Explanation of Drawings

[0017]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Modes for Carrying Out the Invention

[0018] Hereinafter, a plurality of embodiments of the present disclosure will be described with reference to the drawings. In each embodiment, corresponding components may be denoted by the same reference numerals, and redundant descriptions may be omitted. Further, when only a part of the configuration is described in each embodiment, the configuration of other embodiments described previously can be applied to other parts of the said configuration. Furthermore, not only the combinations of configurations explicitly described in the description of each embodiment, but also the configurations of a plurality of embodiments can be partially combined with each other as long as there is no problem with the combination, even if not explicitly stated.

[0019] (First Embodiment) As shown in FIG. 1, the processing system 1 of the first embodiment performs service-related processing related to user services provided to the user Us. For this purpose, the processing system 1 is connected via a communication network Nc to a mobile terminal Mt carried by a pre-registered user Us, an autonomous driving device Ma that autonomously drives, and an infrastructure system Is that provides infrastructure information. At least one each of the connection elements Mt, Ma, and Is to the processing system 1 is assumed.

[0020] As shown in FIGS. 1 and 2, the mobile terminal Mt is configured to be held and operable by the user Us of the system 1 with a finger, at least when connecting to the processing system 1. The mobile terminal Mt is a small electronic device including a display unit 2 and a sensor unit 3, and is, for example, a smartphone or a tablet terminal. The mobile terminal Mt controls the transmission and reception of data via the communication network Nc by a communication unit. The mobile terminal Mt is operationally controlled by a control unit according to a control command from the system 1 via the communication network Nc so as to perform service-related processing in cooperation with the processing system 1.

[0021] Under the operation control by the control unit of the mobile terminal Mt, on a display unit 2 such as a liquid crystal panel or an organic EL panel, for example, a display necessary for service-related processing is realized as shown in FIG. 2. At the same time, under the operation control of the mobile terminal Mt, the sensor unit 3 acquires sensing information by sensing processing according to service-related processing. As the sensor unit 3, at least one type is adopted from, for example, a camera, an inertial sensor, a GNSS (Global Navigation Satellite System) sensor, a touch sensor, and a microphone. With such a configuration, the sensor unit 3 can acquire an input from the same user Us representing the intention of the user Us as sensing information.

[0022] The autonomous driving device Ma shown in FIG. 1 is an autonomous driving vehicle or an autonomous driving robot that can autonomously drive in any direction of front, rear, left, and right by electrically driving the wheels 5 based on sensing information by the sensor unit 4. The autonomous driving device Ma controls the transmission and reception of data via the communication network Nc by the communication unit. The autonomous driving device Ma is driven and controlled by the control unit according to a control command from the system 1 via the communication network Nc so as to perform service-related processing in cooperation with the processing system 1. Under this driving control, the sensor unit 4 acquires sensing information by sensing processing according to service-related processing. As the sensor unit 4, at least one type is adopted from, for example, a camera, an inertial sensor, a GNSS sensor, a LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging), and a sonar.

[0023] The infrastructure system Is is a basic system that shares the three-dimensional space information Ti stored in the infrastructure database Di as a common foundation for a distributed architecture. The infrastructure system Is controls the transmission and reception of data via the communication network Nc by means of a communication unit. The infrastructure system Is collects at any time the three-dimensional space information Ti provided to individual distributed systems such as the processing system 1 that constitutes the distributed architecture, and updates the stored information in the infrastructure database Di to the latest information by means of a control unit.

[0024] Here, in particular, the infrastructure system Is virtually divides the three-dimensional space to be the target of information storage into a plurality of three-dimensional voxels Vi (i.e., a three-dimensional grid) assumed to be in a three-dimensional array as shown in FIG. 3, and manages the information. The infrastructure system Is stores, as the three-dimensional space information Ti, a data set associated by metadata for each space ID individually assigned to each voxel Vi in the infrastructure database Di as shown in FIG. 1. Here, the three-dimensional space information Ti may of course include two-dimensional grid information associated only with, for example, the lower surface of the voxel Vi that constitutes the lowermost layer of the two-dimensional array along the ground in the three-dimensional array.

[0025] The data for constructing the three-dimensional space information Ti may be collected from at least one of the mobile terminal Mt that cooperates with the processing system 1, other mobile terminals, and wearable terminals such as HMDs or smart glasses. The data for constructing the three-dimensional space information Ti may be collected from at least one of the autonomous driving device Ma that cooperates with the processing system 1 and other moving bodies. The data for constructing the three-dimensional space information Ti may be collected from at least one of, for example, communication base stations, smart poles, and smart street lights equipped with infrastructure sensors such as cameras and / or LiDAR. The data for constructing the three-dimensional space information Ti may be collected from a service provider that provides at least one of, for example, map services, weather services, communication services, traffic management services, ground object management services, and aviation management services, and the user services described later handled by the processing system 1.

[0026] The construction data collected in this way to construct the three-dimensional space information Ti may be at least one type of image data among, for example, videos, still images, point cloud images, etc. The construction data of the three-dimensional space information Ti is generated by image processing of such image data for a human including the user Us of the processing system 1 in the target space for information accumulation, such as position data, motion data, posture data, gaze data, behavior data, and crowd flow data, and intention data representing the intention of the user Us described later, etc., and may be at least one type of secondary data with information security guaranteed. The construction data of the three-dimensional space information Ti may be at least one type of secondary data generated by image processing of image data for human carried items including the mobile terminal Mt of the user Us in the information accumulation target space, such as position data, motion data, and posture data, etc. The construction data of the three-dimensional space information Ti may be at least one type of secondary data generated by image processing of image data for a moving body including the autonomous driving device Ma in the information accumulation target space, such as position data, motion data, and posture data, etc.

[0027] The construction data of the three-dimensional space information Ti may be voice data collected from the voices of humans including the user Us of the processing system 1 within the information storage target space. The construction data of the three-dimensional space information Ti may be at least one type of secondary data with guaranteed information security among, for example, position data, motion data, behavior data, pedestrian flow data, conversation data, and intention data representing the intention of the user Us described later, which are analyzed regarding humans including the user Us by voice recognition processing of such voice data. The construction data of the three-dimensional space information Ti may be at least one type among, for example, two-dimensional and / or three-dimensional map data, GIS (Geographic Information System) data, road network data, weather data, communication data, line data, traffic data, ground object management data, BIM (Building Information Modelling) data, POI (Point of Interest) data, aviation management data, and time data. The construction data of the three-dimensional space information Ti may be service data related to at least one type among, for example, a photography service, a trash can guidance service, a delivery service, and a guidance service, etc., which will be described later as user services handled by the processing system 1.

[0028] As shown in FIG. 1, the processing system 1 is a distributed computer system constructed including at least one type among, for example, a cloud server and an edge server, etc., equipped with a communication system 10 and a control system 20. At least a part of each of the systems 10, 20 in the processing system 1 may be respectively constituted by a communication unit and a control unit of the mobile terminal Mt. At least a part of each of the systems 10, 20 in the processing system 1 may be respectively constituted by a communication unit and a control unit of the autonomous driving device Ma. At least a part of each of the systems 10, 20 in the processing system 1 may be respectively constituted by a communication unit and a control unit of the infrastructure system Is.

[0029] The communication system 10 is mainly composed of communication devices for constructing a communication network Nc. The control system 20 is connected to the communication system 10 via at least one of a wired communication line and a wireless communication line. The control system 20 is constructed to include at least one dedicated computer. The dedicated computer constructing the control system 20 has at least one memory 22 and at least one processor 24.

[0030] In such a processing system 1, the control system 20 executes a plurality of instructions of a processing program stored in the memory 22 by the processor 24. As a result, a plurality of functional blocks for performing service-related processing related to user services provided to the user Us are constructed. The constructed functional blocks include a recognition block 200, a monitoring block 210, a drive control block 220, and a display control block 230 as shown in FIG. 4.

[0031] The processing method by which the processing system 1 performs service-related processing through the cooperation of these blocks 200, 210, 220, 230 is executed according to the processing flow shown in FIG. 5. This processing flow is executed in response to the service request input, which is the intention of the user Us to request user service provision, being acquired by the sensor unit 3 of the mobile terminal Mt as sensing information. Note that each "S" in this processing flow means a plurality of steps executed by a plurality of instructions included in the processing program of the first embodiment.

[0032] In S200 of the processing flow, the recognition block 200 (see FIG. 4) acquires content information Ci that recognizes service content Cs and display content Cd intended by the user Us, based on the sensing information of the mobile terminal Mt carried by the same user Us. The service content Cs recognized in this S200 means the type of user service that can be provided to the user Us of the processing system 1. Here, as the types of user services corresponding to the service content Cs, in the first embodiment, at least two types among a shooting service, a trash can guidance service, a delivery service, and a guidance service are prepared.

[0033] The display content Cd recognized in S200 means the type of the XR image 2a among the XR images 2a, 2d, 2e to be displayed on the display unit 2 of the mobile terminal Mt as shown in FIGS. 2 and 6 along with the provision of the user service corresponding to the service content Cs. Here, the XR image 2a in which the type intended by the user Us is recognized as the display content Cd is prepared with a plurality of types of character images representing virtual or real organisms such as monsters, animals, plants, and insects in two dimensions or three dimensions. On the other hand, the XR images 2d, 2e given a common theme corresponding to the type of the XR image 2a are prepared with a plurality of types of character images representing virtual or real organisms such as monsters, animals, plants, and insects, as well as inanimate objects such as prehistoric buildings, service vehicles, and imaginary objects in two dimensions or three dimensions.

[0034] In S200 of FIG. 5, the sensing information for recognizing the service content Cs and the display content Cd is input to the sensor unit 3 as the intention of the user Us, and is acquired through the communication unit of the mobile terminal Mt. At this time, on the display unit 2 of the mobile terminal Mt, an input reception image 2b for receiving a touch input by the sensor unit 3 may be displayed as a selection input from the user Us for the service content Cs and the display content Cd, as shown in FIG. 7. As the selection input from the user Us for the service content Cs and the display content Cd, a gesture input, a gaze input, an expression input, or a voice input by the user Us may be sensed by the sensor unit 3. Such gesture input, gaze input, expression input, or voice input may be recognized based on the three-dimensional space information Ti acquired from the infrastructure database Di.

[0035] In S210 of the processing flow shown in FIG. 5, the monitoring block 210 (see FIG. 4) monitors a background area Ab that becomes the display background of the XR image 2a on the display unit 2 as shown in FIG. 8 when the user Us faces the mobile terminal Mt. At this time, the background area Ab is monitored in the service space As where user services of a type corresponding to the service content Cs are provided as shown in FIG. 8, among the information accumulation target spaces where a plurality of voxels Vi are assumed by the infrastructure system Is as shown in FIG. 3. Therefore, among the sensing information by the sensor unit 3 of the mobile terminal Mt, monitoring of the background area Ab that spreads in the direction in which the mobile terminal Mt faces is executed based on at least one of, for example, camera information and inertial information.

[0036] In S220 of the processing flow shown in FIG. 5, the drive control block 220 (see FIG. 4) provides a user service of a type corresponding to the service content Cs among the content information Ci by driving control of the autonomous driving device Ma in the current background area Ab. At this time, the user service is provided according to the drive control pattern by reading out the drive control pattern of the autonomous driving device Ma adapted to the service content Cs from the memory 22.

[0037] Such a shooting service by S220 is provided by a driving control pattern in which the autonomous driving device Ma having the imaging unit is driven from the standby position to the current position of the user Us or a landmark around the user Us, the user Us is photographed from the surroundings, and then the device returns to the standby position. At this time, in the shooting service, the shooting timing or the shooting period may be presented to the user Us by voice output from the voice unit of the autonomous driving device Ma. In the shooting service, the shooting timing or the shooting period may be presented to the user Us by display output from the display unit of the autonomous driving device Ma.

[0038] In the shooting service, a two-dimensional or three-dimensional still image or moving image may be photographed so as to include the user Us by the autonomous driving device Ma driven around the user Us as exemplified by the white arrow in FIG. 9. In the shooting service, the focus at the time of shooting may be adjusted based on the distance to the user Us sensed by the sensor unit 4 such as LiDAR in the autonomous driving device Ma. In the shooting service, the incident azimuth of external light at the time of shooting may be adjusted based on the three-dimensional space information Ti acquired from the infrastructure database Di or the sensing information from the sensor unit 4 such as a camera in the autonomous driving device Ma.

[0039] The trash can guiding service by S220 is provided by a driving control pattern in which the autonomous driving device Ma having the trash can is driven from the standby position to the current position of the user Us, accepts the disposal of the trash intended by the user Us into the trash can, and then returns to the standby position. At this time, in the trash can guiding service, the disposal of trash into the trash can may be instructed to the user Us by voice output from the voice unit of the autonomous driving device Ma. In the trash can guiding service, the disposal of trash into the trash can may be instructed to the user Us by display output from the display unit of the autonomous driving device Ma.

[0040] The delivery service by S220 is provided by a drive control pattern in which the autonomous driving device Ma having a delivery box is driven from a standby position to the current position of the user Us, accepts the loading of the luggage intended by the user Us into the delivery box, and then returns to the standby position. At this time, in the delivery service, the loading of luggage into the delivery box may be instructed to the user Us by voice output from the voice unit of the autonomous driving device Ma. In the delivery service, the loading of luggage into the delivery box may be instructed to the user Us by display output from the display unit of the autonomous driving device Ma.

[0041] The guidance service by S220 is provided by a drive control pattern in which the autonomous driving device Ma is driven from a standby position to the current position of the person to be guided, such as the user Us or a registered family member, performs route guidance to guide the person to be guided, and then returns to the standby position. At this time, in the guidance service, the route for guiding the person to be guided may be instructed by voice output from the voice unit of the autonomous driving device Ma. In the guidance service, the route for guiding the person to be guided may be instructed by display output from the display unit of the autonomous driving device Ma. Further, prior to such route instruction, on the mobile terminal Mt, an input reception image for receiving an input from the user Us of route-related information such as, for example, the destination and / or the transportation means desired to be used may be displayed on the display unit 2 by the sensor unit 3. Also, along with the route instruction, the autonomous driving device Ma may be given a drive control pattern for accommodating and transporting the carry-on luggage of the person to be guided in the delivery box it holds.

[0042] S230 in the processing flow shown in FIG. 5 is executed in parallel with S220. In S230, the display control block 230 (see FIG. 4) causes the background video 2c of the background area Ab where the autonomous driving device Ma that provides user services by S220 enters as shown in FIGS. 8 and 9 to be displayed on the display unit 2 of the mobile terminal Mt as shown in FIGS. 2 and 6. Here, the background video 2c is acquired based on, for example, sensing information by the sensor unit 4 such as a camera or three-dimensional space information Ti acquired from the infrastructure database Di. At this time, the primary video data obtained by photographing the background area Ab may be used for the background video 2c, or secondary video data obtained by processing a part such as the ground or the sky in the primary data may be used.

[0043] Therefore, the display control block 230 in S230 of FIG. 5 superimposes and displays the XR image 2a corresponding to the display content Cd among the content information Ci at the position Pm of the autonomous driving device Ma on the background video 2c displayed on the display unit 2 as shown in FIGS. 2 and 6. At this time, the superimposed display position of the XR image 2a is adjusted according to the position Pm of the autonomous driving device Ma on the background video 2c according to the driving control pattern of the autonomous driving device Ma by S220. As a result, the XR image 2a is replaced and displayed at the position Pm of the autonomous driving device Ma on the background video 2c.

[0044] The display control block 230 in S230 also superimposes and displays the XR image 2d as shown in FIG. 2 and FIG. 6 at the position Po of another object Oo that enters the background area Ab as shown in FIG. 8 together with the autonomous driving device Ma. At this time, the superimposed display position of the XR image 2d is adjusted according to the position Po of another object Oo. Another object Oo in which the XR image 2d is replaced and displayed at the position Po on the background video 2c is at least one of, for example, a person other than the user Us of the processing system 1, a moving body other than the autonomous driving device Ma cooperating with the processing system 1, and a ground feature.

[0045] In S230, the display control block 230 acquires, from the infrastructure database Di, three-dimensional space information Ti in which position-related information related to the respective existence positions Pm and Po of the autonomous driving device Ma and other objects Oo is added to a plurality of voxels Vi obtained by virtually dividing the service space As as shown in FIG. 3. Therefore, the display control block 230 in S230 superimposes and displays XR images 2a and 2d on the respective existence positions Pm and Po based on the acquired three-dimensional space information Ti. As a result, particularly in the first embodiment, the character images related to the display content Cd among the content information Ci are superimposed and displayed at the respective existence positions Pm and Po as the XR images 2a and 2d.

[0046] The display control block 230 in S230 may superimpose and display the XR image 2e as shown in FIG. 2 on a scenery other than the respective existence positions Pm and Po of the autonomous driving device Ma and other objects Oo in the background area Ab. In this case, in the first embodiment, the character image related to the display content Cd may also be superimposed and displayed as the XR image 2e. At the same time, the superimposition display position of the XR image 2e may be adjusted based on the three-dimensional space information Ti acquired from the infrastructure database Di so as to be outside the respective superimposition positions of the XR images 2a and 2d.

[0047] During the shooting service by such S230, at least the background video 2c with the XR images 2a and 2d superimposed on the existence position Pm of the autonomous driving device Ma driven to shoot the user Us and the existence position Po of the surrounding objects Oo is displayed and output to the user Us. At this time, a still image or a moving image as the shooting image 2f of the user Us and the background area shown in FIG. 9 is switched and displayed from the background video 2c shown in FIG. 2 as shown in FIG. 10, so that the XR image 2a may be superimposed and displayed at the existence position Ps in the shooting image 2f of at least the former among the user Us and the companion. In this way, the image data with the XR image 2a superimposed on the shooting image 2f of the user Us and the background area may be stored in a storage medium inside or outside the mobile terminal Mt.

[0048] During the garbage bin guidance service by S230, a background video 2c with at least XR images 2a and 2d superimposed on the presence position Pm of the autonomous driving device Ma driven to receive garbage disposal from the user Us and the presence position Po of the surrounding objects Oo is displayed and output to the user Us. At this time, the XR image 2a corresponding to the autonomous driving device Ma may be a character image imitating the garbage disposal opening of the garbage bin as the mouth of a living being. The XR image 2a corresponding to the autonomous driving device Ma may include an instruction image indicating the disposal opening corresponding to the type of garbage to be disposed of among the plurality of disposal openings of the garbage bin.

[0049] During the delivery service by S230, a background video 2c with at least XR images 2a and 2d superimposed on the presence position Pm of the autonomous driving device Ma driven to receive the loading of luggage from the user Us and the presence position Po of the surrounding objects Oo is displayed and output to the user Us. At this time, the XR image 2a corresponding to the autonomous driving device Ma may be a character image imitating the loading opening of the delivery box as the mouth of a living being.

[0050] During the guidance service by S230, a background video 2c with at least XR images 2a and 2d superimposed on the presence position Pm of the autonomous driving device Ma driven to guide the person to be guided and the presence position Po of the surrounding objects Oo is displayed and output to the person to be guided. At this time, a still image or a moving image as a photographed image of the background area of the guiding route may be switched and displayed from the background video 2c. In this way, the image data of the photographed image of the background area of the guiding route may be stored in the internal or external storage medium of the mobile terminal Mt.

[0051] The processing flow of the first embodiment shown in FIG. 5 above is completed when the provision of the user service according to the service content Cs among the content information Ci ends by S220 and S230. After that, when the next service request input is acquired through the sensor unit 3 of the mobile terminal Mt, the next execution of the processing flow is started.

[0052] (Function and effect) The effects of the first embodiment described so far will be described below.

[0053] According to the first embodiment, content information Ci that recognizes service content Cs intended by user Us is acquired based on the sensing information of mobile terminal Mt carried by user Us. At the same time, background area Ab that becomes the display background is monitored by user Us turning the mobile terminal Mt towards it. Therefore, user services of a type corresponding to service content Cs are provided by driving control of autonomous driving device Ma in the current background area Ab. This means that among diversified user services, it is possible to provide user services that satisfy user preferences through cooperation with autonomous driving device Ma.

[0054] Moreover, according to the first embodiment, content information Ci that recognizes display content Cd intended by user Us is also acquired based on the sensing information of mobile terminal Mt. Therefore, in mobile terminal Mt that displays background video 2c of background area Ab where autonomous driving device Ma that provides the above-mentioned user service enters, XR image 2a corresponding to display content Cd is superimposed and displayed at the existence position Pm of autonomous driving device Ma. This means that it is possible to enhance the satisfaction of user preferences by superimposing and displaying XR image 2a corresponding to display content Cd that matches service content Cs provided by autonomous driving device Ma.

[0055] According to the first embodiment, XR image 2d is also superimposed and displayed at the existence position Po of other object Oo that enters background area Ab together with autonomous driving device Ma. According to this, it is possible to realize the superimposed display of XR images 2a and 2d corresponding to display content Cd that matches service content Cs provided by autonomous driving device Ma in a wide range in background video 2c of background area Ab, and enhance the satisfaction of user preferences.

[0056] According to the first embodiment, three-dimensional spatial information Ti with position-related information related to the respective existing positions Pm and Po of the autonomous driving device Ma and other objects Oo is obtained from the infrastructure database Di for a plurality of three-dimensional voxels Vi obtained by virtually dividing the service space As where user services are provided. Therefore, XR images 2a and 2d that can satisfy user preferences in the background video 2c of the background area Ab can be replaced and superimposed at appropriate positions on the respective existing positions Pm and Po of the autonomous driving device Ma and other objects Oo based on the obtained three-dimensional spatial information Ti. Therefore, it is possible to increase the satisfaction of user preferences while suppressing user discomfort caused by displacement of the superimposed display.

[0057] According to the first embodiment, user services are provided by the drive control pattern of the autonomous driving device Ma adapted to the service content Cs among the content information Ci. According to this, it is possible to increase the satisfaction of user preferences through the drive of the autonomous driving device Ma in cooperation with the user service. At the same time, according to the first embodiment, the superimposed display position of the XR image 2a with respect to the background video 2c is adjusted in accordance with the existing position Pm of the autonomous driving device Ma following the drive control pattern. According to this, the XR image 2a corresponding to the display content Cd matching the service content Cs can be superimposed in the background video 2c following the drive of the autonomous driving device Ma, so that it is possible to further increase the satisfaction of user preferences. Here, particularly in the first embodiment, by being based on the three-dimensional spatial information Ti from the infrastructure database Di, the followability of the XR image 2a with respect to the drive of the autonomous driving device Ma can be increased, so that it is also possible to enhance the effect of suppressing user discomfort caused by displacement of the superimposed display.

[0058] According to the first embodiment, the character images related to the display content Cd among the content information Ci are superimposed and displayed as XR images 2a and 2d on the existing positions Pm and Po of the autonomous driving device Ma and other objects Oo. According to this, a character image that can satisfy the user preference in the background video 2c of the background area Ab can be replaced from the display of the autonomous driving device Ma and other objects Oo and made visible to the user Us. Therefore, it is possible to particularly enhance the satisfaction degree of the user preference for the display content Cd.

[0059] (Second Embodiment) The second embodiment is a modification of the first embodiment.

[0060] As shown in FIG. 11, the processing system 2001 of the second embodiment is connected via a communication network Nc to a wearable terminal Wt worn by a pre-registered user Us, an autonomous driving device Ma, and an infrastructure system Is. At least one each is assumed as the connection elements Wt, Ma, and Is to the processing system 1. Note that, as described in the first embodiment, the mobile terminal Mt carried by the user Us may or may not be connected to the processing system 2001 via the communication network Nc.

[0061] The wearable terminal Wt is configured such that, at least when connected to the processing system 2001, the user Us of the system 1 can wear it on the face and operate it hands - free. As shown in FIG. 12, the wearable terminal Wt is an optical see - through type electronic device including a display unit 2002 and a sensor unit 2003, such as an HMD or smart glasses. The wearable terminal Wt controls the transmission and reception of data via the communication network Nc by the communication unit. The wearable terminal Wt is operation - controlled by the control unit according to a control command from the system 2001 via the communication network Nc so as to perform service - related processing in cooperation with the processing system 2001. Note that FIG. 12 regarding the wearable terminal Wt schematically shows, in an image drawing, how the visual field area Av looks from the user Us side at a portion corresponding to one of the user Us's two eyes.

[0062] Under the operation control of the control unit of the wearable terminal Wt, the virtual - image projection type or retina - projection type display unit 2002 superimposes, as shown in FIG. 12, a display necessary for service - related processing on the real image of the visual field area Av visually recognized by the user Us through, for example, glasses or lenses, and makes it visually recognized by the user Us. At the same time, under the operation control of the wearable terminal Wt, the sensor unit 2003 acquires sensing information by performing sensing processing according to service - related processing. As the sensor unit 2003, at least one type is adopted from, for example, a camera, an inertial sensor, a GNSS sensor, an electro - oculogram sensor, a gaze sensor, an infrared sensor, a geomagnetic sensor, a motion sensor, a touch sensor, and a microphone. With such a configuration, the sensor unit 2003 can acquire, as sensing information, an input from the user Us representing the intention of the user Us.

[0063] The autonomous driving device Ma shown in Fig. 11 is configured in the same manner as the processing system 1 of the first embodiment, except that it is driven and controlled by a control unit according to a control command from the system 2001 via a communication network Nc so as to perform service-related processing in cooperation with the processing system 2001. At least a part of the control system 2020 of the processing system 2001 may be constituted by a communication unit and a control unit of the wearable terminal Wt, and is configured in the same manner as the control system 20 of the first embodiment, except that it executes a processing flow of the wearable terminal Wt utilization type according to the functional block configuration described later.

[0064] Function blocks constructed to perform service-related processing in the control system 2020 include a recognition block 2200, a monitoring block 2210, a drive control block 2220, and a display control block 2230, as shown in Fig. 13. The processing method by which the processing system 2001 performs service-related processing through the cooperation of these blocks 2200, 2210, 2220, 2230 is executed according to the processing flow shown in Fig. 14. This processing flow is executed in response to the sensor unit 2003 of the wearable terminal Wt acquiring service request input as sensing information, which is the intention of the user Us to request user service provision. Each "S" in this processing flow respectively means a plurality of steps executed by a plurality of instructions included in the processing program of the second embodiment.

[0065] In S2200 of the processing flow, the recognition block 2200 (see FIG. 13) acquires content information Ci, which is obtained by recognizing service content Cs and display content Cd intended by user Us, based on the sensing information regarding the face of the wearable terminal Wt worn on the face of the same user Us. Therefore, the sensing information is input to the sensor unit 2003 as the user's intention and is acquired through the communication unit of the wearable terminal Wt. At this time, the display unit 2002 of the wearable terminal Wt may superimpose and display the input reception image 2b according to the first embodiment on the real image of the visual field area Av in order to receive a line-of-sight input according to the change in the direction of the line of sight or an expression input according to the change in the expression of the face as a selection input of the content Cs, Cd from the user Us to the sensor unit 2003. Except for the above points, S2200 is executed according to S200 of the first embodiment.

[0066] In S2210 of the processing flow shown in FIG. 14, the monitoring block 2210 (see FIG. 13) monitors the visual field area Av visually recognized by the user Us through the wearable terminal Wt. At this time, the visual field area Av is monitored in the service space As where user services of a type corresponding to the service content Cs are provided as shown in FIG. 12 among the information accumulation target spaces where a plurality of voxels Vi are assumed by the infrastructure system Is. Therefore, based on at least one of, for example, camera information and inertial information among the sensing information by the sensor unit 2003 of the wearable terminal Wt, the monitoring of the visual field area Av that spreads through, for example, glasses or lenses in the direction of the face or line of sight of the user Us or the direction in which the wearable terminal Wt faces is executed.

[0067] In S2220 of the processing flow shown in FIG. 14, the drive control block 2220 (see FIG. 13) provides user services of a type corresponding to the service content Cs among the content information Ci by driving and controlling the autonomous driving device Ma in the current visual field area Av. Except for this point, S2220 is executed according to S220 of the first embodiment.

[0068] S2230 in the processing flow shown in FIG. 14 is executed in parallel with S2220. In S2230, the display control block 2230 (see FIG. 13) causes the wearable terminal Wt, in which the autonomous driving device Ma providing the user service by S2220 enters the visual field area Av, to superimpose and display the XR image 2a corresponding to the display content Cd among the content information Ci at the existence position Pm of the autonomous driving device Ma, as shown in FIG. 12. At this time, the superimposed display position of the XR image 2a is adjusted according to the existence position Pm in the visual field area Av according to the driving control pattern of the autonomous driving device Ma by S2220. Thereby, in the visual field area Av, the XR image 2a forms a three-dimensional superimposed display that overlaps the entire autonomous driving device Ma at the existence position Pm so as to hide the entire autonomous driving device Ma at the existence position Pm from the front in an opaque or semi-transparent manner.

[0069] The display control block 2230 in S2230 also superimposes and displays the XR image 2d as shown in FIG. 12 at the existence position Po of another object Oo that enters the visual field area Av together with the autonomous driving device Ma. At this time, the superimposed display position of the XR image 2d is adjusted according to the existence position Po of the other object Oo. In this way, the other object Oo on which the XR image 2d is superimposed at the existence position Po in the visual field area Av is at least one type among, for example, a person other than the user Us of the processing system 2001, a moving body other than the autonomous driving device Ma cooperating with the processing system 2001, and a ground feature.

[0070] Except for these points, S2230 is executed according to S230 of the first embodiment. As a result, in various user services by S2230 according to the first embodiment, the XR images 2a and 2d that are visually recognized at least superimposed on the existence position Pm of the autonomous driving device Ma and the existence position Po of the surrounding object Oo are displayed and output to the user Us.

[0071] In the processing flow of the second embodiment shown in FIG. 14 above, when the provision of the user service corresponding to the service content Cs among the content information Ci ends by S2220 and S2230, the completion of the current execution is achieved. Thereafter, when the next service request input is acquired through the sensor unit 2003 of the wearable terminal Wt, the next execution of the processing flow is started.

[0072] (Function and effect) The function and effect of the second embodiment described so far will be described below.

[0073] According to the second embodiment, the content information Ci that recognizes the service content Cs intended by the user Us is acquired based on the sensing information of the wearable terminal Wt worn by the user Us. At the same time, the visual field area Av visually recognized by the user Us through the wearable terminal Wt is monitored. Therefore, a user service of a type corresponding to the service content Cs is provided by driving control of the autonomous driving device Ma in the current visual field area Av. This means that among various user services, it is possible to provide a user service that satisfies user preference through cooperation with the autonomous driving device Ma.

[0074] Moreover, according to the second embodiment, the content information Ci that recognizes the display content Cd intended by the user Us is also acquired based on the sensing information of the mobile terminal Mt. Therefore, in the wearable terminal Wt where the autonomous driving device Ma that provides the user service as described above enters the visual field area Av, the XR image 2a corresponding to the display content Cd is superimposed and displayed at the existence position Pm of the autonomous driving device Ma. This means that it is possible to enhance the satisfaction degree of user preference by superimposing and displaying the XR image 2a corresponding to the display content Cd that matches the service content Cs provided by the autonomous driving device Ma.

[0075] According to the second embodiment, the XR image 2d is also superimposed and displayed on the position Po of the existence of another object Oo that enters the visual field area Av together with the autonomous driving device Ma. According to this, the superimposed display of the XR images 2a and 2d corresponding to the display content Cd that matches the service content Cs provided by the autonomous driving device Ma can be realized over a wide range of the visual field area Av, making it possible to increase the satisfaction of user preferences.

[0076] According to the second embodiment, content information Ci is acquired based on the sensing information regarding the face of the user Us with the wearable terminal Wt worn thereon. According to this, the service content Cs and the display content Cd intended by the user Us can be automatically recognized from the face with the wearable terminal Wt worn thereon as the content information Ci. Therefore, it is also possible to improve the convenience in enhancing the satisfaction of user preferences.

[0077] According to the second embodiment, the three-dimensional space information Ti with the position-related information related to the respective existence positions Pm and Po of the autonomous driving device Ma and the other object Oo added thereto is acquired from the infrastructure database Di for a plurality of three-dimensional voxels Vi obtained by virtually dividing the service space As where the user service is provided. Therefore, the XR images 2a and 2d that can satisfy the user preferences in the visual field area Av can be appropriately overlapped and superimposed from the front position on the respective existence positions Pm and Po of the autonomous driving device Ma and the other object Oo. Therefore, it is possible to increase the satisfaction of user preferences while suppressing the user discomfort caused by the positional deviation of the superimposed display.

[0078] According to the second embodiment, a user service will be provided according to the drive control pattern of the autonomous driving device Ma adapted to the service content Cs among the content information Ci. According to this, it is possible to enhance the satisfaction of user preferences through the driving of the autonomous driving device Ma in cooperation with the user service. At the same time, according to the second embodiment, the superimposed display position with respect to the visual field area Av of the XR image 2a is adjusted according to the position Pm of the autonomous driving device Ma existing according to the drive control pattern. According to this, the XR image 2a corresponding to the display content Cd that matches the service content Cs can be superimposed and displayed within the visual field area Av following the driving of the autonomous driving device Ma, so that it is possible to further enhance the satisfaction of user preferences. Here, particularly in the second embodiment, by being based on the three-dimensional space information Ti from the infrastructure database Di, the followability of the XR image 2a with respect to the driving of the autonomous driving device Ma can be enhanced, so that it is also possible to enhance the effect of suppressing user discomfort caused by displacement of the superimposed display position.

[0079] According to the second embodiment, the character image related to the display content Cd among the content information Ci will be superimposed and displayed as the XR images 2a and 2d at the existing positions Pm and Po of the autonomous driving device Ma and the other object Oo. According to this, it is possible to overlap and visually recognize by the user Us a character image that can satisfy user preferences in the visual field area Av in front of the autonomous driving device Ma and the other object Oo. Therefore, it is particularly possible to enhance the satisfaction of user preferences with respect to the display content Cd.

[0080] (Other embodiments) Although a plurality of embodiments have been described above, the present disclosure is not to be construed as being limited to those embodiments, and can be applied to various embodiments and combinations without departing from the gist of the present disclosure.

[0081] In a modification regarding the first and second embodiments, the dedicated computer constituting the control systems 20, 2020 of the processing systems 1, 2001 may have at least one of a digital circuit and an analog circuit as a processor. Here, the digital circuit is, for example, at least one type among ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), SOC (System on a Chip), PGA (Programmable Gate Array), and CPLD (Complex Programmable Logic Device). Further, such a digital circuit may have a memory storing a program.

[0082] In a modification regarding the first and second embodiments, the processing system 1, 2001 may be non-connected to the infrastructure system Is. In this case, instead of the three-dimensional space information Ti acquired from the infrastructure database Di, sensing information by the connection elements Mt, Ma, Wt to the processing system 1, 2001 may be used.

[0083] In a modification regarding the first and second embodiments, three-dimensional space information Ti not associated with the voxel Vi may be acquired from the infrastructure database Di by the processing system 1, 2001. In a modification regarding the first and second embodiments, instead of the three-dimensional space information Ti, two-dimensional grid information may be acquired from the infrastructure database Di by the processing system 1, 2001.

[0084] In a modification of the first embodiment, a smartwatch or a video see-through type or non-transmissive type wearable terminal may be used as the mobile terminal Mt. In a modification of the second embodiment, when the mobile terminal Mt is connected to the processing system 2001 and functions as a part of the wearable terminal Wt, a processing flow may be executed in response to the sensor unit 3 constituting the part acquiring a service request input as sensing information. In a modification of the second embodiment, when the mobile terminal Mt is connected to the processing system 2001 and functions as a part of the wearable terminal Wt, content information Ci that recognizes the service content Cs and the display content Cd in S2200 of the processing flow may be acquired based on the sensing information of the sensor unit 3 constituting the part.

[0085] In a modification related to S230 of the first embodiment and S2230 of the second embodiment, an XR image 2d may not be superimposed on the position Po of the presence of another object Oo entering the areas Ab, Av together with the autonomous driving device Ma. In a modification related to S2200 of the second embodiment, according to S200 of the first embodiment, a gesture input or a voice input by the user Us may be sensed by the sensor unit 2003. In a modification related to S2200 of the second embodiment, according to S200 of the first embodiment, a gesture input, a gaze input, an expression input, or a voice input by the user Us may be recognized based on the three-dimensional space information Ti acquired from the infrastructure database Di.

[0086] In addition to the above-described explanatory forms, the above-described embodiments and modifications may be implemented in the form of a semiconductor device (for example, a semiconductor chip, etc.) as a processing system 1, 2001 having at least one processor 24 and memory 22 each in the control systems 20, 2020.

[0087] (Addendum) This specification discloses a plurality of technical ideas listed below and a plurality of combinations thereof.

[0088] (Technical Idea 1) A processing system having a processor (24) and performing service-related processing related to user services provided to a user, The processor Based on the sensing information of the mobile terminal (Mt) carried by the user, obtain content information (Ci) that recognizes the service content (Cs) and display content (Cd) intended by the user, Monitor the background area (Ab) that becomes the display background when the user turns the mobile terminal, Provide user services of a type corresponding to the service content among the content information by driving control of the autonomous driving device (Ma) in the current background area, In a mobile terminal that displays the background video (2c) of the background area where the autonomous driving device that provides the user service enters, superimpose and display the XR image (2a) corresponding to the display content among the content information at the existence position (Pm) of the autonomous driving device. It is configured to execute.

[0089] (Technical idea 2) Superimposing and displaying the XR image The processing system according to Technical Idea 1, including superimposing and displaying the XR image (2d) at the existence position (Po) of another object (Oo) that enters the background area together with the autonomous driving device.

[0090] (Technical idea 3) A processing system having a processor (24) and performing service-related processing related to user services provided to a user, The processor Based on the sensing information of the wearable terminal (Wt) worn by the user, obtain content information (Ci) that recognizes the service content (Cs) and display content (Cd) intended by the user, Monitor the visual field area (Av) visually recognized by the user through the wearable terminal, Among the content information, provide user services of a type corresponding to service content in the current field of view by driving control of the autonomous driving device (Ma). In a wearable terminal where an autonomous driving device that provides user services enters the field of view, an XR image (2a) corresponding to display content among the content information is superimposed and displayed at the existence position (Pm) of the autonomous driving device.

[0091] (Technical idea 4) Superimposing and displaying the XR image The processing system according to Technical Idea 3, including superimposing and displaying an XR image (2d) at the existence position of another object (Oo) that enters the field of view together with the autonomous driving device.

[0092] (Technical idea 5) Performing sensing The processing system according to Technical Idea 3 or 4, including acquiring content information based on sensing information regarding the face of the user wearing the wearable terminal.

[0093] (Technical idea 6) The processor is further configured to acquire three-dimensional space information (Ti) with position-related information related to the existence position added to a plurality of three-dimensional voxels (Vi) obtained by virtually dividing the service space (As) where the user service is provided from the infrastructure database (Di). Superimposing and displaying the XR image The processing system according to any one of Technical Ideas 1 to 5, including superimposing and displaying the XR image at the existence position based on the acquired three-dimensional space information.

[0094] (Technical idea 7) Providing user services The processing system according to any one of Technical Ideas 1 to 5, including providing user services according to the driving control pattern of the autonomous driving device in accordance with the service content among the content information.

[0095] (Technical idea 8) Superimposing an XR image A processing system according to Technical Idea 7, which includes adjusting the superimposition display position of the XR image according to the existence position according to the drive control pattern.

[0096] (Technical idea 9) Providing a user service includes providing a user service according to the drive control pattern of the autonomous driving device according to the service content among the content information, Superimposing an XR image A processing system according to Technical Idea 6, which includes adjusting the superimposition display position of the XR image according to the existence position according to the drive control pattern.

[0097] (Technical idea 10) Superimposing an XR image A processing system according to any one of Technical Ideas 1 to 9, which includes superimposing a character image related to the display content among the content information as an XR image at the existence position.

[0098] In addition, the above Technical Ideas 1 to 10 may be realized in each form of a method and a program.

Explanation of symbols

[0099] 1,2001: Processing system, 2a, 2d, 2e: XR image, 2c: Background video, 22: Memory, 24: Processor, Ab: Background area, As: Service space, Av: Field of view area, Cd: Display content, Ci: Content information, Cs: Service content, Di: Infrastructure database, Ma: Autonomous driving device, Mt: Mobile terminal, Oo: Other object, Pm, Po, Ps: Existence position, Ti: Three-dimensional space information, Vi: Three-dimensional voxel, Wt: Wearable terminal

Claims

1. A processing system having a processor (24) and performing service-related processing related to user services provided to a user, wherein the processor acquires content information (Ci) recognizing service content (Cs) and display content (Cd) intended by the user based on sensing information of a mobile terminal (Mt) carried by the user; monitors a background area (Ab) that becomes a display background when the user turns the mobile terminal; provides, in the current background area, the user service of the type corresponding to the service content among the content information by driving control of an autonomous driving device (Ma); In the mobile terminal that displays a background video (2c) of the background area into which the autonomous driving device that provides the user service enters, superimposes and displays an XR image (2a) corresponding to the display content among the content information at the existence position (Pm) of the autonomous driving device. A processing system configured to perform.

2. The superimposed display of the XR image includes superimposing and displaying the XR image (2d) at the existence position (Po) of another object (Oo) that enters the background area together with the autonomous driving device. The processing system according to claim 1.

3. A processing system having a processor (24) and performing service-related processing related to user services provided to a user, wherein the processor acquires content information (Ci) recognizing service content (Cs) and display content (Cd) intended by the user based on sensing information of a wearable terminal (Wt) worn by the user; monitors a visual field area (Av) visually recognized by the user through the wearable terminal; Providing the user service of the type corresponding to the service content among the content information by driving control of the autonomous driving device (Ma) in the current visual area; A processing system configured to execute, on the wearable terminal when the autonomous driving device that provides the user service enters the visual area, superimposing and displaying an XR image (2a) corresponding to the display content among the content information at the presence position (Pm) of the autonomous driving device.

4. The superimposing and displaying of the XR image The processing system according to claim 3, wherein the superimposing and displaying of the XR image (2d) at the presence position of another object (Oo) entering the visual area together with the autonomous driving device is included.

5. The obtaining of the content information The processing system according to claim 3, wherein the obtaining of the content information is included based on the sensing information regarding the face of the user wearing the wearable terminal.

6. The processor is further configured to execute obtaining, from the infrastructure database (Di), three-dimensional space information (Ti) with position-related information related to the presence position added thereto in a plurality of three-dimensional voxels (Vi) obtained by virtually dividing the service space (As) where the user service is provided, The superimposing and displaying of the XR image The processing system according to any one of claims 1 to 5, wherein the superimposing and displaying of the XR image at the presence position based on the three-dimensional space information is included.

7. The providing of the user service The processing system according to any one of claims 1 to 5, wherein the providing of the user service is included according to the driving control pattern of the autonomous driving device in accordance with the service content among the content information.

8. Superimposing and displaying the XR image includes The processing system according to claim 7, further comprising adjusting a superimposition display position of the XR image according to the existing position according to the drive control pattern. **Claim 9** Providing the user service includes providing the user service according to a drive control pattern of the autonomous driving device adapted to the service content among the content information, Superimposing and displaying the XR image includes The processing system according to claim 6, further comprising adjusting a superimposition display position of the XR image according to the existing position according to the drive control pattern. **Claim 10** Superimposing and displaying the XR image includes The processing system according to any one of claims 1 to 5, further comprising superimposing and displaying, as the XR image, a character image related to the display content among the content information at the existing position. **Claim 11** A processing method executed by a processor (24) to perform service-related processing related to a user service provided to a user, acquiring content information (Ci) recognizing service content (Cs) and display content (Cd) intended by the user based on sensing information of a mobile terminal (Mt) carried by the user; monitoring a background area (Ab) that becomes a display background when the user turns the mobile terminal; providing, in the current background area, the user service of a type corresponding to the service content among the content information by drive control of an autonomous driving device (Ma); In the mobile terminal that displays the background video (2c) of the background area where the autonomous driving device that provides the user service enters, superimposing and displaying an XR image (2a) corresponding to the display content among the content information at the presence position (Pm) of the autonomous driving device. A processing method including this.

12. A processing method executed by a processor (24) to perform service-related processing related to a user service provided to a user, Obtaining content information (Ci) that recognizes service content (Cs) and display content (Cd) intended by the user based on sensing information of a wearable terminal (Wt) worn by the user; Monitoring a visual field area (Av) visually recognized by the user through the wearable terminal; Providing the user service of the type corresponding to the service content among the content information by driving control of an autonomous driving device (Ma) in the current visual field area; In the wearable terminal when the autonomous driving device that provides the user service enters the visual field area, superimposing and displaying an XR image (2a) corresponding to the display content among the content information at the presence position (Pm) of the autonomous driving device. A processing method including this.

13. A processing program stored in a storage medium (22) and including instructions executed by a processor (24) to perform service-related processing related to a user service provided to a user, The instructions are Causing content information (Ci) that recognizes service content (Cs) and display content (Cd) intended by the user to be obtained based on sensing information of a mobile terminal (Mt) carried by the user; Causing a background area (Ab) that becomes a display background when the user faces the mobile terminal to be monitored; Causing the user service of the type corresponding to the service content among the content information to be provided by driving control of the autonomous driving device (Ma) in the current background area; In the mobile terminal that displays the background video (2c) of the background area into which the autonomous driving device that provides the user service enters, causing an XR image (2a) corresponding to the display content among the content information to be superimposed and displayed at the presence position (Pm) of the autonomous driving device. A processing program including this.

14. A processing program stored in a storage medium (22) for performing service-related processing related to a user service provided to a user and including instructions executed by a processor (24), The instructions are, Obtaining content information (Ci) that recognizes service content (Cs) and display content (Cd) intended by the user based on sensing information of a wearable terminal (Wt) worn by the user; Monitoring a visual field area (Av) visually recognized by the user through the wearable terminal; Causing the user service of the type corresponding to the service content among the content information to be provided by driving control of the autonomous driving device (Ma) in the current visual field area; In the wearable terminal when the autonomous driving device that provides the user service enters the visual field area, causing an XR image (2a) corresponding to the display content among the content information to be superimposed and displayed at the presence position (Pm) of the autonomous driving device. A processing program including this.

Images