Processing system, processing method, and processing program
The processing system enhances user service selection by using XR-enhanced images to highlight matching autonomous mobile devices within the user's field or background, addressing the challenge of reflecting user preferences and improving search efficiency and accuracy.
Patent Information
- Application Number
- PCT/JP2024/015344
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-29
- Filing Date
- 2024-04-18
- Publication Date
- 2024-12-05
AI Technical Summary
Existing systems fail to efficiently reflect user preferences when providing services using autonomous mobile devices, making it difficult for users to find devices that meet their specific needs, especially in diversified content scenarios.
A processing system that uses a wearable terminal to obtain desired content and service capability information, displaying an XR-enhanced image to highlight matching autonomous mobile devices within the user's visual field or background area, allowing users to select devices that meet their preferences.
Improves the search efficiency and accuracy of autonomous mobile devices that can provide user services tailored to individual preferences by visually highlighting matching devices, enabling intuitive selection without prior reservations.
Smart Images

Figure JP2024015344_05122024_PF_FP_ABST
Abstract
Description
Processing system, processing method, and processing program CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is based on Patent Application No. 2023-87917 filed in Japan on May 29, 2023, and the contents of the original application are incorporated by reference in their entirety.
[0002] The present disclosure relates to processing techniques for performing service-related processing related to user services provided to users.
[0003] The technology disclosed in Patent Document 1 aims to provide appropriate guidance to users by reserving autonomous driving devices that fall within the service provision range for multiple service areas divided into routes where guidance services are provided to users by autonomous driving devices.
[0004] Patent No. 6956422
[0005] In the technology disclosed in Patent Document 1, autonomous mobile devices whose service provision range falls within the divided service ranges are automatically reserved, making it difficult to reflect user preferences in the reservation. However, while reflecting user preferences is an especially important factor when providing user services of diversified content, not limited to guidance services, using autonomous mobile devices, it complicates users' searches for autonomous mobile devices themselves.
[0006] An object of the present disclosure is to improve the search efficiency of an autonomous mobile device that can provide user services that satisfy user preferences.
[0007] The technical means of the present disclosure for solving the problems will be described below.
[0008] A first aspect of the present disclosure is a processing system having a processor that performs service-related processing related to a user service provided to a user, wherein the processor is configured to: acquire desired content of a user service desired by the user; acquire service capability information that represents the provision capability of the user service, linked to an autonomous mobile device waiting within a field of view visible to the user through a wearable terminal worn by the user; search for an autonomous mobile device with a provision capability that matches the desired content from the provision capabilities represented by the service capability information for each autonomous mobile device within the field of view, and superimpose an XR-enhanced image on the field of view that highlights the searched autonomous mobile device; and provide the user service by driving the autonomous mobile device selected by the user within the field of view in response to the superimposed display of the XR-enhanced image.
[0009] A second aspect of the present disclosure is a processing method executed by a processor to perform service-related processing related to a user service provided to a user, the method including: acquiring desired content of a user service desired by the user; acquiring service capability information indicating the provision capability of the user service, linked to an autonomous mobile device waiting within a field of view visible to the user through a wearable terminal worn by the user; searching for an autonomous mobile device having a provision capability matching the desired content from the provision capabilities indicated by the service capability information for each autonomous mobile device within the field of view, and superimposing an XR-enhanced image on the field of view in which the searched autonomous mobile device has a provision capability matching the desired content; and providing the user service by driving the autonomous mobile device selected by the user within the field of view in response to the superimposed display of the XR-enhanced image.
[0010] A third aspect of the present disclosure is a processing program stored in a storage medium for performing service-related processing related to a user service provided to a user, the processing program including instructions for causing a processor to execute the service-related processing, the processing program including instructions for performing the following: acquiring desired content of a user service desired by the user; acquiring service capability information representing the provision capability of the user service, linked to an autonomous mobile device waiting within a field of view visible to the user through a wearable terminal worn by the user; searching for an autonomous mobile device having a provision capability matching the desired content from the provision capabilities represented by the service capability information for each autonomous mobile device within the field of view, and superimposing an XR-enhanced image on the field of view in which the searched autonomous mobile device has a provision capability matching the desired content; and providing the user service by driving the autonomous mobile device selected by the user within the field of view in response to the superimposed display of the XR-enhanced image.
[0011] According to the first to third aspects, service capability information indicating the capability to provide a user service is acquired by linking it to a standby autonomous mobile device within a field of view visible to the user through a wearable device worn by the user. The desired content of the user service desired by the user is acquired, and then, from the provision capabilities indicated by the service capability information for each autonomous mobile device within the field of view, attention is focused on an autonomous mobile device with a provision capability matching the desired content. This allows an XR-enhanced image that highlights and searches for autonomous mobile devices with a capability matching the desired content, and is superimposed on the field of view to guide the user in selecting the device. This improves the efficiency of searching for autonomous mobile devices that can provide a user service that satisfies the user's preferences by operating within the field of view.
[0012] A fourth aspect of the present disclosure is a processing system having a processor that performs service-related processing related to a user service provided to a user, wherein the processor is configured to: acquire desired content of a user service desired by the user; acquire service capability information that indicates the provision capability of the user service from a mobile terminal carried by the user, linked to an autonomous mobile device waiting in a background area that serves as a display background for the user; search for an autonomous mobile device having a provision capability that matches the desired content from the provision capabilities indicated by the service capability information for each autonomous mobile device in the background area, and superimpose an XR-enhanced image that highlights the searched autonomous mobile device on a background video showing the background area; and provide the user service by operating the autonomous mobile device selected by the user in the background area in response to the superimposed display of the XR-enhanced image.
[0013] A fifth aspect of the present disclosure is a processing method executed by a processor to perform service-related processing related to a user service provided to a user, the method including: acquiring desired content of a user service desired by the user; acquiring service capability information representing the provision capability of the user service from a mobile terminal carried by the user, linked to an autonomous mobile device waiting in a background area that serves as a display background for the user; searching for an autonomous mobile device having a provision capability that matches the desired content from the provision capabilities represented by the service capability information for each autonomous mobile device in the background area, and superimposing an XR-enhanced image on a background video showing the background area; and providing the user service by operating the autonomous mobile device selected by the user within the background area in response to the superimposed display of the XR-enhanced image.
[0014] A sixth aspect of the present disclosure is a processing program stored in a storage medium for performing service-related processing related to a user service provided to a user, the processing program including instructions for causing a processor to execute the service-related processing, the processing program including instructions for performing the following: acquiring desired content of a user service desired by the user; acquiring service capability information representing the provision capability of the user service from a mobile terminal carried by the user, linked to an autonomous mobile device waiting in a background area that serves as a display background for the user; searching for an autonomous mobile device having a provision capability that matches the desired content from the provision capabilities represented by the service capability information for each autonomous mobile device in the background area, and superimposing an XR-enhanced image on a background video showing the background area; and providing the user service by operating an autonomous mobile device selected by the user within the background area in response to the superimposed display of the XR-enhanced image.
[0015] According to the fourth to sixth aspects, service capability information indicating the capability to provide a user service is acquired from a mobile terminal carried by a user in a background area that serves as a display background for the user, and is associated with a waiting autonomous mobile device. Then, by acquiring the desired content of a user service desired by the user, attention is focused on an autonomous mobile device with a capability to provide the desired content from the provision capabilities indicated by the service capability information for each autonomous mobile device in the background area. This allows an XR-enhanced image that highlights an autonomous mobile device with a capability matching the desired content to be superimposed on a background video showing the background area, allowing the user to select the device. This improves the efficiency of searching for an autonomous mobile device that can provide a user service that satisfies the user's preferences by operating within the background area.
[0016] FIG. 1 is an overall configuration diagram showing a network connection environment of a processing system according to a first embodiment. FIG. 2 is a schematic diagram showing a wearable terminal according to the first embodiment. FIG. 3 is a schematic diagram for explaining three-dimensional voxels assumed in the infrastructure system according to the first embodiment. FIG. 4 is a block diagram showing a processing system according to the first embodiment. FIG. 5 is a flowchart showing a processing flow according to the first embodiment. FIG. 6 is a flowchart showing a processing flow according to the first embodiment. FIG. 7 is a schematic diagram showing a display state of a wearable terminal according to the first embodiment. FIG. 8 is a schematic diagram showing a display state of a wearable terminal according to the first embodiment. FIG. 9 is a schematic diagram showing a display state of a wearable terminal according to the first embodiment. FIG. 10 is an overall configuration diagram showing a network connection environment of a processing system according to a second embodiment. FIG. 11 is a schematic diagram showing a mobile terminal according to the second embodiment. FIG. 12 is a block diagram showing a processing system according to the second embodiment. FIG. 13 is a flowchart showing a processing flow according to the second embodiment. FIG. 14 is a flowchart showing a processing flow according to the second embodiment. FIG. 15 is a schematic diagram showing a display state of a mobile terminal according to the second embodiment. FIG. 16 is a schematic diagram showing a display state of a mobile terminal according to the second embodiment.
[0017] Hereinafter, multiple embodiments of the present disclosure will be described with reference to the drawings. Note that corresponding components in each embodiment are designated by the same reference numerals, and redundant description may be omitted. Furthermore, when only a portion of the configuration is described in each embodiment, the configuration of another previously described embodiment may be applied to the remaining portions of the configuration. Furthermore, in addition to the combinations of configurations explicitly stated in the description of each embodiment, configurations of multiple embodiments may be partially combined together even if not explicitly stated, provided that there is no particular problem with the combination.
[0018] 1 performs service-related processing related to user services provided to a user Us. To this end, the processing system 1 is connected to a wearable device Wt worn by a pre-registered user Us, an autonomous mobile device Ma, and an infrastructure system 6 via a communication network Nc. It is assumed that multiple wearable devices Wt and multiple autonomous mobile devices Ma are connected to the processing system 1. It is assumed that at least one infrastructure system 6 is connected to the processing system 1.
[0019] The wearable terminal Wt is configured so that a user Us of the processing system 1 can wear the wearable terminal Wt on their face and operate it hands-free, at least when connected to the processing system 1. As shown in Fig. 2, the wearable terminal Wt is an optical see-through electronic device equipped with a display unit 2 and a sensor unit 3, such as an HMD (Head Mounted Display) or smart glasses. Note that Fig. 2 of the wearable terminal Wt is a schematic image diagram showing how a visual field area Av appears from the user Us's side in a portion corresponding to one of the user Us's eyes.
[0020] The wearable terminal Wt uses a communication unit to control the transmission and reception of data via the communication network Nc as shown in Fig. 1. The wearable terminal Wt is operated and controlled by the control unit in accordance with control commands from the processing system 1 via the communication network Nc so as to perform service-related processing in cooperation with the processing system 1.
[0021] Under operational control by the control unit of the wearable device Wt, the display unit 2 (shown in FIG. 2 ), such as a virtual image projection or retinal projection type, superimposes a display (see FIGS. 7 to 9 , described below) required for service-related processing on a real image of the field of view Av viewed by the user Us through glasses or lenses, allowing the user Us to view it. At the same time, under operational control of the wearable device Wt, the sensor unit 3 (shown in FIG. 2 ) acquires sensing information through sensing processing in accordance with the service-related processing. The sensor unit 3 may be, for example, at least one of a camera, an inertial sensor, a Global Navigation Satellite System (GNSS) sensor, an electrooculography sensor, a gaze sensor, an infrared sensor, a geomagnetic sensor, a motion sensor, a touch sensor, and a microphone. This configuration enables the sensor unit 3 to acquire input representing the user Us's intentions as sensing information.
[0022] The autonomous mobile device Ma shown in FIG. 1 is an autonomous mobile vehicle or an autonomous mobile robot that can autonomously travel in any direction, forward, backward, leftward, or rightward, by electrically driving wheels 5 based on sensing information from a sensor unit 4. The autonomous mobile device Ma uses a communication unit to control the transmission and reception of data via a communication network Nc. The autonomous mobile device Ma is driven and controlled by a control unit in accordance with control commands from a processing system 1 via the communication network Nc so as to perform service-related processing in cooperation with the processing system 1. Under this drive control, the sensor unit 4 acquires sensing information through sensing processing in accordance with the service-related processing. The sensor unit 4 may be at least one of a camera, an inertial sensor, a GNSS sensor, a LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging), a sonar, or the like.
[0023] The infrastructure system 6 is a base system that shares the three-dimensional spatial information Ti stored in the infrastructure database Di as a common foundation for the distributed architecture. The infrastructure system 6 controls the transmission and reception of data via the communication network Nc using a communication unit. The infrastructure system 6 collects the three-dimensional spatial information Ti, which is provided to individual distributed systems such as the processing system 1 that make up the distributed architecture, using a control unit as needed, and updates the information stored in the infrastructure database Di to the latest information.
[0024] In particular, the infrastructure system 6 manages information by virtually dividing the three-dimensional space to be stored into a plurality of three-dimensional voxels Vi (i.e., a three-dimensional grid) assumed to be arranged in a three-dimensional array as shown in Fig. 3. The infrastructure system 6 stores a data set linked by metadata for each space ID individually assigned to each voxel Vi as three-dimensional spatial information Ti in the infrastructure database Di as shown in Fig. 1. Of course, the three-dimensional spatial information Ti may include two-dimensional grid information linked only to, for example, the bottom surface of voxels Vi constituting the lowest layer of a two-dimensional array along the ground in the three-dimensional array.
[0025] The data constituting the three-dimensional spatial information Ti may be collected from at least one of a wearable device Wt that cooperates with the processing system 1, other wearable devices, and mobile devices such as smartphones or tablet devices. The data constituting the three-dimensional spatial information Ti may be collected from at least one of an autonomous mobile device Ma that cooperates with the processing system 1, and other moving bodies.
[0026] The data constituting the three-dimensional spatial information Ti may be collected from at least one of a communication base station, a smart pole, a smart street light, etc., equipped with infrastructure sensors such as a camera and / or LiDAR. The data constituting the three-dimensional spatial information Ti may be collected from a service provider that provides at least one of a map service, a weather service, a communication service, a traffic management service, a feature management service, an aviation management service, and a user service (described later) handled by the processing system 1.
[0027] The construction data collected in this manner to construct the three-dimensional space information Ti may be at least one type of image data, for example, video, still images, point cloud images, etc. The construction data of the three-dimensional space information Ti may be at least one type of secondary data, for which information security is guaranteed, generated by image processing of such image data regarding people, including the user Us of the processing system 1, within the target space for information accumulation, such as position data, movement data, posture data, gaze data, behavior data, and people flow data, as well as intention data representing the intention of the user Us, which will be described later.
[0028] The construction data of the three-dimensional space information Ti may be at least one type of secondary data, such as position data, movement data, and posture data, generated by image processing of image data regarding human belongings, including the wearable terminal Wt of the user Us, within the information accumulation target space.The construction data of the three-dimensional space information Ti may be at least one type of secondary data, such as position data, movement data, and posture data, generated by image processing of image data regarding moving objects, including the autonomous mobile device Ma, within the information accumulation target space.
[0029] The construction data of the three-dimensional space information Ti may be voice data obtained by collecting voices of people, including the user Us of the processing system 1, in the information storage target space. The construction data of the three-dimensional space information Ti may be at least one type of secondary data, for which information security is guaranteed, analyzed regarding people, including the user Us, through voice recognition processing of such voice data, such as position data, movement data, behavior data, people flow data, and conversation data, as well as intention data expressing the intention of the user Us, which will be described later.
[0030] The construction data of the three-dimensional spatial information Ti may be at least one of, for example, two-dimensional and / or three-dimensional map data, GIS (Geographic Information System) data, road network data, weather data, communication data, line data, traffic data, feature management data, BIM (Building Information Modeling) data, POI (Point of Interest) data, aviation management data, and time data. The construction data of the three-dimensional spatial information Ti may also be service data related to at least one of, for example, a guidance service, a transportation service, and a photography service, which will be described later as user services handled by the processing system 1.
[0031] 1 , the processing system 1 is a distributed computer system equipped with a communication system 10 and a control system 20, and constructed to include at least one of, for example, a cloud server and an edge server. At least a portion of each of the systems 10 and 20 in the processing system 1 may be configured by a communication unit and a control unit of a wearable terminal Wt. At least a portion of each of the systems 10 and 20 in the processing system 1 may be configured by a communication unit and a control unit of an autonomous mobile device Ma. At least a portion of each of the systems 10 and 20 in the processing system 1 may be configured by a communication unit and a control unit of an infrastructure system 6.
[0032] The communication system 10 is mainly composed of communication devices for constructing the communication network Nc. The control system 20 is connected to the communication system 10 via at least one of a wired communication line and a wireless communication line. The control system 20 is constructed to include at least one dedicated computer. The dedicated computer that constructs the control system 20 has at least one memory 22 and one processor 24.
[0033] In the processing system 1, the control system 20 executes a plurality of instructions of a processing program stored in the memory 22 using the processor 24. As a result, a plurality of functional blocks for performing service-related processing related to the user service provided to the user Us are constructed in the control system 20. The constructed functional blocks include a recognition block 200, a search block 210, and a drive control block 220, as shown in FIG.
[0034] The processing method in which the processing system 1 performs service-related processing through the cooperation of these blocks 200, 210, and 220 is executed according to the processing flow shown in Figures 5 and 6. This processing flow is executed in response to the sensor unit 3 of the wearable device Wt acquiring, as sensing information, a service request input indicating the user Us's intention to request the provision of a user service. Note that each "S" in this processing flow represents multiple steps executed by multiple commands included in the processing program of the first embodiment.
[0035] In S100 of the processing flow, the recognition block 200 (see FIG. 4) recognizes the field of view area Av visually recognized by the user Us through the wearable device Wt. At this time, the field of view area Av is recognized so as to overlap with a waiting area Aw where multiple autonomous mobile devices Ma can wait, as shown in FIG. 2, in an information storage target space of the infrastructure system 6, such as a station, terminal, facility entrance, or landmark.
[0036] Therefore, in S100, if there is no autonomous mobile device Ma in the waiting area Aw that overlaps with the field of view Av, the current execution of the processing flow ends. In this case, an XR notification image may be superimposed on the field of view Av to notify the user that the search was unsuccessful due to the absence of an available autonomous mobile device Ma.
[0037] The recognition process of the field of view area Av in S100 is based on at least one of sensing information, such as camera information and inertial information, obtained by the sensor unit 3 of the wearable device Wt. As a result, the field of view area Av is recognized as an area extending in the direction of the user Us's face or line of sight, or in the direction of the wearable device Wt.
[0038] 5, the recognition block 200 acquires the desired content Cd (see FIG. 7) of the user service desired by the user Us. At this time, the desired content Cd is recognized based on sensing information about the face obtained by the wearable device Wt worn on the face of the user Us. Therefore, the sensing information for recognizing the desired content Cd is acquired through the wearable device Wt at the timing when the sensing information is input to the sensor unit 3 as the user Us's intention data.
[0039] As the input of the desired content Cd in S110, a gesture input, a gaze input, a facial expression input, or a voice input by the user Us may be sensed by the sensor unit 3. At this time, on the display unit 2 of the wearable device Wt, an XR reception image Ir for receiving the input of the desired content Cd from the user Us by the sensor unit 3 may be superimposed on the field of view Av as illustrated in Fig. 7. Note that the desired content Cd may be recognized based on three-dimensional space information Ti obtained from the infrastructure database Di via the communication system 10.
[0040] 7, the items of the desired content Cd recognized in S110 include at least the types of user services that can be provided to the user Us of the processing system 1. As the types of user services corresponding to the desired content Cd, at least one of a guidance service, a transportation service, a photography service, etc. is prepared.
[0041] In S110, regardless of the type of service the desired content Cd recognized may include at least one type of voice-related item, such as whether or not voice output is required, whether or not a voice recognition function is required, whether or not a dialogue function is required, etc. If the type of desired content Cd is either a guidance service or a transportation service, the desired content Cd may include at least one type of route-related item, such as a route including a destination, a moving pace, an arrival time, and a service provision time (i.e., a guidance time or a transportation time).
[0042] If the type of desired content Cd is a guidance service, the content Cd may include at least one type of tourism-related item for a destination such as a tourist spot, such as an acceptable level of congestion, the presence or absence of a desired facility, the presence or absence of nature, etc. If the type of desired content Cd is a guidance service, the content Cd may include an optional item indicating a desire to use other types of services in combination as an optional function.
[0043] Here, among the desired contents Cd, the optional items indicating the desire to use a transportation service in combination with the guidance service as a transportation function may include at least one of the items related to luggage, such as the number of luggage, size, weight, type, whether temperature control is required, etc. This also applies to the case of a transportation service independent of the guidance service.
[0044] Among the desired content Cd, the optional items indicating the desire to use a photography service in combination with the guidance service as a photography function may include at least one photography-related item from among, for example, a photography schedule, photography timing, photography location, whether or not to transmit images or data, whether or not to print photos, etc. This also applies to photography services that are independent of the guidance service.
[0045] 5, the search block 210 (see FIG. 4) acquires service capability information that indicates a user service provision capability Pc (see FIG. 8) associated with the autonomous mobile device Ma waiting in the waiting area Aw within the field of view Av recognized in S100. In this case, the service capability information may be information that can identify the capability possessed by each autonomous mobile device Ma waiting in the waiting area Aw within the field of view Av as the provision capability Pc required to provide the types of user services prepared in the processing system 1.
[0046] Here, the provision capability Pc for various services may include at least one type of driving-related capability, such as remaining battery power, cruising range, maximum possible driving speed, minimum possible driving road width, possible road surface condition, obstacle avoidance level according to congestion level, etc. The provision capability Pc for various services may include at least one type of voice-related capability, such as whether voice output is possible, whether a voice recognition function is available, and whether a dialogue function is available.
[0047] The capacity Pc for the transportation service provided as a transportation function in the guidance service may include at least one type of transportation-related capacity, for example, the presence or absence of a loading compartment, the loadable size, the total loadable luggage weight, the loadable luggage type, the presence or absence of a refrigeration function, the presence or absence of a freezing function, etc. This also applies to the capacity Pc for the transportation service that is independent of the guidance service.
[0048] The provision capability Pc for the photography service provided as a photography function in the guidance service may include at least one type of photography-related capability, for example, the presence or absence of a camera function, the presence or absence of an image or video data transmission function, the presence or absence of a photo print function, etc. This also applies to the provision capability Pc for the replay service independent of the guidance service.
[0049] In S120, service capability information regarding such provision capability Pc may be acquired from the communication unit of the autonomous mobile device Ma waiting in the waiting area Aw within the field of view Av, via the communication system 10. The service capability information may be acquired based on three-dimensional space information Ti obtained from the infrastructure database Di via the communication system 10. The service capability information may be acquired by having a sensor unit 3, such as a camera, in the wearable device Wt capture an image of a QR code (registered trademark) displayed on the display unit of the autonomous mobile device Ma waiting in the waiting area Aw within the field of view Av.
[0050] 5, the search block 210 searches for an autonomous mobile device Ma having a provision capability Pc that matches the desired content Cd obtained in S110 from the provision capabilities Pc represented by the service capability information obtained for each autonomous mobile device Ma within the field of view Av in S120. At this time, the matching degree R (see FIG. 8) of the provision capability Pc for the desired content Cd is obtained.
[0051] In S130, the matching degree R is obtained as an index correlated with the relevance rate, which is scored for each item other than the type included in the desired content Cd, with respect to the provision capability Pc for providing a user service of the type represented by the desired content Cd. For example, as shown in the following equation 1, the matching degree R may be obtained by normalizing the integrated value of the relevance rate ri, which is scored for each item with index i, by the integrated value of the maximum relevance rate rmi for each item. Note that the matching degree R may be obtained by standardizing the integrated value of the relevance rate ri for each item, or by inputting the relevance rate ri for each item into a machine learning model.
[0052] In S130, autonomous driving devices Ma whose acquired matching degree R falls within the recommended range are extracted as search results for matching devices Mam that possess provision capabilities Pc that match the desired content Cd. The recommended range in this case is defined as a range in which the compatibility rate is equal to or greater than a threshold value and equal to or less than a maximum value, so that the matching degree R for the desired content Cd falls within a range that can be recommended to the user Us. Here, when the matching degree R is acquired by the normalization described above, the threshold value that determines the lower limit of the recommended range may be set to, for example, a specific value between 50% and 80% (when the maximum value is normalized to 100%) or a specific value between 0.5 and 0.8 (when the maximum value is normalized to 1).
[0053] 5, the search block 210 causes the display unit 2 of the wearable device Wt to superimpose on the field of view Av an XR-enhanced image Ipe that highlights the matching device Mam found in S130 as shown in Fig. 8. To achieve this, in S140, the provision capacity Pc for the user service corresponding to the desired content Cd obtained in S110 is extracted from the provision capacity Pc represented by the service capacity information obtained for each autonomous mobile device Ma in S120.
[0054] In S140, as shown in FIG. 8 , the display unit 2 superimposes an XR capability image Ip, which is a candidate for the XR-enhanced image Ipe, on the field of view Av to notify each autonomous mobile device Ma of the provision capability Pc for the user service corresponding to the desired content Cd. In particular, the XR capability image Ip is superimposed and displayed in the field of view Av, aligned with the corresponding autonomous mobile device Ma, to notify the matching degree R obtained in S130 together with the provision capability Pc. Therefore, the superimposed display position of the XR capability image Ip is adjusted so that it is visible around the autonomous mobile device Ma, for example, based on sensing information from at least one of the sensor units 3 and 4 and / or three-dimensional space information Ti obtained from the infrastructure database Di. At this time, the field of view Av on which the XR capability image Ip is superimposed may be fixed to the area recognized in S100 or updated in accordance with S100 as the user Us's viewpoint moves.
[0055] In S140, of the XR capability images Ip superimposed and displayed in the field of view Av, the image Ip highlighting the matching device Mam whose matching degree R of the provision capability Pc for the desired content Cd is within the recommended range is set as the XR-enhanced image Ipe. Therefore, the XR capability image Ip set as the XR-enhanced image Ipe is emphasized by a difference in at least one of the following types of text display and / or the window display that forms the background thereof, for example, color, size, line width, etc., relative to the XR capability images Ip that are not set as the XR-enhanced image Ipe.
[0056] In such S140, if S130 has searched for fewer matching devices Mam than autonomous mobile devices Ma present in the field of view Av, the set and non-set XR capability images Ip will be superimposed on the XR-enhanced image Ipe. On the other hand, in S140, if S130 has searched for the same number of matching devices Mam as the autonomous mobile devices Ma present in the field of view Av, only the set XR capability images Ip will be superimposed on the XR-enhanced image Ipe. Furthermore, in S140, if no matching devices Mam have been searched for by S130, only the non-set XR capability images Ip will be superimposed on the XR-enhanced image Ipe.
[0057] 5 , in response to the superimposed display in S140, the search block 210 determines whether any of the autonomous mobile devices Ma in the field of view Av has been selected by the user Us. At this time, whether or not a selection has been made by the user Us is recognized based on facial sensing information acquired by the wearable device Wt. The sensing information for recognizing whether or not a selection has been made by the user Us is input to the sensor unit 3 as intention data of the user Us and is acquired through the wearable device Wt.
[0058] The selection input for the autonomous mobile device Ma in S150 may be a gesture input, a gaze input, a facial expression input, or a voice input by the user Us, which may be sensed by the sensor unit 3. This selection input may be an input for selecting the location of the autonomous mobile device Ma, or an input for selecting images Ip, Ipe corresponding to the autonomous mobile device Ma. For either selection input, the selection position may be recognized based on, for example, sensing information from at least one of the sensor units 3, 4 and / or three-dimensional spatial information Ti obtained from the infrastructure database Di. The recognition of whether or not the user Us has made a selection and the selection position may be realized based on three-dimensional spatial information Ti obtained from the infrastructure database Di via the communication system 10.
[0059] If a selection by the user Us is recognized in S150 and a positive determination is made, S160 is executed as shown in FIG. 5. In S160, the drive control block 220 (see FIG. 4) controls the drive of the autonomous mobile device Ma selected in S150 within the field of view area Av, thereby providing a user service of a type corresponding to the desired content Cd. At this time, a drive control pattern for the autonomous mobile device Ma that matches the desired content Cd is read from the memory 22, and the user service is provided in accordance with the drive control pattern.
[0060] Here, the guidance service according to S160 is provided by a drive control pattern in which the autonomous mobile device Ma is driven from a standby position to a position in front of a guidance recipient, such as the user Us or a registered family member, performs route guidance to lead the guidance recipient to the destination, and then returns to the standby area Aw. In this guidance service, the route along which the guidance recipient will be guided may be presented by superimposing an XR presentation image on the field of view Av of the wearable device Wt or by display output from a display unit possessed by the autonomous mobile device Ma. In the guidance service, the route along which the guidance recipient will be guided may be presented by audio output from an audio unit possessed by the wearable device Wt or the autonomous mobile device Ma.
[0061] The transportation service according to S160 is provided by a drive control pattern in which the autonomous mobile device Ma, which has a loading compartment, is driven from a standby position to a position directly in front of the user Us, receives a request from the user Us to load luggage into the loading compartment, transports the luggage to the destination, and then returns to the standby area Aw. In this case, the transportation service may present to the user Us a method for loading luggage into the loading compartment by superimposing an XR presentation image on the field of view Av of the wearable device Wt or by display output from a display unit possessed by the autonomous mobile device Ma. In the transportation service, the loading of luggage into the loading compartment may be presented to the user Us by audio output from an audio unit possessed by the wearable device Wt or the autonomous mobile device Ma. Such a transportation service may be provided as a transportation function of a guidance service when the desired content Cd is included.
[0062] The photography service according to S160 is provided by a drive control pattern in which the autonomous mobile device Ma, which has an imaging unit, is driven from a standby position to a position in front of the user Us or to a landmark in the user Us's vicinity, photographs the user Us from the surrounding area, and then returns to the standby area Aw. In this photography service, the photography schedule or photography timing may be presented to the user Us by superimposing an XR presentation image on the field of view Av of the wearable device Wt, or by display output from a display unit possessed by the autonomous mobile device Ma. In the photography service, the photography schedule or photography timing may be presented to the user Us by audio output from an audio unit possessed by the wearable device Wt or the autonomous mobile device Ma.
[0063] In the photography service by S160, the autonomous mobile device Ma driven around the user Us may capture two-dimensional or three-dimensional still images or videos that include the user Us. In the photography service, the focus during photography may be adjusted based on the distance to the user Us sensed by a sensor unit 4 such as a LiDAR in the autonomous mobile device Ma.
[0064] In the photography service performed by S160, the incident direction of external light during photography may be adjusted based on three-dimensional spatial information Ti obtained from the infrastructure database Di or sensing information from a sensor unit 4, such as a camera, in the autonomous mobile device Ma. In the photography service, image data of the user Us may be stored in the memory 22 of the control system 20 or in a storage medium in the wearable device Wt. The photography service described above may be provided as a photography function of a guidance service when the desired content Cd is included in the desired content Cd.
[0065] The viewpoint of the user Us moves as the user service progresses under the drive control of the autonomous mobile device Ma. Therefore, in S160, the field of view Av is updated in accordance with S100, so that the autonomous mobile device Ma that provides the user service can be driven within the updated field of view Av as needed.
[0066] In the above-described S150, a negative determination is made, for example, if the user Us does not perform a selection input between the start of execution and the elapse of a set time, or if the user U inputs an intention not to select any autonomous mobile devices Ma, etc. In this case, it is determined that the selection by the user U has been rejected for the autonomous mobile devices Ma within the field of view Av, and S170 is executed as shown in FIGS.
[0067] 6, the search block 210 determines whether or not there is an autonomous mobile device Ma waiting within a set distance from the user Us outside the recognized field of view Av. At this time, the presence or absence of the autonomous mobile device Ma within the set distance outside the field of view Av may be recognized based on three-dimensional space information Ti obtained from the infrastructure database Di via the communication system 10.
[0068] If a positive determination is made in S170, S180 is executed as shown in Figure 6. In S180, the search block 210 associates with the autonomous mobile device Ma whose presence was recognized in S170 and that is waiting outside the field of view Av, and acquires service capability information that indicates the user service provision capability Pc as additional capability information. At this time, the acquisition of the additional capability information is realized in accordance with S120.
[0069] 6, the search block 210 searches for a matching device Mam whose provision capability Pc of the additional capability information acquired for each autonomous mobile device Ma outside the field of view Av in S180 matches the desired content Cd acquired in S110. This search is also realized based on the matching degree R acquired in accordance with S130.
[0070] 6, the search block 210 notifies the providing capability Pc represented by the additional capability information acquired for each autonomous mobile device Ma outside the field of view Av in S180. In S200, the additional capability image Ia shown in FIG. 9 is superimposed on the field of view Av by the display unit 2 of the wearable device Wt to realize such notification in accordance with the XR capability image Ip.
[0071] In particular, in S200, the additional capability image Ia is superimposed on, for example, an air area or a ground area in the field of view Av to notify the matching degree R acquired in S190 together with the provision capability Pc. Here, the additional capability image Ia may notify at least one of the following regarding the location of the corresponding autonomous mobile device Ma: the direction from the user Us, the distance from the user Us, and the time it takes to reach the user Us when called. For the matching device Mam found in S190 (in the example of FIG. 9 , the autonomous mobile device Ma whose location is displayed as "3m to the right, robot"), the additional capability image Ia may be set as an additional emphasis image Iae that emphasizes the device Mam in accordance with the XR-emphasized image Ipe. Such additional capability image Ia may be superimposed and displayed within the field of view Av at a fixed relative position to the glasses or lenses of the display unit 2, or may be superimposed and displayed within the field of view Av as an XR capability image Ip (including an XR-enhanced image Ipe for the matching device Mam) at a fixed relative position to the autonomous mobile device Ma. In either case, the field of view Av on which the additional capability image Ia is superimposed may be fixed to the area recognized by S100, or updated in accordance with S100 as the user Us's viewpoint moves.
[0072] In S210 shown in FIG. 6 , the search block 210 determines whether any autonomous mobile device Ma outside the field of view Av has been selected by the user Us in response to the superimposed display in S200, in a manner similar to S150. As a result, if a positive determination is made due to the recognition of the selection by the user Us, S220 is executed. In S220, the drive control block 220 controls the autonomous mobile device Ma selected in S210 to drive into the field of view Av in order to provide a user service corresponding to the desired content Cd in a manner similar to S160. In other words, in S220, the drive control of the selected autonomous mobile device Ma into the field of view Av enables the service of the desired content Cd to be provided.
[0073] 5 and 6, the current execution of the process flow is completed when the provision of the user service corresponding to the desired content Cd is completed in either S160 or S220. On the other hand, if a negative determination is made in either S170 or S210, the current execution of the process flow may be completed after an XR notification image notifying the user of the unsuccessful search is superimposed on the field of view Av. In either of these cases, after the current execution is completed, the next service request input is acquired via the sensor unit 3 of the wearable device Wt, and the next execution of the process flow will be initiated.
[0074] (Operational Effects) The operational effects of the first embodiment described above will be described below.
[0075] According to the first embodiment, service capability information indicating the user service provision capability Pc is acquired in association with a standby autonomous mobile device Ma within a field of view Av visible to the user Us through a wearable device Wt worn by the user Us. Then, the desired content Cd of the user service desired by the user Us is acquired, and from the provision capabilities Pc indicated by the service capability information for each autonomous mobile device Ma within the field of view Av, attention is focused on an autonomous mobile device Ma having a provision capability Pc that matches the desired content Cd. This results in an XR-enhanced image Ipe that highlights an autonomous mobile device Ma (specifically, a matching device Mam) with a capability Pc that matches the desired content Cd, and is superimposed on the field of view Av to encourage the user Us to select that device Ma. This improves the efficiency of searching for an autonomous mobile device Ma that can provide a user service that satisfies the user's preferences by operating within the field of view Av.
[0076] Furthermore, according to the first embodiment, the XR capability image Ip is aligned with and superimposed on the field of view Av for each autonomous mobile device Ma so as to notify each autonomous mobile device Ma within the field of view Av of the provision capability Pc corresponding to the desired content Cd. Therefore, among the XR capability images Ip for each autonomous mobile device Ma, the XR capability image Ip highlighting the autonomous mobile device Ma (specifically, the matching device Mam) whose matching degree R of the provision capability Pc for the desired content Cd falls within the recommended range is set as the XR-enhanced image Ipe. This allows the user Us to appropriately select an autonomous mobile device Ma automatically searched for based on the matching degree R between the desired content Cd and the provision capability Pc, without, for example, making a reservation in advance, from among the autonomous mobile devices Ma within the field of view Av notified of the provision capability Pc. Therefore, not only is the search efficiency for an autonomous mobile device Ma capable of providing a user service that satisfies the user's preferences improved, but the search accuracy can also be improved by suppressing the gap between the desired content Cd and the provision capability Pc.
[0077] According to the first embodiment, the XR capability image Ip is superimposed on the field of view Av to notify the matching degree R together with the provision capability Pc. This allows the user Us to intuitively select, from among the autonomous mobile devices Ma notified of their provision capabilities Pc within the field of view Av, an autonomous mobile device Ma that is highlighted along with a notification of the matching degree of the provision capability Pc for the desired content Cd. This can provide reliability to a highly efficient search for an autonomous mobile device Ma that can provide a user service that satisfies the user's preferences.
[0078] Furthermore, according to the first embodiment, when it is determined that the user Us's selection from among the autonomous mobile devices Ma within the field of view Av has been rejected, the service capability information of the autonomous mobile device Ma waiting outside the field of view Av is acquired as additional capability information. An additional capability image Ia notifying the user of the provision capability Pc represented by the additional capability information is then superimposed on the field of view Av. This allows the user Us to be notified of the search results for the device Ma expanded outside the field of view Av, even if an autonomous mobile device Ma capable of providing a user service that satisfies the user's preferences is not found within the field of view Av. This ensures fail-safety in the highly efficient search for the autonomous mobile device Ma, giving the user Us a sense of security.
[0079] Furthermore, according to the first embodiment, the autonomous mobile device Ma selected by the user Us in response to the superimposed display of the additional capability image Ia is driven into the field of view Av to provide the user service. This allows the user U to select the autonomous mobile device Ma notified by a highly efficient search outside the field of view Av, and the user U can receive a user service that satisfies his or her preferences from the device Ma that is automatically called into the field of view Av.
[0080] Second Embodiment The second embodiment is a modification of the first embodiment.
[0081] 10 , a processing system 2001 according to the second embodiment is connected to a mobile terminal Mt carried by a pre-registered user Us, an autonomous mobile device Ma that performs autonomous driving, and an infrastructure system 6 that provides infrastructure information, via a communication network Nc. Here, it is assumed that a plurality of mobile terminals Mt are connected to the processing system 1.
[0082] The mobile terminal Mt is configured so that it can be held and operated by a user Us of the processing system 2001 with his / her fingers, at least when connected to the system 2001. As shown in Fig. 11 , the mobile terminal Mt is a small electronic device equipped with a display unit 2002 and a sensor unit 2003, such as a smartphone or a tablet terminal. The mobile terminal Mt controls the transmission and reception of data via the communication network Nc as shown in Fig. 10 by a communication unit. The mobile terminal Mt is operated and controlled by the control unit in accordance with control commands from the processing system 2001 via the communication network Nc so as to perform service-related processing in cooperation with the processing system 2001.
[0083] Under the operational control of the control unit of the mobile terminal Mt, a display unit 2002 shown in Fig. 11, such as a liquid crystal panel or organic EL panel, realizes the screen display required for service-related processing (see Figs. 15 and 16 described below). At the same time, under the operational control of the mobile terminal Mt, a sensor unit 2003 shown in Fig. 11 acquires sensing information through sensing processing corresponding to the service-related processing. The sensor unit 2003 may be, for example, at least one of a camera, an inertial sensor, a GNSS sensor, a touch sensor, a microphone, or the like. This configuration enables the sensor unit 2003 to acquire input representing the user Us's intention as sensing information.
[0084] 10 is configured in the same manner as the processing system 1 of the first embodiment, except that the autonomous mobile device Ma is driven and controlled by a control unit in accordance with control commands from a processing system 2001 via a communication network Nc so as to carry out service-related processing in cooperation with the processing system 2001. A control system 2020 of the processing system 2001 is configured in the same manner as the control system 20 of the first embodiment, except that at least a part of the control system 2020 may be configured by the control unit of the mobile terminal Mt, and that it executes a processing flow using the mobile terminal Mt, which will be described later.
[0085] In order to perform service-related processing in the control system 2020, the processing system 2001 performs service-related processing in cooperation with blocks 200, 210, and 220 constructed as shown in FIG. 12, and the processing method is executed in accordance with the processing flows shown in FIGS. 13 and 14.
[0086] In S2100 of the processing flow, the recognition block 200 (see FIG. 12 ) recognizes a background area Ab, which serves as the display background for the user Us from the display unit 2002, by pointing the mobile terminal Mt carried by the user Us. At this time, the background area Ab is recognized as an area Ab extending in the direction in which the mobile terminal Mt is facing, based on at least one of sensing information, such as camera information from the sensor unit 2003 of the mobile terminal Mt and inertial information. In particular, the background area Ab may be recognized so as to overlap with a waiting area Aw in which multiple autonomous mobile devices Ma can wait, as shown in FIG. 11 , in accordance with the field of view area Av recognized in S100 of the first embodiment. Processing in the case where no autonomous mobile device Ma is present in the waiting area Aw overlapping with the background area Ab is similar to S100 of the first embodiment.
[0087] 13, the recognition block 200 acquires the desired content Cd desired by the user Us based on sensing information from the mobile terminal Mt carried by the user Us. The sensing information at this time is acquired through the mobile terminal Mt at the timing when the sensing information is input to the sensor unit 2003 as the user Us's intention data, and is used to recognize the desired content Cd. The items of the desired content Cd and their input are similar to S110 in the first embodiment.
[0088] 13, the search block 210 (see FIG. 12) acquires service capability information that indicates a user service provision capability Pc (see FIG. 15) associated with the autonomous mobile device Ma that is waiting in the waiting area Aw within the background area Ab recognized in S2100. Details of the provision capability Pc are similar to S120 of the first embodiment.
[0089] 13 , the search block 210 searches for an autonomous mobile device Ma having a provision capability Pc that matches the desired content Cd obtained in S2110, from the provision capabilities Pc represented by the service capability information obtained for each autonomous mobile device Ma in the background area Ab in S2120. The acquisition of the matching degree R (see FIG. 15 ) and the extraction of the matching device Mam based thereon are similar to S130 in the first embodiment.
[0090] 13, the search block 210 displays an XR-enhanced image Ipe, which emphasizes the matching device Mam found in S2130, on the display unit 2002 of the mobile terminal Mt, as shown in FIG. 15. At this time, the XR-enhanced image Ipe is superimposed on a background video Ib showing the background area Ab. Here, the background video Ib is acquired based on sensing information obtained by a sensor unit 4 such as a camera, or three-dimensional space information Ti obtained from an infrastructure database Di. Primary video data obtained by capturing the background area Ab may be used for the background video Ib, or secondary video data obtained by processing the primary data.
[0091] 15 , an XR capability image Ip that is a candidate for the XR-enhanced image Ipe is superimposed on the background video Ib by the display unit 2002 so as to notify each autonomous mobile device Ma of the provision capability Pc for the user service corresponding to the desired content Cd. In particular, the XR capability image Ip is superimposed and aligned with the corresponding autonomous mobile device Ma in the background video Ib so as to notify the matching degree R obtained in S2130 together with the provision capability Pc.
[0092] The superimposed display position of the XR capability image Ip in S2140 is adjusted to a position surrounding the position where the autonomous driving device Ma appears in the background video Ib, based on, for example, sensing information from at least one of the sensor units 2003, 2004 and / or three-dimensional space information Ti obtained from the infrastructure database Di. At this time, the background area Ab shown in the background video Ib superimposed with the XR capability image Ip may be fixed to the area recognized in S2100 or updated in accordance with movement of the user Us's viewpoint in accordance with S2100. In the above S2140, the extraction of the provision capability Pc corresponding to the desired content Cd and the setting and display of the images Ip and Ipe are similar to S140 in the first embodiment.
[0093] 13 , the search block 210 determines whether any of the autonomous mobile devices Ma in the background area Ab has been selected by the user Us in response to the superimposed display in S2140. At this time, whether or not a selection has been made by the user Us is recognized based on sensing information from the mobile terminal Mt. Therefore, the sensing information for recognizing whether or not a selection has been made by the user Us is input to the sensor unit 2003 as intention data of the user Us, and is thereby acquired through the mobile terminal Mt. Details of the selection input of the autonomous mobile device Ma are similar to S150 in the first embodiment.
[0094] If a selection by the user Us is recognized in S2150 and a positive determination is made, S2160 is executed as shown in Figure 13. In S2160, the drive control block 220 (see Figure 12) controls the drive of the autonomous mobile device Ma selected in S2150 within the background area Ab, thereby providing a user service of a type corresponding to the desired content Cd. The details of the user service provided and the drive are similar to those of S160 in the first embodiment.
[0095] In the above-described S2150, a negative determination is made, for example, if the user Us does not perform a selection input between the start of execution and the elapse of a set time, or if the user U inputs an intention not to select any autonomous mobile devices Ma, etc. In this case, it is determined that the selection by the user U has been rejected for the autonomous mobile devices Ma in the background area Ab, and S2170 is executed as shown in FIGS.
[0096] 14, the search block 210 determines whether or not an autonomous mobile device Ma is waiting outside the recognized background area Ab within a set distance from the user Us. The recognition of the presence or absence here is similar to S170 in the first embodiment.
[0097] If a positive determination is made in S2170, S2180 is executed as shown in Fig. 14. In S2180, the search block 210 associates the additional capability information with the autonomous mobile device Ma that is waiting outside the background area Ab and whose presence was recognized in S2170, and acquires additional capability information that indicates the user service provision capability Pc, in accordance with S120 and S180 of the first embodiment.
[0098] 14 , the search block 210 searches for a matching device Mam whose provision capability Pc of the additional capability information acquired for each autonomous mobile device Ma outside the background area Ab in S2180 matches the desired content Cd acquired in S2110. This search conforms to S190 of the first embodiment, which is based on the matching degree R.
[0099] 14, the search block 210 notifies the providing capability Pc represented by the additional capability information acquired for each autonomous mobile device Ma outside the background area Ab in S2180. In S2200, to realize such notification in accordance with the XR capability image Ip, the additional capability image Ia shown in FIG. 16 is superimposed on the background video Ib of the background area Ab by the display unit 2002 of the mobile terminal Mt.
[0100] In particular, in S2200, the additional capability image Ia is superimposed on, for example, an air area or a ground area shown in the background video Ib to notify the matching degree R acquired in S2190 together with the provision capability Pc. Here, the additional capability image Ia may notify at least one of the following regarding the location of the corresponding autonomous mobile device Ma: the direction from the user Us, the distance from the user Us, and the time it takes to reach the user Us when called. For the matching device Mam found in S2190 (in the example of FIG. 16 , the autonomous mobile device Ma whose location is displayed as "3m to the right, robot"), the additional capability image Ia may be set as an additional emphasis image Iae that emphasizes the device Mam in accordance with the XR-emphasized image Ipe. Such additional capability image Ia may be superimposed and displayed at a fixed relative position on the screen of the display unit 2002 that displays the background video Ib of the background area Ab, or may be superimposed and displayed as an XR capability image Ip (including an XR-enhanced image Ipe for the matching device Mam) at a fixed relative position on the autonomous mobile device Ma that is displayed in the background video Ib of the background area Ab. In either case, the background area Ab that is displayed in the background video Ib that is superimposed with the additional capability image Ia may be fixed to the area recognized by S2100, or may be updated in accordance with S2100 as the user Us's viewpoint moves.
[0101] 14, the search block 210 determines, in response to the superimposed display in S2200, whether any of the autonomous mobile devices Ma outside the background area Ab has been selected by the user Us, in accordance with S2150. As a result, if a positive determination is made due to the recognition of the selection by the user Us, S2220 is executed. In S2220, the drive control block 220 controls the autonomous mobile device Ma selected in S2210 to drive into the background area Ab in order to provide a user service corresponding to the desired content Cd in accordance with S2160. In other words, in S2220, the drive control of the selected autonomous mobile device Ma into the background area Ab enables the provision of the desired content Cd service.
[0102] 13 and 14, the current execution of the process flow is completed when the provision of the user service corresponding to the desired content Cd is completed in either S2160 or S2220. On the other hand, if a negative determination is made in either S2170 or S2210, the current execution of the process flow may be completed after an XR notification image notifying the user of the search failure is superimposed on the background video Ib in the background area Ab. In either of these cases, after the current execution is completed, the next service request input is acquired via the sensor unit 2003 of the mobile terminal Mt, and the next execution of the process flow is initiated.
[0103] (Operations and Effects) Operations and effects of the second embodiment described above will be described below.
[0104] According to the second embodiment, service capability information indicating the provision capability Pc of a user service is acquired from the mobile terminal Mt carried by the user Us, linked to a waiting autonomous mobile device Ma in a background area Ab that serves as a display background for the user Us. Then, by acquiring the desired content Cd of the user service desired by the user Us, attention is focused on the provision capability Pc indicated by the service capability information for each autonomous mobile device Ma in the background area Ab, which matches the desired content Cd. This results in an XR-enhanced image Ipe that searches for and highlights an autonomous mobile device Ma (specifically, a matching device Mam) with a capability Pc that matches the desired content Cd, and the XR-enhanced image Ipe is superimposed on the background video Ib showing the background area Ab, in preparation for the user Us's selection of the device Ma. This improves the efficiency of searching for an autonomous mobile device Ma that can provide a user service that satisfies the user's preferences by operating within the background area Ab.
[0105] Furthermore, according to the second embodiment, an XR capability image Ip is superimposed on the background video Ib of the background area Ab for each autonomous mobile device Ma, so as to notify the provision capability Pc corresponding to the desired content Cd for each autonomous mobile device Ma in the background area Ab. Therefore, among the XR capability images Ip for each autonomous mobile device Ma, an XR capability image Ip highlighting an autonomous mobile device Ma (specifically, a matching device Mam) whose matching degree R of the provision capability Pc for the desired content Cd falls within the recommended range is set as the XR-enhanced image Ipe. This allows the user Us to appropriately select an autonomous mobile device Ma automatically searched for based on the matching degree R between the desired content Cd and the provision capability Pc, without, for example, making a reservation in advance, from among the autonomous mobile devices Ma in the background area Ab for which the provision capability Pc is notified. Therefore, not only is the search efficiency for an autonomous mobile device Ma capable of providing a user service that satisfies the user's preferences improved, but the search accuracy can also be improved by suppressing the gap between the desired content Cd and the provision capability Pc.
[0106] According to the second embodiment, the XR capability image Ip is superimposed on the background video Ib of the background area Ab to notify the matching degree R together with the provision capability Pc. This allows the user Us to intuitively select, from among the autonomous mobile devices Ma notified of their provision capabilities Pc within the background area Ab, an autonomous mobile device Ma that is highlighted along with a notification of the matching degree of the provision capability Pc for the desired content Cd. This can provide reliability to a highly efficient search for an autonomous mobile device Ma that can provide a user service that satisfies the user's preferences.
[0107] Furthermore, according to the second embodiment, when it is determined that the user Us's selection from among the autonomous mobile devices Ma within the background area Ab has been rejected, the service capability information of the autonomous mobile device Ma waiting outside the background area Ab is acquired as additional capability information. An additional capability image Ia notifying the user of the provision capability Pc represented by the additional capability information is then superimposed on the background video Ib of the background area Ab. This allows the user Us to be notified of the search results for the device Ma expanded outside the background area Ab, even if an autonomous mobile device Ma capable of providing a user service that satisfies the user's preferences is not found within the background area Ab. This ensures fail-safety in the highly efficient search for the autonomous mobile device Ma, giving the user U a sense of security.
[0108] Furthermore, according to the second embodiment, the autonomous mobile device Ma selected by the user Us in response to the superimposed display of the additional capability image Ia is driven into the background area Ab to provide the user service. This allows the user U to select the autonomous mobile device Ma notified by a highly efficient search outside the background area Ab, and the user U can receive a user service that satisfies his or her preferences from the device Ma that is automatically called into the background area Ab.
[0109] (Other Embodiments) Although multiple embodiments have been described above, the present disclosure should not be construed as being limited to those embodiments, and can be applied to various embodiments and combinations within the scope that does not deviate from the gist of the present disclosure.
[0110] In the modified examples of the first and second embodiments, the dedicated computer constituting the control system 20, 2020 of the processing system 1, 2001 may have at least one of a digital circuit and an analog circuit as a processor. Here, the digital circuit is at least one of the following: an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a system-on-a-chip (SOC), a programmable gate array (PGA), and a complex programmable logic device (CPLD). Furthermore, such a digital circuit may have a memory that stores a program.
[0111] In a modification of the first and second embodiments, the processing system 1, 2001 may be disconnected from the infrastructure system 6. In this modification, sensing information from connection elements Wt, Ma, and Mt with the processing system 1, 2001 may be used instead of the three-dimensional spatial information Ti obtained from the infrastructure database Di.
[0112] In the modified examples of the first and second embodiments, three-dimensional spatial information Ti that is not linked to the voxel Vi may be acquired from the infrastructure database Di by the processing system 1, 2001. In the modified examples of the first and second embodiments, two-dimensional grid information may be acquired from the infrastructure database Di by the processing system 1, 2001 instead of the three-dimensional spatial information Ti.
[0113] In a modification of the first embodiment, when a mobile terminal Mt as in the second embodiment is connected to the processing system 1 and functions as part of a wearable terminal Wt, the processing flow may be executed in response to the sensor unit 2003 constituting the part acquiring a service request input as sensing information. In a modification of the first embodiment, when a mobile terminal Mt as in the second embodiment is connected to the processing system 1 and functions as part of a wearable terminal Wt, the desired content Cd in S110 of the processing flow may be acquired based on the sensing information of the sensor unit 2003 constituting the part.
[0114] In a modification of the second embodiment, a smart watch or a video see-through or non-transparent wearable terminal may be used as the mobile terminal Mt. In the modification of the second embodiment, when the wearable terminal Wt as in the first embodiment is connected to the processing system 2001 and functions as part of the mobile terminal Mt, the processing flow may be executed in response to the sensor unit 3 constituting the part acquiring a service request input as sensing information. In the modification of the second embodiment, when the wearable terminal Wt as in the first embodiment is connected to the processing system 2001 and functions as part of the mobile terminal Mt, the desired content Cd in S2110 of the processing flow may be acquired based on sensing information from the sensor unit 3 constituting the part.
[0115] In the modified examples of the first and second embodiments, S170 to S220 and S2170 to S2220 may be skipped, and the current execution of the processing flow may be terminated in the event of a negative determination at S150 or S2150. In S140 and S2140 according to the modified examples of the first and second embodiments, the matching degree R may be excluded from the notification target of the XR capability image Ip.
[0116] 17 (which shows a modification of the first embodiment), only the XR capability image Ip notifying the provision capability Pc of the matching device Mam, i.e., only the XR capability image Ip set in the XR-enhanced image Ipe, may be displayed. However, in the display of such a modification, if the matching device Mam is not found by S130, S2130, S140 to S160 and S2140 to S2160 may be skipped, and S170 and S2170 may be executed.
[0117] In S200 and S2200 according to the modified examples of the first and second embodiments, as shown in Fig. 18 (which is a modified example of the first embodiment), only the additional capability image Ia notifying the provision capability Pc of the matching device Mam, i.e., only the additional capability image Ia set in the additional emphasis image Iae, may be displayed. However, in the display of such a modified example, if the matching device Mam is not found by S190 and S2190, S200 to S2200 and S2200 to S2220 may be skipped, and the current execution of the processing flow may be completed.
[0118] In addition to the forms described so far, the above-mentioned embodiments and variations may be implemented in the form of a semiconductor device (e.g., a semiconductor chip) as a processing system 1, 2001 having at least one processor 24 and one memory 22 in a control system 20, 2020.
[0119] (Additional Remarks) This specification discloses the following technical ideas and combinations thereof.
[0120] (Technical Idea 1) A processing system having a processor (24) that performs service-related processing related to a user service provided to a user, wherein the processor is configured to: acquire desired content (Cd) of the user service desired by the user; acquire service capability information that indicates the provision capability (Pc) of the user service, linked to an autonomous mobile device (Ma) waiting within a field of view (Av) visible to the user through a wearable terminal (Wt) worn by the user; search for and highlight an autonomous mobile device having a provision capability that matches the desired content from the provision capabilities indicated by the service capability information for each autonomous mobile device within the field of view (Av); superimpose an XR-enhanced image (Ipe) on the field of view that highlights the autonomous mobile device; and provide the user service by driving the autonomous mobile device selected by the user within the field of view in response to the superimposed display of the XR-enhanced image.
[0121] (Technical Idea 2) The processing system described in Technical Idea 1 includes superimposing and displaying the XR-enhanced image, and including XR capability images (Ip) that are superimposed and aligned with the field of view for each autonomous driving device in the field of view so as to notify the provision capability corresponding to the desired content for each autonomous driving device in the field of view, and that highlight the autonomous driving devices whose matching degree (R) of the provision capability for the desired content is within a recommended range, as the XR-enhanced image.
[0122] (Technical Idea 3) The processing system described in Technical Idea 2, wherein superimposing and displaying the XR-enhanced image includes superimposing and displaying the XR capability image on the field of view area so as to notify the matching degree together with the provision capability.
[0123] (Technical Idea 4) A processing system according to Technical Idea 2 or 3, wherein acquiring the service capability information includes acquiring the service capability information of the autonomous mobile device waiting outside the field of view as additional capability information when it is determined that the user's selection for the autonomous mobile device within the field of view has been rejected; and superimposing the XR-enhanced image includes superimposing an additional capability image (Ia) on the field of view that notifies the user of the provision capability represented by the additional capability information; and driving the autonomous mobile device selected by the user in response to the superimposed display of the additional capability image into the field of view to provide the user service.
[0124] (Technical Idea 5) A processing system having a processor (24) that performs service-related processing related to a user service provided to a user, wherein the processor is configured to execute the following: acquire desired content (Cd) of the user service desired by the user; acquire service capability information that indicates a provision capability (Pc) of the user service, linked to an autonomous mobile device (Ma) waiting in a background area (Ab) that serves as a display background for the user, from a mobile terminal (Mt) carried by the user; search for and highlight an autonomous mobile device having a provision capability that matches the desired content from the provision capabilities indicated in the service capability information for each autonomous mobile device in the background area, and superimpose the highlighted XR-enhanced image (Ipe) on a background video (Ib) showing the background area; and provide the user service by driving the autonomous mobile device selected by the user within the background area in response to the superimposed display of the XR-enhanced image.
[0125] (Technical Idea 6) The processing system described in Technical Idea 5 includes superimposing and displaying the XR-enhanced image, and includes setting, as the XR-enhanced image, an XR capability image (Ip) that is superimposed and aligned with the background video for each autonomous driving device in the background area so as to notify each autonomous driving device of the provision capability corresponding to the desired content, the XR capability image that highlights the autonomous driving device whose matching degree (R) of the provision capability for the desired content is within a recommended range.
[0126] (Technical Idea 7) The processing system described in Technical Idea 6, wherein superimposing and displaying the XR-enhanced image includes superimposing and displaying the XR capability image on the background video so as to notify the matching degree together with the provision capability.
[0127] (Technical Idea 8) The processing system described in Technical Idea 6 or 7, wherein acquiring the service capability information includes acquiring the service capability information of the autonomous mobile device waiting outside the background area as additional capability information when it is determined that the user's selection of the autonomous mobile device in the background area has been rejected, and superimposing the XR-enhanced image includes superimposing an additional capability image (Ia) on the background video that notifies the user of the provision capability represented by the additional capability information, and driving the autonomous mobile device selected by the user in response to the superimposed display of the additional capability image into the background area to provide the user service.
[0128] (Technical Idea 9) A processing system described in any one of Technical Ideas 2 to 4, 6 to 8, in which the superimposed display of the XR-enhanced image includes obtaining the matching degree correlated with the compatibility rate scored for each item other than the type included in the desired content, with respect to the ability to provide the user service for the type represented by the desired content.
[0129] The above-mentioned technical concepts 1 to 9 may be understood as the respective technical concepts of the method and the program.
Claims
1. A processing system having a processor (24) and performing service-related processing related to a user service provided to a user, wherein the processor is configured to execute the following: acquire desired content (Cd) of the user service desired by the user; acquire service capability information indicating a provision capability (Pc) of the user service, linked to an autonomous mobile device (Ma) waiting within a field of view (Av) visible to the user through a wearable terminal (Wt) worn by the user; search for and highlight an autonomous mobile device having a provision capability matching the desired content from the provision capabilities represented by the service capability information for each autonomous mobile device within the field of view, and superimpose an XR-enhanced image (Ipe) on the field of view; and provide the user service by driving the autonomous mobile device selected by the user within the field of view in response to the superimposed display of the XR-enhanced image.
2. The processing system of claim 1, wherein the superimposing and displaying of the XR-emphasized image includes setting, as the XR-emphasized image, an XR capability image (Ip) that is superimposed and aligned with respect to the field of view for each autonomous driving device in the field of view so as to notify each autonomous driving device in the field of view of the provision capability corresponding to the desired content, the XR capability image that highlights an autonomous driving device whose matching degree (R) of the provision capability for the desired content is within a recommended range.
3. The processing system according to claim 2, wherein the step of superimposing and displaying the XR enhanced image includes: superimposing and displaying the XR capability image on the field of view area so as to notify the matching degree together with the provision capability.
4. The processing system of claim 2 or 3, wherein acquiring the service capability information includes acquiring the service capability information of the autonomous driving device waiting outside the field of view as additional capability information when it is determined that the user's selection of the autonomous driving device within the field of view has been rejected, and superimposing the XR enhanced image includes superimposing, on the field of view, an additional capability image (Ia) notifying the provision capability represented by the additional capability information, and driving the autonomous driving device selected by the user in response to the superimposed display of the additional capability image into the field of view to provide the user service.
5. A processing system having a processor (24) and performing service-related processing related to a user service provided to a user, wherein the processor is configured to execute the following operations: acquire desired content (Cd) of the user service desired by the user; acquire service capability information indicating a provision capability (Pc) of the user service from a mobile terminal (Mt) carried by the user, linked to an autonomous mobile device (Ma) waiting in a background area (Ab) that serves as a display background for the user; search for and highlight an autonomous mobile device having a provision capability matching the desired content from the provision capabilities represented by the service capability information for each autonomous mobile device in the background area, and superimpose the highlighted XR-enhanced image (Ipe) on a background video (Ib) showing the background area; and provide the user service by driving the autonomous mobile device selected by the user within the background area in response to the superimposed display of the XR-enhanced image.
6. The processing system of claim 5, wherein the superimposing and displaying of the XR-emphasized image includes setting, as the XR-emphasized image, an XR capability image (Ip) that is superimposed and aligned with the background video for each autonomous driving device in the background area so as to notify the provision capability corresponding to the desired content for each autonomous driving device in the background area, the XR capability image that highlights an autonomous driving device whose matching degree (R) of the provision capability for the desired content is within a recommended range.
7. The processing system according to claim 6, wherein the superimposing and displaying the XR enhanced image includes superimposing and displaying the XR capability image on the background video so as to notify the matching degree together with the provision capability.
8. The processing system of claim 6 or 7, wherein acquiring the service capability information includes acquiring, as additional capability information, the service capability information of the autonomous driving device waiting outside the background area when it is determined that the user's selection of the autonomous driving device in the background area has been rejected; and superimposing the XR enhanced image includes superimposing, on the background video, an additional capability image (Ia) notifying the provision capability represented by the additional capability information; and driving the autonomous driving device selected by the user in response to the superimposed display of the additional capability image into the background area to provide the user service.
9. A processing system as claimed in any one of claims 2, 3, 6 and 7, wherein the superimposed display of the XR enhanced image includes obtaining the degree of matching correlated with the compatibility rate scored for each item other than the type included in the desired content with respect to the ability to provide the user service for the type represented by the desired content.
10. A processing method executed by a processor (24) to perform service-related processing related to a user service provided to a user, the processing method including: acquiring desired content (Cd) of the user service desired by the user; acquiring service capability information representing a provision capability (Pc) of the user service, linked to an autonomous mobile device (Ma) waiting within a visual field (Av) visible to the user through a wearable terminal (Wt) worn by the user; searching for and highlighting an autonomous mobile device having a provision capability matching the desired content from the provision capabilities represented by the service capability information for each autonomous mobile device within the visual field; superimposing an XR-enhanced image (Ipe) on the visual field; and providing the user service by driving the autonomous mobile device selected by the user within the visual field in response to the superimposed display of the XR-enhanced image.
11. A processing program stored in a storage medium (22) for performing service-related processing related to a user service provided to a user, the processing program including instructions for causing a processor (24) to execute the service-related processing, the processing program including the instructions for executing the following: acquiring desired content (Cd) of the user service desired by the user; acquiring service capability information representing a provision capability (Pc) of the user service, linked to an autonomous mobile device (Ma) waiting within a visual field (Av) visible to the user through a wearable terminal (Wt) worn by the user; searching for and highlighting an autonomous mobile device having a provision capability matching the desired content from the provision capabilities represented by the service capability information for each autonomous mobile device within the visual field; superimposing an XR-enhanced image (Ipe) on the visual field; and providing the user service by driving the autonomous mobile device selected by the user within the visual field in response to the superimposed display of the XR-enhanced image.
12. A processing method executed by a processor (24) to perform service-related processing related to a user service provided to a user, the processing method including: acquiring desired content (Cd) of the user service desired by the user; acquiring service capability information indicating a provision capability (Pc) of the user service from a mobile terminal (Mt) carried by the user, linked to an autonomous mobile device (Ma) waiting in a background area (Ab) that serves as a display background for the user; searching for and highlighting an autonomous mobile device having a provision capability matching the desired content from the provision capabilities represented by the service capability information for each autonomous mobile device in the background area, and superimposing the highlighted XR-enhanced image (Ipe) on a background video (Ib) showing the background area; and providing the user service by driving the autonomous mobile device selected by the user within the background area in response to the superimposed display of the XR-enhanced image.
13. A processing program stored in a storage medium (22) for performing service-related processing related to a user service provided to a user, the processing program including instructions for causing a processor (24) to execute the service-related processing, the processing program including the instructions for executing the following: acquiring desired content (Cd) of the user service desired by the user; acquiring service capability information indicating a provision capability (Pc) of the user service from a mobile terminal (Mt) carried by the user, linked to an autonomous mobile device (Ma) waiting in a background area (Ab) that serves as a display background for the user; searching for and highlighting an autonomous mobile device having a provision capability matching the desired content from the provision capabilities represented by the service capability information for each autonomous mobile device in the background area, and displaying the highlighted XR-enhanced image (Ipe) superimposed on a background video (Ib) showing the background area; and providing the user service by driving the autonomous mobile device selected by the user in the background area in response to the superimposed display of the XR-enhanced image.
Citation Information
Patent Citations
Virtual reality service providing system and virtual reality service providing method
JP2014164537A
System and method for providing augmented virtual reality content in autonomous vehicles
JP2017204261A
Information providing device, information providing method, and information providing program
JP2020034421A
Robot control system and robot control method
JP2020138255A
Robot guidance system
WO2019202778A1
