Service determination device, service determination system, mobile body system, service determination method, and service determination program

Abstract

To provide a service determination device, service determination system, mobile body system, service determination method, and service determination program that enable a user of a service targeting an object to use an appropriate service with fewer efforts.SOLUTION: In a management system 1, a service determination device 9 includes: an acquisition unit 10; a determination unit 13; and an output unit 14. The acquisition unit 10 acquires image data representing an image obtained by imaging an object by an imaging device such as a portable apparatus 4. The determination unit 13 determines the type of service targeting the object on the basis of the image data acquired by the acquisition unit 10 from the multiple types of services. The output unit 14 outputs service information including information on the type of service determined by the determination unit 13 to a provider of the service.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a service determination device, a service determination system, a mobile system, a service determination method, and a service determination program. [Background technology]

[0002] Patent Document 1 discloses an example of a document management system. The document management system manages packages that store documents. The document management system includes a mobile terminal and an information processing device. The mobile terminal includes an imaging unit, a position recognition unit, and a mobile terminal control unit. The imaging unit obtains image data of a management label. The management label is written with information on the administrator and storage period of the document, and is assigned with identification information. The position recognition unit recognizes its own position information. The mobile terminal control unit transmits storage location information of the package recognized based on the position information and the image data. The information processing device includes a storage unit and an information processing device control unit. The storage unit stores management information that associates administrator information and storage period recognized from the image data with storage location information and identification information for each piece of identification information. The information processing device control unit compares the storage period with the current date and time for each piece of management information, and issues a warning to the administrator when there is management information whose date and time is approaching the storage period. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-13337 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the system of Patent Document 1, each item on the management label that is the source of management information is filled in by the user with a pen or the like. Therefore, a user who wishes to use a service such as document management for objects such as packages must first photograph the object after completing tasks such as filling in the label. This can be time-consuming to use the service.

[0005] The present disclosure relates to solving such problems. The present disclosure provides a service determination device, a service determination system, a mobile system, a service determination method, and a service determination program that enable a user of a service targeting an object to use an appropriate service with less effort. [Means for solving the problem]

[0006] The service determination device of the present disclosure includes an acquisition unit that acquires first image data representing an image obtained by an imaging device photographing an object, a determination unit that determines a type of service targeting the object from among a plurality of types of services based on the first image data acquired by the acquisition unit, and an output unit that outputs service information including information on the type of service determined by the determination unit to a service provider.

[0007] The service determination system according to the present disclosure includes an imaging device that captures an image, an acquisition unit that acquires first image data representing an image obtained by the imaging device capturing an object, a determination unit that determines a type of service targeted at the object from among a plurality of types of services based on the first image data acquired by the acquisition unit, and an output unit that outputs service information including information on the type of service determined by the determination unit to a service provider.

[0008] The mobile body system according to the present disclosure includes an acquisition unit that acquires first image data representing an image obtained by an imaging device photographing an object, a determination unit that determines a type of service targeted at the object from among a plurality of types of services based on the first image data acquired by the acquisition unit, an autonomous mobile body that provides at least one of the plurality of types of services, and an output unit that outputs service information to the autonomous mobile body including information on the type of service determined by the determination unit.

[0009] The service determination method of the present disclosure is a method in which a computer executes the following steps: acquire first image data representing an image obtained by an imaging device photographing an object; determine a type of service targeted at the object from among a plurality of types of services based on the acquired first image data; and output service information including information on the type of service determined as the type of service targeted at the object to a service provider.

[0010] The service determination program of the present disclosure causes a computer to acquire first image data representing an image obtained by an imaging device photographing an object, determine a type of service targeted at the object based on the acquired first image data from among a plurality of types of services, and output service information including information on the type of service determined as the type of service targeted at the object to a service provider. Effect of the Invention

[0011] According to the service determination device, service determination system, mobile system, service determination method, or service determination program of the present disclosure, a user of a service targeting an object can use an appropriate service with less effort. [Brief description of the drawings]

[0012] [Figure 1] 1 is a configuration diagram of a management system according to a first embodiment. [Diagram 2]4 is a diagram illustrating an example of learning of a determination model in the management system according to the first embodiment. FIG. [Diagram 3] FIG. 4 is a diagram for explaining an example of service determination in the management system according to the first embodiment. [Figure 4] 5 is a flowchart showing an example of an operation of the management system according to the first embodiment. [Diagram 5] 5 is a flowchart showing an example of an operation of the management system according to the first embodiment. [Figure 6] 1 is a hardware configuration diagram of a main part of a management system according to a first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The embodiments for carrying out the subject of the present disclosure will be described with reference to the accompanying drawings. In each drawing, the same or corresponding parts are given the same reference numerals, and duplicated descriptions are appropriately simplified or omitted. Note that the subject of the present disclosure is not limited to the following embodiments, and any component of the embodiment may be modified or omitted within the scope of the gist of the present disclosure.

[0014] Embodiment 1 FIG. 1 is a configuration diagram of a management system 1 according to the first embodiment.

[0015] The management system 1 is applied to a facility. The management system 1 is a system that manages the operation of the facility. The facility is, for example, an indoor facility, an outdoor facility, or a facility that combines these. The facility is, for example, made up of one or more buildings. The facility may be, for example, a part of a building.

[0016] One or more autonomous mobile bodies 2 operate in the facility. Each autonomous mobile body 2 is a robot, a drone, or other mobile equipment that moves autonomously in the facility. The autonomous mobile body 2 may move in the facility based on a movement instruction from a mobile body management device (not shown), for example. The mobile body management device is composed of, for example, one or more server computers. Each autonomous mobile body 2 and the mobile body management device are connected to a local network such as a LAN (Local Area Network) in the facility, or a wide area network such as the Internet. Each autonomous mobile body 2 moves in the facility and provides services such as cleaning, collection, or lost property collection. Each autonomous mobile body 2 is equipped with a function that can provide one or more services out of multiple types of services such as cleaning, collection, or lost property collection.

[0017] Each of the autonomous mobile bodies 2 constitutes a mobile body system that provides services to users 3 in the facility. A part or all of the mobile body system may be an internal system included in the management system 1, or an external system applied to the management system 1 from the outside. In the facility of this example, a plurality of autonomous mobile bodies 2 are in operation. The plurality of autonomous mobile bodies 2 include autonomous mobile bodies 2 that can provide different types of services.

[0018] In the management system 1, the services provided by the autonomous mobile body 2 and the like are performed on an object. For example, the cleaning service provided by the autonomous mobile body 2 and the like includes a service of removing garbage or dirt in a facility as an object. For example, the collection service provided by the autonomous mobile body 2 and the like includes a service of collecting a delivery object such as a document or package in a facility as an object and collecting the object in a dispatch center or the like. A dispatch center is, for example, a place where delivery items are sent outside the facility. For example, the lost property collection service provided by the autonomous mobile body 2 and the like includes a service of collecting a lost object such as a lost or forgotten item in a facility as an object and collecting the object in a manager's room or the like of the facility. A manager's room is, for example, a place where a manager who performs management tasks such as the operation of the facility is stationed. In the mobile body system, the autonomous mobile body 2 may provide other services.

[0019] A facility user 3 uses a service provided by an autonomous mobile body 2 or the like through a mobile device 4 that the user possesses. The mobile device 4 is, for example, a portable information processing device such as a smartphone. The mobile device 4 is equipped with a wireless communication function. The wireless communication includes, for example, the IEEE 802.11 standard and the IEEE 802.15.1 standard, as well as successor standards, alternative standards, related standards, or similar standards. The mobile device 4 connects to a network such as the Internet or a telephone line network. The mobile device 4 is equipped with a device having a function of taking an image, such as a camera. The mobile device 4 is an example of an imaging device. In this example, the mobile device 4 is equipped with a function of acquiring information on its own position and direction. The mobile device 4 may acquire information on its own position by, for example, a satellite positioning system such as a global positioning system (GPS), may acquire information on its own position by an autonomous navigation such as pedestrian dead reckoning (PDR) using an acceleration sensor, may acquire information on its own position by an indoor positioning system using a wireless signal such as an ultra-wide band (UWB) signal, or may acquire information on its own position by other methods. The mobile device 4 may acquire information on its own direction by, for example, an acceleration sensor, may acquire information on its own direction by a method using a phase difference of a received wireless signal such as an angle of arrival (AoA), or may acquire information on its own direction by other methods. The mobile device 4 may be an internal device included in the management system 1, or may be an external device outside the management system 1. The mobile device 4 realizes a function in the management system 1 by operating based on a program such as application software preinstalled, for example.

[0020] The management system 1 includes one or more wireless communication devices 5. The wireless communication device 5 is a device capable of communicating with the mobile device 4 and the like by wireless signals. The wireless communication device 5 is, for example, a wireless beacon that transmits a wireless signal to the mobile device 4. The wireless communication device 5 may be equipped with a function of receiving a wireless signal from the mobile device 4. The wireless communication device 5 is disposed in a facility. The wireless communication device 5 is connected to, for example, a local network such as a LAN within the facility, or a wide-area network such as the Internet.

[0021] The management system 1 includes a position detection device 6, an image analysis device 7, an image management device 8, and a service determination device 9. Each of the position detection device 6, the image analysis device 7, the image management device 8, and the service determination device 9 is configured by, for example, one or more server computers. A part or all of the position detection device 6, the image analysis device 7, the image management device 8, and the service determination device 9 may be provided in a facility or outside the facility. A part or all of the functions of the position detection device 6, the image analysis device 7, the image management device 8, and the service determination device 9 may be implemented by processing or storage resources on a cloud service. A part or all of the functions of the position detection device 6, the image analysis device 7, the image management device 8, and the service determination device 9 may be mounted on the same hardware. A part or all of the functions of the position detection device 6, the image analysis device 7, the image management device 8, and the service determination device 9 are realized, for example, by each server computer executing processing according to a program. The program may be a software package or the like including a plurality of components that operate each of a plurality of server computers. Each of the position detection device 6, image analysis device 7, image management device 8, and service determination device 9 is connected to a local network such as a LAN within a facility, or a wide area network such as the Internet. The management system 1 has a function of determining the type of service provided by the autonomous mobile object 2 or the like to the user 3 of the facility. The management system 1 is an example of a service determination system. The user 3 of the facility uses a service such as a mobile object system by transmitting image data of an image of an object photographed via a mobile device 4 or the like to the management system 1.

[0022] The position detection device 6 is a device equipped with a function to detect the position of the mobile device 4 carried by the facility user 3. The position detection device 6 may detect the direction of the mobile device 4. For example, the position detection device 6 may detect the position of the mobile device 4 based on the signal strength or propagation time of a wireless signal communicated between the mobile device 4 and one or more wireless communication devices 5, or may detect the position of the mobile device 4 using other methods. For example, the position detection device 6 may detect the direction of the mobile device 4 based on the phase difference of a wireless signal communicated between the mobile device 4 and one or more wireless communication devices 5, or may detect the direction of the mobile device 4 using other methods. The position detection device 6 detects the position and direction of the mobile device 4 using the position information of each wireless communication device 5 acquired in advance. Note that, in the management system 1, in a case where it is assumed that the mobile device 4 has a function to acquire information on its own position and direction, the management system 1 may not be provided with the position detection device 6. In this case, the management system 1 performs necessary information processing, etc. using the position and direction information acquired by the mobile device 4 itself.

[0023] The image analysis device 7 is a device equipped with a function of analyzing an image represented by image data transmitted by the user 3. The image analysis device 7 acquires, for example, image data of an image taken by the user 3 using the mobile device 4 directly from the mobile device 4 by wireless communication or through a network such as a LAN in a facility or the Internet. The image analysis device 7 is equipped with a function of detecting a subject included in an image by image processing technology or the like. The subject included in an image includes, for example, garbage or dirt, dispatched items such as documents or packages, or lost items such as lost or forgotten items, which may be objects that are the subject of a service provided by the autonomous mobile body 2 or the like. The image analysis device 7 may have a function of detecting the background of the subject. The image analysis device 7 may extract features of the image as a result of the image analysis.

[0024] The image management device 8 is a device equipped with a function for managing image data transmitted by the user 3. The image management device 8 stores, for example, image data transmitted by the user 3. The image management device 8 stores, for example, together with the image data, information on the position of the mobile device 4 when the mobile device 4 photographed the image of the image data as photographing position information. The image management device 8 stores, for example, together with the image data, information on the direction of the mobile device 4 when the mobile device 4 photographed the image of the image data as photographing direction information. The information on the position and direction of the mobile device 4 may be, for example, information acquired by the mobile device 4 itself, or information detected by the position detection device 6. In addition, the image management device 8 stores, for example, together with the image data, information on the analysis result of the image of the image data by the image analysis device 7. The information on the analysis result by the image analysis device 7 may be information on the detected subject, etc., may be an image in which the position, size, orientation, and color tone of the detected subject are normalized, or may be information on the feature amount of the extracted image.

[0025] The service determination device 9 is a device that performs information processing such as determining the type of service to be provided by the autonomous mobile body 2 to the facility user 3. The service determination device 9 includes an acquisition unit 10, a learning unit 11, a storage unit 12, a determination unit 13, and an output unit 14.

[0026] The acquisition unit 10 is a part equipped with a function of acquiring image data representing an image obtained by an imaging device such as the mobile device 4 capturing an image of an object. The acquisition unit 10 acquires, for example, image data managed by the image management device 8 from the image management device 8 via a network such as a LAN within a facility or the Internet. The image data acquired by the acquisition unit 10 may include, for example, information representing an image, shooting position information, shooting direction information, and information on the analysis result by the image analysis device 7. The image data is an example of the first image data.

[0027] The learning unit 11 is a part that has a function of learning the judgment model. The judgment model is, for example, a learned model that receives image data as input and outputs the type of service that targets the subject of the image of the image data. The learning unit 11 learns the judgment model by a machine learning method such as supervised learning using a training data set, for example. The training data set includes a plurality of training data. Each training data is data that includes, for example, a combination of image data representing an image and the type of service that targets the subject of the image. The learning unit 11 may store training data used to learn the judgment model. The learning unit 11 may learn the judgment model using other machine learning methods, etc.

[0028] The storage unit 12 is a part equipped with a function of storing information. The storage unit 12 stores, for example, a judgment model generated in advance by the learning unit 11 through learning. The storage unit 12 may store a judgment model that has been learned in advance outside the management system 1.

[0029] The determination unit 13 is a part equipped with a function of determining the type of service to be provided to the user 3 from among a plurality of types of services provided by the autonomous moving body 2 or the like in the facility. The determination unit 13 determines the type of service targeted at a subject included in the image of the image data based on the image data acquired by the acquisition unit 10. For example, the determination unit 13 may use a learned determination model stored in advance in the storage unit 12 to input the image data into the determination model and determine the type of service based on an output obtained by the input. The determination unit 13 may also use the shooting position information and the shooting direction information when determining the type of service. The determination unit 13 may determine the type of service and also determine the location where the service is provided by the autonomous moving body 2 or the like.

[0030] The output unit 14 is a part equipped with a function of outputting service information about a service to be provided to the user 3 to a service provider. The service information includes, for example, information on the type of service determined by the determination unit 13. The service information may include, for example, information on the location of the service determined by the determination unit 13. The output unit 14 may have, for example, a function of determining an output destination of the service information. The output unit 14 outputs, for example, the service information to an information processing terminal installed in a manager's room or the like. The manager of the facility can quickly respond to the requested service by referring to the input service information. The output unit 14 selects, for example, an autonomous mobile body 2 based on the type of service included in the service information, and outputs the service information to the selected autonomous mobile body 2. The output unit 14 selects an autonomous mobile body 2 that can provide the type of service included in the service information from among the autonomous mobile bodies 2 operating in the facility. When the autonomous mobile body 2 is controlled by a mobile body control device or the like, the output unit 14 may output the service information to the autonomous mobile body 2 via the mobile body control device. The autonomous mobile body 2 that receives the input of the service information moves through the facility and performs the service corresponding to the service information. The autonomous moving body 2 and the facility manager are examples of service providers. Note that, for example, the facility manager does not need to provide the service directly. The facility manager may provide the service to the user 3 through, for example, a service provider.

[0031] Next, an example of learning a judgment model will be described with reference to FIG. FIG. 2 is a diagram for explaining an example of learning of a determination model in the management system 1 according to the first embodiment.

[0032] The image analysis device 7 is equipped with a subject detection model. The subject detection model is, for example, a trained model that receives image data as input and outputs a subject in the image of the image data. The subject detection model may be trained in advance outside the management system 1, or may be trained in the management system 1. The image analysis device 7 detects a subject in the image of the input image data using, for example, the subject detection model.

[0033] Next, for the image represented by the image data, a type of service targeted at the subject of the image is determined. This determination is made, for example, by a manager of a facility that uses the management system 1 or by a manager of the management system 1 itself. At this time, the information on the subject detected by the image analysis device 7 may or may not be used to determine the type of service.

[0034] The learning unit 11 accumulates a combination of image data and the type of service determined for the image data as training data. The training data may include shooting position information and shooting direction information for the image data. The training data may include information on the detection result of the subject of the image of the image data. The learning unit 11 accumulates a plurality of training data as a training data set in, for example, a database.

[0035] The learning unit 11 is equipped with a learning engine. The learning engine outputs a judgment model using a sufficiently accumulated training data set by a machine learning method such as supervised learning. The learning engine may use an AI technology (AI: Artificial Intelligence) such as a decision tree or deep learning. The training data may not include information on the detection result of the subject of the image of the image data. In this case, the judgment model may internally have a function of detecting the subject of the image, or may directly output the type of service from the image data.

[0036] The management system 1 is operated using the judgment model learned in this way. The management system 1 may accumulate a new training data set using data such as image data obtained during operation, and update the judgment model as needed. In this case, the management system 1 may increase the learning efficiency and learning accuracy by reducing the weight of old training data or deleting old training data.

[0037] Furthermore, during operation of the management system 1, the type of service determined by the determination unit 13 may be presented to the mobile device 4 of the user 3. At this time, the user 3 may correct the type of service if the presented type of service is incorrect. The corrected type of service is then stored in the learning unit 11 as training data together with the image data. Furthermore, the type of service that is not corrected may be stored in the learning unit 11 as training data together with the image data as it is. This enables the management system 1 to further improve the accuracy of service determination during operation.

[0038] Next, an example of determining the type of service will be described with reference to FIG. FIG. 3 is a diagram for explaining an example of service determination in the management system 1 according to the first embodiment.

[0039] For example, the user 3 takes an image of a delivery packaged in a cardboard box using the mobile device 4 in the office area on the 10th floor where the user 3 works in the facility. At this time, the mobile device 4 acquires the current location as the image capturing position information. The user 3 transmits image data representing the captured image to the image management device 8 via the image analysis device 7 or the like through the mobile device 4. The image data includes the image capturing position information. The acquisition unit 10 acquires image data representing an image including a package such as a cardboard box as a subject and including information on the office area on the 10th floor as the image capturing position information of the mobile device 4. At this time, the determination unit 13 determines, for example, based on a determination model, that the type of service to be provided is a collection service in which the package is collected as an object and collected at a dispatch center. The output unit 14 generates service information to be output based on the determination result. The service information includes image capturing position information corresponding to the location where the object was placed at the time of image capturing as the location where the service is performed. The output unit 14 selects the autonomous moving body 2 to be output based on the service information. The output unit 14 selects, for example, an autonomous mobile body 2 capable of implementing a collection service. The output unit 14 outputs service information to the selected autonomous mobile body 2. The autonomous mobile body 2 that has received the service information moves to the office area on the 10th floor where the service is to be implemented, and collects the delivery item, which is the target item. Thereafter, the autonomous mobile body 2 transports the collected delivery item to a shipping center.

[0040] For example, the user 3 finds a delivery packaged in a cardboard box in a toilet area on the 10th floor of the facility. Since deliveries waiting for collection are usually not placed in the toilet area, the user 3, who believes that the delivery is a lost item, takes an image of the lost item, the cardboard box, using the mobile device 4. At this time, the mobile device 4 acquires the current location as the shooting position information. The user 3 transmits image data representing the captured image to the image management device 8 via the image analysis device 7 or the like through the mobile device 4. The image data includes the shooting position information. The acquisition unit 10 acquires image data representing an image including a package such as a cardboard box as a subject, and including information on the toilet area on the 10th floor as the shooting position information of the mobile device 4. At this time, the judgment unit 13 judges, for example, based on a judgment model, that the type of service to be provided is a lost item collection service that collects the lost item as an object and collects it in the manager's room. The output unit 14 generates service information to be output based on the judgment result. The service information includes, as a location where the service is to be performed, shooting location information corresponding to the location where the object was placed at the time of shooting. The output unit 14 selects the autonomous moving body 2 to which the service information is to be output based on the service information. The output unit 14 selects, for example, an autonomous moving body 2 capable of performing a lost item recovery service. The output unit 14 outputs the service information to the selected autonomous moving body 2. The autonomous moving body 2 that has received the service information moves to the toilet area on the 10th floor, where the service is to be performed, and recovers the lost item, which is the object. The autonomous moving body 2 then transports the recovered lost item to the manager's room.

[0041] For example, the user 3 finds a smartphone in a toilet area on the 13th floor of the facility. Since a smartphone that is not a delivery item is usually not left unattended in a facility, the user 3, who believes that the delivery item is a lost item, takes an image of the lost smartphone using the mobile device 4. At this time, the mobile device 4 acquires the current location as the shooting location information. The user 3 transmits image data representing the captured image to the image management device 8 via the image analysis device 7 or the like through the mobile device 4. The image data includes the shooting location information. The acquisition unit 10 acquires image data representing an image including a lost item such as a smartphone as a subject, and including information on the toilet area on the 13th floor as the shooting location information of the mobile device 4. At this time, the judgment unit 13 judges, for example, based on a judgment model, that the type of service to be provided is a lost property collection service that collects lost items as objects and collects them in the manager's room. The output unit 14 generates service information to be output based on the judgment result. The service information includes shooting location information corresponding to the location where the object was placed at the time of shooting as the implementation location of the service. The output unit 14 selects the destination autonomous moving body 2 based on the service information. The output unit 14 selects, for example, an autonomous moving body 2 capable of implementing a lost-item recovery service. The output unit 14 outputs the service information to the selected autonomous moving body 2. The autonomous moving body 2 that has received the service information moves to the toilet area on the 13th floor, where the service is to be implemented, and recovers the lost item, which is the target item. The autonomous moving body 2 then transports the recovered lost item to the manager's room.

[0042] For example, a user 3 who has spilled a drink in a dining area on the fifth floor of a facility takes an image of the spilled drink using a mobile device 4. At this time, the mobile device 4 acquires a current location as shooting position information. The user 3 transmits image data representing the captured image to an image management device 8 via an image analysis device 7 or the like through the mobile device 4. The image data includes shooting position information. The acquisition unit 10 acquires image data representing an image including a spilled drink on a subject and including information on the dining area on the fifth floor as shooting position information of the mobile device 4. At this time, the determination unit 13 determines, for example, based on a determination model, that the type of service to be provided is a cleaning service that removes stains caused by the spilled drink as an object. The output unit 14 generates service information to be output based on the determination result. The service information includes shooting position information corresponding to the location where the object was placed at the time of shooting as the implementation location of the service. The output unit 14 selects an autonomous moving body 2 to which the service is to be output based on the service information. The output unit 14 selects, for example, an autonomous moving body 2 that can perform a cleaning service. The output unit 14 outputs the service information to the selected autonomous moving body 2. The autonomous moving body 2 that has received the service information moves to the dining area on the fifth floor, where the service is to be performed, and removes the dirt, which is the target object.

[0043] Next, an example of the operation of the management system 1 will be described with reference to FIGS. 4 and 5 are flowcharts showing an example of the operation of the management system 1 according to the first embodiment.

[0044] Fig. 4 shows an example of operations when learning a judgment model, etc. The operations shown in Fig. 4 may be performed in advance before starting operation of the management system 1, may be performed in parallel with the operation, or may be performed periodically during operation.

[0045] In step S101, the portable device 4 carried by the user 3 or the manager takes an image of the service object as a subject in response to an input operation. Then, in step S102, the portable device 4 transmits the taken image of the object to the image management device 8 through the wireless communication device 5, the position detection device 6, the image analysis device 7, and the like provided in the facility in response to the input operation. Then, in step S103, the position detection device 6 adds the shooting position information and shooting direction information of the portable device 4 when taking the image to the image data of the image.

[0046] Thereafter, in step S104, the acquisition unit 10 of the service determination device 9 acquires image data from the image management device 8. The acquisition unit 10 acquires information such as detection information of the subject of the image and the determination result of the type of service that targets the subject from, for example, a portable device 4 possessed by a manager or other information processing device. At this time, the acquisition unit 10 may acquire information such as whether or not the type of service has been corrected by the user 3, and the corrected type of service if there has been a correction, from the portable device 4 possessed by the user 3. Thereafter, the processing of the management system 1 proceeds to step S105.

[0047] In step S105, the learning unit 11 combines the determination result such as the type of service with image data including the shooting position information and the shooting direction information, and accumulates the combined data in a training data set as training data. After that, the process of the management system 1 proceeds to step S106.

[0048] In step S106, the learning unit 11 judges whether the data accumulated as the training data set is sufficient. The learning unit 11 may make this judgment based on, for example, the amount of data in the training data set, or based on the comprehensiveness of the data in the training data set. If it is judged that the accumulated data is sufficient, the processing of the management system 1 proceeds to step S107. On the other hand, if it is judged that the accumulated data is not sufficient, the management system 1 continues to collect training data. Thereafter, the processing from step S101 is repeated.

[0049] In step S107, the learning unit 11 uses the accumulated training data set and the learning engine to output a judgment model. The storage unit 12 stores the output judgment model. After that, the process of the management system 1 for learning the judgment model and the like is terminated.

[0050] 5 shows an example of the operation of the management system 1. In this example, the autonomous moving body 2 is equipped with a device having a function of taking images, such as a camera. Furthermore, the shooting position information and shooting direction information when the autonomous moving body 2 takes an image are acquired in the same manner as the shooting position information and shooting direction information when the mobile device 4 takes an image, for example.

[0051] In step S201, the portable device 4 carried by the user 3 takes an image of the object of the service as a subject in response to the operation of the user 3. Then, in step S202, the portable device 4 transmits the captured image of the object to the image management device 8 through the wireless communication device 5, the position detection device 6, the image analysis device 7, and the like provided in the facility in response to the operation of the user 3. Then, in step S203, the position detection device 6 adds, to the image data of the image, the shooting position information and shooting direction information of the portable device 4 when the image was taken.

[0052] After that, in step S204, the acquisition unit 10 of the service judgment device 9 acquires the image data from the image management device 8. After that, the process of the management system 1 proceeds to step S205.

[0053] In step S205, the determination unit 13 determines the type of service based on the output obtained by inputting image data including shooting position information and shooting direction information into the determination model stored in the storage unit 12. After that, the processing of the management system 1 proceeds to step S206.

[0054] In step S206, the output unit 14 determines an output destination of the service information including information on the type of service determined by the determination unit 13. In this example, the output unit 14 selects, as the output destination, any one of the autonomous moving bodies 2 operating in the facility based on the type of service. The output unit 14 outputs the service information to the selected autonomous moving body 2. After that, the processing of the management system 1 proceeds to step S207.

[0055] In step S207, the autonomous mobile body 2 that has received the service information moves to the location where the service is provided. The autonomous mobile body 2 photographs an object specified in the service information at the location where the service is provided. The autonomous mobile body 2 transmits image data representing the image of the photographed object to the service determination device 9. The image data transmitted at this time includes photographing position information and photographing direction information when the autonomous mobile body 2 photographs the image. The image data is an example of second image data. Thereafter, the processing of the management system 1 proceeds to step S208.

[0056] In step S208, the acquisition unit 10 of the service determination device 9 acquires image data representing an image captured by the autonomous moving body 2. The output unit 14 determines whether the subject of the image captured by the autonomous moving body 2 matches the object for which the type of service has been determined. At this time, the output unit 14 may determine the match with the object by image analysis or the like. In addition, the output unit 14 may determine the match with the object by using the shooting position information and shooting direction information of the image data captured by the portable device 4 and the shooting position information and shooting direction information of the image data captured by the autonomous moving body 2. For example, when the shooting position information or shooting direction information does not match within a preset range, the output unit 14 determines that the subject of the image captured by the autonomous moving body 2 does not match the object of the service even if the captured image itself is similar. When it is determined that the subject of the image captured by the autonomous moving body 2 does not match the object of the service, the processing of the management system 1 for the provision of the service during operation is terminated. At this time, the output unit 14 may output information to the autonomous moving body 2 to stop the provision of the service. On the other hand, if it is determined that the subject of the image captured by the autonomous moving body 2 matches the target of the service, the processing of the management system 1 proceeds to step S209.

[0057] In step S209, the output unit 14 outputs information for executing the service provision to the autonomous mobile body 2. The autonomous mobile body 2 that has received the information executes the provision of the service targeted at the target object. After the provision of the service is completed, the processing of the management system 1 for the provision of the service during operation ends.

[0058] As described above, the service determination device 9 according to the first embodiment includes the acquisition unit 10, the determination unit 13, and the output unit 14. The acquisition unit 10 acquires image data representing an image obtained by an imaging device such as the mobile device 4 photographing an object. The determination unit 13 determines the type of service targeted at the object from among a plurality of types of services based on the image data acquired by the acquisition unit 10. The output unit 14 outputs service information including information on the type of service determined by the determination unit 13 to the service provider. With this configuration, the user 3 can use an appropriate service targeted at the object simply by photographing and transmitting an image of the object using the mobile device 4. In addition, the service provider can obtain the service information in a state in which the type of service to be provided is automatically determined, which makes the work of providing the service more efficient.

[0059] The determination unit 13 determines the location of the service based on the photographing location information, using the location of the mobile device 4 when the object is photographed as the photographing location information. The output unit 14 outputs the information of the implementation location included in the service information. With this configuration, the service can be provided promptly at the implementation location where the service is to be provided. For example, when the autonomous mobile body 2 provides a service, the autonomous mobile body 2 can be moved to the implementation location, thereby enabling the service to be provided promptly. Also, when a staff member of a service provider provides a service, the implementation location can be notified to the staff member, thereby enabling the service to be provided promptly. Note that the service information does not necessarily include information on the implementation location of the service. For example, when the content or implementation location of the service to be provided is clear from the type of service, or when the type of service is, for example, for the entire facility, the service information does not need to include information on the implementation location of the service.

[0060] Further, the determination unit 13 determines the type of service for the object based on the image data and the photographing position information. With this configuration, even for the same object such as a package, an appropriate service is provided according to the location where it is placed. For example, when a package such as a cardboard box is in an office area, it is determined that a collection service should be provided, and when a package such as a cardboard box is in a toilet area, it is determined that a lost property recovery service should be provided, and so on. In addition, the determination unit 13 may determine the type of service without using the photographing position information, etc. For example, when the content of the service to be provided is clear from the subject of the image, the determination unit 13 may determine the type of service without using the photographing position information, etc. For example, the determination unit 13 may determine that a cleaning service should be provided for an image having a spilled drink as a subject.

[0061] Furthermore, the output unit 14 outputs the service information to the autonomous moving body 2 that provides the service. The output unit 14 outputs the service information to the autonomous moving body 2 selected based on the type of service included in the service information. With this configuration, an autonomous moving body 2 suitable for a service targeting an object is selected, and the service is provided to the user 3. This further enhances the convenience for the user 3.

[0062] The acquisition unit 10 also acquires image data representing an image captured by the autonomous moving body 2 that has received the service information. When the subject of the image matches the object of the service, the output unit 14 outputs information for providing the service to the autonomous moving body 2. On the other hand, when the subject of the image does not match the object of the service, the output unit 14 does not output information for providing the service to the autonomous moving body 2. At this time, the output unit 14 may output information for stopping the provision of the service to the autonomous moving body 2. With this configuration, the service is provided after it is confirmed that the autonomous moving body 2 faces the object of the service. This prevents the autonomous moving body 2 from performing the service on the wrong object.

[0063] In addition, the determination of whether the subject of the image captured by the autonomous moving body 2 matches the target of the service is performed based on the shooting position information or shooting direction information of the image data representing the image captured by the mobile device 4, and the shooting position information or shooting direction information of the image data representing the image captured by the autonomous moving body 2. Here, in this determination, one or both of the shooting position information and the shooting direction information may be used. With this configuration, it is possible to more reliably prevent the autonomous moving body 2 from performing a service on a wrong target. For example, since the position of the subject is specified by the shooting position information and the shooting direction information, even if a similar object that looks similar to the target is placed near the target, the similar object and the target object are more reliably distinguished. Note that the determination of whether the subject of the image captured by the autonomous moving body 2 matches the target of the service may be performed without using the shooting position information or the shooting direction information.

[0064] The service determination device 9 also includes a storage unit 12. The storage unit 12 stores a determination model. The determination model is a trained model that is trained in advance using a set of image data representing an image and a type of service targeting the subject of the image, and that inputs image data and outputs a type of service. The determination unit 13 inputs image data and uses the determination model to determine the type of service targeting the target object. This configuration makes it possible to determine the type of service using a technique such as machine learning, so that a determination that is more suited to the actual conditions of the facility can be made.

[0065] The imaging device may not be the mobile device 4. The imaging device may be, for example, a surveillance camera installed in a facility. In this case, in the management system 1, information on the location where the surveillance camera is installed may be treated as the imaging location information. In addition, when the imaging device is a surveillance camera, there is a possibility that there is a distance between the location of the imaging device itself and the location of the subject. In this case, the determination unit 13 may set the location calculated by comparing the image data and the environmental map information as the imaging location information of the image data. The environmental map information may be, for example, information on the location and appearance of architectural structures such as walls, floors, ceilings, columns, and beams in the facility, or information on the location and appearance of objects such as equipment and furniture installed in the facility. The environmental map information may be, for example, photo data of the architectural structures or objects. The determination unit 13 may acquire the location of the object by, for example, comparing the background of the image captured by the imaging device with the environmental map information.

[0066] Next, an example of the hardware configuration of the management system 1 will be described with reference to FIG. FIG. 6 is a hardware configuration diagram of a main part of the management system 1 according to the first embodiment.

[0067] Each function of the management system 1 may be realized by a processing circuit. The processing circuit includes at least one processor 100a and at least one memory 100b. The processing circuit may include at least one dedicated hardware 200 in addition to or in place of the processor 100a and the memory 100b.

[0068] When the processing circuit includes a processor 100a and a memory 100b, each function of the management system 1 is realized by software, firmware, or a combination of software and firmware. At least one of the software and firmware is written as a program. The program is stored in the memory 100b. The processor 100a realizes each function of the management system 1 by reading and executing the program stored in the memory 100b.

[0069] The processor 100a is also called a CPU (Central Processing Unit), a processing device, an arithmetic device, a microprocessor, a microcomputer, or a DSP. The memory 100b is composed of a non-volatile or volatile semiconductor memory such as a RAM, a ROM, a flash memory, an EPROM, or an EEPROM.

[0070] Where the processing circuitry comprises dedicated hardware 200, the processing circuitry may be implemented, for example, as a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0071] Each function of the management system 1 can be realized by a processing circuit. Alternatively, each function of the management system 1 can be realized collectively by a processing circuit. Some of the functions of the management system 1 may be realized by dedicated hardware 200, and the other parts may be realized by software or firmware. In this way, the processing circuit realizes each function of the management system 1 by dedicated hardware 200, software, firmware, or a combination of these.

[0072] To summarize the above explanation, possible configurations of the technology according to the present disclosure include the configurations listed below as appendices. (Appendix 1) an acquisition unit that acquires first image data representing an image obtained by an imaging device capturing an image of an object; a determination unit that determines a type of service for the object from among a plurality of types of services based on the first image data acquired by the acquisition unit; an output unit that outputs service information including information on the type of service determined by the determination unit to a service provider; A service determination device comprising: (Appendix 2) The determination unit determines a location where the service is to be provided based on the image capturing position information of the first image data, the location of the image capturing device when the object is captured, and The output unit outputs the information on the location of the service together with the service information. 2. A service determination device as described in claim 1. (Appendix 3) The determination unit determines a location where the service will be provided based on the photographing location information, the location being calculated by comparing the first image data with the environmental map information as photographing location information of the first image data; The output unit outputs the information on the location of the service together with the service information. 2. A service determination device as described in claim 1. (Appendix 4) The determination unit determines a type of service for the object based on the first image data and the shooting position information of the first image data. 4. A service determination device according to claim 2 or 3. (Appendix 5) the output unit outputs the service information to an autonomous moving object that provides at least one of the plurality of types of services. 5. A service determination device according to claim 1. (Appendix 6) the output unit outputs the service information to an autonomous moving object selected from a plurality of autonomous moving objects each providing at least one of the plurality of types of services based on a type of service included in the service information. 6. A service determination device as described in claim 5. (Appendix 7) the acquisition unit acquires second image data representing an image captured by an autonomous moving body that has received the output of the service information from the output unit, when a subject of the image of the second image data matches the target object, the output unit outputs information to the autonomous moving body for providing the service with the target object as a target, and when a subject of the image of the second image data does not match the target object, the output unit does not output information to the autonomous moving body for providing the service. 7. A service determination device according to claim 5 or 6. (Appendix 8) The determination of whether or not the subject of the image of the second image data matches the target object is performed based on imaging direction information of the first image data representing the orientation of the imaging device when the image of the first image data was captured and imaging direction information of the second image data representing the orientation of the autonomous moving body when the image of the second image data was captured. 8. A service determination device as described in claim 7. (Appendix 9) A storage unit that stores a judgment model that is learned in advance based on a combination of image data representing an image and a type of service targeting the subject of the image, and that receives image data as input and outputs a type of service. Equipped with The determination unit determines a type of service for the object by using the determination model with the first image data as an input. 9. A service determination device according to any one of claims 1 to 8. (Appendix 10) An imaging device that captures an image; an acquisition unit that acquires first image data representing an image obtained by the imaging device capturing an image of an object; a determination unit that determines a type of service for the object from among a plurality of types of services based on the first image data acquired by the acquisition unit; an output unit that outputs service information including information on the type of service determined by the determination unit to a service provider; A service determination system comprising: (Appendix 11) an acquisition unit that acquires first image data representing an image obtained by an imaging device capturing an image of an object; a determination unit that determines a type of service for the object from among a plurality of types of services based on the first image data acquired by the acquisition unit; an autonomous moving body that provides at least one of the plurality of types of services; an output unit that outputs service information including information on the type of service determined by the determination unit to the autonomous moving body; A mobile system comprising: (Appendix 12) The computer acquiring first image data representing an image obtained by capturing an image of an object with an imaging device; determining a type of service targeted at the object from among a plurality of types of services based on the acquired first image data; outputting service information including information on the type of service determined as the type of service targeting the object to a service provider; A service determination method for performing the above. (Appendix 13) On the computer, acquiring first image data representing an image obtained by capturing an image of an object with an imaging device; determining a type of service targeted at the object from among a plurality of types of services based on the acquired first image data; outputting service information including information on the type of service determined as the type of service targeting the object to a service provider; A service judgment program that executes the above. [Explanation of symbols]

[0073] REFERENCE SIGNS LIST 1 Management system, 2 Autonomous mobile object, 3 User, 4 Portable device, 5 Wireless communication device, 6 Position detection device, 7 Image analysis device, 8 Image management device, 9 Service determination device, 10 Acquisition unit, 11 Learning unit, 12 Storage unit, 13 Determination unit, 14 Output unit, 100a Processor, 100b Memory, 200 Dedicated hardware

Claims

1. an acquisition unit that acquires first image data representing an image obtained by an imaging device capturing an image of an object; a determination unit that determines a type of service for the object from among a plurality of types of services based on the first image data acquired by the acquisition unit; an output unit that outputs service information including information on the type of service determined by the determination unit to a service provider; A service determination device comprising:

2. The determination unit determines a location where the service is to be provided based on the image capturing position information of the first image data, the location of the image capturing device when the object is captured, and The output unit outputs the information on the location of the service together with the service information. The service determination device according to claim 1 .

3. The determination unit determines a location where the service will be provided based on the photographing position information, the location being calculated by comparing the first image data with the environmental map information as photographing position information of the first image data, and The output unit outputs the information on the location of the service together with the service information. The service determination device according to claim 1 .

4. The determination unit determines a type of service for the object based on the first image data and the shooting position information of the first image data.

4. The service determination device according to claim 2 or 3.

5. the output unit outputs the service information to an autonomous moving object that provides at least one of the plurality of types of services. The service determination device according to any one of claims 1 to 3.

6. the output unit outputs the service information to an autonomous moving object selected from a plurality of autonomous moving objects each providing at least one of the plurality of types of services based on a type of service included in the service information. The service determination device according to claim 5.

7. the acquisition unit acquires second image data representing an image captured by an autonomous moving body that has received the output of the service information from the output unit, the output unit outputs, to the autonomous moving body, information for providing the service with the object as a target when a subject of the image of the second image data matches the target object, and does not output, to the autonomous moving body, information for providing the service when a subject of the image of the second image data does not match the target object. The service determination device according to claim 5.

8. The determination of whether or not the subject of the image of the second image data matches the target object is performed based on imaging direction information of the first image data representing the orientation of the imaging device when the image of the first image data was captured and imaging direction information of the second image data representing the orientation of the autonomous moving body when the image of the second image data was captured. The service determination device according to claim 7.

9. A storage unit that stores a judgment model that is learned in advance based on a combination of image data representing an image and a type of service targeting the subject of the image, and that receives image data as input and outputs a type of service. Equipped with The determination unit determines a type of service for the object by using the determination model with the first image data as an input. The service determination device according to any one of claims 1 to 3.

10. An imaging device that captures an image; an acquisition unit that acquires first image data representing an image obtained by the imaging device capturing an image of an object; a determination unit that determines a type of service for the object from among a plurality of types of services based on the first image data acquired by the acquisition unit; an output unit that outputs service information including information on the type of service determined by the determination unit to a service provider; A service determination system comprising:

11. an acquisition unit that acquires first image data representing an image obtained by an imaging device capturing an image of an object; a determination unit that determines a type of service for the object from among a plurality of types of services based on the first image data acquired by the acquisition unit; an autonomous moving body that provides at least one of the plurality of types of services; an output unit that outputs service information including information on the type of service determined by the determination unit to the autonomous moving body; A mobile system comprising:

12. The computer acquiring first image data representing an image obtained by capturing an image of an object with an imaging device; determining a type of service for the object from among a plurality of types of services based on the acquired first image data; outputting service information including information on the type of service determined as the type of service targeting the object to a service provider; A service determination method for performing the above.

13. On the computer, acquiring first image data representing an image obtained by capturing an image of an object with an imaging device; determining a type of service for the object from among a plurality of types of services based on the acquired first image data; outputting service information including information on the type of service determined as the type of service targeting the object to a service provider; A service judgment program that executes the above.

Images