Service determination device, system, mobile body system, service determination method, and program

Abstract

Provided are a service determination device, a system, a mobile body system, a service determination method, and a program, which enable a user of a service for which an object is a target to use an appropriate service with less effort. A service determination device (9) is provided with an acquisition unit (10), a determination unit (13), and an output unit (14). The acquisition unit (10) acquires image data indicating an image obtained by imaging an object by an imaging device such as a portable device (4). On the basis of the image data acquired by the acquisition unit (10), a determination unit (13) determines, from among a plurality of types of services, the type of service for which the object is the target. An output unit (14) outputs, to a provider of the service, service information including information on the service type determined by the determination unit (13).

Description

Technical Field

[0001] The present invention relates to a service judgment device, a service judgment system, a mobile system, a service judgment method and a service judgment program. Background Art

[0002] Patent document 1 discloses an example of a document management system. The document management system manages the packaging body for storing documents. The document management system has a portable terminal and an information processing device. The portable terminal has a photographing unit, a position identification unit, and a portable terminal side control unit. The photographing unit obtains photographing data of a management tag. The manager information and storage period of the document are recorded in the management tag, and identification information is given. The position identification unit identifies its own position information. The portable terminal side control unit sends the storage location information and photographing data of the packaging body identified based on the position information. The information processing device has a storage unit and an information processing device side control unit. The storage unit stores management information that corresponds the manager information and storage period identified based on the photographing data, the storage location information, and the identification information for each identification information. The information processing device side control unit compares the storage period with the current date and time for each management information, and issues a warning to the manager if there is management information whose date and time is close to the storage period.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2020-13337

[0004] In the system of Patent Document 1, each item of the management label, which is the basis of the management information, is written by the user using a pen, etc. Therefore, a user who wants to use a service such as document management for an object such as a package body needs to perform a photographing operation of the object after performing operations such as writing the label, etc. Therefore, it may take time to use the service. Summary of the invention

[0005] The present invention relates to solving such problems and provides a service determination device, a service determination system, a mobile system, a service determination method, and a service determination program, which enable users of services targeting objects to use appropriate services with less effort.

[0006] The service judgment device of the present invention comprises: an acquisition unit, which acquires first image data representing an image obtained by photographing an object by a photographing device; a judgment unit, which judges the type of service with the object as the object from a plurality of services based on the first image data acquired by the acquisition unit; and an output unit, which outputs service information including information on the service type judged by the judgment unit to a service provider.

[0007] The service judgment system of the present invention comprises: a photographing device that photographs an image; an acquisition unit that acquires first image data representing an image of an object photographed by the photographing device; a judgment unit that judges the type of service with the object as the object from a plurality of services based on the first image data acquired by the acquisition unit; and an output unit that outputs service information including information on the type of service judged by the judgment unit to a service provider.

[0008] The mobile body system of the present invention comprises: an acquisition unit, which acquires first image data representing an image obtained by photographing an object by a photographing device; a judgment unit, which judges the type of service with the object as the object from a plurality of services based on the first image data acquired by the acquisition unit; and an autonomous mobile body, which provides at least any one of the plurality of services; and an output unit, which outputs service information including information on the service type judged by the judgment unit to the autonomous mobile body.

[0009] The service judgment method of the present invention is a method in which a computer performs the following processing: obtaining first image data representing an image obtained by a photographing device photographing an object; judging the type of service with the object as the object from a plurality of services based on the obtained first image data; and outputting service information including information on the type of service judged as the type of service with the object as the object to a service provider.

[0010] The service judgment program of the present invention enables a computer to perform the following processing: obtaining first image data representing an image obtained by a camera capturing an object; judging the type of service with the object as the object from a plurality of services based on the obtained first image data; and outputting service information including information on the type of service judged as the type of service with the object as the object to a service provider.

[0011] According to the service determination device, service determination system, mobile system, service determination method or service determination program of the present invention, it is possible for users of services targeting objects to use appropriate services with less effort. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a structural diagram of the management system according to the first implementation mode.

[0013] Figure 2 This is a diagram for explaining an example of learning of a judgment model in the management system according to the first embodiment.

[0014] Figure 3 This is a diagram for explaining an example of determination of a service in the management system according to the first embodiment.

[0015] Figure 4This is a flowchart showing an example of the operation of the management system according to the first embodiment.

[0016] Figure 5 This is a flowchart showing an example of the operation of the management system according to the first embodiment.

[0017] Figure 6 This is a hardware configuration diagram of the main components of the management system according to the first embodiment.

[0018] Description of symbols

[0019] 1: Management system; 2: Autonomous mobile body; 3: User; 4: Portable device; 5: Wireless communicator; 6: Position detection device; 7: Image analysis device; 8: Image management device; 9: Service judgment device; 10: Acquisition unit; 11: Learning unit; 12: Storage unit; 13: Judgment unit; 14: Output unit; 100a: Processor; 100b: Memory; 200: Dedicated hardware. DETAILED DESCRIPTION

[0020] Referring to the accompanying drawings, the method for implementing the object of the present invention is described. In each figure, the same or corresponding parts are marked with the same reference numerals, and repeated descriptions are appropriately simplified or omitted. In addition, the object of the present invention is not limited to the following embodiments, and any structural element of the embodiment can be deformed or omitted within the scope of the present invention.

[0021] Implementation Method 1

[0022] Figure 1 This is a configuration diagram of the management system 1 according to the first embodiment.

[0023] The management system 1 is applied to a facility. The management system 1 is a system for managing the operation of the facility. The facility is, for example, an indoor facility or an outdoor facility, or a facility composed of these facilities. The facility is, for example, composed of one or more buildings. The facility may also be a part of a building.

[0024] In the facility, one or more autonomous mobile bodies 2 are operating. Each autonomous mobile body 2 is a robot, drone, or other movable device that moves autonomously in the facility. The autonomous mobile body 2 may also move in the facility according to movement instructions from a mobile body management device (not shown), for example. The mobile body management device is composed of, for example, one or more server computers, etc. Each autonomous mobile body 2 and the mobile body management device, etc. are connected to a local network such as a LAN (Local Area Network) in the facility or a wide area network such as the Internet. Each autonomous mobile body 2 moves in the facility, for example, to provide services such as cleaning, collection, or lost property collection. Each autonomous mobile body 2 is equipped with a function that can provide one or more of a variety of services such as cleaning, collection, or lost property collection.

[0025] Each autonomous mobile body 2 constitutes a mobile body system that provides services to users 3 in the facility. Part or all of the mobile body system may be an internal system included in the management system 1, or an external system applied to the management system 1 from the outside. In the facility of this example, a plurality of autonomous mobile bodies 2 operate. The plurality of autonomous mobile bodies 2 include autonomous mobile bodies 2 that can provide different types of services.

[0026] The services provided by the autonomous mobile body 2 and the like in the management system 1 are carried out with objects as objects. For example, the cleaning service provided by the autonomous mobile body 2 and the like includes a service of removing garbage or dirt in the facility as objects. For example, the collection service provided by the autonomous mobile body 2 and the like includes a service of collecting transported objects such as documents or packages in the facility as objects and collecting the objects to a transportation center. A transportation center is, for example, a place where transported objects are transported outside the facility. For example, the lost property collection service provided by the autonomous mobile body 2 and the like includes a service of collecting lost objects such as dropped objects or forgotten objects in the facility as objects and collecting the objects to the administrator's room of the facility. The administrator's room is, for example, a place where administrators who perform management business such as the operation of the facility reside. In the mobile body system, the autonomous mobile body 2 may also provide other services.

[0027] The user 3 of the facility uses the services provided by the autonomous mobile body 2, etc. via the portable device 4 he / she possesses. The portable device 4 is, for example, a mobile information processing device such as a smartphone. The portable device 4 is equipped with a wireless communication function. Wireless communication includes, for example, the IEEE 802.11 standard and the IEEE 802.15.1 standard and their successor standards, alternative standards, related standards, or similar standards. The portable device 4 is connected to a network such as the Internet or a telephone line network. The portable device 4 is equipped with a device having a function of taking pictures, such as a camera. The portable device 4 is an example of a photographing device. In this example, the portable device 4 is equipped with a function of obtaining its own position and direction information. The portable device 4 can obtain its own location information through a satellite positioning system such as GPS (Global Positioning System), or through autonomous navigation such as PDR (Pedestrian Dead Reckoning) using an accelerometer or the like, or through an indoor positioning system using a wireless signal such as UWB (Ultra-Wide Band) signal, or by other methods. The portable device 4 can obtain its own direction information by using an accelerometer or the like, or by using a method using a phase difference of a wireless signal to be received such as AoA (Angle of Arrival), or by other methods. The portable device 4 can be an internal device included in the management system 1, or an external device outside the management system 1. The portable device 4 operates based on a program such as pre-installed application software, thereby realizing the functions in the management system 1.

[0028] The management system 1 has one or more wireless communicators 5. The wireless communicator 5 is a device that can communicate with the portable device 4 or the like using wireless signals. The wireless communicator 5 is, for example, a wireless beacon that transmits wireless signals to the portable device 4. The wireless communicator 5 may also be equipped with a function of receiving wireless signals from the portable device 4. The wireless communicator 5 is arranged in the facility. The wireless communicator 5 is connected to a local area network such as a LAN in the facility or a wide area network such as the Internet.

[0029] The management system 1 includes a position detection device 6, an image analysis device 7, an image management device 8, and a service judgment device 9. The position detection device 6, the image analysis device 7, the image management device 8, and the service judgment device 9 are each composed of, for example, one or more server computers. Part or all of the position detection device 6, the image analysis device 7, the image management device 8, and the service judgment device 9 may be installed in the facility or outside the facility. Part or all of the functions of the position detection device 6, the image analysis device 7, the image management device 8, and the service judgment device 9 may also be installed by processing or storage resources on a cloud service. Part or all of the functions of the position detection device 6, the image analysis device 7, the image management device 8, and the service judgment device 9 may also be mounted on the same hardware. Part or all of the functions of the position detection device 6, the image analysis device 7, the image management device 8, and the service judgment device 9 are realized by, for example, each server computer executing processing according to a program. The program may also be a software package including a plurality of components that cause a plurality of server computers to operate respectively. The position detection device 6, the image analysis device 7, the image management device 8, and the service judgment device 9 are each connected to, for example, a local area network such as a LAN in the facility or a wide area network such as the Internet. The management system 1 has a function of determining the type of service provided by the autonomous mobile body 2 or the like to the facility user 3. The management system 1 is an example of a service determination system. The facility user 3 transmits image data of an object image captured by a portable device 4 or the like to the management system 1, thereby utilizing the service of the mobile body system or the like.

[0030] The position detection device 6 is a device equipped with a function of detecting the position of the portable device 4 held by the user 3 of the facility. The position detection device 6 may also detect the direction of the portable device 4. The position detection device 6 may detect the position of the portable device 4 based on the signal strength or propagation time of the wireless signal communicated between one or more wireless communicators 5 and the portable device 4, or may detect the position of the portable device 4 using other methods. The position detection device 6 may detect the direction of the portable device 4 based on the phase difference of the wireless signal communicated between one or more wireless communicators 5 and the portable device 4, or may detect the direction of the portable device 4 using other methods. The position detection device 6 detects the position and direction of the portable device 4 using the position information of each wireless communicator 5 acquired in advance. In addition, in the management system 1, if it is assumed that the portable device 4 has a function of acquiring its own position and direction information, the management system 1 may not have the position detection device 6. In this case, the management system 1 uses the position and direction information acquired by the portable device 4 itself to perform necessary information processing.

[0031] The image analysis device 7 is a device equipped with a function of analyzing an image represented by image data sent by the user 3. The image analysis device 7 obtains image data of an image captured by the user 3 using the portable device 4, for example, directly from the portable device 4 by wireless communication or through a network such as a LAN or the Internet within the facility. The image analysis device 7 is equipped with a function of detecting a subject included in an image by using image processing technology or the like. The subject included in the image may include, for example, garbage or dirt, transported objects such as documents or packages, or lost objects such as dropped objects or forgotten objects, which may be the objects of the service provided by the autonomous mobile body 2 or the like. The image analysis device 7 may also have a function of detecting the background of the subject, etc. The image analysis device 7 may also extract a feature value of the image, etc., as the analysis result of the image.

[0032] The image management device 8 is a device equipped with a function of managing the image data sent by the user 3. The image management device 8 stores, for example, the image data sent by the user 3. The image management device 8 stores, for example, the position information of the portable device 4 when the portable device 4 captured the image of the image data together with the image data as the shooting position information. The image management device 8 stores, for example, the direction information of the portable device 4 when the portable device 4 captured the image of the image data together with the image data as the shooting direction information. The position and direction information of the portable device 4 may be, for example, information obtained by the portable device 4 itself, or information detected by the position detection device 6. In addition, the image management device 8 stores, for example, information on the analysis result of the image of the image data by the image analysis device 7 together with the image data. The information on the analysis result of the image analysis device 7 may be information on the detected subject, an image obtained by normalizing the position, size, direction, color tone, etc. of the detected subject, or information on the feature quantity of the extracted image.

[0033] The service determination device 9 is a device that performs information processing such as determination of the type of service provided by the autonomous mobile body 2 to the facility user 3. The service determination device 9 includes an acquisition unit 10, a learning unit 11, a storage unit 12, a determination unit 13, and an output unit 14.

[0034] The acquisition unit 10 is a part equipped with a function of acquiring image data representing an image of an object captured by an image capturing device such as the portable device 4. The acquisition unit 10 acquires image data managed by the image management device 8 from the image management device 8 via a network such as a LAN in a facility or the Internet. The image data acquired by the acquisition unit 10 may include, for example, information representing the image, information on the capturing position and the capturing direction, and information on the analysis result of the image analysis device 7. This image data is an example of the first image data.

[0035] The learning unit 11 is a part equipped with a function of learning a judgment model. The judgment model is, for example, a learned model that takes image data as input and takes the service type with the subject of the image of the image data as output. The learning unit 11 learns the judgment model by a machine learning method such as supervised learning, using, for example, a training data set. The training data set includes a plurality of training data. Each training data is, for example, data including a combination of image data representing an image and a service type with the subject of the image as the object. The learning unit 11 may also store training data used for learning the judgment model. The learning unit 11 may also use other machine learning methods to learn the judgment model.

[0036] The storage unit 12 is a part equipped with a function of storing information. The storage unit 12 stores, for example, a judgment model etc. generated in advance by learning by the learning unit 11. The storage unit 12 may also store a judgment model etc. learned in advance outside the management system 1.

[0037] The judgment unit 13 is a part equipped with a function of judging the type of service provided to the user 3 from among a plurality of services provided by the autonomous mobile body 2 and the like in the facility. The judgment unit 13 judges the type of service for the subject included in the image of the image data based on the image data acquired by the acquisition unit 10. The judgment unit 13 may, for example, use a learned judgment model pre-stored in the storage unit 12 and judge the type of service by using the output obtained by inputting the image data into the judgment model. The judgment unit 13 may also use the shooting position information and the shooting direction information together when judging the type of service. The judgment unit 13 may also judge the type of service and also judge the implementation location where the autonomous mobile body 2 and the like implement the service.

[0038] The output unit 14 is a part equipped with a function of outputting service information about the service provided to the user 3 to the provider of the service. The service information includes, for example, information on the type of service determined by the determination unit 13. The service information may also include, for example, information on the implementation location determined by the determination unit 13. The output unit 14 may also have a function of determining the output destination of the service information. The output unit 14 outputs the service information to, for example, an information processing terminal installed in an administrator's room or the like. The administrator of the facility refers to the input service information, thereby being able to quickly respond to the requested service. The output unit 14 selects, for example, an autonomous mobile body 2 based on the type of service included in the service information, and outputs the service information to the selected autonomous mobile body 2. The output unit 14 selects an autonomous mobile body 2 that can provide the type of service included in the service information from among the autonomous mobile bodies 2 operating in the facility. When the autonomous mobile body 2 is controlled by a mobile body control device or the like, the output unit 14 may output the service information to the autonomous mobile body 2 via the mobile body control device. The autonomous mobile body 2 that receives the input of the service information moves in the facility and implements the service corresponding to the service information. The autonomous mobile body 2 and the administrator of the facility are examples of service providers. In addition, for example, the administrator of the facility does not need to provide the service directly, but may provide the service to the user 3 through a service provider or the like.

[0039] Next, use Figure 2 , an example illustrating the learning of the judgment model.

[0040] Figure 2 This is a diagram for explaining an example of learning of a judgment model in the management system 1 according to the first embodiment.

[0041] The image analysis device 7 is equipped with a subject detection model. The subject detection model is, for example, a learned model that takes image data as input and outputs the subject of the image of the image data. The subject detection model may be pre-learned outside the management system 1 or may be learned in the management system 1. The image analysis device 7 detects the subject of the image of the image data with respect to the input image data, for example, using the subject detection model.

[0042] Next, the type of service for the subject of the image is determined with respect to the image represented by the image data. This determination is performed, for example, by an administrator of a facility using the management system 1 or an administrator of the management system 1 itself. At this time, the information of the subject detected by the image analysis device 7 may be used to determine the type of service, or may not be used to determine the type of service.

[0043] The learning unit 11 accumulates a combination of image data and a service type determined with respect to the image data as training data. The training data may also include shooting position information and shooting direction information with respect to the image data. The training data may also include detection result information of a subject of the image of the image data. The learning unit 11 accumulates a plurality of training data as a training data set in, for example, a database.

[0044] The learning unit 11 is equipped with a learning engine. The learning engine uses, for example, a training data set that is fully accumulated through a machine learning method such as supervised learning to output a judgment model. The learning engine may also use, for example, AI technologies (AI: Artificial Intelligence) such as decision trees or deep learning. In addition, the training data may not include the detection result information of the subject of the image of the image data. In this case, the judgment model may have a function of detecting the subject of the image internally, or may directly output the service type from the image data.

[0045] The management system 1 uses the judgment model learned in this way to operate. In addition, the management system 1 can also use the image data and other data obtained during operation to accumulate new training data sets and update the judgment model at any time. At this time, the management system 1 can also improve the learning efficiency and learning accuracy by reducing the weight of the old training data or deleting the old training data.

[0046] In addition, when the management system 1 is running, the service type determined by the determination unit 13 may be presented to the portable device 4 of the user 3. At this time, the user 3 may correct the service type if the presented service type is incorrect. The corrected service type is stored in the learning unit 11 together with the image data as training data after being corrected. In addition, the service type that has not been corrected may also be directly stored in the learning unit 11 together with the image data as training data. In this way, the management system 1 can further improve the judgment accuracy of the service during operation.

[0047] Next, use Figure 3 , an example of determining the type of service is described below.

[0048] Figure 3 This is a diagram for explaining an example of determination of a service in the management system 1 according to the first embodiment.

[0049] For example, the user 3 uses the portable device 4 to capture an image of a transport object packaged in a corrugated cardboard box in an office area on the 10th floor where the user 3 works in a facility. At this time, the portable device 4 obtains the current position as the shooting position information. The user 3 transmits image data representing the image captured by the portable device 4 to the image management device 8 via the image analysis device 7 or the like. The image data includes the shooting position information. The acquisition unit 10 obtains image data representing an image including a package such as a corrugated cardboard box in the subject and including information of the office area on the 10th floor as the shooting position information of the portable device 4. At this time, the judgment unit 13 judges that the type of service to be provided is a collection service that collects the package as an object and collects it at a delivery center, for example, based on a judgment model or the like. The output unit 14 generates service information to be output based on the judgment result. The service information includes the shooting position information corresponding to the place where the object was placed when the image was captured as the place where the service was implemented. The output unit 14 selects the autonomous mobile body 2 of the output destination based on the service information. The output unit 14 selects, for example, the autonomous mobile body 2 that can implement the collection service. The output unit 14 outputs the service information to the selected autonomous mobile body 2. The autonomous mobile body 2 that receives the service information moves to the office area on the 10th floor, which is the place where the service is implemented, and collects the transported object. Then, the autonomous mobile body 2 transports the collected transported object to the transport center.

[0050] For example, user 3 finds a transported object packaged in a corrugated cardboard box in the restroom area on the 10th floor of the facility. Usually, transported objects waiting for collection are not placed in the restroom area. Therefore, user 3, who thinks that the transported object is a forgotten object, uses portable device 4 to take an image of the corrugated cardboard box as a forgotten object. At this time, portable device 4 obtains the current position as shooting position information. User 3 sends image data representing the image captured by portable device 4 to image management device 8 through image analysis device 7 or the like. The image data includes shooting position information. Acquisition unit 10 acquires image data representing an image including a packaging body such as a corrugated cardboard box in the subject and including information of the restroom area on the 10th floor as shooting position information of portable device 4. At this time, judgment unit 13, for example, determines the type of service to be provided as a lost property collection service that collects the forgotten object as an object and collects it in the administrator's room based on a judgment model or the like. Output unit 14 generates service information to be output based on the judgment result. The service information includes shooting position information corresponding to the place where the object was placed when shooting as the implementation place of the service. The output unit 14 selects an autonomous mobile body 2 as the output destination based on the service information. For example, the output unit 14 selects an autonomous mobile body 2 that can provide lost property recovery service. The output unit 14 outputs the service information to the selected autonomous mobile body 2. The autonomous mobile body 2 that receives the service information moves to the restroom area on the 10th floor, which is the place where the service is provided, and recovers the lost property that is the target. Then, the autonomous mobile body 2 transports the recovered lost property to the administrator's office.

[0051] User 3, for example, finds a smartphone in the restroom area on the 13th floor of a facility. Usually, a smartphone that is not a transported object is not placed in a facility, so user 3, who thinks that the transported object is a dropped object, uses portable device 4 to capture an image of the smartphone as a dropped object. At this time, portable device 4 obtains the current position as shooting position information. User 3 sends image data representing an image captured by portable device 4 to image management device 8 through image analysis device 7 or the like. The image data includes shooting position information. Acquisition unit 10 acquires image data representing an image containing a dropped object such as a smartphone in the subject and including information of the restroom area on the 13th floor as shooting position information of portable device 4. At this time, judgment unit 13, for example, determines the type of service to be provided as a lost property collection service that collects dropped objects as objects and gathers them in the administrator's room based on a judgment model or the like. Output unit 14 generates service information to be output based on the judgment result. The service information includes shooting position information corresponding to the place where the object was placed when the object was captured as the implementation place of the service. The output unit 14 selects an autonomous mobile body 2 as an output destination based on the service information. For example, the output unit 14 selects an autonomous mobile body 2 that can provide lost property recovery service. The output unit 14 outputs the service information to the selected autonomous mobile body 2. The autonomous mobile body 2 that receives the service information moves to the restroom area on the 13th floor, which is the place where the service is provided, and recovers the lost property that is the target. Then, the autonomous mobile body 2 transports the recovered lost property to the administrator's office.

[0052] For example, a user 3 who spilled a drink in a dining area on the fifth floor of a facility uses a portable device 4 to take an image of the spilled drink. At this time, the portable device 4 obtains the current position as the shooting position information. The user 3 sends image data representing the image captured by the portable device 4 to the image management device 8 through the image analysis device 7 or the like. The image data includes the shooting position information. The acquisition unit 10 obtains image data representing an image including the spilled drink in the subject and including information of the dining area on the fifth floor as the shooting position information of the portable device 4. At this time, the judgment unit 13 judges that the type of service to be provided is a cleaning service that removes dirt generated by the spilled drink as an object based on, for example, a judgment model. The output unit 14 generates service information to be output based on the judgment result. The service information includes the shooting position information corresponding to the place where the object was placed when the image was captured as the place where the service was implemented. The output unit 14 selects the autonomous mobile body 2 as the output destination based on the service information. The output unit 14 selects, for example, an autonomous mobile body 2 that can implement the cleaning service. The output unit 14 outputs the service information to the selected autonomous mobile body 2. The autonomous mobile body 2 that receives the service information moves to the dining area on the fifth floor, which is the place where the service is to be performed, and removes dirt from the object.

[0053] Next, use Figure 4 and Figure 5 , an example of the operation of management system 1 is described.

[0054] Figure 4 and Figure 5 This is a flowchart showing an example of the operation of the management system 1 according to the first embodiment.

[0055] exist Figure 4 , an example of the operation during learning of the judgment model etc. is shown. Figure 4 The operations shown may be performed in advance before starting the operation of the management system 1 , may be performed in parallel with the operation, or may be performed periodically during the operation.

[0056] In step S101, the portable device 4 held by the user 3 or the administrator captures an image with the service object as the subject according to the input operation. Then, in step S102, the portable device 4 transmits the captured image of the object to the image management device 8 through the wireless communication device 5, the position detection device 6, and the image analysis device 7 installed in the facility according to the input operation. Then, in step S103, the position detection device 6 adds the capture position information and the capture direction information of the portable device 4 when capturing the image to the image data of the image.

[0057] Then, in step S104, the acquisition unit 10 of the service determination device 9 acquires the image data from the image management device 8. The acquisition unit 10 acquires information such as detection information of the subject of the image and the determination result of the service type with the subject as the object from the portable device 4 or other information processing device held by the administrator or the like. At this time, the acquisition unit 10 may also acquire information such as whether the user 3 has corrected the service type and the corrected service type if there is a correction from the portable device 4 held by the user 3. Then, the processing of the management system 1 proceeds to step S105.

[0058] In step S105, the learning unit 11 combines the judgment result of the service type and the like with the image data including the shooting position information and the shooting direction information and accumulates them in the training data set as training data. Then, the processing of the management system 1 proceeds to step S106.

[0059] In step S106, the learning unit 11 determines whether the data accumulated as the training data set is sufficient. The learning unit 11 may make this determination based on, for example, the amount of data in the training data set, or may make this determination based on the completeness of the data in the training data set. If it is determined that the accumulated data is sufficient, the management system 1 proceeds to step S107. On the other hand, if it is determined that the accumulated data is insufficient, the management system 1 continues to collect training data. Then, the process from step S101 is repeated.

[0060] In step S107, the learning unit 11 outputs the judgment model using the accumulated training data set and the learning engine. The storage unit 12 stores the judgment model to be output. Then, the management system 1 completes the process of learning the judgment model and the like.

[0061] exist Figure 5 , an example of the operation when the management system 1 is operated is shown. In this example, the autonomous mobile body 2 is equipped with a device having a function of capturing images, such as a camera. In addition, the capturing position information and the capturing direction information when the autonomous mobile body 2 captures an image are obtained in the same manner as the capturing position information and the capturing direction information when the portable device 4 captures an image.

[0062] In step S201, the portable device 4 held by the user 3 captures an image with the service object as the subject according to the operation of the user 3. Then, in step S202, the portable device 4 transmits the captured image of the object to the image management device 8 through the wireless communication device 5, the position detection device 6, and the image analysis device 7 installed in the facility according to the operation of the user 3. Then, in step S203, the position detection device 6 adds the shooting position information and shooting direction information of the portable device 4 when the image was captured to the image data of the image.

[0063] Then, in step S204, the acquisition unit 10 of the service determination device 9 acquires the image data from the image management device 8. Then, the process of the management system 1 proceeds to step S205.

[0064] In step S205, the determination unit 13 determines the service type using the output obtained by inputting the image data including the shooting position information and the shooting direction information into the determination model stored in the storage unit 12. Then, the processing of the management system 1 proceeds to step S206.

[0065] In step S206, the output unit 14 determines the output destination of the service information including the information of the service type determined by the determination unit 13. In this example, the output unit 14 selects any autonomous mobile body 2 operating in the facility as the output destination according to the service type. The output unit 14 outputs the service information to the selected autonomous mobile body 2. Then, the processing of the management system 1 proceeds to step S207.

[0066] In step S207, the autonomous mobile body 2 that receives the service information moves to the place where the service is implemented. The autonomous mobile body 2 photographs the object specified in the service information at the place where the service is implemented. The autonomous mobile body 2 sends image data representing the image of the photographed object to the service determination device 9. The image data sent at this time includes the shooting position information and the shooting direction information when the autonomous mobile body 2 photographed the image. This image data is an example of the second image data. Then, the processing of the management system 1 enters step S208.

[0067] In step S208, the acquisition unit 10 of the service determination device 9 acquires image data representing the image captured by the autonomous mobile body 2. The output unit 14 determines whether the subject of the image captured by the autonomous mobile body 2 is consistent with the object for which the service type is determined. At this time, the output unit 14 may determine the consistency with the object by image analysis or the like. In addition, the output unit 14 may determine the consistency with the object by using the shooting position information and shooting direction information of the image data captured by the portable device 4 and the shooting position information and shooting direction information of the image data captured by the autonomous mobile body 2. For example, when these shooting position information or shooting direction information are inconsistent within a preset range, the output unit 14 determines that the subject of the image captured by the autonomous mobile body 2 is inconsistent with the object of the service even if the captured images themselves are similar. When it is determined that the subject of the image captured by the autonomous mobile body 2 is inconsistent with the object of the service, the processing of the management system 1 for providing the service at runtime ends. At this time, the output unit 14 may output information for terminating the provision of the service to the autonomous mobile body 2. On the other hand, when it is determined that the subject of the image captured by the autonomous mobile object 2 matches the target object of the service, the processing of the management system 1 proceeds to step S209.

[0068] In step S209, the output unit 14 outputs information on the execution of service provision to the autonomous mobile body 2. The autonomous mobile body 2 receiving the information executes the service provision for the object. After the service provision is completed, the processing of the management system 1 for providing the service at runtime ends.

[0069] As described above, the service determination device 9 of embodiment 1 includes an acquisition unit 10, a determination unit 13, and an output unit 14. The acquisition unit 10 acquires image data representing an image obtained by a portable device 4 or other imaging device when the object is photographed. The determination unit 13 determines the type of service for the object from a plurality of services based on the image data acquired by the acquisition unit 10. The output unit 14 outputs service information including information on the type of service determined by the determination unit 13 to the provider of the service. According to such a structure, the user 3 can use appropriate services for the object only by using the portable device 4 to capture and send an image of the object. In addition, the provider of the service can obtain the service information in a state where the type of service to be provided is automatically determined, thereby making the business of providing the service more efficient.

[0070] In addition, the judgment unit 13 uses the position of the portable device 4 when the object is photographed as the shooting position information, and judges the implementation place of the service based on the shooting position information. The output unit 14 outputs the implementation place information included in the service information. According to such a structure, the service is quickly provided at the implementation place where the service is to be provided. For example, in the case where the autonomous mobile body 2 provides the service, the autonomous mobile body 2 is moved to the implementation place, thereby enabling the service to be quickly provided. In addition, in the case where the service is provided by the staff of the service provider, the service can be quickly provided by contacting the implementation place with the staff. In addition, the service information may not necessarily include the information of the implementation place of the service. For example, in the case where the content or implementation place of the service to be implemented is clear according to the type of service, or in the case where the type of service is, for example, for the entire facility, the service information may not include the information of the implementation place of the service.

[0071] In addition, the judgment unit 13 judges the type of service for the object based on the image data and the shooting location information. According to such a structure, appropriate services are also provided for the same object such as a package body according to the place where it is placed. For example, appropriate services are provided in the following manners: when a package body such as corrugated paper is located in an office area, it is judged that a collection service should be provided, and when a package body such as corrugated paper is located in a toilet area, it is judged that a lost property recovery service should be provided. In addition, the judgment unit 13 may also judge the type of service without using the shooting location information. For example, when the content of the service to be implemented is clear based on the subject of the image, the judgment unit 13 may also judge the type of service without using the shooting location information. The judgment unit 13 may also judge, for example, that a cleaning service should be provided for an image in which a spilled beverage is set as the subject.

[0072] Furthermore, the output unit 14 outputs the service information to the autonomous mobile body 2 that provides the service. The output unit 14 outputs the service information to the autonomous mobile body 2 selected according to the service type included in the service information. According to such a configuration, the autonomous mobile body 2 suitable for the service with the object as the target is selected, and the service is provided to the user 3. As a result, the convenience of the user 3 is further improved.

[0073] In addition, the acquisition unit 10 acquires image data of an image captured by the autonomous mobile body 2 indicating the output of the received service information. When the subject of the image coincides with the object of the service, the output unit 14 outputs the information of providing the service to the autonomous mobile body 2. On the other hand, when the subject of the image coincides with the object of the service, the information of providing the service is not output to the autonomous mobile body 2. At this time, the output unit 14 may output information for terminating the provision of the service to the autonomous mobile body 2. According to such a structure, the service is provided after confirming that the autonomous mobile body 2 and the object of the service are facing each other. Thereby, the autonomous mobile body 2 is prevented from performing the service on the wrong object.

[0074] In addition, based on the shooting position information or shooting direction information of the image data representing the image shot by the portable device 4 and the shooting position information or shooting direction information of the image data representing the image shot by the autonomous mobile body 2, it is determined whether the subject of the image shot by the autonomous mobile body 2 is consistent with the object of the service. Here, one or both of the shooting position information and the shooting direction information can also be used in the determination. According to such a structure, it is more reliably suppressed that the autonomous mobile body 2 performs the service on the wrong object. Since the position of the object is determined by the shooting position information and the shooting direction information, for example, even when a similar object that is similar in appearance to the object is placed near the object, the similar object and the object can be more reliably distinguished. In addition, it is also possible to determine whether the subject of the image shot by the autonomous mobile body 2 is consistent with the object of the service without using the shooting position information or the shooting direction information.

[0075] In addition, the service determination device 9 includes a storage unit 12. The storage unit 12 stores a determination model. The determination model is a learned model that is pre-learned by a combination of image data representing an image and a service type with a subject of the image as an object, and takes the image data as an input and the service type as an output. The determination unit 13 uses the determination model by taking the image data as an input, thereby determining the service type with the object as an object. According to such a structure, it is possible to determine the service type by a method such as machine learning, thereby making a determination that is more in line with the actual situation of the facility.

[0076] In addition, the photographing device may not be the portable device 4. The photographing device may be, for example, a surveillance camera installed in a facility. In this case, the information of the location where the surveillance camera is installed may also be processed in the management system 1 as the shooting location information. In addition, in the case where the photographing device is a surveillance camera, there may be a distance between the location of the photographing device itself and the location of the subject. In this case, the judgment unit 13 may also set the location calculated by comparing the image data with the environmental map information as the shooting location information of the image data. The environmental map information may be, for example, the location and appearance information of architectural structures such as walls, floors, ceilings, columns and beams in the facility, or the location and appearance information of configuration objects such as equipment and furniture installed in the facility. The environmental map information may also be, for example, photograph data of architectural structures or configuration objects. The judgment unit 13 may also obtain the location of the object by, for example, comparing the background of the image captured by the photographing device with the environmental map information.

[0077] Next, use Figure 6 , an example of the hardware structure of the management system 1 is described.

[0078] Figure 6 This is a hardware configuration diagram of main components of the management system 1 according to the first embodiment.

[0079] Each function of the management system 1 can be realized by a processing circuit. The processing circuit has at least one processor 100a and at least one memory 100b. The processing circuit may also have at least one dedicated hardware 200 together with the processor 100a and the memory 100b or instead of them.

[0080] When the processing circuit has a processor 100a and a memory 100b, each function of the management system 1 is implemented by software, firmware, or a combination of software and firmware. At least one of the software and firmware is recorded as a program. The program is stored in the memory 100b. The processor 100a implements each function of the management system 1 by reading and executing the program stored in the memory 100b.

[0081] The processor 100a is also called a CPU (Central Processing Unit), a processing device, an arithmetic device, a microprocessor, a microcomputer, or a DSP. The memory 100b is composed of a nonvolatile or volatile semiconductor memory such as a RAM, a ROM, a flash memory, an EPROM, or an EEPROM.

[0082] In the case where the processing circuit has dedicated hardware 200, the processing circuit is implemented by, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0083] Each function of the management system 1 can be implemented by a processing circuit respectively. Alternatively, each function of the management system 1 can also be implemented uniformly by a processing circuit. Regarding each function of the management system 1, a part can also be implemented by dedicated hardware 200, and another part can be implemented by software or firmware. In this way, the processing circuit implements each function of the management system 1 through dedicated hardware 200, software, firmware or a combination thereof.

[0084] To summarize the above description, the configurations that can be adopted by the technology of the present invention include the configurations shown below as supplementary notes.

[0085] (Note 1)

[0086] A service determination device, wherein the service determination device comprises:

[0087] an acquisition unit that acquires first image data representing an image obtained by an image pickup device capturing an object;

[0088] a determination unit that determines a type of service for the object from among a plurality of services based on the first image data acquired by the acquisition unit; and

[0089] An output unit outputs service information including information on the service type determined by the determination unit to a service provider.

[0090] (Note 2)

[0091] The service determination device according to Supplement 1, wherein:

[0092] The determination unit uses the position of the imaging device when the object is imaged as the imaging position information of the first image data, and determines the implementation location of the service based on the imaging position information.

[0093] The output unit outputs the service information by including the information of the implementation location.

[0094] (Note 3)

[0095] The service determination device according to Supplement 1, wherein:

[0096] The determination unit uses the position calculated by comparing the first image data with the environment map information as the shooting position information of the first image data, and determines the implementation place of the service based on the shooting position information.

[0097] The output unit outputs the service information by including the information of the implementation location.

[0098] (Note 4)

[0099] The service determination device according to Supplement 2 or Supplement 3, wherein:

[0100] The determination unit determines the type of service for the object based on the first image data and the photographing position information of the first image data.

[0101] (Note 5)

[0102] The service determination device according to any one of Supplementary Notes 1 to 4, wherein:

[0103] The output unit outputs the service information to an autonomous mobile object that provides at least any one of the plurality of services.

[0104] (Note 6)

[0105] The service determination device according to Supplement 5, wherein:

[0106] The output unit outputs the service information to an autonomous mobile object selected according to a service type included in the service information, among a plurality of autonomous mobile objects that respectively provide at least any one of the plurality of services.

[0107] (Note 7)

[0108] The service determination device according to Supplement 5 or Supplement 6, wherein:

[0109] The acquisition unit acquires second image data representing an image captured by the autonomous mobile object that receives output of the service information from the output unit,

[0110] When the subject of the image of the second image data is consistent with the object, the output unit will output the information of providing the service with the object as the object to the autonomous mobile body. When the subject of the image of the second image data is inconsistent with the object, the output unit will not output the information of providing the service to the autonomous mobile body.

[0111] (Note 8)

[0112] The service determination device according to Supplement 7, wherein:

[0113] Based on the shooting direction information of the first image data indicating the orientation of the shooting device when the image of the first image data was captured, and the shooting direction information of the second image data indicating the orientation of the autonomous mobile body when the image of the second image data was captured, it is determined whether the subject of the image of the second image data is consistent with the object.

[0114] (Note 9)

[0115] The service determination device according to any one of Supplementary Notes 1 to 8, wherein:

[0116] The service determination device includes a storage unit that stores a determination model that has been pre-learned using a set of image data representing an image and a service type for a subject of the image, and that takes the image data as input and the service type as output.

[0117] The determination unit determines the type of service for the object by using the determination model by taking the first image data as input.

[0118] (Note 10)

[0119] A service judgment system, wherein the service judgment system has:

[0120] a camera that captures images;

[0121] an acquisition unit that acquires first image data representing an image obtained by capturing an object by the imaging device;

[0122] a determination unit that determines a type of service for the object from among a plurality of services based on the first image data acquired by the acquisition unit; and

[0123] An output unit outputs service information including information on the service type determined by the determination unit to a service provider.

[0124] (Note 11)

[0125] A mobile system, wherein the mobile system has:

[0126] an acquisition unit that acquires first image data representing an image obtained by an image pickup device capturing an object;

[0127] a determination unit that determines a type of service for the object from among a plurality of services based on the first image data acquired by the acquisition unit;

[0128] an autonomous mobile body that provides at least any one of the plurality of services; and

[0129] An output unit outputs service information including information on the service type determined by the determination unit to the autonomous mobile object.

[0130] (Note 12)

[0131] A service determination method, wherein a computer performs the following processing:

[0132] Acquiring first image data representing an image obtained by photographing an object with a photographing device;

[0133] determining a type of service for the object from a plurality of services based on the acquired first image data; and

[0134] The service information including the information of the service type determined as the service type for the object is output to the provider of the service.

[0135] (Note 13)

[0136] A service determination program, wherein the service determination program causes a computer to execute the following processing:

[0137] Acquiring first image data representing an image obtained by photographing an object with a photographing device;

[0138] determining a type of service for the object from a plurality of services based on the acquired first image data; and

[0139] The service information including the information of the service type determined as the service type for the object is output to the provider of the service.

Claims

1. A service determination device, wherein: The service determination device comprises: an acquisition unit that acquires first image data representing an image obtained by an image pickup device capturing an object; a determination unit that determines a type of service for the object from among a plurality of services based on the first image data acquired by the acquisition unit; as well as An output unit outputs service information including information on the service type determined by the determination unit to a service provider.

2. The service determination device according to claim 1, wherein: The determination unit uses the position of the imaging device when the object is imaged as the imaging position information of the first image data, and determines the implementation location of the service based on the imaging position information. The output unit outputs the service information by including the information of the implementation location.

3. The service determination device according to claim 1, wherein: The determination unit uses the position calculated by comparing the first image data with the environment map information as the shooting position information of the first image data, and determines the implementation place of the service based on the shooting position information. The output unit outputs the service information by including the information of the implementation location.

4. The service determination device according to claim 2 or 3, wherein: The determination unit determines the type of service for the object based on the first image data and the photographing position information of the first image data.

5. The service determination device according to any one of claims 1 to 3, wherein: The output unit outputs the service information to an autonomous mobile object that provides at least any one of the plurality of services.

6. The service determination device according to claim 5, wherein: The output unit outputs the service information to an autonomous mobile object selected according to a service type included in the service information, among a plurality of autonomous mobile objects that respectively provide at least any one of the plurality of services.

7. The service determination device according to claim 5, wherein: The acquisition unit acquires second image data representing an image captured by the autonomous mobile object that receives output of the service information from the output unit, When the subject of the image of the second image data is consistent with the object, the output unit will output the information of providing the service with the object as the object to the autonomous mobile body. When the subject of the image of the second image data is inconsistent with the object, the output unit will not output the information of providing the service to the autonomous mobile body.

8. The service determination device according to claim 7, wherein: Based on the shooting direction information of the first image data indicating the orientation of the shooting device when the image of the first image data was captured, and the shooting direction information of the second image data indicating the orientation of the autonomous mobile body when the image of the second image data was captured, it is determined whether the subject of the image of the second image data is consistent with the object.

9. The service determination device according to any one of claims 1 to 3, wherein: The service determination device includes a storage unit that stores a determination model that has been pre-learned using a set of image data representing an image and a service type for a subject of the image, and that takes the image data as input and the service type as output. The determination unit determines the type of service for the object by using the determination model by taking the first image data as input.

10. A service judgment system, wherein: The service judgment system has: a camera that captures images; an acquisition unit that acquires first image data representing an image obtained by capturing an object by the imaging device; a determination unit that determines a type of service for the object from among a plurality of services based on the first image data acquired by the acquisition unit; as well as An output unit outputs service information including information on the service type determined by the determination unit to a service provider.

11. A mobile system, wherein: The mobile system has: an acquisition unit that acquires first image data representing an image obtained by an image pickup device capturing an object; a determination unit that determines a type of service for the object from among a plurality of services based on the first image data acquired by the acquisition unit; an autonomous mobile body providing at least any one of the plurality of services; as well as An output unit outputs service information including information on the service type determined by the determination unit to the autonomous mobile object.

12. A service determination method, wherein: The computer performs the following processing: Acquiring first image data representing an image obtained by photographing an object with a photographing device; determining, based on the acquired first image data, a type of service targeting the object from a plurality of services; as well as The service information including the information of the service type determined as the service type for the object is output to the provider of the service.

13. A service judgment program, wherein: The service determination program causes the computer to execute the following processing: Acquiring first image data representing an image obtained by photographing an object with a photographing device; determining, based on the acquired first image data, a type of service targeting the object from a plurality of services; as well as The service information including the information of the service type determined as the service type for the object is output to the provider of the service.

Images