Control system, control method, and storage medium storing program

Abstract

The present application relates to a control system, a control method, and a storage medium storing a program. The present application provides a control system that evaluates information sent from a user in interactive communication between the user and a mobile body using a dispatch service. The control system includes: a travel control unit that causes the mobile body to travel to a specified place specified by a dispatch request from the user in accordance with the dispatch request; a communication unit that performs interactive communication between the user and the mobile body via a portable terminal held by the user before the mobile body reaches the specified place; an acquisition unit that acquires image data of a surrounding of the specified place taken by a photographing unit after the mobile body reaches the specified place as a result of the interactive communication; and an evaluation unit that evaluates information sent from the user in the interactive communication based on the image data acquired by the acquisition unit.

Description

TECHNICAL FIELD

[0001] The present application relates to a control system capable of performing an interaction with a user, a control method, and a storage medium storing a program. BACKGROUND

[0002] Techniques for voice control of a device such as a vehicle are known. Patent Literature 1 describes that, in a case where an intention is ambiguous in correspondence with a result of voice recognition, the ambiguity is eliminated based on a positional relationship between the vehicle and a user or the like. In addition, techniques for voice control of a device such as a vehicle are used for an automatic deployment service of a vehicle, and also for an interaction between a user who utilizes the service and the vehicle.

[0003] PATENT LITERATURE

[0004] Patent Literature 1: Japanese Patent Application Publication No. 2017-156511 SUMMARY

[0005] PROBLEMS TO BE SOLVED BY THE INVENTION

[0006] However, in Patent Literature 1, there is no mention of a mechanism for evaluating information sent from a user based on a result of an interaction between the user and a vehicle utilizing a deployment service.

[0007] The present application provides a control system, a control method, and a storage medium storing a program, which evaluate information sent from a user in an interactive communication between the user and a mobile body utilizing a deployment service.

[0008] MEANS FOR SOLVING THE PROBLEMS

[0009] The control system according to the present application includes a travel control unit that causes a mobile body to travel to a specified place specified by a deployment request from a user in accordance with the deployment request, a communication unit that performs an interactive communication between the user and the mobile body via a portable terminal held by the user before the mobile body reaches the specified place, an acquisition unit that acquires image data of a periphery of the specified place photographed by a photographing unit after the mobile body reaches the specified place as a result of the interactive communication, and an evaluation unit that evaluates information sent from the user in the interactive communication based on the image data acquired by the acquisition unit.

[0010] The control method according to the present application is a control method in which, in accordance with a deployment request from a user, a mobile body is caused to travel to a specified place specified by the deployment request, an interactive communication between the user and the mobile body is performed via a portable terminal held by the user before the mobile body reaches the specified place, after the mobile body reaches the specified place as a result of the interactive communication, image data taken by a photographing unit of a periphery of the specified place is acquired, and information from the user in the interactive communication is evaluated based on the acquired image data.

[0011] The storage medium according to the present application stores a program which causes a computer to function in such a manner that, in accordance with a deployment request from a user, a mobile body is caused to travel to a specified place specified by the deployment request, an interactive communication between the user and the mobile body is performed via a portable terminal held by the user before the mobile body reaches the specified place, after the mobile body reaches the specified place as a result of the interactive communication, image data taken by a photographing unit of a periphery of the specified place is acquired, and information from the user in the interactive communication is evaluated based on the acquired image data.

[0012] Effects of the Invention

[0013] According to the present application, it is possible to evaluate information from a user in an interactive communication between the user and a mobile body using a deployment service. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a diagram showing a system configuration of a deployment service.

[0015] Figure 2 is a diagram showing a flow of a deployment service.

[0016] Figure 3 is a flowchart showing a process of an interaction.

[0017] Figure 4 is a flowchart showing a process of an inquiry.

[0018] Figure 5 is a flowchart showing a process of an evaluation.

[0019] Figure 6 is a flowchart showing a process of an evaluation.

[0020] Figure 7 is a flowchart showing a process of a charge.

[0021] Figure 8 is a flowchart showing a process of changing a threshold value of S425.

[0022] Figure 9is a diagram showing an example of an interaction. DETAILED DESCRIPTION

[0023] Hereinafter, the embodiments will be described in detail with reference to the drawings. Note that the following embodiments are not intended to limit the technical scope of the application, and all combinations of features described in the embodiments are not essential to the application. Two or more of the features described in the embodiments can be arbitrarily combined. Also, the same reference numerals are used to designate the same or similar components, and repetitive description is omitted.

[0024] Figure 1 is a diagram showing an example of a system configuration of the dispatch service in this embodiment. As shown in Figure 1 , the system includes a vehicle 100, a portable terminal 200 held by a user, and a server 300 that provides the dispatch service. The user installs an application program capable of enjoying the dispatch service from the server 300 to the portable terminal 200, and registers information related to the user (user information) on the server 300 via the application program. In addition, the application program can be a Web application program launched on a Web browser. On the other hand, the server 300 manages identification information such as a user name, a mail address, and the like as information registered from the user.

[0025] In a state where the user registers the user information on the server 300, the user can delegate the dispatch of the vehicle 100 from the portable terminal 200 via the application program, for example, in a street. The vehicle 100 is, for example, an autonomous vehicle for the purpose of a taxi or a carpool, and the user can enjoy the dispatch service by riding the dispatched vehicle 100. In addition, at an appropriate timing such as after the user gets off the vehicle 100, charging for the use of the dispatch service is performed.

[0026] When the vehicle 100 goes to the place of the dispatch from the user in accordance with the dispatch delegation from the user, a dialogue (interaction) for the vehicle 100 to reach the place of the dispatch (or find the user) is sometimes performed between the vehicle 100 and the user. For example, in response to a question "where?" from the vehicle 100, an answer "at the corner of the intersection" or the like from the user is performed. The dispatch time of the vehicle 100 greatly affects whether such an interaction is smoothly successful. In a street, there are many blind spots, and there are many scenes in which various kinds of signs are mixed, and sometimes the interaction cannot be smoothly successful even if the user tries to convey the user's own position to the vehicle 100. Therefore, in a case where the dispatch of the vehicle 100 takes time because the interaction cannot be smoothly successful, measures such as discount of the use fee of the service are sometimes taken because the intention of the user cannot be smoothly recognized. However, on the other hand, a case where the user intentionally delays the arrival of the vehicle 100 for the purpose of the discount of the use fee is also assumed.

[0027] In the present embodiment, it is possible to increase the likelihood that the case where the user seriously intends to convey the user's own current position to the vehicle 100 is targeted for the discount of the usage charge, and to reduce the likelihood that the case where the user intentionally causes the recognition of the vehicle 100 to be delayed is targeted for the discount.

[0028] The configuration of the vehicle 100 will be described.

[0029] The control section 120 collectively controls the travel of the vehicle 100, for example, by a plurality of ECUs. Further, the travel of the vehicle 100 herein refers to travel including the participation of the driver of the vehicle 100, and travel performed by automatic driving without the participation of the driver.

[0030] The outside recognition section 121 recognizes the outside information of the vehicle 100 on the basis of signals from the outside recognition camera and the outside recognition sensor. Here, the outside recognition camera is included in the camera group 145, and is a camera that photographs the surroundings of the vehicle 100. Further, the outside recognition sensor is included in the sensor group 144, and is, for example, a LIDAR (Light Detection and Ranging) that detects an object in the surroundings of the vehicle 100 or measures the distance to the object. The outside recognition section 121 recognizes, for example, a scene such as an intersection, a railroad crossing, a tunnel, and a free space such as a shoulder on the basis of the signals from the outside recognition camera and the outside recognition sensor, and recognizes the behavior (speed, direction of travel, and the like) of other vehicles.

[0031] The own position recognition section 122 recognizes the current position of the vehicle 100 on the basis of a signal from a GPS sensor included in the sensor group 144.

[0032] The in-vehicle recognition section 123 recognizes the occupant of the vehicle 100 and recognizes the state of the occupant on the basis of signals from the in-vehicle recognition camera and the in-vehicle recognition sensor. The in-vehicle recognition camera is included in the camera group 145, and is, for example, a near-infrared camera provided on a display device included in the notification section 146. The in-vehicle recognition section 123 detects, for example, the direction of the line of sight of the occupant from the photographed image data. Further, the in-vehicle recognition sensor is included in the sensor group 144, and is, for example, a sensor that detects the biological signals of the occupant and acquires biological information. The biological information refers to, for example, information related to the biological body such as the pulse, the heart rate, the body weight, the body temperature, the blood pressure, the perspiration, and the sleep information. The in-vehicle recognition sensor can also acquire such biological information from, for example, a wearable device of the occupant. The in-vehicle recognition section 123 recognizes, for example, the case where the occupant is in a dozing state, the case where the occupant is in a state during work other than driving, and the like on the basis of these signals.

[0033] The action planning unit 124 plans an action of the vehicle 100 such as an optimal path or a risk avoidance path based on the results of the recognition by the outside recognition unit 121 and the self-position recognition unit 122. The action planning unit 124, for example, performs an action plan based on a prediction result of the behavior of other vehicles based on the entry determination of a start point such as an intersection or a railroad crossing and an end point. The drive control unit 125 controls the drive unit 141, the brake unit 142, and the steering unit 143 based on the action plan by the action planning unit 124. Here, the drive unit 141 corresponds to a power device that outputs a driving force to rotate a drive wheel of the vehicle 100, for example, and the brake unit 142 corresponds to a brake device. In addition, the steering unit 143 corresponds to an electric power steering device that includes a mechanism for steering a wheel according to a steering operation (steering operation) of a driver on the steering wheel. The electric power steering device assists the steering operation or includes a motor that generates a driving force for automatically steering the wheel, a sensor that detects a steering angle, and the like. In a case where the driving state of the vehicle 100 is automatic driving, the control unit 120 automatically controls the drive unit 141, the brake unit 142, and the steering unit 143 and controls the travel of the vehicle 100.

[0034] The device control unit 126 controls devices connected to the control unit 120. For example, the device control unit 126 controls a speaker and a microphone, causes a predetermined voice message such as a warning or a message for navigation to be output, and acquires voice data by detecting a voice signal uttered by an occupant in the vehicle. In addition, for example, the device control unit 126 controls a display device included in the notification unit 146 and causes a predetermined interface screen to be displayed. In addition, for example, the device control unit 126 controls a navigation device and acquires setting information from the navigation device.

[0035] The communication unit 147 is a communication device that can communicate with the outside, and can perform wireless communication with a server that provides map information, traffic information, weather information, and other vehicles in the vicinity, for example. The communication unit 147 has a plurality of types of communication functions, and has a dedicated short-range communication (DSRC: Dedicated Short Range Communication) function and a cellular communication function, for example.

[0036] The control unit 120 can appropriately include Figure 1 functions other than those illustrated in the drawing, and can include an optimal path calculation unit that calculates an optimal path to a destination based on map information acquired from a server outside, for example. In addition, the control unit 120 can acquire information from the camera group 145 and the sensor group 144 other than those illustrated in the drawing, and can acquire information of other vehicles via the communication unit 147, for example. Figure 1

[0037] ​The control section 110 is a control section 110 for realizing the operation of the vehicle 100 in the present embodiment, and can communicate with the storage section 130 and the control section 120. The control section 110 includes a processor 111, a memory 112, a data processing section 113, and a communication interface (I / F) 116. The processor 111 reads and executes a program stored in the memory 112 or the storage section 130 or the like, a computer-readable storage medium, thereby uniformly controlling the control section 110. The operation of the vehicle 110 in the present embodiment is realized, for example, by executing the above-mentioned program by the processor 111.

[0038] The data processing section 113 includes a processing section that processes voice data, image data received from the portable terminal 200. The voice data processing section 114 extracts a feature based on the voice data received from the portable terminal 200, and performs voice recognition based on the feature amount. The voice data here is, for example, voice data converted from a voice signal issued by the user to the microphone 205 of the portable terminal 200 into communication data. In addition, the voice data processing section 114 generates voice data to be notified to the user of the portable terminal 200. The voice data generated in this case includes voice data based on a predetermined format, and voice data generated based on the voice recognition result of the voice data received from the portable terminal 200. The image data processing section 115 extracts a feature based on the captured data of the surrounding image of the vehicle 100 captured by the camera of the vehicle 100, and performs image recognition based on the feature amount. The data processing section 113 can also include modules other than the voice data processing section 114 and the image data processing section 115. For example, it can include a module that processes other kinds of data for the interactive communication of the user of the portable terminal 200 with the vehicle 100, such as text data, and a module that generates a user interface screen for display on the display section 204 of the portable terminal 200.

[0039] The communication I / F 116 is an interface for realizing communication with other devices via the network 400. For example, the communication I / F 116 realizes communication with the portable terminal 200 via the network 400. The storage section 130 stores programs and the like for realizing the operation of the present embodiment, for example, a program for transmitting information received from the portable terminal 200 to the server 300, and a program for performing the interactive communication of the user of the portable terminal 200 with the vehicle 100. The vehicle 100 is not limited to the module configuration shown, and can appropriately include other modules. Figure 1

[0040] ​The portable terminal 200 is, for example, a smartphone or tablet computer-type information processing terminal, and includes a processor 201 and a memory 202. In this embodiment, the operation of the portable terminal 200 is achieved, for example, by the processor 201 executing a program stored in the memory 202. The communication I / F 20 is an interface for enabling communication with other devices via the network 400. The display unit 204 is, for example, a touch panel, which displays various user interface screens and can accept user operations. The microphone 205 receives voice from the user of the portable terminal 200. The speaker 206 provides voice notifications to the user of the portable terminal 200. The portable terminal 200 is not limited to... Figure 1 The module configuration shown can appropriately include other modules. For example, the portable terminal 200 can include a camera capable of taking pictures of the outside.

[0041] The server 300 is configured as a general-purpose information processing device such as a PC, including a processor 301, a memory 302, a communication I / O interface 303, and a storage unit 304. In this embodiment, the server 300 operates, for example, by the processor 301 executing a program stored in the computer-readable storage medium of the memory 302 and the storage unit 304. The communication I / O interface 303 is an interface for communication with other devices via the network 400. The storage unit 304 stores programs and data for implementing dispatch services, as well as applications that can be downloaded to the portable terminal 200. Data for implementing dispatch services includes, for example, the operating information and location information of each vehicle that is subject to dispatch. Furthermore, the storage unit 304 stores registered user information via a device corresponding to a user, such as the portable terminal 200. Additionally, the storage unit 304 forms a database containing map information, traffic information, weather information, etc.

[0042] Server 300 (not limited to) Figure 1 The module configuration shown can appropriately include other modules. Furthermore, while server 300 is described as a single server, it can also be configured as multiple server devices. Additionally, a portion of server 300 can be included within vehicle 100. For example, a database such as map information can be configured as storage unit 130 of vehicle 100, or the evaluation processing described later can be performed in control unit 110 of vehicle 100. Furthermore, a portion of vehicle 100 can be included within server 300. For example, voice data processing unit 114 and image data processing unit 115 can be configured as server 300.

[0043] Figure 2 It means Figure 1 A diagram illustrating the process of allocating services within the system architecture. Figure 2 Before the sequence begins, the user information of the user of the portable terminal 200 is registered on the server 300.

[0044] In S101, the portable terminal 200 starts an application program of the dispatch service of the server 300 by a user instruction, and logs in by a predetermined ID and a password. The application program can be started by a content click operation or the like downloaded from the server 300 in advance, or can be started by specifying a URL corresponding to the server 300. After starting the application program, the portable terminal 200 opens a reservation screen on the application program by a user instruction, and makes a reservation for dispatch. The reservation for dispatch is made, for example, in the manner of "October 20, 15:00, A City, B Block". The specification of the residence can be made, for example, by a user specification on a map. When the reservation information is determined on the application program, the portable terminal 200 transmits the determined reservation information to the server 300.

[0045] In S102, when the server 300 receives the reservation information from the portable terminal 200, dispatch is planned based on the reservation information. In the dispatch planning of S102, for example, a determination is made as to whether dispatch can be made in the date and time and the place specified by the reservation information. In a case where it is determined that dispatch can be made in the dispatch planning, in S103, the server 300 decides a vehicle 100 that makes dispatch in the date and time and the place specified by the reservation information, and transmits notification information of the reservation determination to the portable terminal 200. The vehicle 100 that makes dispatch is decided, for example, as a vehicle located in a place closest to the place of the reservation date and time among vehicles that can operate. In a case where it is determined that dispatch cannot be made in the dispatch planning, in S103, the server 300 transmits notification information indicating that dispatch based on the reservation information cannot be implemented to the portable terminal 200, and ends the processing. Hereinafter, the transmission of the notification information of the reservation determination in S103 is described. Figure 2

[0046] When the user confirms the reception of the notification information of the reservation determination from the server 300 in the portable terminal 200, the user rides to the place specified by the reservation information at the date and time specified by the reservation information, and makes a dispatch request to the portable terminal 200. Further, in the present embodiment, the device that makes the reservation in S101 and the device that makes the dispatch request in S104 can be the same portable terminal 200, or can be different. For example, the processing of S101 and S103 can be executed in a PC different from the portable terminal 200.

[0047] In S104, the portable terminal 200 starts the application program of the dispatch service of the server 300 by a user instruction as described above, and transmits a dispatch request to the server 300 on the application program screen. In S105, when the server 300 receives the dispatch request, the vehicle 100 decided in S102 is instructed to ride to the place specified by the reservation information.

[0048] ​In S106, the vehicle 100 plans a route from the current position of the vehicle 100 to the specified place, as instructed by the server 300. Further, in a case where the vehicle 100 is to travel on a road by using the dispatch service issued by other users at the time point instructed by the server 300, there is a case where the vehicle 100 waits for the dispatch service. In S107, the vehicle 100 starts traveling with the specified place as a destination, based on the route planned in S106. In S108, when the vehicle 100 reaches a predetermined range of the specified destination, the interaction communication with the user is started since the vehicle 100 reaches the place where the user is waiting.

[0049] In S109, the interaction communication is performed between the user and the vehicle 100. Further, in the present embodiment, the interaction communication is performed by voice. The interaction communication is performed, for example, in two stages of finding a marker such as a building located in the vicinity of the user, and finding the user himself or herself. For example, the user sends information that becomes a marker such as a building located in the vicinity of himself or herself by voice, and the vehicle 100 attempts to find the marker based on the voice information. In a case where the marker cannot be found, the vehicle 100 requests the user to provide further information. On the other hand, in a case where the marker can be found, the vehicle 100 notifies the user that the marker can be found, and requests the user to provide information of himself or herself. The user conveys the information of himself or herself (color of clothes, etc.) by voice, and the vehicle 100 attempts to find the user. In a case where the user cannot be found, the vehicle 100 requests the user to provide further information. On the other hand, in a case where the user can be found, the vehicle 100 notifies the user that the user can be found, and stops at a position where the vehicle 100 can stop, such as a space on a shoulder (S110).

[0050] When the vehicle 100 stops in S110, in S111, the user gets on the vehicle 100, and sends a message indicating that the user has gotten on the vehicle 100 to the server 300 via the application program of the portable terminal 200. The information indicating that the user has gotten on the vehicle 100 can be sent by pressing a predetermined button on the screen of the application program. After the user gets on the vehicle 100, traveling to a destination desired by the user is started.

[0051] When the server 300 receives the information indicating that the user has gotten on the vehicle 100, in S112, the information sent from the user in the interaction communication between the user and the vehicle 100 is evaluated. The evaluation in S112 is described later. In S113, the server 300 performs charging to the user based on the evaluation result in S112. Note that, in the present embodiment, a case where the processing of S112 and S113 is performed in the server 300 is described, but at least any one of the above processing can be performed in the control section 110 of the vehicle 100.

[0052] Figure 3 is a flowchart showing the processing of the interaction communication in S109. Figure 3The processing is realized, for example, by the processor 111 executing a program stored in the memory 112. When the vehicle 100 starts running in S107, the processing of S108 is started. Figure 2 The processing of S108 is started when the vehicle 100 starts running in S107. Figure 3

[0053] In S201, the processor 111 determines whether or not the place indicated by the server 300 is reached within a predetermined distance. For example, the processor 111 can determine based on the route planned in S106, and can determine based on whether or not the area reached is within a predetermined distance from the center of the indicated place. In the case where it is determined that the place is not reached within the predetermined distance, the processing of S201 is repeated, and in the case where it is determined that the place is reached within the predetermined distance, S202 is entered.

[0054] In S202, the processor 111 starts interactive communication with the user of the portable terminal 200 through the voice data processing section 114. For example, the processor 111 generates voice data of a message indicating the start of interactive communication, such as "The consigned place will be reached soon. Please indicate the marker in the vicinity.", and transmits it to the portable terminal 200. Further, such a message can be text data, and in the present embodiment, voice data is described as one example. When the user of the portable terminal 200 receives the notification of the above message, the feature of the marker in the vicinity of the user is input to the microphone 205 by voice.

[0055] Figure 9 An example of a message input by the user through the microphone 205 is shown in the upper layer. For example, in the case where the user himself stands in front of a yellow building, a voice such as "in front of a yellow building." is input to the microphone 205. The processor 201 of the portable terminal 200 transmits the input voice signal as voice data to the vehicle 100.

[0056] In S203, the processor 111 extracts the feature of the received voice data based on the feature amount through the voice data processing section 114 via the communication I / F 116, and performs voice recognition based on the feature amount. For example, "yellow building" is extracted as a feature from the voice data indicating the voice of the above user. The feature extracted here is stored in the memory 112, and is used for the evaluation processing of S112.

[0057] In S204, the processor 111 searches for a target object indicating the feature based on the feature extracted in S203 through the image data processing section 115. The processor 111 searches for an object having the feature as a target object candidate based on the image data of the surroundings captured by the camera group 145 of the vehicle 100.

[0058] ​In S205, processor 111 determines whether a target object has been detected. If one target object candidate is retrieved in S205, processor 111 determines that a target object has been detected; if no multiple target object candidates are retrieved or no target object is retrieved, processor 111 determines that no target object has been detected. This determination can also be made using other methods. For example, if multiple target object candidates are retrieved but these targets are close to each other, they can be treated as one and determined to have been detected. If a target object is detected in S205, the process proceeds to S207; if a target object is not detected, the process proceeds to S206.

[0059] In S206, the processor 111 generates a message indicating an inquiry about the target object through the voice data processing unit 114 and sends it to the portable terminal 200.

[0060] Figure 4 This is a flowchart illustrating the processing of the query in S206. (Beginning) Figure 4 Before processing, a variable representing the number of times the process of S206 was executed (the number of queries) is stored in memory 112 and initialized to 0. In S301, processor 111 increments the number of queries. If the result is negative in S205 and the process of S206 is initially executed, the number of queries is increased to "1".

[0061] In S302, the processor 111 generates a message indicating an inquiry about the target object through the voice data processing unit 114.

[0062] Figure 9 The middle layer illustrates an example of an inquiry. In S205, if an object with the characteristic of a "yellow building" cannot be identified from the surrounding image data captured by camera group 145, a message indicating an inquiry about the target object is generated. Such a message is generated as, for example, "Sorry. No yellow building found. Please provide further information." Furthermore, in S205, if multiple objects with the characteristic of a "yellow building" are retrieved from the image data, the same message is generated.

[0063] In S303, the processor 111 sends the message generated in S302 to the portable terminal 200.

[0064] When the user of the portable terminal 200 receives the notification of the above message, the features of the other marker located in the vicinity of the self are input to the microphone 205 by voice. For example, in the case of the message that notifies the user of the portable terminal 200 of "I am sorry. The yellow building was not found. Please provide other information.", the user inputs "The yellow building is located at the intersection." or the like to the microphone 205 by voice. The processor 201 of the portable terminal 200 transmits the input voice signal as voice data to the vehicle 100.

[0065] In S304, the processor 111 extracts the features of the received voice data by the voice data processing section 114 via the communication I / F 116, and performs voice recognition based on the feature amount. For example, "intersection" is extracted as a feature from the voice data representing the voice of the above user. The feature amount extracted here is stored in the memory 112, and is used for the evaluation process of S112.

[0066] After S304, the process of Figure 4 is ended, and is repeated from S204 of Figure 3 . In the above example, in the case of proceeding from S206 to S204, the target object representing the features thereof is searched for based on the feature "intersection" extracted in S304, and the determination of S205 is performed. In the case where it is determined that the target object is found in S205, the processor 111 requests the control section 120 to move the vehicle 100 to the vicinity of the target object.

[0067] In S207, the processor 111 determines whether the vehicle 100 is moved to the vicinity of the target object. The determination of S207 is repeated until it is determined that the vehicle 100 is moved to the vicinity of the target object. In the case where it is determined that the vehicle 100 is moved to the vicinity of the target object, S208 is entered.

[0068] In S208, the processor 111 generates a message representing that the vehicle 100 is moved to the vicinity of the target object by the voice data processing section 114, and transmits it to the portable terminal 200. For example, the processor 111 generates a message such as "The yellow building has been reached. Please inform your features." In addition, in the case where the vehicle 100 is moved to the vicinity of the target object without the process of the inquiry via S206, the message generated in the case where the vehicle 100 is moved to the vicinity of the target object with the process of the inquiry via S206 can be made different. For example, in the case where the vehicle 100 is moved to the vicinity of the target object with the process of the inquiry via S206, a message such as "I am sorry for the delay. Now I finally know which building the yellow building is." and a message such as "Please inform your features." can be generated. Figure 9

[0069] ​When the user of the portable terminal 200 receives the notification of the above message, the user inputs his / her own feature by voice to the microphone 205. For example, the user inputs "wearing blue clothes" or the like to the microphone 205. The processor 201 of the portable terminal 200 transmits the input voice signal as voice data to the vehicle 100. In S209, the processor 111 extracts the feature of the received voice data based on the voice data processing section 114 via the communication I / F 116, and performs voice recognition based on the feature amount. For example, "blue clothes" is extracted as the feature from the voice data representing the voice of the above user. The feature extracted here is stored in the memory 112, and used for the processing of the evaluation in S112.

[0070] In S210, the processor 111 searches for the user representing the feature based on the feature extracted in S209 by the image data processing section 115. The processor 111 searches for the object having the feature as the target candidate based on the image data of the surroundings captured by the camera group 145 of the vehicle 100.

[0071] In S211, the processor 111 determines whether or not the user is found. In the case where it is determined that the user is found in S211, S213 is entered, and in the case where it is determined that the user is not found, S212 is entered.

[0072] Reference Figure 4 The processing of the inquiry of S212 is described. Further, the value of the number of inquiries up to the current time point is stored in the variable area reserved in the memory 112.

[0073] In S301, the processor 111 increases the number of inquiries. In S302, the processor 111 generates a message representing the inquiry to the user by the voice data processing section 114. For example, a message such as "please inform other features." is generated.

[0074] In S303, the processor 111 transmits the message generated in S302 to the portable terminal 200.

[0075] When the user of the portable terminal 200 receives the notification of the above message, the user inputs his / her own feature by voice to the microphone 205. For example, the user inputs "wearing blue clothes" or the like to the microphone 205. The processor 201 of the portable terminal 200 transmits the input voice signal as voice data to the vehicle 100.

[0076] In S304, the processor 111 extracts a feature of the received voice data via the communication I / F 116 by the voice data processing section 114, and performs voice recognition based on the feature amount. For example, "brown bag" is extracted as a feature from the voice data representing the voice of the user described above. The feature extracted here is stored in the storage 112, and used for the evaluation process of S112.

[0077] After S304, the process of Figure 4 is repeated from S210 of Figure 3 In the example described above, in the case of proceeding to S210 from S212, the target object representing the feature "brown bag" extracted in S304 is searched for, and the determination of S211 is performed. Further, it is difficult to assume that the user is in a position that becomes a blind spot from the vehicle 100 at the point in time when the process of S210 is performed, and thus it is possible to perform the determination in S206 without using the feature stored in the storage 112.

[0078] In the case where it is determined that it is found in S211, in S213, the processor 111 requests the control section 120 to stop the vehicle 100 in the vicinity of the user. The process of stopping the vehicle 100 in S213 corresponds to S110 of Figure 2 The processor 111 stores the timing of stopping the vehicle 100 in the storage 112.

[0079] In S214, the processor 111 requests the control section 120 to capture an image of the periphery of the vehicle 100 by the camera group 145. The image data captured in S214 is stored in a storage area such as the storage section 130, and used for the evaluation process of S112. Thus, the periphery of the vehicle 100 is captured over the entire 360 degrees. After S214, the process of Figure 3 is ended.

[0080] Next, the evaluation process of S112 will be described.

[0081] Figure 5 and Figure 6 is a flowchart representing the evaluation process of S112. Figure 5 and Figure 6 The process of

[0082] In S401, the processor 301 acquires the number of inquiries stored in the storage 112 from the vehicle 100. In S402, the processor 301 determines whether the number of inquiries is equal to or more than a predetermined value. In the case where it is determined that it is not equal to or more than the predetermined value, the process of Figure 5 Figure 6 ​the processing of S113. Further, in the charge performed at this time, the fee at the time when the user who registered the server 300 normally uses the deployment service is applied. On the other hand, in a case where it is determined that the predetermined value or more, S403 is entered.

[0083] In S403, the processor 301 acquires the image data acquired in S214 of the vehicle 100. Then, in S404, the processor 301 acquires the features stored in the memory 112 from the vehicle 100. The features here are the features extracted in S203, S209, S304. Figure 3

[0084] In S405, the processor 301 focuses on one feature acquired in S404, and analyzes whether or not the feature corresponds to the feature of the object determined in the image data acquired in S403. Then, in S406, the processor 301 determines whether or not the feature focused on in S405 is a true feature. For example, in a case where the object having the feature of "yellow building" is determined in the image data, it is determined that the feature is a true feature. In this case, in S407, the processor 301 labels the feature with a mark indicating "true". On the other hand, for example, in a case where the object having the feature of "yellow building" is not determined in the image data, it is determined that the feature is a "false" feature. In this case, in S408, the processor 301 labels the feature with a mark indicating "false". After S407 and S408, S409 is entered.

[0085] In S409, the processor 301 determines whether or not the processing of S405 to S408 is performed for all the features acquired in S404. In a case where it is determined that the processing is not performed for all the features, in S410, the processor 301 focuses on the next feature, and repeats the processing from S405. On the other hand, in a case where it is determined that the processing is performed for all the features, S421 of FIG. 42 is entered. Figure 6

[0086] In S421, the processor 301 determines whether or not all the features for which the processing is performed in S405 to S408 are true features. For example, the processor 301 determines whether or not the mark indicating "true" is labeled for all the features. In a case where it is determined that the mark indicating "true" is labeled for all the features, S422 is entered. On the other hand, in a case where it is determined that the mark indicating "true" is not labeled for all the features, that is, in a case where the mark indicating "false" is labeled for the feature, S425 is entered.

[0087] ​​In S422, the processor 301 determines whether there is an object excluding feature. Here, the object excluding feature means a feature amount that can judge that the place designated by the appointment information is unclear even if the object having the feature extracted in the interactive communication is determined within the image data. For example, the place designated by the appointment information is a street. Then, in a case where the determined object is a mountain or the like visible in the distance and above, it is recognized that the feature is an object excluding feature. For example, the processor 301 refers to the map information configured as a database in the storage section 304, and estimates the position of the object having a feature with a mark indicating "true" labeled. Then, if the estimated position satisfies a condition such as a predetermined distance from the place designated by the appointment information, the processor 301 labels a mark expressing "false" to the feature in S424. After S424, S425 is entered. In a case where it is determined in S422 that there is no object excluding feature, S423 is entered.

[0088] In S423, the processor 301 labels an evaluation mark to the user of the portable terminal 200. The evaluation mark here means a mark expressing that the user is an object of discount. After S423, the processing of Figure 5 and Figure 6 is ended, and the processing of the charge in S113 is performed.

[0089] As described above, in a case where the inquiry is performed more than a predetermined number of times in the interactive communication, and the features extracted from the voice data from the user of the portable terminal 200 all correspond to the features of the object determined within the image data, the user is regarded as an object of discount. For example, in a case where the user performs the dispatch request in a complicated place of a street, it is assumed that there are various confusing marks or many blind spots around the place. Even in such a case, if the user wants to convey the place of the user seriously, in the image data around the place of the dispatch request, the features (in other words, the consistency) that all the users want to convey should be confirmed. In the present embodiment, in such a case, since the seriousness of the user is high, the user can be regarded as an object of discount. In addition, it is possible to exclude the following case from the object of discount: the user intentionally conveys a feature that is not visible from the place of the dispatch request or is apparently far from the place although it can be visible.

[0090] In S425, the processor 301 determines whether the proportion of the features labeled with the mark expressing "false" among the features for which the processing is performed in S405 to S408 is below a threshold value. In a case where it is determined that it is not below the threshold value, the processing of Figure 5 and Figure 6In other words, in this case, the user is not targeted for discounting. In a case where it is determined in S425 that the proportion of the features on which the flag indicating "false" is attached is below the threshold value, the process proceeds to S426.

[0091] In S426, the processor 301 performs a similarity determination on the features on which the flag indicating "false" is attached. For example, the processor 301 analyzes whether or not an object whose feature quantity of the feature on which the flag indicating "false" is attached is within a prescribed range is determined in the image data. Then, in S427, the processor 301 determines whether or not an object having a feature similar to the feature on which the flag indicating "false" is attached is determined. In a case where an object having a feature similar to the feature is determined, in S423, the evaluation flag is attached to the user of the portable terminal 200. For example, for similarity in shape, a shape whose curvature is within a predetermined range is determined to have similarity. In addition, for example, for similarity in chroma, a shape whose saturation and brightness are within a predetermined range is determined to have similarity.

[0092] For example, in a case where there are various confusing marks around the user, the user can make a mistake. In the present embodiment, even if the feature on which the flag indicating "false" is attached is constant, in a case where similarity is recognized, since the user's attentiveness is high, the user can be targeted for discounting.

[0093] In a case where an object having a feature similar to the feature is not determined in S427, the process of S426 and S427 is ended. Figure 5 and Figure 6 In this case, the user is not targeted for discounting.

[0094] In S425, a case where even if the feature on which the flag indicating "false" is attached is constant, in a case where similarity is recognized, since the user's attentiveness is high, the user can be targeted for discounting is explained. Here, the threshold value of S425 can be changed according to the number of times the evaluation flag is attached.

[0095] Figure 8 is a flowchart showing the process of changing the threshold value of S425 according to the number of times the evaluation flag is attached. Figure 8 The process of S426 and S427 is realized, for example, by the processor 301 executing a program stored in the storage 302. Figure 8 The process of S426 and S427 is executed, for example, after the evaluation of S112 is performed. In addition, the server 300 stores the number of times the evaluation flag is attached as user information stored in the storage 304.

[0096] In S601, the processor 301 determines whether the number of times the evaluation flag has been assigned to the user is greater than or equal to a predetermined value. If the determination is that the number of times the number of times the evaluation flag has been assigned is less than or equal to the predetermined value, the process ends. Figure 8 The processing involves steps S602 if the value is determined to be above a predetermined value.

[0097] In S602, processor 301 uses the threshold increase pre-defined amount from S425. After S602, the process ends. Figure 8 This kind of processing increases the likelihood of highly attentive users becoming eligible for discounts.

[0098] Figure 7 This is a flowchart illustrating the charging process for S113. Figure 7 The processing is implemented, for example, by the processor 301 executing a program stored in the memory 302.

[0099] In S501, the processor 301 obtains the arrival time of the location specified by the reservation information from the time the vehicle 100 stopped in S213. Then, if the time obtained from the vehicle 100 is less than or equal to the time specified by the reservation information, the processor 301 determines whether the delay is within the predetermined time. If it is determined that the delay is within the predetermined time, the process proceeds to S504 and charges are processed. For example, if the delay within the predetermined time takes into account factors such as traffic conditions, it is not considered as a discount under this embodiment, and the normal charge is applied. On the other hand, if it is determined that the delay is not within the predetermined time, the process proceeds to S502.

[0100] Alternatively, the conditions for determining S501 can be set as follows. For example, in Figure 3 If, in S201, the vehicle is determined to be within a predetermined distance, the processor 111 stores the current time in memory 112, etc. At this time, the processor 111 can infer the arrival time (or required time) to the designated location based on the current position of the vehicle 100, using path information and traffic information. Then, in S501, it can determine whether the difference is a delay within the predetermined time based on the inferred time and the actual arrival time stored in S213. If it is not a delay within the predetermined time, the process proceeds to S502. Furthermore, in the above configuration, when an instruction is received in S105, the processor 111 infers the arrival time to the designated location based on the current position of the vehicle 100, using path information and traffic information.

[0101] In S502, the processor 301 determines whether or not the evaluation mark is marked for the user of the portable terminal 200, which is determined in S423. In a case where it is determined that the evaluation mark is not marked, S504 is entered, and the charge processing is performed. That is, in this case, the user is not targeted for the discount, and the normal charge is performed. On the other hand, in a case where it is determined that the evaluation mark is marked, S503 is entered.

[0102] In S503, the processor 301 determines the discount rate for the user of the portable terminal 200. The discount rate herein can be a predetermined discount rate, or the discount rate can be changed according to the number of times of the evaluation mark being given. For example, if the number of times of the evaluation mark being given is equal to or more than a predetermined number of times, the discount rate can be made larger. In addition, a plurality of stages can be set for the predetermined number of times, and the larger the number of times of the evaluation mark being given, the larger the discount rate.

[0103] In S504, the processor 301 performs the charge processing for the user of the portable terminal 200. For example, as the charge processing, the transmission of the notification of the requested amount to the mail address of the user, or the like is performed. In addition, in a case where the discount rate is determined in S503, the discount rate is applied to the charge processing. After S504, the processing ends. Figure 7

[0104] As described above, according to the present embodiment, the seriousness of the user in the interactive communication of the deployment service is judged. Then, the discount is performed based on the judgment result. With such a configuration, it is possible to appropriately narrow down the users targeted for the discount in a case where the delay of the vehicle 100 is equal to or more than a predetermined time. In addition, in the present embodiment, a case where the discount rate is determined in S503 is described, but processing other than determining the discount rate can be performed. For example, processing of increasing the priority of the next deployment request can be performed. In the present embodiment, the vehicle 100 is described as an autonomous vehicle. However, the vehicle 100 is not limited to an autonomous vehicle, and a micro mobile body in which a passenger does not ride or an autonomous mobile robot can be used as the vehicle 100. Even with such a configuration, the same effects as those in the present embodiment are exerted.

[0105] <Summary of Embodiments>

[0106] ​The control system of the above-described embodiment has a travel control unit (120) that causes a mobile body to travel to a specified place specified by a user's arrangement request, a communication unit (110) that performs interactive communication between the user and the mobile body via a portable terminal held by the user before the mobile body reaches the specified place, an acquisition unit (110, 145) that acquires image data of a periphery of the specified place taken by a photographing unit after the mobile body reaches the specified place as a result of the interactive communication, and an evaluation unit (300) that evaluates information sent from the user in the interactive communication based on the image data acquired by the acquisition unit.

[0107] With such a configuration, it is possible to evaluate information sent from a user in interactive communication.

[0108] In addition, the control system further has an identification unit (113) that identifies information sent from the user based on data received from the portable terminal. The identification unit identifies information sent from the user based on voice data received from the portable terminal.

[0109] With such a configuration, it is possible to evaluate voice information sent from a user in interactive communication.

[0110] In addition, the evaluation unit evaluates information sent from the user based on coincidence between the information sent from the user and information extracted from the image data. The evaluation unit evaluates whether or not a discount is made in a charge processing for the user Figure 5 , Figure 6 ).

[0111] With such a configuration, it is possible to evaluate whether or not a user is a target of a discount based on information sent from the user in interactive communication.

[0112] In addition, the information is information indicating a feature, and the evaluation unit evaluates that a discount is made in the charge processing when all features sent from the user coincide with features extracted from the image data Figure 5 , Figure 6 ). The evaluation unit evaluates that a discount is made in the charge processing when a proportion of features sent from the user that coincide with features extracted from the image data is equal to or higher than a threshold value Figure 5 , Figure 6 ).

[0113] With such a configuration, it is possible to evaluate information sent from a user based on a taken image of a periphery of a specified place when a mobile body reaches the specified place.

[0114] Further, the evaluation is made on the basis of whether the feature from the user is similar to the feature extracted from the image data (S2). Figure 6 ) The similarity is a similarity in at least any one of a shape and a color.

[0115] With such a configuration, it is possible to make an evaluation taking into account a misrecognition or the like of the user.

[0116] Further, the control system further includes a storage unit that stores a number of times of evaluation by the evaluation unit that a discount is made in the charge processing, and a change unit that changes the threshold value when the number of times is equal to or more than a predetermined value.

[0117] With such a configuration, it is possible to increase the positivity of correct communication of information.

[0118] Further, the evaluation by the evaluation unit is made when arrival of the mobile body to the specified place satisfies a condition. The condition includes a case where the mobile body arrives at the specified place more delayed than a required time estimated. The required time is estimated on the basis of a position of the mobile body and the specified place.

[0119] With such a configuration, it is possible to make an evaluation of information from a user when the mobile body arrives at the specified place more delayed.

[0120] Further, the control system is mounted on the mobile body.

[0121] With such a configuration, it is possible to configure a system that evaluates information from a user in interactive communication in a mobile body.

[0122] The present invention is not limited to the above-described embodiments, and various modifications and changes can be made within the scope of the present invention.

Claims

1. A control system, wherein the control system comprises: a travel control unit that causes a mobile body to travel to a specified place specified by a user's arrangement request; a communication unit that performs interactive communication between the user and the mobile body via a portable terminal held by the user before the mobile body reaches the specified place; an acquisition unit that acquires image data of a periphery of the specified place taken by a photographing unit after the mobile body reaches the specified place as a result of the interactive communication; and an evaluation unit that evaluates information sent from the user in the interactive communication based on the image data acquired by the acquisition unit.

2. The control system according to claim 1, wherein the control system further comprises an identification unit that identifies the information sent from the user based on data received from the portable terminal.

3. The control system according to claim 2, wherein the identification unit identifies the information sent from the user based on voice data received from the portable terminal.

4. The control system according to claim 1, wherein the evaluation unit evaluates the information sent from the user based on coincidence between the information sent from the user and information extracted from the image data.

5. The control system according to claim 4, wherein the evaluation unit evaluates whether or not a discount is made in a charge processing for the user.

6. The control system according to claim 5, wherein the information is information indicating a feature, in a case where all of the features sent from the user coincide with the features extracted from the image data, the evaluation unit evaluates that the discount is made in the charge processing.

7. The control system according to claim 5, wherein in a case where a proportion of the features sent from the user that coincide with the features extracted from the image data is equal to or higher than a threshold value, the evaluation unit evaluates that the discount is made in the charge processing.

8. The control system according to claim 7, wherein in a case where an inconsistent feature among the features sent from the user is similar to the features extracted from the image data, the evaluation unit evaluates that the discount is made in the charge processing.

9. The control system according to claim 8, wherein the evaluation unit determines whether or not the inconsistent feature among the features sent from the user is similar to the features extracted from the image data, and evaluates that the discount is made in the charge processing based on a result of the determination.

10. The control system according to claim 8, wherein the similarity is similarity with respect to at least any one of a shape and a color.

11. The control system according to claim 7, wherein the control system further comprises: a storage unit that stores a number of times that the discount is evaluated to be made in the charge processing by the evaluation unit; and a change unit that changes the threshold value in a case where the number of times is equal to or higher than a predetermined value.

12. The control system according to claim 1, wherein ​ In a case where the condition is satisfied in that the mobile body reaches the specified place, evaluation made by the evaluation unit is performed.

13. The control system according to claim 12, wherein the condition includes a delay of a required time that is inferred for the mobile body to reach the specified place.

14. The control system according to claim 13, wherein the required time is inferred based on a position of the mobile body and the specified place.

15. The control system according to claim 1, wherein the control system is constituted in the mobile body.

16. A control method, wherein in the control method: a mobile body is caused to travel to a specified place specified by a user's arrangement request; before the mobile body reaches the specified place, interactive communication between the user and the mobile body is performed via a portable terminal held by the user; after the mobile body reaches the specified place as a result of the interactive communication, image data captured by a photographing unit of a periphery of the specified place is acquired; and based on the acquired image data, information sent from the user in the interactive communication is evaluated.

17. A computer-readable storage medium storing a program, wherein the program causes a computer to perform: a mobile body is caused to travel to a specified place specified by a user's arrangement request; before the mobile body reaches the specified place, interactive communication between the user and the mobile body is performed via a portable terminal held by the user; after the mobile body reaches the specified place as a result of the interactive communication, image data captured by a photographing unit of a periphery of the specified place is acquired; and based on the acquired image data, information sent from the user in the interactive communication is evaluated. ​ ​

Images