Information processing device, recording medium, and information processing method

Abstract

The present disclosure relates to an information processing device, a recording medium, and an information processing method. An information processing device capable of communicating with a user terminal and a camera device includes a control unit and a communication unit, wherein the control unit performs the following operations: receiving location information from the user terminal via the communication unit; selecting a camera device around the user terminal based on the location information; sending an image capture instruction to the selected camera device via the communication unit; receiving one or more captured images from the camera device via the communication unit; identifying a user of the user terminal from the one or more captured images; and sending the captured image of the identified user to a vehicle assigned to the user via the communication unit.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, a recording medium, and an information processing method. Background Art

[0002] Conventionally, there is known a technique for acquiring a user's surrounding image from a user terminal having a camera unit and transmitting the acquired surrounding image to a taxi (for example, Japanese Patent Application Laid-Open No. 2002-032897). Summary of the Invention

[0003] The surrounding images sent to the taxi are images from the user's viewpoint, so the user himself is not captured in the surrounding images. Therefore, it may be difficult for the taxi driver to determine the location of the user.

[0004] The present disclosure has been made in view of the above circumstances, and an object of the present disclosure is to provide an information processing device, a recording medium, and an information processing method that can support identification of a user's location.

[0005] An information processing device according to a first aspect of the present disclosure is an information processing device capable of communicating with a user terminal and a camera device, including:

[0006] Control unit and communication unit,

[0007] The control unit performs:

[0008] receiving location information from the user terminal via the communication unit;

[0009] selecting a camera device around the user terminal according to the location information;

[0010] sending an image capturing instruction to the selected imaging device via the communication unit;

[0011] receiving one or more captured images from the imaging device via the communication unit;

[0012] identifying a user of the user terminal from the one or more camera images; and

[0013] The captured image in which the user is recognized is transmitted to a vehicle assigned to the user via the communication unit.

[0014] A recording medium according to a second aspect of the present disclosure records a program for causing a computer, which is an information processing device capable of communicating with a user terminal and an imaging device, to execute operations, the operations including:

[0015] receiving location information from the user terminal;

[0016] selecting a camera device around the user terminal according to the location information;

[0017] Sending an image capturing instruction to the selected camera device;

[0018] receiving one or more camera images from the camera device;

[0019] identifying a user of the user terminal from the one or more camera images; and

[0020] The camera image in which the user is recognized is sent to a vehicle assigned to the user.

[0021] The information processing method according to the third aspect of the present disclosure is an information processing method executed by an information processing device capable of communicating with a user terminal and a camera device, including:

[0022] receiving location information from the user terminal;

[0023] selecting a camera device around the user terminal according to the location information;

[0024] Sending an image capturing instruction to the selected camera device;

[0025] receiving one or more camera images from the camera device;

[0026] identifying a user of the user terminal from the one or more camera images; and

[0027] The camera image in which the user is recognized is sent to a vehicle assigned to the user.

[0028] According to the information processing device, the recording medium, and the information processing method according to each aspect of the present disclosure, it is possible to support identification of a user's location. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will be described below with reference to the accompanying drawings, in which like reference numerals represent like elements, and in which:

[0030] Figure 1 This is a schematic diagram of an information processing system.

[0031] Figure 2 It is a block diagram showing the structure of an information processing device.

[0032] Figure 3 It is a block diagram showing the structure of a vehicle.

[0033] Figure 4 is a block diagram showing the structure of a user terminal.

[0034] Figure 5 It is a diagram showing the data structure of the feature information database (DB).

[0035] Figure 6 It is a diagram showing the data structure of the reservation information DB.

[0036] Figure 7 A diagram showing an example of imaging.

[0037] Figure 8 A diagram showing an example of a captured image.

[0038] Figure 9 It is a sequence diagram showing the operation of the information processing system. DETAILED DESCRIPTION

[0039] Figure 1 1 is a schematic diagram of an information processing system S according to this embodiment. The information processing system S includes an information processing device 1, a vehicle 2, a user terminal 3, and an imaging device 4 that can communicate with each other via a network NW. The network NW includes, for example, a mobile communication network and the Internet.

[0040] exist Figure 1 For simplicity of description, the diagram shows one information processing device 1, one vehicle 2, one user terminal 3, and one camera 4. However, the number of information processing devices 1, vehicles 2, user terminals 3, and cameras 4 is not limited to this. For example, the processing performed by the information processing device 1 of this embodiment can be performed by multiple information processing devices 1 in a distributed configuration. Multiple user terminals 3 can be operated by user U01.

[0041] The following describes an overview of the processing performed by the information processing device 1 of the present embodiment. The control unit 11 of the information processing device 1 receives one or more camera images from the camera device 4, and identifies the user from the one or more camera images using an arbitrary image analysis method. The control unit 11 sends the camera image in which the user has been identified to the vehicle 2 assigned to the user. With this structure, the control unit 11 can provide the camera image in which the user is captured to the vehicle 2, so that the driver of the vehicle 2 can be assisted in determining the correct location of the user. From another perspective, the accuracy of location information is sometimes low in places with many buildings, etc. However, it is easy to determine the location of the user by providing the camera image in which the user is captured. Therefore, the driver of the vehicle 2 can reliably construct a pick-up route to the waiting place of the user, so that the number of pick-ups from the wrong direction can be reduced.

[0042] The information processing device 1 is installed in a facility such as a data center. The information processing device 1 is a computer such as a server in a cloud computing system or other computing system. Alternatively, the information processing device 1 may be installed in a vehicle 2 .

[0043] Reference Figure 2 , the internal structure of the information processing device 1 is described in detail.

[0044] The information processing device 1 includes a control unit 11, a communication unit 12, and a storage unit 13. The components of the information processing device 1 are connected to each other via, for example, a dedicated line so as to be communicable with each other.

[0045] The control unit 11 includes, for example, one or more general-purpose processors including a central processing unit (CPU) or a micro processing unit (MPU). The control unit 11 may include one or more dedicated processors dedicated to specific processing. The control unit 11 may also include not a processor but one or more dedicated circuits. The dedicated circuit may be, for example, a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The control unit 11 may also include an electronic control unit (ECU). The control unit 11 sends and receives arbitrary information via the communication unit 12.

[0046] The communication unit 12 includes a communication module corresponding to one or more wired or wireless local area network (LAN) standards for connecting to the network NW. The communication unit 12 may include a module corresponding to one or more mobile communication standards including Long Term Evolution (LTE), 4th Generation (4G), or 5th Generation (5G). The communication unit 12 may include a communication module corresponding to one or more short-range communication standards or specifications including Bluetooth (registered trademark), AirDrop (registered trademark), IrDA, ZigBee (registered trademark), Felica (registered trademark), or RFID. The communication unit 12 sends and receives arbitrary information via the network NW.

[0047] The storage unit 13 includes, for example, semiconductor memory, magnetic memory, optical memory, or a combination of at least two of these types, but is not limited to these. Semiconductor memory is, for example, RAM or ROM. RAM is, for example, SRAM or DRAM. ROM is, for example, EEPROM. The storage unit 13 can also function as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 13 can store information resulting from analysis or processing by the control unit 11. The storage unit 13 can store various information related to the operation or control of the information processing device 1. The storage unit 13 can store system programs, application programs, embedded software, etc. The storage unit 13 includes the feature information DB and reservation information DB described later.

[0048] Vehicle 2 includes, for example, any type of car such as a micromobility vehicle, a gasoline vehicle, a diesel vehicle, an HV, a PHV, an EV, or an FCV. The various components of vehicle 2 are communicatively connected to each other via an on-board network such as a Controller Area Network (CAN) or a dedicated line. "HV" is the abbreviation for Hybrid Vehicle. "PHV" is the abbreviation for Plug-in Hybrid Vehicle. "EV" is the abbreviation for Electric Vehicle. "FCV" is the abbreviation for Fuel Cell Vehicle. Vehicle 2 of this embodiment is driven by a driver. As an alternative, vehicle 2 can automate its driving at any level. The level of automation is, for example, any level from level 1 to level 5 in the SAE classification. "SAE" is the abbreviation for the Society of Automotive Engineers. Vehicle 2 can also be a MaaS dedicated vehicle. "MaaS" is the abbreviation for Mobility as a Service. The vehicle 2 may be, for example, a bicycle, a bicycle with an engine, or a two-wheeled motor vehicle (motorcycle).

[0049] Reference Figure 3 , describing the internal structure of vehicle 2 in detail.

[0050] The vehicle 2 includes a control unit 21, a communication unit 22, a storage unit 23, an imaging unit 24, and a display unit 25. The components of the vehicle 2 are connected to each other via, for example, a dedicated line so as to be communicable with each other.

[0051] The hardware configuration of the control unit 21, communication unit 22, and storage unit 23 of the vehicle 2 may be the same as the hardware configuration of the control unit 11, communication unit 12, and storage unit 13 of the information processing device 1. The description thereof will be omitted here.

[0052] The imaging unit 24 includes a camera. The imaging unit 24 can capture images of the surroundings. The imaging unit 24 can store the captured images in the storage unit 23 or transmit them to the control unit 21 for image analysis.

[0053] The display unit 25 is, for example, a display. The display is, for example, an LCD or an organic EL display. "LCD" is the abbreviation of Liquid Crystal Display. "EL" is the abbreviation of Electro Luminescence. The display unit 25 displays information obtained through the operation of the vehicle 2. The display unit 25 may not be provided in the vehicle 2, but may be connected to the vehicle 2 as an external output device. As a connection method, for example, any method such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used. The display unit 25 can, for example, display a camera image and location identification information received from the information processing device 1.

[0054] The user terminal 3 is a terminal operated by the user U01 and is a mobile device such as a mobile phone, a smartphone, a wearable device, or a tablet.

[0055] Reference Figure 4 , describes the internal structure of user terminal 3 in detail.

[0056] The user terminal 3 includes a control unit 31, a communication unit 32, a storage unit 33, a positioning unit 34, and an input / output unit 35. The components of the user terminal 3 are connected to each other via, for example, a dedicated line so as to be communicable with each other.

[0057] The hardware configuration of the control unit 31 , the communication unit 32 , and the storage unit 33 of the user terminal 3 in this embodiment may be the same as the hardware configuration of the control unit 21 , the communication unit 22 , and the storage unit 23 of the vehicle 2 .

[0058] The positioning unit 34 includes at least one GNSS receiver. "GNSS" is the abbreviation of Global Navigation Satellite System. In GNSS, for example, it includes at least any one of GPS, QZSS, BeiDou, GLONASS, and Galileo. "GPS" is the abbreviation of Global Positioning System. "QZSS" is the abbreviation of Quasi-Zenith Satellite System. The satellites of QZSS are called Quasi-Zenith Satellites. "GLONASS" is the abbreviation of Global Navigation Satellite System. The positioning unit 34 measures the position of the user terminal 3. The result of the measurement is obtained by the control unit 31 as the position information of the user terminal 3. "Position information" refers to information that can determine the position of the user terminal 3. The position information includes, for example, address, latitude, longitude, or altitude.

[0059] The input-output unit 35 includes at least one input interface. The input interface is, for example, a physical key, an electrostatic capacitance key, a pointing device, a touch screen integrated with the display, or a microphone. The input-output unit 35 accepts operations for inputting information used for the actions of the user terminal 3. The input-output unit 35 may also be connected to the user terminal 3 as an external input device instead of being provided in the user terminal 3. As a connection method, for example, any method such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used. "USB" is the abbreviation of Universal Serial Bus. "HDMI (registered trademark)" is the abbreviation of High-Definition Multimedia Interface.

[0060] The input / output unit 35 further includes at least one output interface. The output interface is, for example, a display or a speaker. The display is, for example, an LCD or an organic EL display. "LCD" is the abbreviation for Liquid Crystal Display. "EL" is the abbreviation for Electro Luminescence. The input / output unit 35 outputs information obtained through the operation of the user terminal 3. The input / output unit 35 may not be provided in the user terminal 3, but may be connected to the user terminal 3 as an external output device. As a connection method, for example, any method such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

[0061] The imaging device 4 includes a camera. The imaging device 4 is capable of capturing images of the surrounding area. For example, the imaging device 4 may be a surveillance camera, an anti-theft camera, or a weather camera. The imaging device 4 may be fixed or movable. The imaging device 4 may be a driving recorder mounted on any vehicle. The imaging device 4 transmits the captured images to the information processing device 1 for image analysis.

[0062] The following describes in detail the processing executed by the information processing system S of this embodiment. Here, a scenario in which a user U01 of the user terminal 3 requests the dispatch of a vehicle 2 will be described.

[0063] The control unit 31 of the user terminal 3 receives input of feature information from the user U01 via the input / output unit 35. The feature information is information related to the features of the user U01. The feature information may include information related to the physical features of the user U01. Physical features include features such as the face (e.g., eyes, nose, mouth), height, or outline.

[0064] The control unit 31 sends the characteristic information to the information processing device 1 via the communication unit 32. Figure 5 As shown in FIG. 1 , the control unit 11 of the information processing device 1 associates the feature information with the user ID and stores it in the feature information DB.

[0065] The control unit 31 receives input of reservation information from the user U01 via the input / output unit 35. The reservation information is information regarding the reservation of the vehicle 2. The reservation information includes the user ID of the user U01 who made the reservation and the reservation date and time when the user U01 wishes to dispatch the vehicle.

[0066] The control unit 31 sends the reservation information to the information processing device 1 via the communication unit 32. Figure 6 As shown in FIG. 1 , the information processing device 1 associates the reservation information with the reservation ID and stores it in the reservation information DB.

[0067] User U01 waits at the assigned location of vehicle 2. Upon receiving a current location transmission instruction or detecting that the current date and time is approaching the scheduled date and time, control unit 31 transmits the current location information of user terminal 3 to information processing device 1 via communication unit 32.

[0068] The storage unit 13 stores position information of one or more imaging devices 4. Upon receiving information on the current position of the user terminal 3, the information processing device 1 selects one or more imaging devices 4 around the current position (eg, within a predetermined distance from the current position).

[0069] The control unit 11 sends an image capturing instruction to one or more selected image capturing devices 4. Figure 7 As shown, user U01 operating user terminal 3 is present at the corner where building BL is located among the four corners of the intersection. User U02 is present at the corner where user U01 is present. Camera 4 transmits the captured image to information processing device 1.

[0070] The control unit 11 of the information processing device 1 acquires the feature information of the user U01 from the feature information DB. The control unit 11 performs image recognition on the captured image using the acquired feature information. Figure 8 An example of a captured image in which the user U01 is recognized is shown. The control unit 11 recognizes the user U01 from the captured image. The control unit 11 identifies the position of the user U01 and generates location identification information such as the following.

[0071] User U01 is on a specific road or at a specific intersection

[0072] User U01 is in a corner where a specific object (such as a building BL, a tree, or a billboard) exists.

[0073] User U01 is on the sidewalk on the north side of a specific road

[0074] like Figure 8 As shown, the control unit 11 superimposes the location identification information on the captured image in which the user U01 is recognized. The location identification information may include a highlight display 81 of the user U01. The location identification information may include an annotation 82 that identifies the location of the user.

[0075] The control unit 11 specifies the vehicle (here, the vehicle 2 ) assigned to the user U01 and transmits the captured image on which the location identification information is superimposed to the vehicle 2 via the communication unit 12 .

[0076] After receiving the captured image with the location identification information superimposed thereon, the vehicle 2 displays the captured image on the display unit 25. The driver of the vehicle 2 can identify the location of the user U01 by visually recognizing the location identification information and arrive near the user U01.

[0077] As an alternative example, the control unit 11 may generate the place identification information in the form of voice. The control unit 11 transmits the place identification information to the vehicle 2 via the communication unit 12. The vehicle 2 may output the place identification information in the form of voice from a speaker.

[0078] The above embodiment describes the deployment of vehicle 2. However, as an alternative, the information processing system S can be applied to scenarios where it is necessary to identify the location or person that the user initially visits. Examples of such scenarios include home delivery, contactless delivery, or mail delivery. In this case, the characteristic information may be information identifying the door of user U01's home or the vehicle parked in front of their home. As another alternative, the information processing system S can also be applied to scenarios where users meet each other or provide pick-up and drop-off services.

[0079] Reference Figure 9 , describing an information processing method using the information processing system S of this embodiment.

[0080] In step S1, the control unit 11 of the information processing device 1 receives feature information and reservation information from the user terminal 3 via the communication unit 12. The feature information and reservation information may be received simultaneously or separately.

[0081] In step S2 , the control unit 11 registers the feature information and reservation information in the storage unit 13 .

[0082] In step S3 , the control unit 11 receives the location information of the user terminal 3 from the user terminal 3 .

[0083] In step S4 , the control unit 11 selects one or more imaging devices 4 around the user terminal 3 based on the position information.

[0084] In step S5 , the control unit 11 transmits an image capturing instruction to the selected imaging device 4 .

[0085] In step S6 , the imaging device 4 captures an image.

[0086] In step S7 , the imaging device 4 transmits the captured image to the information processing device 1 .

[0087] In step S8 , the control unit 11 recognizes the user U01 of the user terminal 3 from the captured image.

[0088] In step S9, the control unit 11 specifies the location of the user U01. The control unit 11 superimposes the location specifying information on the captured image. Whether or not to superimpose the location specifying information is optional.

[0089] In step S10 , the control unit 11 transmits the location identification information and the captured image to the vehicle 2 .

[0090] As described above, according to this embodiment, the control unit 11 receives one or more camera images from the camera device 4 and recognizes the user U01 from the one or more camera images. The control unit 11 sends the camera image in which the user U01 is recognized to the vehicle 2 assigned to the user U01. With this structure, the control unit 11 can provide the camera image of the user U01 to the vehicle 2, so that the driver of the vehicle 2 can be assisted in determining the correct location of the user U01. From another perspective, the accuracy of the location information is sometimes low in places with many buildings, etc. However, it is easy to determine the location of the user U01 by providing the camera image of the user U01. Therefore, the driver of the vehicle 2 can reliably build a pick-up route to the waiting place of the user U01, so that the pick-up from the wrong direction can be reduced.

[0091] Furthermore, according to this embodiment, the control unit 11 further executes: receiving characteristic information of the user U01 from the user terminal 3; and using the characteristic information to identify the user U01 from one or more captured images. The characteristic information includes information related to the physical characteristics of the user U01. With this configuration, the information processing device 1 can improve the recognition accuracy of the user U01.

[0092] Furthermore, according to this embodiment, the control unit 11 further executes the following operations: generating location identification information from a captured image in which the user U01 is recognized, identifying the location where the user U01 is present; overlaying the location identification information onto the captured image; and transmitting the captured image to the vehicle 2. The location identification information includes information about roads, intersections, buildings, trees, or billboards. The location identification information also includes a highlighted display of the user U01. This configuration facilitates identification of the location where the user U01 is present.

[0093] Furthermore, according to this embodiment, the control unit 11 further generates location identification information in the form of voice, identifying the location of the user U01 from a captured image in which the user U01 is recognized, and transmits the location identification information to the vehicle 2. This configuration facilitates identification of the location of the user U01 even when the vehicle 2 does not include a display unit.

[0094] While the present disclosure has been described with reference to the accompanying drawings and embodiments, various modifications and variations can be made by those skilled in the art based on the present disclosure. Furthermore, modifications can be made without departing from the spirit of the present disclosure. For example, the functions included in each unit or step can be rearranged without logical contradiction, and multiple units or steps can be combined into one or divided.

[0095] For example, in the above-mentioned embodiment, the program for executing all or part of the functions or processes of the information processing device 1 can be pre-recorded in a recording medium that can be read by a computer. The recording medium that can be read by a computer includes a non-temporary computer-readable medium, such as a magnetic recording device, an optical disc, an optical magnetic recording medium, or a semiconductor memory. The circulation of the program can be carried out, for example, by selling, transferring, or renting a removable recording medium such as a digital versatile disc (DVD) or a compact disc read-only memory (CD-ROM) on which the program is recorded. In addition, the circulation of the program can also be carried out by storing the program in a storage device of any server and sending the program from any server to other computers. In addition, the program can also be provided as a program product. The present disclosure can also be implemented as a program executable by a processor.

[0096] The computer temporarily stores a program recorded on a removable recording medium or a program transmitted from a server in a main storage device, for example. Then, the computer uses a processor to read the program stored in the main storage device and uses the processor to execute processing in accordance with the read program. The computer can also read the program directly from the removable recording medium and execute processing in accordance with the program. The computer can also successively execute processing in accordance with the received program each time the program is transmitted from the server to the computer. It is also possible to perform processing using a so-called ASP-type service that realizes functions only by executing instructions and obtaining results, instead of transmitting the program from the server to the computer. "ASP" is an abbreviation for application service provider. The program includes information used in the processing performed by the electronic computer and is information in accordance with the program. For example, data that is not a direct instruction to the computer but has the nature of specifying the processing of the computer is equivalent to "information in accordance with the program."

Claims

1. An information processing device capable of communicating with a user terminal and a camera device, comprising: Control Department; as well as Ministry of Communications, The control unit executes: receiving location information from the user terminal via the communication unit; receiving characteristic information of the user from the user terminal; selecting a camera device around the user terminal according to the location information; sending an image capturing instruction to the selected imaging device via the communication unit; receiving one or more captured images from the imaging device via the communication unit; using the feature information, identifying the user of the user terminal from the one or more captured images; and The captured image in which the user is recognized is transmitted to a vehicle assigned to the user via the communication unit.

2. The information processing device according to claim 1, wherein The characteristic information includes information related to the user's physical characteristics.

3. The information processing device according to claim 1 or 2, wherein: The control unit also performs: generating location identification information for identifying a location where the user is located based on the camera image in which the user is identified; overlaying the location identification information onto the camera image; as well as The camera image is sent to the vehicle.

4. The information processing device according to claim 3, wherein: The location determination information includes information about roads, intersections, buildings, trees, or billboards. The information processing apparatus according to claim 3 , wherein: The location identification information includes a highlighted display of the user. The information processing apparatus according to claim 3 , wherein: The control unit also performs: generating location identification information for identifying a location of the user in the form of sound based on a captured image in which the user is recognized; and The location identification information is sent to the vehicle. 7 . A vehicle equipped with the information processing device according to claim 1 .

8. A recording medium having a program recorded thereon, the program causing a computer, which is an information processing device capable of communicating with a user terminal and an imaging device, to execute an operation, the operation comprising: receiving location information from the user terminal; receiving characteristic information of the user from the user terminal; selecting a camera device around the user terminal according to the location information; Sending an image capturing instruction to the selected camera device; receiving one or more camera images from the camera device; using the feature information, identifying the user of the user terminal from the one or more captured images; and The camera image in which the user is recognized is sent to a vehicle assigned to the user.

9. The recording medium according to claim 8, wherein The characteristic information includes information related to the user's physical characteristics.

10. The recording medium according to claim 8 or 9, wherein The program causes the computer to further execute: generating location identification information for identifying a location where the user is located based on the camera image in which the user is identified; overlaying the location identification information onto the camera image; as well as The camera image is sent to the vehicle.

11. The recording medium according to claim 10, wherein The location determination information includes information about roads, intersections, buildings, trees, or billboards.

12. The recording medium according to claim 10, wherein The location identification information includes a highlighted display of the user.

13. An information processing method, performed by an information processing device capable of communicating with a user terminal and a camera device, the information processing method comprising: receiving location information from the user terminal; receiving characteristic information of the user from the user terminal; selecting a camera device around the user terminal according to the location information; Sending an image capturing instruction to the selected camera device; receiving one or more camera images from the camera device; using the feature information, identifying the user of the user terminal from the one or more captured images; and The camera image in which the user is recognized is sent to a vehicle assigned to the user.

14. The information processing method according to claim 13, wherein: The characteristic information includes information related to the user's physical characteristics.

15. The information processing method according to claim 13 or 14, further comprising: generating location identification information for identifying a location where the user is located based on the camera image in which the user is identified; overlaying the location identification information onto the camera image; as well as The camera image is sent to the vehicle.

16. The information processing method according to claim 15, wherein: The location determination information includes information about roads, intersections, buildings, trees, or billboards.

17. The information processing method according to claim 15, wherein: The location identification information includes a highlighted display of the user.

Images