Information processing apparatus, information processing system, and information processing method

Abstract

The invention provides an information processing device, an information processing system, and an information processing method, which can enable a user in a place different from a moving body to perform more appropriate simulated riding. An information management device processes information provided from a first device mounted on a moving body to a second device used by a user at a location different from the moving body. The information processing apparatus includes an acquisition unit, a providing unit, a receiving unit, and a brightness changing unit. The acquisition unit acquires information including moving image data transmitted by the first device. The provision unit provides the moving image data to the second device on the basis of the information acquired by the acquisition unit. The reception unit receives a first operation performed by the user on a region of a portion of the presented moving image on the basis of the moving image data supplied to the second device by the supply unit. The brightness changing unit changes the brightness of a region of a portion of the moving image on the basis of the first operation received by the receiving unit.

Description

Technical Field

[0001] This invention relates to an information processing device, an information processing system, and an information processing method. Background Technology

[0002] In recent years, efforts have intensified to provide sustainable transportation systems that also take into account vulnerable groups among transport participants, particularly the elderly, people with disabilities, and children. To achieve this, research and development are particularly focused on further improving the safety and convenience of transportation through the development of mobility facilities geared towards vulnerable transport users.

[0003] Relatedly, it is known that technology can automatically adjust the brightness of the projected image based on the ambient brightness (for example, see Patent Document 1 (Japanese Patent Application Publication No. 2005-321535)). Summary of the Invention

[0004] [The problem the invention aims to solve]

[0005] However, when using devices located in a different place from the mobile vehicle to provide a simulated experience as if riding in the mobile vehicle (hereinafter referred to as "simulated ride"), it is a problem that this can burden users who are taking simulated rides in a different place from the mobile vehicle.

[0006] This application aims to address the aforementioned issues by providing users in locations different from the actual vehicles they are in with a more appropriate simulated ride. This, in turn, contributes to the development of sustainable transportation systems and supports better connections between urban areas and surrounding regions.

[0007] [Methods used to solve problems]

[0008] The information processing device, information processing system, and information processing method involved in this invention adopt the following structure.

[0009] (1): One aspect of the present invention relates to an information processing apparatus that processes information provided from a first device mounted on a mobile body to a second device used by a user at a location different from the mobile body, wherein the information processing apparatus comprises: an acquisition unit that acquires information including dynamic image data sent by the first device; a supply unit that provides dynamic image data to the second device based on the information acquired by the acquisition unit; a receiving unit that receives a first operation, the first operation being an operation performed by the user on a region of a indicated dynamic image based on the dynamic image data provided to the second device by the supply unit; and a brightness changing unit that changes the brightness of the region of the dynamic image based on the first operation received by the receiving unit.

[0010] (2): Based on the above (1) scheme, the first device may also include a camera device, and the brightness changing unit may also generate control information to change the brightness of the camera device based on the first operation received by the receiving unit.

[0011] (3): Based on the above (1) scheme, the receiving unit can also notify the mobile body of the first operation that the brightness has been changed based on the first operation received.

[0012] (4): Based on the above (2) scheme, the camera device can also adjust the brightness of a portion of the area based on the surrounding brightness.

[0013] (5): Based on the above (1) scheme, the information processing device may also include a detection unit that detects the gaze area of ​​the dynamic image that the user is looking at, and the brightness change unit may also change the brightness of the gaze area of ​​the dynamic image based on the gaze area detected by the detection unit.

[0014] (6): Based on the above (1) scheme, the information processing device may also include a sound recognition unit that detects and recognizes sound. The brightness adjustment unit may also adjust the brightness of the associated POI information when the sound recognized by the sound recognition unit is associated with the POI (Point of Interest) information around the moving body.

[0015] (7): Based on the above (1) scheme, the information processing device may also include a detection unit that detects the area of ​​the dynamic image that the user is looking at. The receiving unit may also accept a second operation if the detection unit detects that the user is looking at a part of the dynamic image for a predetermined time or more. The brightness changing unit may also change the brightness of the area of ​​the dynamic image based on the second operation received by the receiving unit.

[0016] (8): Based on the above (7) scheme, the second operation may also include blinking, posture, and speaking.

[0017] (9): Based on the above (7) scheme, the second operation may contain more actions than the first operation.

[0018] (10): Based on the above (1) scheme, the brightness adjustment unit may also give priority to the second device that is initially connected to the information processing device among the plurality of second devices for brightness adjustment.

[0019] (11): Based on the above (1) scheme, the brightness change unit may also notify each of the conflicting second devices when the first operation received by the receiving unit from the multiple second devices is conflicted.

[0020] (12): Based on the above (1) scheme, the brightness change unit may also perform processing to detect the POI information of the area surrounding a part of the dynamic image in which the first operation was performed by the users of the multiple second devices in the case of conflict of the first operation from multiple second devices, and adjust the brightness of the area where the POI information is detected or the POI information when the POI information can be detected.

[0021] (13): Based on the above (1) scheme, the brightness change unit may also perform processing to detect the POI information around a region of the dynamic image in which the first operation was performed by the user of each of the multiple second devices that had the first operation received by the receiving unit. If multiple POI information can be detected, the unit may notify the user of each of the multiple second devices that had the first operation conflict of the summary of the multiple POI information.

[0022] (14): One aspect of the present invention relates to an information processing system, wherein the information processing system comprises: a first device mounted on a mobile body; a second device used by a user in a location different from the mobile body; an acquisition unit that acquires information including dynamic image data sent by the first device; a providing unit that provides dynamic image data to the second device based on the information acquired by the acquisition unit; a receiving unit that receives a first operation, the first operation being an operation performed by the user on a region of a indicated dynamic image based on the dynamic image data provided to the second device by the providing unit; and a brightness changing unit that changes the brightness of the region of the dynamic image based on the first operation received by the receiving unit.

[0023] (15): One aspect of the present invention relates to an information processing method, wherein an information processing device processes information provided from a first device mounted on a mobile body to a second device used by a user in a location different from the mobile body, the information processing method causing the information processing device to perform the following processing: acquiring information containing dynamic image data sent by the first device; providing dynamic image data to the second device based on the acquired information; receiving a first operation, the first operation being an operation performed by the user on a region of a prompted dynamic image based on the dynamic image data provided to the second device; and changing the brightness of the region of the dynamic image based on the received first operation.

[0024] [Invention Effects]

[0025] According to the above schemes (1) to (15), users in locations different from the moving vehicle can more appropriately simulate riding a vehicle. Attached Figure Description

[0026] Figure 1 This is a diagram showing the usage environment of the information processing system 1 and the management server 300.

[0027] Figure 2 This is a diagram representing an example of the content of user data 360.

[0028] Figure 3 This is a structural diagram of the first device 100.

[0029] Figure 4 This is a diagram showing an example of the configuration of a portion of the first device 100 in the mobile body M.

[0030] Figure 5 This is a structural diagram of the second device 200.

[0031] Figure 6 It is a diagram used to illustrate the image corresponding to the pointing direction.

[0032] Figure 7 This is a diagram illustrating a first example of the functional structure of the first control device 170 and the second control device 270.

[0033] Figure 8 This is a second example of a diagram showing the functional structure of the first control device 170 and the second control device 270.

[0034] Figure 9 This is a sequence diagram representing an example of the processing performed by information processing system 1.

[0035] Figure 10 This is an example of an image IM10 representing a query.

[0036] Figure 11 This is an example of an image IM20 showing the result of a request.

[0037] Figure 12A This is a sequence diagram representing the first example of the processing performed by information processing system 1.

[0038] Figure 12B This is a sequence diagram representing the second example of the processing performed by information processing system 1.

[0039] Figure 12C This is a sequence diagram representing the third example of the processing performed by information processing system 1.

[0040] Figure 12D This is a sequence diagram representing the fourth example of the processing performed by information processing system 1.

[0041] Figure 13 This is a diagram used to illustrate an example of providing information.

[0042] [Explanation of Labels in the Attached Image]

[0043] 1. Information Processing System

[0044] 100 First Device

[0045] 110 First Communication Device

[0046] 120 First Microphone

[0047] 125 External Sensors

[0048] 130 camera unit

[0049] 132 Indoor Camera

[0050] 134 Outdoor Camera

[0051] 140 First loudspeaker

[0052] 150 User display devices

[0053] 160 HMI

[0054] 170 First Control Device

[0055] 190 Controlled Object Equipment

[0056] 200 Second Device

[0057] 210 Second Communication Device

[0058] 220 Second Microphone

[0059] 230 Detection device

[0060] 232 Direction Detection Device

[0061] 233 Gaze area detection device

[0062] 234 Head position detection device

[0063] 236 Motion Sensors

[0064] 240 Second speaker

[0065] 250 Moving Body Image Display Device

[0066] 260 HMI

[0067] 270 Second Control Device

[0068] 300 Management Server

[0069] 310 Communication device

[0070] 315 Obtained by the Department

[0071] 320 Matching Processing Unit

[0072] 330 Supply Department

[0073] 332 Receiving Department

[0074] 334 Brightness Change Section

[0075] 340 Expense Management Department

[0076] 350 Storage Unit

[0077] M is a moving body. Detailed Implementation

[0078] Hereinafter, embodiments of the information processing apparatus, information processing system, and information processing method of the present invention will be described with reference to the accompanying drawings. It should be noted that the following description applies to the information processing system of the information processing apparatus. The information processing system includes, for example, a first device mounted on a mobile body carried by occupants, a second device used by a user in a different location from the mobile body, and an information processing apparatus. The mobile body refers to, for example, a vehicle, but can be any mobile body that occupants can ride on (e.g., a ship, an aircraft). The occupants are primarily the driver of the mobile body, but can also be occupants other than the driver.

[0079] The first and second devices operate in a state where a telephone conversation is conducted by transmitting sound collected by a microphone to the other party and playing it through a speaker. Furthermore, a portion of the image captured by the camera unit of the first device is displayed on the second device, thus providing MR (Mixed Reality) to the second device. Therefore, the user of the second device, even though in a different location from the mobile device, can experience the feeling of actually riding in the mobile device (simulated ride experience). Moreover, the user can engage in conversation with the simulated user of the mobile device via the first device, thus gaining the feeling that the user is actually riding in the mobile device together. Alternatively, a portion of the image captured by the camera unit of the first device can be displayed on the second device, thus providing VR (Virtual Reality) to the second device. Therefore, the user of the second device, even though in a different location from the mobile device, can experience the feeling of actually riding in the mobile device (simulated ride experience). The user can engage in conversation with the simulated user of the mobile device via the first device, thus gaining the feeling that the user is actually riding in the mobile device together. Furthermore, the user can be fully immersed in the world of images while using the application. As mentioned above, the simulated experience where the user feels as if they are actually riding in a moving vehicle is sometimes referred to as "simulated ride." The first and second devices do not need to be in a one-to-one relationship; rather, one of multiple first devices can be matched with multiple second devices in a one-to-many relationship, thereby functioning as an information processing system. In this case, for example, it is possible for one passenger to communicate simultaneously with multiple users or for one passenger to communicate sequentially with multiple users.

[0080] <Basic Structure>

[0081] Figure 1 This diagram illustrates the usage environment of the information processing system 1 and the management server 300. The information processing system 1 includes a first device (mobile device) 100 mounted on a mobile body M carried by passenger P, and a second device (user device) 200 used by user U at a location different from the mobile body M (excluding occasional nearby locations). The first device 100, the second device 200, and the management server 300 communicate with each other via a network NW. The information processing system 1 may or may not include the management server 300.

[0082] Network NW includes, for example, at least one of the following: Internet, WAN (Wide Area Network), LAN (Local Area Network), mobile communication network, cellular network, etc. Management server 300 is an example of an "information processing device." It should be noted that management server 300 can also be implemented by server devices and storage devices assembled in a cloud computing system. In this case, the functions of management server 300 can also be implemented by multiple server devices and storage devices in the cloud computing system. The first device 100 mounted on the mobile body M can also be implemented by multiple units.

[0083] The management server 300 manages the information provided to the first device 100 and the second device 200 respectively, and manages the communication between the first device 100 and the second device 200. The management server 300 includes, for example, a communication device 310, an acquisition unit 315, a matching processing unit 320, a providing unit 330, a receiving unit 332, a brightness adjustment unit 334, a cost management unit 340, and a storage unit 350. The acquisition unit 315, the matching processing unit 320, the providing unit 333, the receiving unit 332, the brightness adjustment unit 334, and the cost management unit 340 are implemented, for example, by executing programs (software) using a hardware processor such as a CPU (Central Processing Unit).

[0084] Some or all of these components can be implemented using hardware (including the circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or through the coordinated use of software and hardware. The program can be pre-stored on storage devices such as HDD (Hard Disk Drive) and flash memory (storage devices with non-transitory storage media), or on removable storage media such as DVDs and CD-ROMs (non-transitory storage media), and installed via the storage media mounted on a drive. The program can also be stored on the management server 300 and updated appropriately via a connection to the network NW.

[0085] The communication device 310 is a communication interface for connecting to a network NW. It should be noted that the communication between the communication device 310 and the first device 100, as well as the communication between the communication device 310 and the second device 200, is performed, for example, using TCP / IP (Transmission Control Protocol / Internet Protocol).

[0086] The acquisition unit 315 acquires various information sent by the first device 100, the second device 200, and other external devices via the network NW. This information may include dynamic image data, driver's voice data, and vehicle data (such as METER (instrument panel) and IVI (in-vehicle infotainment) system display content).

[0087] The matching processing unit 320 is implemented, for example, by executing a program (command group) stored in a storage medium using a processor such as a CPU. For instance, when the communication device 310 receives a matching request from user U via the second device 200 or from passenger P via the first device 100, the matching processing unit 320 performs a matching between user U and passenger P by referring to user data 360. Furthermore, the matching processing unit 320 uses the communication device 310 to send communication identification information of passenger P's first device 100 to the second device 200 of the successfully matched user U, and also sends communication identification information of user U's second device 200 to the first device 100 of the successfully matched passenger P. Upon receiving this information, the first device 100 and the second device 200 can perform high-real-time communication, for example, using UDP (User Datagram Protocol).

[0088] The providing unit 330 generates information to be provided to the first device 100 and the second device 200 respectively based on various information obtained by the obtaining unit 315, and sends the generated information to the target device. The providing unit 330 generates information such as the processing result processed by the matching processing unit 320 and the cost information (accounting information) managed by the cost management unit 340, and provides the generated information to the target device.

[0089] The receiving unit 332 receives first operation information and second operation information sent from the second device 200. The first and second operation information are based on operations performed by the user U on a portion of a displayed dynamic image, according to dynamic image data provided by the providing unit 330 to the second device 200. Furthermore, the receiving unit 332 receives text information sent from the second device 200. The receiving unit 332 also receives information indicating the gaze area sent from the second device 200.

[0090] The brightness adjustment unit 334 adjusts the brightness of a region of the moving image based on the first operation information received by the receiving unit 332. For example, the brightness adjustment unit 334 generates control information based on the first operation information to adjust the brightness of the camera in the camera unit 130 that captures a region of the moving image. The brightness adjustment unit 334 adjusts the brightness of a region of the moving image based on the second operation information received by the receiving unit 332. For example, the brightness adjustment unit 334 generates control information based on the second operation information to adjust the brightness of the camera in the camera unit 130 that captures a region that the user U gazes at for a predetermined time or more.

[0091] The brightness adjustment unit 334 determines, based on the text information received by the receiving unit 332, whether the POI (Point of Interest) information registered in the provision information DB362 stored in the storage unit 350 includes POI information associated with the text information. If the POI information registered in the provision information DB362 includes POI information associated with the text information, the brightness adjustment unit 334 determines, among the cameras included in the camera unit 130, the camera that captured the aforementioned associated POI information based on the information (moving image data) provided by the providing unit 330 to the second device 200, and generates control information to change the brightness of the determined camera.

[0092] The brightness adjustment unit 334 adjusts the brightness of the gaze area of ​​the moving image based on the information representing the gaze area received by the receiving unit 332. For example, the brightness adjustment unit 334 generates control information based on the information representing the gaze area to adjust the brightness of the camera in the camera unit 130 that captures the gaze area being viewed by the user U.

[0093] The brightness adjustment unit 334 sends the generated control information to the first device 100.

[0094] The Fee Management Department 340 manages the fees charged to user U based on information provided to user U, and the fees charged to passenger P based on information provided to passenger P of the mobile vehicle M. The Fee Management Department 340 may also manage, for example, the remuneration paid to user U and passenger P based on information provided from user U and passenger P. The Fee Management Department 340 may also perform processing related to the accounting of user U and passenger P.

[0095] The storage unit 350 can be implemented using various storage devices described above, or SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory). The storage unit 350 stores, for example, user data 360, information DB362, programs, and other various types of information.

[0096] Figure 2 This diagram illustrates an example of the content of user data 360. User data 360 includes a passenger list 360A and a user list 360B. In passenger list 360A, for example, the passenger ID of passenger P of mobile body M, their communication identification information (IP address, etc.), the user ID of user U who is the matching target, the mobile body information of the passenger, and the passenger's settings regarding whether or not they can provide information are mutually correlated. The mobile body information includes, for example, equipment information mounted on mobile body M (equipment information), and vehicle specification information indicating the size and shape of mobile body M. The mobile body information may also include information transmitted from mobile body M at predetermined intervals related to the current location, destination, and surrounding conditions of mobile body M (e.g., while traveling on a coastal road).

[0097] In the user list 360B, a correspondence is established between user ID, their communication identification information (IP address, etc.), the passenger P who is matched, and user information. User information may include physical-related information (e.g., height, sitting height) and information that can predict physical characteristics (e.g., age). The "can be provided" information is information that the mobile body M can or cannot provide, for example, set by the passenger P. The "can be provided" information can also be set according to the equipment carried by the mobile body M or according to the user U. Examples of "can be provided" information include, but are not limited to, "permitted image provision," "disallowed sound provision," "permitted indoor image provision but not outdoor image provision," "disallowed passenger image provision," and "disallowed use of navigation device." The "can be provided" information may also include the cost (service provision fee) for making the provision possible. User data 360 is not limited to this as long as it contains this information. Figure 2 The shape shown can also be generated in any shape.

[0098] The information provided in DB362 stores various information provided to the user U or passenger P. This information includes, for example, map information, POI (Point of Interest) information, and computer-generated images (e.g., CG images of people, symbols, icons, etc.). POI information, for example, includes information related to various shops, theme parks, and landmarks at each location, and may be included in the map information. Audio information may also be included. The information provided in DB362 may include advertising information. This advertising information may include, for example, advertisements related to the mobile vehicle M, advertisements related to the user U or passenger P, and other advertisements related to the goods and services of shops. It should be noted that inserted advertising information may be managed separately from indoor and outdoor information, and may differ from the inserted advertising information during real-time distribution when it is archived and distributed later (e.g., closed shops become the latest shops, and introduced menus become the latest menus). Advertising information may include, for example, animated images and audio.

[0099] Figure 3 This is a structural diagram of the first device 100. The first device 100 includes, for example, a first communication device 110, a first microphone 120, an external sensor 125, a camera unit 130, a first speaker 140, a user display device 150, an HMI (Human Machine Interface) 160, and a first control device 170. The first control device 170 is connected to a control object device 190 mounted on the mobile body M.

[0100] The first communication device 110 is a communication interface for communicating with the communication device 310 of the management server 300 and the second communication device 210 of the second device 200 (described later) via a network NW.

[0101] The first microphone 120 collects at least the sound emitted by the occupant P. The first microphone 120 may also be installed indoors within the mobile body M and has the sensitivity to also collect external sounds from the mobile body M. The first microphone 120 may also include a microphone installed indoors within the mobile body M and a microphone installed outside the mobile body M. Hereinafter, the sound information obtained by the microphone installed indoors will sometimes be referred to as "indoor sound information." The sound collected by the first microphone 120 is transmitted, for example, via the first control device 170 to the management server 300 or the second device 200 by the first communication device 110. In cases where a microphone cannot be installed outside the mobile body M, the indoor sound information can be processed to simulate outdoor sound information based on driving information (vehicle speed, acceleration / deceleration, road vibration, etc.) and the surrounding driving environment. Alternatively, the speaker's position relative to the mobile body M (whether the speaker is indoors or outdoors) can be recorded, and the collected sound can be processed according to this positional relationship.

[0102] An external sensor 125 detects the position of objects around the moving body M. The external sensor 125 can be, for example, a radar device, a LIDAR (Light Detection and Ranging) sensor, or various other proximity sensors. The radar device emits millimeter-wave or other electromagnetic waves around the moving body M and detects the reflected waves (reflected waves) to at least detect the object's position (distance and orientation). The radar device can also detect the object's position and velocity using FM-CW (Frequency Modulated Continuous Wave). The LIDAR sensor illuminates the moving body M with light (or electromagnetic waves with wavelengths similar to light) and measures the scattered light, detecting the distance to the object based on the time from emission to reception. The illuminated light can be, for example, a pulsed laser. The radar device or LIDAR sensor can be mounted at any location on the moving body M. The external sensor 125 can also detect surrounding objects using images captured by the outdoor camera 134 of the camera unit 130.

[0103] Camera unit 130 is an example of a camera device, such as having an indoor camera 132 and an outdoor camera 134. Camera unit 130 obtains control information from management server 300 via first communication device 110 and adjusts the brightness based on the obtained control information. First speaker 140 outputs sound emitted by user U obtained via first communication device 110. Details regarding the configuration of camera unit 130 and first speaker 140 will be provided later. Figure 4 To explain.

[0104] The user display device 150 displays the user U as if the user were present inside the moving body M. For example, the user display device 150 displays a hologram showing the user U in a portion corresponding to the reflector or window of the moving body M.

[0105] HMI160 includes a touch panel, a voice response device (smart agent device), etc. HMI160 receives various instructions from occupant P to the first device 100 and provides various information to occupant P.

[0106] The first control device 170 includes, for example, a processor such as a CPU, and a storage medium connected to the processor and storing a program (command group), and controls various parts of the first device 100 by executing the command group through the processor.

[0107] The control device 190 is, for example, a navigation device mounted on the mobile body M that guides the user to a destination, or a driving support device that controls one or both of the steering and speed of the mobile body M to support the occupant P's driving. The control device 190 includes, for example, a seat drive mechanism capable of adjusting the seat's position (forward, backward, left, right), orientation, and height. When viewing images via the second device 200, if the camera unit 130 of the first device 100 is mounted on the seat, the impact on the image can be suppressed by preventing seat movement. Even when seat movement is permitted, processing such as changing the viewing angle can be performed to avoid affecting the image when the seat has moved. If the user U wishes to view images outside the viewing angle of the current camera unit 130, the seat drive mechanism can be controlled as a request from the second device 200.

[0108] Figure 4 This diagram illustrates a configuration example of a portion of the first device 100 in the mobile body M. An interior camera 132 is mounted, for example, with the aid of accessory 132A, to the neck pillow of the co-driver's seat S2 (an example of a "prescribed seat"), and is positioned slightly offset from the backrest of the co-driver's seat S2 towards the direction of travel of the mobile body M. The interior camera 132 has a wide-angle lens, capable of capturing the area indicated by the shaded region 132B in the diagram. The interior camera 132 can capture images not only of the interior of the mobile body M but also of the exterior through the windows. In the following description, the co-driver's seat S2 is assumed to be a prescribed seat, but the prescribed seat could also be a rear seat or other seats.

[0109] The outdoor camera 134 may include, for example, multiple sub-outdoor cameras 134-1 to 134-4. By combining the images captured by the multiple sub-outdoor cameras 134-1 to 134-4, a panoramic image of the exterior of the moving body M can be obtained. The outdoor camera 134 may also include, instead of these (or in addition to), a wide-angle camera mounted on the roof of the moving body M.

[0110] It should be noted that an additional camera capable of capturing images from behind the co-pilot's seat S2 can be added to the indoor camera 132. The moving body image described later can be generated into a 360-degree panoramic image by combining images captured by one or more indoor cameras 132 through the first control device 170, or by appropriately combining images captured by indoor cameras 132 with images captured by outdoor cameras 134.

[0111] The first speaker 140 outputs the sound of the user U obtained via the first communication device 110. The first speaker 140 includes, for example, a plurality of sub-first speakers 140-1 to 140-5. For example, sub-first speaker 140-1 is located in the center of the dashboard. Sub-first speaker 140-2 is located at the left end of the dashboard. Sub-first speaker 140-3 is located at the right end of the dashboard. Sub-first speaker 140-4 is located at the lower part of the left door. Sub-first speaker 140-5 is located at the lower part of the right door.

[0112] When the first control device 170 outputs the user U's voice through the first speaker 140, it outputs sound from the sub-first speakers 140-2 and 140-4 at the same volume level, and shuts off the other sub-first speakers. Thus, the first control device 170 performs sound image localization for the occupant P seated in the driver's seat S1, so that the sound can be heard from the front passenger seat S2. The sound image localization method is not limited to volume adjustment; it can also be performed by staggering the phase of the sounds output from each sub-first speaker. For example, when performing sound image localization so that the sound is heard from the left, the timing of the sound output from the left sub-first speaker can be slightly earlier than the timing of the same sound output from the right sub-first speaker.

[0113] When the first control device 170 outputs the user U's voice through the first speaker 140, it can also perform acoustic image localization so that the occupant P can hear the sound from a position on the co-pilot's seat S2 at a height corresponding to the height of the user U's head. In this case, the first speaker 140 needs to have multiple sub-first speakers 140-k (k is a plurality of natural numbers) with different heights.

[0114] Figure 5This is a structural diagram of the second device 200. The second device 200 includes, for example, a second communication device 210, a second microphone 220, a detection device 230, a second speaker 240, a moving body image display device 250, an HMI 260, and a second control device 270. The detection device 230 includes, for example, a pointing direction detection device 232, a gaze area detection device 233, a head position detection device 234, and a motion sensor 236.

[0115] The second communication device 210 is a communication interface for communicating with the communication device 310 of the management server 300 and the first communication device 110 of the first device 100 via the network NW.

[0116] The second microphone 220 collects the sound emitted by the user U. The sound collected by the second microphone 220 is transmitted to the first communication device 110 via the second communication device 210 through the second control device 270.

[0117] The pointing direction detection device 232 is a device for detecting the pointing direction. The pointing direction refers to the orientation of the user U's face or line of sight, or an orientation derived from both. Hereinafter, the pointing direction is defined as an angle in the horizontal plane, i.e., an angle without a vertical component, but the pointing direction can also be an angle that includes a vertical component. The pointing direction detection device 232 may also include physical sensors (e.g., accelerometer, gyroscope, etc.) mounted on the VR glasses described later. The pointing direction detection device 232 may also be an infrared sensor that detects multiple positions of the user U's head, or a camera that captures images of the user U's head. In any case, the second control device 270 calculates the pointing direction based on the information input from the pointing direction detection device 232. Various techniques are known in this regard, therefore detailed descriptions are omitted.

[0118] The gaze area detection device 233 is a device for detecting the gaze area. The gaze area refers to the area that the user U is carefully observing. The gaze area detection device 233 may also be a camera installed on the VR glasses described later to capture the user's gaze. The second control device 270 derives the gaze area based on the information input from the gaze area detection device 233. Various techniques are known in this regard, so detailed descriptions are omitted.

[0119] The head position detection device 234 is a device for detecting the position (height) of the user U's head. For example, one or more infrared sensors or optical sensors positioned around the chair where the user U sits are used as the head position detection device 234. In this case, the second control device 270 detects the position of the user U's head based on the presence or absence of detection signals detected by one or more infrared sensors or optical sensors. The head position detection device 234 can also be an accelerometer sensor mounted on VR glasses (head-mounted display (HMD), etc.). In this case, the second control device 270 detects the position of the user U's head by integrating the component obtained by subtracting gravitational acceleration from the output of the accelerometer sensor. The head position information obtained in this way is provided to the second control device 270 as height information.

[0120] The user's head position can also be obtained based on the user's (U) operations on the HMI260. For example, the user (U) can input their height digitally into the HMI260, or use a DIP switch included in the HMI260. In these cases, the head position, i.e., height information, is calculated based on the height. The user (U) can also input a discrete value for their physique, such as large / medium / small, instead of a continuous value. In this case, height information is obtained based on the physique information. Alternatively, the user's head height can be easily obtained based on a typical adult physique (which may be gender-specific) without specifically obtaining the user's head height.

[0121] Motion sensor 236 is a device for recognizing blinking and gestures performed by user U. For example, a camera that captures images of user U's upper body is used as motion sensor 236. In this case, the second control device extracts feature points of user U's face (eyes, nose, mouth) and body (fingertips, wrists, elbows, etc.) based on the images captured by the camera, and recognizes user U's blinking and gestures based on the movement of these feature points.

[0122] The second speaker 240 outputs the sound emitted by occupant P, obtained via the second communication device 210. The second speaker 240, for example, has a function to change the direction of the heard sound. The second control device 270 causes the second speaker to output sound in a manner that allows the user U to hear the sound from the position of occupant P as observed from the co-pilot's seat S2. The second speaker 240 includes a plurality of sub-second speakers 240-n (n being a plurality of natural numbers), and the second control device 270 can also perform sound image localization by adjusting the volume of each sub-second speaker 240-n. If headphones are included with the VR glasses, sound image localization can also be performed using the headphones.

[0123] The moving body image display device 250 displays an image from the perspective of the co-driver's seat that corresponds to the pointing direction in the image captured by the camera unit 130 (or an image that has undergone the aforementioned combination processing, hereinafter referred to as the moving body image). Figure 6 This is a diagram illustrating the image corresponding to the pointing direction. In the example shown, the VR glasses 255 include a physical sensor serving as a pointing direction detection device 232 and a head position detection device 234, as well as a moving body image display device 250. The second control device 270, for example, sets the center of the user U's head or the center of the VR glasses 255 as Ω, and uses a pre-calibrated direction as a reference direction to detect the direction in which the VR glasses 255 is facing as the pointing direction φ. Various methods are known regarding the functions involved, therefore detailed descriptions are omitted.

[0124] The moving body image display device 250 displays an image A2 of the moving body image A1 (which has an angle of about 240 degrees in the figure, but the screen angle can also be expanded by combining processing as described above) with the positive and negative α angle range centered on the pointing direction φ, facing the user U.

[0125] HMI260 is a touch panel, a voice-responsive device (smart device), or a switch as described above. HMI260 accepts various instructions from user U to the second device 200.

[0126] The second control device 270 includes, for example, a processor such as a CPU, and a storage medium connected to the processor and storing a program (command set), and controls various parts of the second device 200 by executing the command set through the processor. It should be noted that the second device 200 may also be... Figure 5 All the functions shown are integrated into the VR glasses.

[0127] <Functional Structure of First Control Device 170 and Second Control Device 270>

[0128] The functional structures of the first control device 170 and the second control device 270 are described below. It should be noted that the first and second examples shown below mainly involve the generation and transmission of information provided between the first control device 170 and the second control device 270, while the management server 300 manages the matching of simulated rides between passenger P and user U.

[0129] [First example]

[0130] Figure 7This diagram illustrates a first example of the functional structure of the first control device 170 and the second control device 270. In this first example, the first control device 170 includes a matching request / acceptance unit 171, a sound output control unit 172, an image transmission unit 173, a device cooperation unit 174, and a brightness adjustment unit 175. The second control device 270 includes a matching request / acceptance unit 271, a sound output control unit 272, a pointing direction detection unit 273, a head position detection unit 274, a posture input detection unit 275, an image editing unit 276, a moving body image display control unit 277, a sound recognition unit 279, and an operation information transmission unit 280. These functional units are implemented, for example, by executing programs (command groups) using a processor such as a CPU. Some or all of these components can also be implemented using hardware (including circuitry) such as LSI, ASIC, FPGA, and GPU, or through the coordinated use of software and hardware.

[0131] The matching request / approval unit 171 uses HMI 160 to accept input of a matching request (simulated ride request) from passenger P and sends it to management server 300. It also uses HMI 160 to accept input of approval for the matching request received from management server 300 and sends it to management server 300. The matching request / approval unit 171 controls the first communication device 110 to make the second device 200 of the user U, which enables the matching, a communication partner. For example, the matching request / approval unit 171 can also, upon approval of a matching request received from management server 300, receive settings from passenger P indicating whether or not information is permitted to be provided.

[0132] The sound output control unit 172 controls the first speaker 140 as described above.

[0133] After initiating communication with the second device 200, the image transmitting unit 173 transmits a moving body image A1 to the second device 200 using the first communication device 110. The moving body image A1 may include, for example, an indoor image captured by the indoor camera 132 and / or an outdoor image captured by the outdoor camera 134.

[0134] The mounted equipment coordination unit 174 controls the controlled object device 190 based on the instruction signal input from the second device 200. It should be noted that the mounted equipment coordination unit 174 can also control the controlled object device 190 if the user U has given permission from the occupant P.

[0135] The brightness adjustment unit 175 adjusts the brightness of the moving object image captured by the camera unit 130 based on the control information received from the management server 300 that changes the brightness.

[0136] The matching request / approval unit 271 uses HMI 260 to accept input of a matching request (simulated ride request) from user U and sends it to management server 300. It also uses HMI 260 to accept input of approval for the matching request received from management server 300 and sends it to management server 300. The matching request / approval unit 271 controls the second communication device 210 to make the first device 100 of the matched passenger P a communication partner. For example, the matching request / approval unit 271 can also, after prompting user U based on the matching result (request result) obtained from management server 300 to allow user U to confirm what information to provide, receive an instruction from user U on whether to simulate a ride.

[0137] The sound output control unit 272 controls the second speaker 240 as described above.

[0138] The pointing direction detection unit 273 detects the pointing direction φ based on the output of the pointing direction detection device 232. The head position detection unit 274 detects the height of the user U's head based on the output of the head position detection device 234. The head position can be represented as three-dimensional coordinates, or only the head height can be detected as the head position.

[0139] The gesture input detection unit 275 detects the user U’s blinking and gesture input based on the output of the motion sensor 236.

[0140] Image editing unit 276, for example, performs processing to extract image A2 from moving body image A1, corresponding to the pointing direction φ as viewed from the co-driver's seat S2. Figure 6 It should be noted that, in Figure 6 In the example, the pointing direction φ for left and right movements toward the user U's head is shown, but it is also possible to perform image cropping corresponding to the pointing direction for up and down movements toward the head.

[0141] The moving body image display control unit 277 causes the moving body image display device 250 to display the image A2 captured by the image editing unit 276. At this time, the image editing unit 276 can also cause the moving body image display device 250 to display an image corresponding to the pointing direction φ when viewed from the height indicated by the height information of the user U's head. For example, the image editing unit 276 switches between displaying an indoor image and an outdoor image based on the distance of the user U's head from the reference position (e.g., the amount of upward movement or the amount of lateral movement).

[0142] The gaze region detection unit 278 derives the gaze region based on the output of the gaze region detection device 233. For example, the gaze region detection unit 278 may define the gaze region as the point where the user U's line of sight intersects with the dynamic image displayed on the moving image display device 250, and a certain range from that point.

[0143] The voice recognition unit 279 acquires the voice (speech) collected by the second microphone 220, identifies the acquired voice, and converts the voice into text information.

[0144] The operation information sending unit 280 receives input for a first operation, which is an operation performed by the user U using the HMI 260 on at least a portion of the dynamic image displayed on the mobile image display device 250, and sends it to the management server 300. The operation information sending unit 280 receives text information converted by the voice recognition unit 279 and sends the received text information to the management server 300. The operation information sending unit 280 receives information indicating the gaze area derived by the gaze area detection unit 278 and sends the received gaze area information to the management server 300. The operation information sending unit 280 receives information indicating the gaze area derived by the gaze area detection unit 278. If, based on the received gaze area information, the operation information sending unit 280 determines that the user U has gazed at the gaze area for a predetermined time or more, and receives information indicating at least one of a blink or a gesture operation recognized by the gesture input detection unit 275, the operation information sending unit 280 receives input for a second operation and sends it to the management server 300.

[0145] [Second example]

[0146] Figure 8 This is a second example of a diagram illustrating the functional structure of the first control device 170 and the second control device 270. (Compared to...) Figure 7 The difference between the first example and the second example is that the first control device 170 includes an image editing unit 176, while the second control device 270 does not include an image editing unit 276 but includes a direction sending unit 281. The other components have essentially the same function as in the first example, so further explanation is omitted.

[0147] The pointing direction transmitting unit 281 uses the second communication device 210 to transmit the pointing direction φ detected by the pointing direction detecting unit 273 to the first device 100.

[0148] The image editing unit 176 performs processing to extract image A2 from the moving body image A1, which corresponds to the pointing direction φ (the pointing direction sent from the second device 200) when viewed from the co-driver's seat. Figure 6At this time, the image editing unit 176 can also obtain the height information of the user U's head from the second device 200 in advance, and perform processing to extract the image A2 corresponding to the pointing direction φ when viewed from the height shown in the height information.

[0149] In the second example, the image transmitting unit 173 uses the first communication device 110 to transmit the image A2, which was captured by the image editing unit 176, to the second device 200. Furthermore, the moving body image display control unit 277 causes the moving body image display device 250 to display the image A2 transmitted from the first device 100.

[0150] <Processing related to simulated rides>

[0151] Next, in the implementation method, a series of processes performed by the information processing system 1 will be specifically described in the case where user U simulates riding in a moving body M (passenger P allows user U to simulate riding in a moving body M). Figure 9 This is a sequence diagram representing an example of the processing performed by information processing system 1. Figure 9 In this example, a first device 100 mounted on a mobile body M carried by passenger P, a second device 200 used by user U, and processing within a management server 300 are shown. Here, user U and passenger P register in advance with the management server 300 and with user data 360 in order to receive services provided in the information processing system 1.

[0152] exist Figure 9 In the example, the second device 200 generates a request for simulating a ride (simulated ride request) (step S100) and sends the generated simulated ride request to the management server 300 (step S102). The simulated ride request may include, for example, information pre-specified about the passenger P, or information specifying the current location, destination, or surrounding conditions of the moving vehicle M. More specific examples of simulated ride requests include requests such as "I want to simulate a ride on a vehicle driven by Ms. / Mr. A" or "I want to simulate a ride on a vehicle traveling on a coastal road." These are examples of the matching requests described above.

[0153] The matching processing unit 320 of the management server 300 receives a simulation experience request from the second device 200 and performs matching processing based on user data 360 (step S104). Furthermore, if there is more than one successfully matched mobile body M, the providing unit 330 generates inquiry information for all mobile bodies M asking whether to allow the simulation ride (step S106) and sends the generated information to the first device 100 of the successfully matched mobile body M (step S108).

[0154] The first device 100 displays the query information sent from the management server 300 on the screen (step S110). Figure 10 This is an example of an image IM10 representing an inquiry. It should be noted that the display content, layout, and other display formats included in image IM10 are not limited to this. The same applies to the descriptions of other images thereafter. In image IM10, the inquiry screen includes, for example, a user information display area AR11, an area AR12 indicating whether an option is available, and a switch display area AR13. The user information display area AR11 displays information related to the user U who has made a simulated ride request. The information related to user U may be the user ID or other information that passenger P can use to identify user U.

[0155] The AR12 area displays an image of the content used by occupant P to select information from the information provided to user U. Figure 10 In the example, regarding indoor images, outdoor images, audio communication, and navigation information (information provided by the navigation device), a screen is displayed showing whether to grant permission to provide information to occupant P. Audio communication is an example of "indoor audio information." For example, occupant P can also communicate and provide images to user U, but in a specific case where they do not want the other party to know the destination, they can grant permission to provide audio communication, indoor images, and outdoor images, but refuse to provide navigation information.

[0156] Therefore, in cases where user U is actually riding in a mobile vehicle M, the destination can be easily known based on navigation information. However, navigation information is not provided during the simulated ride, thus allowing user U to travel to the destination while keeping it confidential and providing a unique and surprising experience. For example, for a user U who is not very close to you, by refusing to provide a call (voice), you can drive without talking.

[0157] It should be noted that the information that can be provided is not limited to... Figure 10 Examples may include, for instance, information regarding the selectability of navigation operations performed by user U, operations to other control object devices 190, and information regarding whether to provide an image including occupant P. It should be noted that the selectability of the information provided, as selected by the selectability area AR12, can also be changed during the simulated ride. The switch display area AR13 includes, for example, icons IC11 and IC12. Icon IC11 is a switch that accepts occupant P's permission for user U to ride the simulated ride. Conversely, icon IC12 is a switch that accepts occupant P's refusal to accept user U's simulated ride.

[0158] return Figure 9 The first device 100 accepts based on the above. Figure 10The image IM10 shown is used to determine whether the user U can provide the setting (step S112), and the received setting information is sent to the management server 300 (step S114). It should be noted that, in order to ensure driving safety, the processing in steps S110 to S112 can also be performed when the moving body M is stopped, or the simulated ride request can be notified by sound instead of the screen display (or in addition to the screen display), and the setting can be accepted. The provisioning unit 330 of the management server 300 generates a request result (response result) based on the setting information obtained from the first device 100 (step S116), and sends information indicating the generated request result to the second device 200 (step S118).

[0159] The second device 200 receives the request result from the management server 300 and displays the received information representing the request result on the screen (step S120). Figure 11 This diagram illustrates an example of an image IM20 representing the result of a request. The image IM20, as a screen displaying the request result, includes, for example, an occupant information display area AR21, an information overview display area AR22, and a switch display area AR23. This image IM20 can also be generated based on a licensed occupant P (moving body M) simulating a vehicle ride, and can further display an image representing an overview of the request result.

[0160] The passenger information display area AR21 displays information related to passenger P who has granted permission for a simulated ride request. This information may include the passenger ID or other information that allows user U to identify passenger P. The information granted by passenger P is displayed in a summary view in the information provision overview display area AR22. The fee (service usage fee) based on the granted information may also be displayed in the information provision overview display area AR22. The switch display area AR23 includes icons IC21-IC23, for example. Icon IC21 is a switch for accepting the information provided in the information provision overview display area AR22 to simulate a ride with the mobile body M of passenger P, which is displayed in the passenger information display area AR21. Conversely, icon IC22 is a switch for accepting no simulated ride. Icon IC23 is a switch for accepting the search for other passengers (mobile bodies). When searching for other passengers, users can select from other passengers included in the request results, or enter other request conditions to cause the management server 300 to perform matching processing again.

[0161] return Figure 9 The second device 200 accepts based on the above. Figure 11The image IM20 shown receives a selection instruction from user U (step S122) and sends the received instruction information to management server 300 (step S124). Hereinafter, it will be explained that user U has given an instruction to simulate a ride on the mobile body M of passenger P. Management server 300 manages the communication status between first device 100 and second device based on the instruction from second device 200 (step S126), and sends information indicating the execution of the simulated ride to both first device 100 and second device 200 (steps S128, S130). It should be noted that during the processing of steps S126 to S130, if user U selects not to simulate a ride, the providing unit 330 of management server 300 sends information indicating this intention to first device 100 and notifies user U.

[0162] Next, the first device 100 and the second device 200 communicate with each other to perform a simulated ride (step S132). When the simulated ride ends, a message indicating the end of communication is sent to the management server 300 (steps S134, S136). It should be noted that during the processing in step S132, the second device 200 can also notify the passenger P from the user U before providing information from the first device 100 to the second device 200. Thus, the user U can grant communication at their own time to start a call, receive dynamic images, and other information. Next, the cost management unit 340 of the management server 300 performs cost calculation processing related to the service utilization of the first device 100 and the second device 200. Specifically, the cost management unit 340 differentiates the costs related to information provision based on at least one of the following: the type of passenger P providing information, the type of user U, the type of mobile vehicle M, the type of information that can be provided to the user U, and the amount of information. For example, if passenger P and user U are celebrities, the usage fee will be set higher than usual; similarly, if the mobile vehicle M is a new or rare vehicle, the usage fee will be set higher than usual. Furthermore, the fee management department 340 will perform accounting processing corresponding to the set fee. This concludes the processing of this sequence. It should be noted that regarding... Figure 9 The processing of step S112 shown (whether setting processing can be provided) can also be preset by the occupant P according to each registered user U.

[0163] <Processing related to brightness adjustment>

[0164] Next, in the implementation method, a series of processes performed by the information processing system 1 will be specifically described in the case where the user U simulates riding in a vehicle M and the brightness is adjusted based on the user U's request. Figure 12AThis is a sequence diagram representing the first example of processing performed by information processing system 1. The first device 100 and the second device 200 communicate with each other to perform a simulated ride.

[0165] exist Figure 12A In the example, the second device 200 receives input of a first operation performed by the user U (step S200) and sends first operation information to the management server 300 based on the received first operation (step S202). The first operation may, for example, be a click operation performed by the user U using the HMI 260 on at least a portion of a region of a moving image displayed on the mobile image display device 250. For example, the user U clicks on any location included in the moving image displayed on the mobile image display device 250. The location may also be a location that the user U wants to observe remotely. The HMI 260 receives the input of the click operation performed by the user U. The operation information sending unit 280 receives the click operation received by the HMI 260, generates first operation information containing information representing the region where the click operation was performed, and sends the generated first operation information to the management server 300. An example of the information representing the region is the coordinates of at least a portion of the outer contour of the region in the mobile image display device 250.

[0166] Next, the brightness change unit 334 of the management server 300 generates control information for changing the brightness of a region of the dynamic image based on the first operation information from the second device 200 (step S204), and sends the generated control information to the first device 100 (step S206).

[0167] For example, receiving unit 332 obtains and receives first operation information from communication device 310 and from second device 200. Brightness adjustment unit 334 obtains the first operation information from receiving unit 332 and determines, based on the obtained first operation information, the camera that captured a portion of the moving image that underwent the first operation. Here, brightness adjustment unit 334 may also store camera moving image region association information that associates information representing the multiple cameras included in camera unit 130 with information representing the regions of the moving images captured by each of the multiple cameras. For example, the area of ​​the front glass may be divided into three parts and set as left, center, and right areas to set the regions of the moving images captured by each of the multiple cameras, or it may be divided into six parts. In this case, brightness adjustment unit 334 obtains information representing a camera that is associated with information in the first operation information representing a portion of the moving image that underwent the first operation, and generates control information to change the brightness of that camera based on the obtained information representing the camera.

[0168] Next, the camera unit 130 of the first device 100 adjusts the brightness based on control information from the management server 300 (step S208). For example, either the indoor camera 132 or the outdoor camera 134 obtains control information from the first control device 170 and adjusts the EV value (Exposure Value) based on the obtained control information, thereby adjusting the brightness. Afterwards, either the indoor camera 132 or the outdoor camera 134 can also adjust the brightness based on the ambient brightness. Thus, for example, even if the external environment changes due to driving and the ambient brightness changes, the brightness can be dynamically adjusted so that the location indicated by the user U is always easily visible.

[0169] Figure 12B This is a sequence diagram representing a second example of the processing performed by the information processing system 1. The first device 100 and the second device 200 communicate with each other to perform a simulated ride.

[0170] exist Figure 12B In the example, the second device 200 collects the user U's voice (speech), performs voice recognition on the collected voice (step S300), and sends it to the management server 300 (step S302). For example, the second microphone 220 collects the user U's voice (speech). The voice recognition unit 279 acquires the voice (speech) collected by the second microphone 220, recognizes the acquired voice, and converts it into text information. The operation information sending unit 280 receives the text information converted by the voice recognition unit 279 and sends the received text information to the management server 300.

[0171] Next, the brightness adjustment unit 334 of the management server 300 determines whether there is POI information associated with the text information based on the text information from the second device 200 (step S304). If POI information exists, it generates control information to change the brightness of the area containing the POI information in a portion of the moving image (step S306) and sends it to the first device 100 (step S308). For example, the receiving unit 332 obtains and receives text information from the second device 200 from the communication device 310. The brightness adjustment unit 334 obtains the text information from the receiving unit 332 and determines whether the POI information registered in the provision information DB362 stored in the storage unit 350 contains POI information associated with the text information. If the POI information registered in the information provision DB362 includes POI information associated with text information, the brightness adjustment unit 334 determines, from the cameras included in the camera unit 130, the camera that captured the aforementioned associated POI information based on the dynamic image data provided by the information provision unit 330 to the second device 200, and generates control information to change the brightness of the determined camera. If the POI information registered in the information provision DB362 does not include POI information associated with text information, the brightness adjustment unit 334 terminates.

[0172] Step S310 and Figure 12A Step S208 is the same, so its description is omitted. Afterwards, either the indoor camera 132 or the outdoor camera 134 can adjust the brightness based on the ambient light. Thus, for example, even if the external environment changes due to driving and the ambient light changes, the brightness can be dynamically adjusted to ensure that the POI information associated with the text information is always easily visible.

[0173] Figure 12C This is a sequence diagram representing the third example of the processing performed by the information processing system 1. The first device 100, the second device 200-1, and the second device 200-2 communicate with each other to perform a simulated ride.

[0174] exist Figure 12CIn the example, the second device 200-1 detects the gaze area (hereinafter referred to as the "first gaze area") being gazed at by user U1 (step S400) and sends information indicating the detected first gaze area to the management server 300 (step S402). The second device 200-2 detects the gaze area (hereinafter referred to as the "second gaze area") being gazed at by user U2 (step S404) and sends information indicating the detected second gaze area to the management server 300 (step S406). For example, in the second device 200-1, the gaze area detection unit 278 derives the first gaze area being gazed at by user U1. The operation information sending unit 280 receives information indicating the first gaze area derived by the gaze area detection unit 278 and sends the received information indicating the first gaze area to the management server 300. In the second device 200-2, the gaze area detection unit 278 derives the second gaze area being gazed at by user U2. The operation information sending unit 280 receives information representing the second gaze region derived by the gaze region detection unit 278, and sends the received information representing the second gaze region to the management server 300.

[0175] Next, the brightness adjustment unit 334 of the management server 300 determines whether the viewing areas are the same based on information representing the first viewing area from the second device 200-1 and information representing the second viewing area from the second device 200-2 (step S408). If they are the same, it generates control information to change the brightness of a portion of the moving image (step S410) and sends the generated control information to the first device 100 (step S412). For example, the receiving unit 332 obtains and receives information representing the first viewing area from the second device 200-1 and information representing the second viewing area from the second device 200-2 from the communication device 310. The brightness adjustment unit 334 obtains information representing the first viewing area and information representing the second viewing area from the receiving unit 332 and determines whether the viewing areas are the same. Here, "the viewing areas are the same" means that at least a portion of the first viewing area and the second viewing area are the same. The process ends when the viewing areas are different. When the viewing areas are the same, the brightness adjustment unit 334 generates camera brightness adjustment control information for the camera in the camera unit 130 that captures the viewing area being viewed by the user U. Here, the brightness adjustment unit 334 may also store camera dynamic image region association information that associates information representing the multiple cameras included in the camera unit 130 with information representing the regions of dynamic images captured by each of the multiple cameras. In this case, the brightness adjustment unit 334 obtains information representing a camera based on the camera dynamic image region association information, associates the camera with information representing the viewing area being viewed by the user U, and generates control information to change the brightness of the camera based on the obtained information representing the camera.

[0176] Step S414 and Figure 12A Step S208 is the same, so its description is omitted. Afterwards, either the indoor camera 132 or the outdoor camera 134 can adjust the brightness based on the ambient light. Thus, for example, when the external environment changes due to driving and the ambient light changes, the brightness can be dynamically adjusted so that the area being gazed at by the user U is always easily visible.

[0177] Figure 12D This is a sequence diagram representing the fourth example of the processing performed by the information processing system 1. The first device 100 and the second device 200 communicate with each other to perform a simulated ride.

[0178] exist Figure 12D In the example, the second device 200 detects the area of ​​gaze that the user U is looking at (step S500) and determines whether a predetermined time has elapsed while the user U is looking at the area of ​​gaze (step S502). If the predetermined time has not elapsed (step S502: "No"), it returns to step S500. If the predetermined time has elapsed (step S502: "Yes"), it determines whether at least one of the user U's blinking and posture has been detected (step S504).

[0179] For example, the gaze area detection unit 278 identifies the gaze area being gazed upon by the user U. The operation information transmission unit 280 obtains information representing the gaze area from the gaze area detection device 233 and determines, based on the obtained information representing the gaze area, whether the user U has been gazing at the gaze area for a predetermined time. The motion sensor 236 identifies blinking and gesture operations performed by the user U. The operation information transmission unit 280 obtains the identification results of the user U's blinking and gesture operations from the motion sensor 236 and determines whether it has received information representing at least one of the blinking and gesture operations.

[0180] Next, the operation information sending unit 280 of the second device 200 accepts the input of the second operation (step S506) upon receiving information indicating at least one of blinking and posture operations, and sends it to the management server 300 (step S508). An example of the second operation is blinking at a certain interval, closing the eyes for more than one second, and saying "adjust".

[0181] The brightness change unit 334 of the management server 300 generates control information that changes the brightness of a region of the dynamic image based on the second operation information from the second device 200 (step S510), and sends the generated control information to the first device 100 (step S512).

[0182] For example, receiving unit 332 obtains and receives second operation information from communication device 310 and from second device 200. Brightness adjustment unit 334 obtains the second operation information from receiving unit 332, obtains information indicating the viewing area based on the obtained second operation information, and generates control information to change the brightness of the camera in the camera unit 130 that captures the viewing area being viewed by user U. Here, brightness adjustment unit 334 may also store camera dynamic image area association information that associates information indicating multiple cameras included in camera unit 130 with information indicating the area of ​​dynamic images captured by each of the multiple cameras. In this case, brightness adjustment unit 334 obtains information indicating a camera based on the camera dynamic image area association information, associates the camera with information indicating the viewing area being viewed by user U, and generates control information to change the brightness of that camera based on the obtained information indicating the camera.

[0183] Step S514 and Figure 12A Step S208 is the same, therefore its description is omitted. Afterwards, either the indoor camera 132 or the outdoor camera 134 can adjust the brightness based on the ambient light. Thus, for example, even if the external environment changes due to travel and the ambient light changes, the brightness can be dynamically adjusted to ensure that the area being gazed upon by the user U is always easily visible. Figure 12D As an example, it is shown that when there are two second devices 200, it is also applicable when there are three or more devices.

[0184] <Information provided during the simulated ride>

[0185] Next, we will explain the information provided to user U during the simulated ride. Figure 13 This is a diagram used to illustrate an example of providing information. The following example is... Figure 12A The processing in step S208 shown, Figure 12B The processing in step S310 shown Figure 12C The processing of step S414 shown and Figure 12D The information provided in step S514 of the process shown includes a brightness change area BAA in which the brightness of a portion of the moving image V has been changed. The output format of the information provided to the second device 200 varies depending on the action performed by the user U and the type or amount of information that can be provided to the user U. It should be noted that, in the following example, it is assumed that the occupant P has authorized the provision of indoor images, outdoor images, and audio communication.

[0186] In the aforementioned embodiment, the brightness adjustment unit 334 can also assign priority to the second device connected to the management server 300 for brightness adjustment when any operation information from the first operation information and the second operation information obtained from multiple second devices is received. Therefore, even if any operation information from the first operation information and the second operation information from multiple second devices conflicts, brightness adjustment can be performed without determining which second device to assign priority.

[0187] In the aforementioned embodiment, the brightness adjustment unit 334 may also notify each of the conflicting second devices of the conflicting information if there is a conflict in the information representing a portion of the moving image that has undergone the first operation, which is included in the first operation information from the multiple second devices. Thus, the users U of the multiple conflicting second devices can communicate with each other to decide which user U's first operation information to use.

[0188] In the aforementioned embodiments, the brightness adjustment unit 334 may also, in the event of conflicting first operation information from multiple second devices, determine whether POI information associated with the periphery of each of the multiple conflicting first operation information regions representing a portion of the moving image is included. For example, the brightness adjustment unit 334 may obtain POI information from the provision information DB362 of the storage unit 350. The brightness adjustment unit 334 may also adjust the brightness of the region where the POI information is detected or the brightness of the POI information if POI information associated with the periphery of one of the multiple regions of the moving image is included.

[0189] Here, the brightness adjustment unit 334 can also notify each of the multiple POI information summaries to multiple second devices that conflict with the first operation information, when the POI information includes POI information associated with the periphery of each of multiple regions in multiple regions that are part of the moving image. As a result, the users U of the multiple second devices that have been notified of the summaries of the multiple POI information can decide which user U's first operation information to adopt by communicating with each other.

[0190] <Summary>

[0191] According to the information processing system 1 configured as described above, the information processing device processes information provided from a first device 100 mounted on a mobile body to a second device 200 used by a user U in a location different from the mobile body. The information processing device includes an acquisition unit 315, a supply unit 330, a receiving unit 332, and a brightness adjustment unit 334. The acquisition unit 315 acquires information including moving image data sent by the first device 100. The supply unit 330 provides the moving image data to the second device 200 based on the information acquired by the acquisition unit 315. The receiving unit 332 receives a first operation, which is an operation performed by the user U on a portion of a displayed moving image based on the moving image data provided by the supply unit 330 to the second device 200. The brightness adjustment unit 334 adjusts the brightness of a portion of the moving image based on the first operation received by the receiving unit 332.

[0192] The second device 200 can remotely perform processing to make the moving image data (video data) transmitted by the first device 100 easily viewable, thus improving convenience. Instead of the first device 100 distributing the moving image data, the second device 200, which receives and views the moving image data, can adjust the brightness, thus improving convenience compared to situations where only the first device 100 can adjust the brightness. The ability to distribute images with adjusted brightness avoids overexposure and underexposure in desired viewing locations.

[0193] In the information processing apparatus, the first device 100 includes a camera device. The brightness adjustment unit 334 generates control information to change the brightness of the camera device based on the first operation received by the receiving unit 332.

[0194] The system can generate control information for changing the brightness of the camera device based on the first operation, and thus can control the brightness of the camera device included in the first device 100.

[0195] In the information processing device, the receiving unit 332 notifies the mobile body of the situation that a first operation to change the brightness has been performed based on the received first operation.

[0196] When a mobile vehicle is notified that a brightness change has been initiated, it can provide topics of conversation by informing the occupants. This allows for smoother communication among the occupants. For example, if a brightness change is initiated at a scenic location, topics related to that location can be discussed.

[0197] In information processing devices, camera devices adjust the brightness of a portion of the area based on the ambient brightness.

[0198] Even when the external environment changes due to the movement of the moving object and the surrounding brightness changes, the brightness can be dynamically adjusted so that the location indicated by the user U is always easy to see.

[0199] The information processing device also includes a detection unit that detects the gaze area of ​​the user U looking at the moving image. A brightness adjustment unit 334 adjusts the brightness of the gaze area of ​​the moving image based on the gaze area detected by the detection unit.

[0200] The processing of dynamic image data (video data) sent by the first device 100 can be performed remotely without being operated by the second device 200, thus further improving convenience compared to changing the brightness through operation.

[0201] The information processing device also includes a sound recognition unit 279, which detects and recognizes sound. When the sound recognized by the sound recognition unit 279 is associated with POI information around the moving object, the brightness adjustment unit 334 adjusts the brightness of the associated POI information.

[0202] The device can perform processing remotely to make the dynamic image data (video data) transmitted by the first device 100 easily visible without operation by the second device 200, and can adjust the brightness of the POI information around the sound-related moving object. This further improves convenience compared to changing the brightness through operation.

[0203] The information processing device also includes a detection unit that detects the area of ​​gaze of the user U looking at the moving image. The receiving unit 332 receives a second operation if the detection unit detects that the user is looking at a portion of the moving image for a predetermined time or more. The brightness adjustment unit 334 adjusts the brightness of a portion of the moving image based on the second operation received by the receiving unit 332. The second operation includes blinking, gestures, and speaking. Therefore, the second operation includes a greater number of actions than the first operation.

[0204] Processing can be performed remotely to make the dynamic image data (image data) transmitted by the first device 100 easily visible, provided that a portion of a dynamic image has been viewed for a specified time and a second operation has been performed.

[0205] In the information processing device, the brightness adjustment unit 334 assigns priority to the second device 200 that is initially connected to the information processing device among a plurality of second devices 200 for brightness adjustment.

[0206] The convenience is improved because the second device 200, which is given priority through the initial connection, can perform the processing remotely to make the dynamic image data (video data) sent by the first device 100 easily visible.

[0207] In the information processing device, if a first operation conflict is received from multiple second devices 200 by the receiving unit 332, the brightness change unit 334 notifies each of the conflicting second devices 200.

[0208] The users U of the multiple conflicting second devices communicate with each other and decide which user U's first operation information to adopt. Therefore, even if the first operation information from multiple second devices conflicts, the second device 200 can remotely perform processing to make the dynamic image data (video data) sent by the first device 100 easily viewable. This improves convenience.

[0209] In the information processing device, when a first operation from multiple second devices 200 conflicts, the brightness adjustment unit 334 performs processing to detect the POI information around a region of a dynamic image in which the user U of each of the conflicting second devices 200 has performed the first operation, and adjusts the brightness of the region where the POI information is detected or the POI information itself when the POI information can be detected.

[0210] It is possible to adjust the brightness of the area or POI information around the region of a portion of the dynamic image that has undergone the first operation, thereby enabling processing to make the dynamic image data (image data) sent by the first device 100 more easily visible.

[0211] In the information processing device, when a first operation conflict is received by the receiving unit 332 from multiple second devices 200, the brightness change unit 334 performs processing to detect the POI information around a region of a dynamic image in which the user U of each of the conflicting second devices 200 has performed the first operation. If multiple POI information can be detected, the brightness change unit 334 notifies the user U of each of the conflicting second devices 200 of a summary of the multiple POI information.

[0212] It can detect the POI information surrounding a region of a dynamic image in which a user U of each of the multiple conflicting second devices has performed a first operation, and notify the user U of each of the multiple second devices 200 of a summary of the multiple POI information, thereby improving convenience.

[0213] <Example of use>

[0214] Information processing system 1 can be used in the following form.

[0215] (A) The occupant P and the user U are family members, friends, etc., and the occupant P provides the user U with a virtual driving mode. The user U can talk to the occupant P while watching images and talking about the scenery around the mobile body M.

[0216] (B) Passenger P is a general user, and user U is a provider of road guidance services, driving instruction services, etc. User U can provide road guidance and driving operation instructions in places that are not easily known in navigation devices or are not displayed on maps while viewing the scenery around the mobile body M.

[0217] (C) Passenger P is a celebrity, and user U is a general user. A business-based virtual driving experience is provided to user U. In this case, a correspondence is established simultaneously between one passenger P and multiple users U. For example, the transmission of voice from user U can be turned off. Users U can also pay an additional fee to have a scheduled conversation with passenger P.

[0218] The above describes specific embodiments of the present invention, but the present invention is not limited to such embodiments in any way. Various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. An information processing apparatus that processes information provided from a first apparatus mounted on a moving body to a second apparatus used by a user at a different place from the moving body, wherein the information processing apparatus includes: an acquisition section that acquires information including dynamic image data transmitted by the first apparatus; a provision section that provides dynamic image data to the second apparatus based on the information acquired by the acquisition section; an acceptance section that accepts a first operation that is an operation performed by the user on a region of a part of a prompted dynamic image based on the dynamic image data provided to the second apparatus by the provision section; and a brightness change section that changes brightness of the region of the part of the dynamic image based on the first operation accepted by the acceptance section. 2.The information processing apparatus according to claim 1, wherein the first apparatus includes an imaging device, the brightness change section creates control information that causes the imaging device to change brightness based on the first operation accepted by the acceptance section. 3.The information processing apparatus according to claim 1, wherein the acceptance section notifies the moving body of the fact that the first operation that changes brightness is performed based on the accepted first operation. 4.The information processing apparatus according to claim 2, wherein the imaging device adjusts brightness of the region of the part based on brightness of the surroundings. 5.The information processing apparatus according to claim 1, wherein the information processing apparatus further includes a detection section that detects a gazing region on which the user gazes at the dynamic image, the brightness change section changes brightness of the gazing region of the dynamic image based on the gazing region detected by the detection section. 6.The information processing apparatus according to claim 1, wherein the information processing apparatus further includes a sound recognition section that detects a sound and performs sound recognition, the brightness change section adjusts brightness of POI (Point of Interest) information associated with the sound recognized by the sound recognition section in a case where the sound is associated with the POI information of the surroundings of the moving body. 7.The information processing apparatus according to claim 1, wherein the information processing apparatus further includes a detection section that detects a gazing region on which the user gazes at the dynamic image, the acceptance section accepts a second operation in a case where the gazing region of the part of the dynamic image is gazed at for a predetermined time or more, the brightness change section changes brightness of the region of the part of the dynamic image based on the second operation accepted by the acceptance section. 8.The information processing apparatus according to claim 7, wherein the second operation includes blinking, a gesture, and speech. 9.The information processing apparatus according to claim 7, wherein the number of times of a behavior included in the second operation is larger than the number of times of a behavior included in the first operation. 10.The information processing apparatus according to claim 1, wherein ​ The luminance changing section gives priority for luminance adjustment to a second device that is initially connected to the information processing device among the plurality of second devices.

11. The information processing apparatus according to claim 1, wherein The luminance changing section, in a case where the first operations from the plurality of second devices are in conflict as accepted by the accepting section, individually notifies the plurality of second devices in conflict.

12. The information processing apparatus according to claim 1, wherein The luminance changing section, in a case where the first operations from the plurality of second devices are in conflict, performs processing of detecting POI information around a region of a part of the dynamic image on which the first operation is performed by the user of each of the plurality of second devices in conflict, in a case where POI information can be detected, adjusts the region of the part of the dynamic image on which the first operation is performed by the user of each of the plurality of second devices in conflict or the luminance of the POI information.

13. The information processing apparatus according to claim 1, wherein The luminance changing section, in a case where the first operations from the plurality of second devices are in conflict as accepted by the accepting section, performs processing of detecting POI information around a region of a part of the dynamic image on which the first operation is performed by the user of each of the plurality of second devices in conflict, in a case where a plurality of POI information can be detected, notifies the user of each of the plurality of second devices in conflict of an outline of the plurality of POI information.

14. An information processing system, wherein The information processing system includes: a first device mounted on a mobile body; a second device used by a user at a place different from the mobile body; an acquisition section that acquires information including dynamic image data transmitted by the first device; a provision section that provides dynamic image data to the second device based on the information acquired by the acquisition section; an accepting section that accepts a first operation that is an operation performed by the user on a region of a part of a prompted dynamic image based on the dynamic image data provided to the second device by the provision section; and a luminance changing section that changes the luminance of the region of the part of the dynamic image based on the first operation accepted by the accepting section.

15. An information processing method, wherein an information processing apparatus processes information provided from a first device mounted on a mobile body to a second device used by a user at a place different from the mobile body, the information processing method causes the information processing apparatus to perform the following processing: acquire information including dynamic image data transmitted by the first device; provide dynamic image data to the second device based on the acquired information; accept a first operation that is an operation performed by the user on a region of a part of a prompted dynamic image based on the dynamic image data provided to the second device; and change the luminance of the region of the part of the dynamic image based on the accepted first operation. ​ ​

Images