Drive record display system and drive record display method

Abstract

In a drive record display service using in-vehicle equipment equipped with a camera, the system will be made capable of additionally recording video data that the user has filmed themselves. [Solution] In the drive record display system, Server 1 acquires first video data and location information captured from a vehicle-mounted device 6 having a camera 61, and generates drive record data for each drive, which includes drive information generated from this location information and the first video data. The drive record data can be viewed by the user's information terminal 8. Server 1 acquires second video data captured by the user and adds the second video data to the drive record data of the drive corresponding to the acquired second video data.

Description

Technical Field

[0001] The technology disclosed herein relates to a drive recording display service using an in-vehicle device having a camera.

Background Art

[0002] In recent years, it has become common to share memories of drives and the like through digital data. As a method of leaving drive memories, there is a method of creating an electronic album using image data taken during a drive.

[0003] Patent Document 1 discloses a navigation album system that creates an electronic album in which digital image data including data taken at a tourist spot located on a moving route is linked to the position information of the moving route specified by a navigation device and displayed on a display means.

[0004] Patent Document 2 discloses a shooting information browsing system in which a server collects and stores shooting location data of shootings by in-vehicle cameras implemented in a plurality of vehicles, and makes the collection and storage results viewable on a terminal device.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] The inventors of this application are considering a service that uses in-vehicle equipment, such as a dashcam, to create and present driving records to the user. In this service, video data captured by the in-vehicle equipment's camera is collected and managed on a server, and can be presented to the user's information terminal along with driving information such as the route taken.

[0007] When driving, users often take photos and videos using the cameras on their personal devices. For example, they might take pictures of scenery, food at restaurants, or family and friends accompanying them while taking a break during a drive. By allowing users to view these photos and videos together with the driving information presented in the aforementioned service, they can more deeply appreciate their travel memories.

[0008] The technology disclosed herein was developed in view of these points, and its purpose is to enable users to additionally record video data they have taken themselves in a drive record display service using a vehicle-mounted device equipped with a camera. [Means for solving the problem]

[0009] To solve the aforementioned problems, the technology disclosed herein provides a drive record display system for displaying records of driving using a vehicle, comprising: a device mounted on the vehicle and having a first camera and capable of acquiring the vehicle's location information; an information terminal having a second camera and owned by the vehicle's user; and a server capable of communicating with the information terminal, wherein the server acquires first video data captured by the first camera of the device and the location information acquired by the device, generates drive information based on the location information, generates drive record data for each drive that includes the generated drive information and at least a portion of the first video data and is viewable by the information terminal, acquires second video data captured by the second camera of the information terminal, and adds the acquired second video data to the drive record data of the drive corresponding to the acquired second video data.

[0010] According to this configuration, the drive record display system comprises a device mounted in the vehicle, equipped with a camera and capable of acquiring the vehicle's location information; an information terminal equipped with a camera and owned by the vehicle's user; and a server capable of communicating with the information terminal. The server acquires first video data captured by the camera on the device and the location information acquired by the device. Based on the acquired location information, it generates drive information and creates drive record data for each drive, which includes the generated drive information and at least a portion of the first video data and is viewable by the information terminal. The server also acquires second video data captured by the camera on the information terminal. The server then adds the acquired second video data to the drive record data of the corresponding drive. This allows the user to add video data captured by the camera on the information terminal to the drive record data of the corresponding drive. Therefore, the user can more deeply enjoy their travel memories.

[0011] Furthermore, the drive information may include the drive trajectory, the drive's starting point, and the drive's destination, and the drive record data may include drawing data of the drive trajectory on the map display and a thumbnail of the first video data.

[0012] This allows users to view the drive trajectory and thumbnails of video data captured by equipment installed in the vehicle on the information terminal screen.

[0013] Furthermore, the second video data may include data on the date and time of shooting and the shooting location as metadata, and the server may identify the drive corresponding to the second video data by referring to the metadata and the drive information of each drive.

[0014] This means that simply by the user uploading the second video data to the server, the content management server will identify the drive corresponding to the second video data and add the second video data to the recorded data on the identified drive.

[0015] Furthermore, in order to solve the aforementioned problems, the technology disclosed herein provides a drive record display method for displaying a record of driving using a vehicle, wherein the vehicle is equipped with a device having a first camera and capable of acquiring the vehicle's location information, the user of the vehicle has an information terminal having a second camera, and a server capable of communicating with the information terminal acquires first video data captured by the first camera of the device and the location information acquired by the device, generates drive information based on the location information, generates drive record data for each drive that includes the generated drive information and at least a part of the first video data and is viewable by the information terminal, acquires second video data captured by the second camera of the information terminal, and adds the acquired second video data to the drive record data of the drive corresponding to the acquired second video data.

[0016] In this configuration, the drive record display method involves a device mounted on the vehicle that has a camera and is capable of acquiring vehicle location information. A server that can communicate with the user's information terminal acquires first video data captured by the device's camera and the location information acquired by the device. Based on the acquired location information, the server generates drive information and creates drive record data for each drive that includes the generated drive information and at least a portion of the first video data, and is viewable by the information terminal. The server also acquires second video data captured by the information terminal's camera. The server then adds the acquired second video data to the drive record data of the corresponding drive. This allows the user to add video data captured by the information terminal's camera to the drive record data of the corresponding drive. Therefore, the user can more deeply enjoy their travel memories. [Effects of the Invention]

[0017] As explained above, the technology disclosed herein allows for the additional recording of video data captured by the user in a drive recording display service using a vehicle-mounted device equipped with a camera. [Brief explanation of the drawing]

[0018] [Figure 1] Conceptual diagram showing an example of the overall configuration of a drive recording display system according to an exemplary embodiment [Figure 2] Schematic configuration diagram of a vehicle [Figure 3] Schematic configuration diagram of a device management server [Figure 4] Diagram showing the configuration of a content management server [Figure 5] Schematic configuration diagram of an information terminal [Figure 6] Diagram showing an example of the screen display of an information terminal [Figure 7] Flowchart for determining the start and end of a drive [Figure 8] Flowchart showing an example of the acquisition process and storage process of video data [Figure 9] Example of the screen of an information terminal, (a) shows a list of drives, (b) shows detailed information of a drive [Figure 10] Flowchart showing the first operation example of an embodiment [Figure 11] (a) shows an example of a screen showing detailed information of a drive, (b) shows an image that the user wants to add a record, (c) shows an example of a screen where the image in (b) is additionally displayed [Figure 12] Example of content information [Figure 13] Flowchart showing the second operation example of an embodiment

Mode for Carrying Out the Invention

[0019] Hereinafter, exemplary embodiments will be described in detail with reference to the drawings

[0020] The following embodiments are illustrative, and there is no intention to limit the content of this disclosure by the presence or absence of descriptions or by the numerical values ​​provided. Furthermore, the components constituting the systems, servers, etc., described herein, and the functions realized by the components such as functional blocks and modules described herein, may be implemented in circuitry or processing circuitry, including general-purpose processors such as CPUs (Central Processing Units) and GPUs (Graphics Processing Units), application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), conventional circuits, and / or combinations thereof, programmed to realize the described functions. A processor includes transistors and other circuits and is considered circuitry or processing circuitry. A processor may be configured to execute a program stored in memory. Memory includes, for example, volatile memory such as RAM, and non-volatile semiconductor memory such as ROM and flash memory.

[0021] (1) Overall configuration of the drive record display system Figure 1 is a conceptual diagram showing an example of the overall configuration of a drive recording and display system M according to an embodiment. The drive recording and display system M comprises a vehicle Q including equipment with a camera, an information terminal 8, a device management server 5, and a content management server 1.

[0022] In Figure 1, vehicle Q is equipped with a drive recorder 6, which has a camera 61. The drive recorder 6 captures and records video data Dv (including the first video data Dv1 described later) of the area around vehicle Q while it is in motion using the camera 61. The video data Dv is recorded periodically during driving, or when an event such as an impact occurs. The recorded video data Dv includes still images and videos (including those shot in time lapse). The drive recorder 6 is a communication-type drive recorder and communicates with a device management server 5 for drive recorders. The drive recorder 6 transmits the recorded video data Dv to the device management server 5, for example, periodically, or at any arbitrary time.

[0023] In this embodiment, the drive recorder 6 has a positioning system 65 (see Figure 2) that acquires the location information of the vehicle Q. The positioning system acquires positioning data Dp. Details of the positioning data Dp will be described later.

[0024] Information terminal 8 is a terminal device carried by user U who is driving vehicle Q or user U who is a passenger in vehicle Q. Information terminal 8 has a camera 81. Information terminal 8 records second video data Dv2 using the camera 81.

[0025] The device management server 5 and the content management server 1 can communicate with each other. The content management server 1 corresponds to the server in this disclosure.

[0026] The device management server 5 receives the video data Dv captured by the drive recorder 6's camera from the drive recorder 6 and temporarily stores it.

[0027] The content management server 1 receives and saves the first video data Dv1 from the device management server 5, which is video data captured by the drive recorder 6's camera 61. The content management server 1 edits the saved first video data Dv1 according to instructions from user U via the information terminal 8, creates, for example, drive memory information, and sends it to the information terminal 8. The content management server 1 may also use the second video data Dv2 sent by user U from the information terminal 8 to edit the memory information. The first video data Dv1 will be described later.

[0028] (2) Vehicles As shown in Figure 2, vehicle Q is equipped with a typical in-vehicle system 7. The drive recorder 6 was retrofitted to vehicle Q by user U.

[0029] The drive recorder 6 is an example of a device capable of acquiring location information of vehicle Q. The drive recorder 6 comprises a camera 61, an input unit 62, an output unit 63, a communication interface 64, a positioning system 65, and a storage unit 66. At predetermined intervals, the drive recorder 6 acquires positioning data Dp using the positioning system 65 and takes pictures of a predetermined range using the camera 61, and transmits the acquired positioning data Dp and captured video data Dv to the device management server 5. The drive recorder 6 may be in the form of a dedicated drive recorder device for the purpose of acquiring driving log data, or it may be in the form of an information terminal such as a smartphone with a mobile application for acquiring driving logs (hereinafter referred to as "mobile app") installed and the information terminal used for drive recorder purposes.

[0030] Camera 61 includes a front camera that captures a predetermined range in front of the vehicle when the housing of the drive recorder 6 (not shown) is installed in a predetermined position on the vehicle (installed state). Alternatively, camera 61 may include a rear camera that captures a predetermined range behind the vehicle when the drive recorder 6 is installed, or it may include an interior camera that captures the interior of the vehicle Q to acquire interior video data. The video data Dv captured by camera 61 is linked to the positioning data Dp acquired by the positioning system 65 and stored in the storage unit 66. Camera 61 corresponds to the first camera.

[0031] The input unit 62 accepts various types of input. Specifically, the input unit 62 includes, for example, a microphone for acquiring audio from inside the vehicle and an operation reception unit for receiving input operations from user U. Examples of the operation reception unit include a push-button switch and a touch-sensor touch panel. It is also possible to receive input operations from user U via voice recognition through the microphone. In other words, the microphone may also serve as the operation reception unit. Furthermore, in this disclosure, the microphone located inside the vehicle may be a "microphone that is standardly installed in the vehicle as part of the in-vehicle system 7", a "microphone of an aftermarket device such as a drive recorder 6 that is retrofitted to the vehicle Q", or a "microphone provided on an information terminal 8 that is brought into the vehicle by user U while driving".

[0032] The output unit 63 is used to output various types of information. Specifically, the output unit 63 may include, for example, a display panel that displays output information on a display screen, a speaker that outputs audio information, or a liquid crystal display unit that presents information by turning on or off.

[0033] The communication interface 64 is a circuit that is connected to the internet and communicates with the device management server 5 via the internet. In this example, the communication interface 64 does not communicate with the content management server 1. Positioning data Dp acquired by the positioning system 65 and video data Dv captured by the camera 61 are exchanged with the device management server 5 via the communication interface 64. The positioning data Dp and video data Dv are transmitted to the device management server 5 at predetermined intervals. The predetermined interval is not particularly limited, but for example, it is 1 second.

[0034] The positioning system 65 can use a conventionally known configuration (e.g., GPS), receive radio waves from satellites, and acquire positioning data Dp based on those satellite radio waves. The positioning data Dp may be stored in the storage unit 66 on its own, or it may be stored in the storage unit 66 in association with first video data Dv1 captured by the camera 61.

[0035] The memory unit 66 is composed of, for example, volatile memory such as RAM, and non-volatile memory such as ROM or flash memory. The memory unit 66 stores the program for operating the drive recorder 6. In addition, the memory unit 66 temporarily records positioning data Dp acquired by the positioning system 65 and video data Dv captured by the camera 61.

[0036] The in-vehicle system 7 includes an input unit 71, an output unit 72, a communication interface 73, a sensor group 74, a controller group 75, an actuator group 76, and a storage unit 77.

[0037] The input unit 71 accepts various types of input. Specifically, the input unit 71 includes, for example, a microphone for acquiring sound inside the vehicle cabin and an operation reception unit for receiving user U input operations. Examples of the operation reception unit include a push-button switch and a touch-sensor touch panel. Alternatively, user U input operations may be accepted via voice recognition through the microphone. In other words, the microphone may also function as the operation reception unit.

[0038] The output unit 72 is used to output various types of information. Specifically, the output unit 72 may include, for example, a display panel that displays output information on a display screen, a speaker that outputs audio information, or a liquid crystal display unit that presents information by turning on / off.

[0039] The communication interface 73 is a circuit or device connected to the internet for communicating with other computers, servers, information terminals, etc. In this embodiment, the communication interface 73 does not communicate with any of the device management server 5, drive recorder 6, or content management server 1.

[0040] The sensor group 74 includes a camera 741 positioned on the vehicle and a positioning system 742 used for autonomous driving or driver assistance. The camera 741 includes an external camera that captures a predetermined area outside the vehicle and an internal camera that captures the interior of the vehicle. Although not shown, the external camera is composed of multiple cameras, and the combined shooting ranges of each camera are arranged to capture a 360-degree area around the vehicle.

[0041] The controller group 75 consists of various controllers that control the in-vehicle system. The controller group 75 consists of, for example, ECUs (Electronic Control Units) located in various places on the vehicle Q, and controls the actuator group 76 based on inputs from the input unit 71 and the sensor group 74. The controller group 75 may also include ADAS (Advanced driver-assistance systems) 751 that assist in driving the vehicle Q. The actuator group 76 includes drive system actuators related to the driving of the vehicle Q, braking system actuators related to braking the vehicle Q, and steering system actuators related to steering the vehicle Q.

[0042] The memory unit 77 includes, for example, memory and storage. The memory includes, for example, volatile memory such as RAM and non-volatile memory such as ROM and flash memory. The storage includes, for example, magnetic recording media such as HDD (Hard Disk Drive), semiconductor recording media such as SDD (Solid State Drive), and optical recording media such as DVD. Positioning data Dp acquired by the positioning system 742 and video data Dv captured by the camera 741 are temporarily recorded in the memory unit 77.

[0043] (3) Device management server As shown in Figure 3, the device management server 5 comprises a communication interface 51, a content management unit 52, and storage 53. The device management server 5 acquires content data Dc (including positioning data Dp and video data Dv) acquired by the managed drive recorder 6 through data linkage. Furthermore, for drive recorders 6 that are linked with the content management server 1, the device management server 5 also links the content data Dc acquired from the drive recorder 6 with the content management server 1. The device management server 5 is a cloud-type server that receives and manages event notifications or content data Dc from the vehicle Q.

[0044] The communication interface 51 is connected to the internet and communicates with the content management server 1 and the drive recorder 6 via the internet. Data exchange with the drive recorder 6 and data exchange with the content management server 1 are performed via the communication interface 51.

[0045] The content management unit 52 manages the content data Dc in the storage 53. Specifically, it saves the content data Dc received from the managed drive recorder 6 to the storage 53. Figure 3 shows an example where content data Dc is received from each of the vehicles Q (Q1, Q2, Q3, ...) and saved to the storage areas 53a, 53b, and 53c of the storage 53, respectively. It also retrieves the content data Dc to be transmitted from the storage 53 and sends it to the content management server 1 via the communication interface 51.

[0046] (4) Content management server As shown in Figure 4, the content management server 1 comprises a device gateway 2, a content manager 3, a backend for frontend 4 (BFF), a messaging basis 11, and a set of functions 14. The content management server 1 is a cloud-type server that receives and manages content data Dc.

[0047] (4-1) Device Gateway 2 is connected to a network such as the Internet and has a communication interface for communicating with other computers, servers, information terminals, etc. Device Gateway 2 communicates with Device Management Server 5.

[0048] When content data Dc is transmitted from a designated vehicle Q to the device management server 5, the device management server 5 transmits the content data Dc of that vehicle Q to the content management server 1. The content management server 1 receives the content data Dc via the device gateway 2. The device gateway 2 separately receives the positioning data Dp and the first video data Dv1 acquired by the drive recorder 6 of the vehicle Q.

[0049] Positioning data Dp is the positioning information of vehicle Q at a predetermined time. Positioning data Dp includes, for example, the date the positioning information was acquired (data acquisition date), the time the positioning information was acquired (data acquisition time), the location information of vehicle Q (e.g., latitude and longitude), the direction of travel information of vehicle Q, the direction and bearing information, the speed of vehicle Q, and some or all of the identification ID. The file format of positioning data Dp is not particularly limited, but for example, it is a text file conforming to the NMEA format.

[0050] The first video data Dv1 is video data captured by the drive recorder 6, in addition to video data captured by events such as impacts. In the drive recording display system M, the first video data Dv1 to be recorded is not particularly limited. The first video data Dv1 may also include audio. For example, the first video data Dv1 includes "video data captured by an onboard camera while the vehicle is in motion and linked to positioning data, capturing a predetermined range outside the vehicle (for example, a predetermined range in the direction of travel of the vehicle)", "video data of the outside or inside of the vehicle captured based on input received directly or indirectly from the driver or passenger", "video data of the outside or inside of the vehicle that is automatically captured when a specific location or condition is observed by a sensor, etc.", and "time-lapse video data of the outside of the vehicle captured semi-automatically and continuously". Examples of specific conditions include "a state where the atmosphere inside the vehicle is lively" and "a state where scenery such as tourist spots or landmark buildings are within the camera's frame". The first video data Dv1 may be linked to positioning data Dp. The first video data Dv1 and the positioning data Dp linked to that first video data Dv1 only need to be related to each other. For example, the positioning data Dp at the time the first video data Dv1 was taken can be said to be the positioning data Dp linked to the first video data Dv1.

[0051] (4-2) Content Management Department The content management unit 3 integrates the first video data Dv1 transmitted from the device management server 5 and the second video data Dv2 transmitted from the information terminal 8, stores them in storage, and manages them centrally. In this centralized management, the content management unit 3 manages "content files" and "content information" separately. Management here refers to enabling each application module 15 of the function group 14 and the information terminal 8 to search, refer to, and retrieve "content files" and "content information" based on predetermined conditions. "Content files" and "content information" are related to each other, for example, through a content ID. This configuration allows for centralized management of the first video data Dv1 and the second video data Dv2. For example, when a mobile application on the information terminal 8 accesses the content management unit 3 to view videos or still images taken while driving, searching becomes easier, and search speed and accuracy can be improved.

[0052] (4-3) BFF BFF4 functions as a data exchange interface with the mobile application implemented on the information terminal 8. Specifically, BFF4 has a communication interface that receives and processes commands from the mobile application implemented on user U's information terminal 8.

[0053] BFF4 executes processing in response to commands from the mobile app on the information terminal 8. For example, BFF4 can receive commands from the mobile app on the information terminal 8 such as "request to view drive logs and videos / still images," "request to create a memory movie," and "instruction to edit drive logs." BFF4 executes processing in response to these commands. Specifically, when BFF4 receives a "request to view drive logs" from the information terminal 8, it notifies the application module 15, which has a "drive log viewing request function," receives the drive log data output from the application module 15, and sends (outputs) it to the information terminal 8 (mobile app).

[0054] (4-4) Messaging infrastructure The messaging infrastructure 11 is a messaging system used for exchanging messages between application modules 15 included in the function group 14. The messaging infrastructure 11 is provided in accordance with the functions available in the function group 14 and is used by the application modules 15 that implement each of those functions.

[0055] (4-5) Functional Groups Function group 14 includes an application module 15 for realizing various functions and services (hereinafter collectively referred to as "functions"). The application module 15 includes memory for storing application programs for realizing various functions and services, and a processor for executing those application programs, and various functions are realized by executing the application programs. Examples of functions realized by function group 14 include, for example, "user information management function," "drive log management function," "video data management function," "vehicle driving trajectory display function," and "memory movie creation and editing function."

[0056] As described above, each application module 15 obtains topic information necessary for realizing its respective function from the messaging infrastructure 11. Various processing requests based on messages may be sent from the messaging infrastructure 11 to the target application module 15. Examples of processing requests include, for example, "a request to create a new record to the drive log management function when a drive start event occurs" and "an instruction to import the drive end point to the drive log management function when a drive end event occurs".

[0057] The application module 15 for displaying the driving trajectory identifies the position of vehicle Q from the acquired positioning data Dp and connects the previously acquired position of vehicle Q with the currently acquired position of vehicle Q with a straight line. The starting point of the driving trajectory is the point where the content management server 1 determines that the drive has started. The ending point of the driving trajectory is the point where the content management server 1 determines that the drive has ended. The driving trajectory is displayed on the information terminal 8 along with map data, at least as long as the vehicle Q is being driven.

[0058] Content management server 1 may be configured to automatically link predetermined data using technologies such as API integration or SFTP integration. While a detailed explanation is omitted as it is outside the scope of the disclosed technology, content management server 1 consists of hardware such as a processor, memory, and storage, and software such as control programs and data implemented in memory. Each function of content management server 1 is realized by the execution of programs stored in memory by a processor such as a CPU or GPU. Note that content management server 1 is not limited to a cloud server; an on-premises server may also be used.

[0059] (5) Information terminals As shown in Figure 5, the information terminal 8 includes a camera 81, a communication interface 82, an input unit 83, an output unit 84, and a storage unit 86.

[0060] Camera 81 is a camera mounted on the main body of the information terminal 8. It captures optical images and generates second video data Dv2. The second video data Dv2 is not particularly limited. The second video data Dv2 may be a still image or a video. Camera 81 acquires the second video data Dv2 upon request from user U. Multiple cameras 81 may be provided on the information terminal 8. Camera 81 corresponds to the second camera.

[0061] The communication interface 82 is connected to the internet and communicates with the content management server 1. The communication interface 82 transmits the second video data Dv2 and positioning data Dp captured by the camera 81 to the content management server 1. The communication interface 82 transmits commands from the mobile application to the content management server 1. The communication interface 82 obtains the location information of vehicle Q from the content management server 1 at predetermined intervals. The predetermined interval is not particularly limited, but is, for example, 1 second. The communication interface 82 receives memory information transmitted from the content management server 1.

[0062] The input unit 83 receives various inputs from user U. The input unit 83 includes a push-button switch, a touch-sensor touch panel, and a microphone for acquiring user U's voice.

[0063] The output unit 84 is used to output various types of information. Examples of the output unit 84 include a display panel that displays output information on a display screen and a speaker that outputs audio information. If the input unit 83 is configured as a touch panel, the output unit 84 and the input unit 83 are configured as the same panel. The output unit 84 reflects the location information and driving trajectory of vehicle Q received from the content management server 1 into map data and displays it together with the map image. The output unit 84 displays the memory information received from the content management server 1. The method of acquiring map data is not particularly limited. For example, location information can be provided to a platform that provides digital maps, and map data corresponding to the provided location information can be acquired.

[0064] Figure 6 shows an example of a screen displayed on the output unit 84 of the information terminal 8. In the example screen in Figure 6, the date and time of the drive, the driving trajectory of vehicle Q, and either the first video data Dv1 or the second video data Dv2 are displayed.

[0065] The storage unit 86 is composed of, for example, volatile memory such as RAM, and non-volatile semiconductor memory such as ROM or flash memory. The storage unit 86 stores the second video data Dv2 and positioning data Dp acquired by the camera 81. The second video data Dv2 and positioning data Dp stored in the storage unit 86 are transmitted to the content management server 1 via the communication interface 82.

[0066] (6) Operation of the drive recording and display system The drive record display system M operates by downloading a mobile application for using the drive record display system M to the information terminal 8 and operating the mobile application on the information terminal 8.

[0067] As part of the initial setup of the mobile app, user U pre-registers information identifying the drive recorder 6 installed in their vehicle Q (manufacturer, model, ID, name of vehicle Q, etc.) with the content management server 1 via the information terminal 8. The content management server 1 identifies the device management server 5 for the drive recorder of that manufacturer and model, and communicates with this device management server 5. The content management server 1 receives the first video data Dv1 recorded by the drive recorder with the ID registered by user U from the device management server 5. The received first video data Dv1 is then edited and processed to create, for example, drive memory information, and sent to user U's information terminal 8.

[0068] Furthermore, when user U changes the drive recorder 6 in their vehicle Q, they register information identifying the new drive recorder 6 (manufacturer, model, ID, name of vehicle Q, etc.) with the content management server 1 via the information terminal 8. The content management server 1 then identifies the device management server 5 for the new drive recorder and communicates with this device management server 5. The content management server 1 receives the first video data Dv1 recorded by the drive recorder with the ID registered by user U from the device management server 5. The received first video data Dv1 is then edited and processed to create, for example, drive memory information, and sent to user U's information terminal 8.

[0069] Next, the sequence of operations of the drive recording display system M from drive start to drive end will be explained with reference to the flowcharts in Figures 7 and 8.

[0070] (6-1) Start and end of drive Figure 7 describes the operation and processing related to drive start and drive end. Figure 7 describes the operation and processing on content management server 1. In Figure 7, the initial state is after the drive end determination, and content management server 1 has not received positioning data Dp.

[0071] When the vehicle Q is powered on, power is supplied to the drive recorder 6, and the drive recorder 6 turns on. The drive recorder 6 starts recording with the camera 61 and acquiring positioning data Dp with the positioning system 65. When the drive recorder 6 is powered on, the drive recorder 6 and the device management server 5 begin coordinating. The drive recorder 6 exchanges data with the device management server 5, sharing the first video data Dv1 recorded by the camera 61 and the content data Dc, which includes the positioning data Dp acquired by the positioning system 65.

[0072] When the drive recorder 6 and the device management server 5 are linked, the device management server 5 and the content management server 1 begin to link. When the device management server 5 receives content data Dc (including first video data Dv1 and positioning data Dp) from the drive recorder 6, which is the target of data exchange with the content management server 1, it sends an event notification to the content management server 1.

[0073] In step S11, when the content management server 1 receives positioning data Dp from the device management server 5, in step S12, the content management server 1 determines that a drive has started. The positioning data Dp includes positioning data Dp associated with the first video data Dv1.

[0074] Next, in step S13, the content management server 1 determines whether the state in which positioning data Dp has not been received at the device gateway 2 (hereinafter referred to as "non-reception state") has exceeded a predetermined time. If the duration of the non-reception state is less than or equal to the predetermined time (NO), the content management server 1 determines in step S15 that the drive is continuing. On the other hand, if the duration of the non-reception state is longer than or equal to the predetermined time (YES), the content management server 1 determines in step S14 that the drive has ended.

[0075] The "predetermined period" for determining the occurrence of drive start and drive end events is not particularly limited. The "predetermined period" may be set in advance, or it may be configured to be set and changed externally. In addition, the content management server 1 may allow user U to set or change the "predetermined period".

[0076] (6-2) Acquisition and storage of video data Figure 8 is a flowchart showing an example of the video data acquisition and saving process. This flowchart is executed repeatedly as long as the drive is running.

[0077] In step S21, when the content management server 1 receives an event notification from the device management server 5, in step S22, it requests the device management server 5 to acquire the first video data Dv1.

[0078] Upon receiving an acquisition request from the content management server 1, the device management server 5 transmits the first video data Dv1 to the content management server 1. In step S23, the content management server 1 receives the transmitted first video data Dv1. At this time, the positioning data Dp and the first video data Dv1 associated with the positioning data Dp are received separately. Note that the timing at which the positioning data Dp and the first video data Dv1, which are associated with each other, are received is not required to be the same; they may be received at different times. The first video data Dv1 received by the device gateway 2 is transmitted to the content management unit 3, and the positioning data Dp is transmitted to the event hub 12 as topic information Dm.

[0079] In step S24, the content management server 1 determines whether the received first video data Dv1 is normal driving video (e.g., time-lapse video) taken during normal vehicle driving, or event video taken at a different time from normal driving video based on a predetermined input.

[0080] If the first video data Dv1 is normal driving video data, the content management server 1 proceeds to step S25. In step S25, the content management server 1 assigns a content ID to the first video data Dv1. The content ID of the first video data Dv1 is sent to the event hub 12 as topic information Dm. The content management server 1 also performs a virus check on the first video data Dv1 to confirm the data's security. Once the data's security is confirmed, the first video data Dv1 is saved to the storage of the content management unit 3. The content management server 1 also executes a process to acquire content information. The acquired content information is saved to the storage of the content management unit 3, linked to the content ID.

[0081] If the first video data Dv1 is event video data, the content management server 1 proceeds to step S26. In step S26, the same processing as in step S25 is performed. In the next step S27, the content management server 1 creates additional content based on the first video data Dv1. The additional content is not particularly limited, but for example, it may be a thumbnail of the first video data Dv1. The path to the additional content and the video data of the additional content are stored in the storage of the content management unit 3. Note that the processing in steps S26 and S27 may be executed in parallel.

[0082] <User-captured photo and video import function> In the drive record display system M described above, when the drive ends, the content management server 1 generates drive information. The drive information is based on the location information contained in the positioning data Dp.

[0083] Drive information includes, for example, the date and time of the drive, the start location (departure point), the end location (destination point), the drive trajectory, the drive duration, the drive distance, and the name of the vehicle Q used for the drive. The start location of the drive is the location indicated by the latitude and longitude when the content management server 1 determines that the drive has started. The end location of the drive is the location indicated by the latitude and longitude when the content management server 1 determines that the drive has ended. The drive trajectory is the path taken by vehicle Q from the start location to the end location. The drive duration is the time from when the drive is determined to have started until when it is determined to have ended. As mentioned above, the start and end of a drive are determined by whether or not positioning data Dp containing the location information of vehicle Q is acquired, so the drive duration can be said to be drive information based on location information. Also, since positioning data Dp includes data that identifies which vehicle is located, the name of vehicle Q can be said to be drive information based on location information. Note that the end location of the drive does not necessarily coincide with the destination of the drive.

[0084] The content management server 1 generates drive record data viewable by the information terminal 8 for each drive. The drive record data includes text data and image data. For example, the content management server 1 generates text data for the date of the drive, the start location of the drive, the end location of the drive, the driving time, and the driving distance. For the driving trajectory, the content management server 1 generates image data, for example, a map display of the actual driving trajectory. It also generates a thumbnail from the first video data Dv1 and pastes it onto the image data of the driving trajectory.

[0085] Users can view drive record data generated by content management server 1 using the mobile application on information terminal 8.

[0086] Figure 9 shows an example of the information terminal 8 screen, where (a) is a screen showing a list of past drives. In Figure 9(a), information such as the date, departure and arrival locations, departure and arrival times, and distance traveled is shown for each drive.

[0087] In the screen shown in Figure 9(a), if the user touches, for example, area 101 to select a drive, detailed information about that drive will be displayed on the screen, as shown in Figure 9(b). In Figure 9(b), the driving trajectory 102 of the drive is shown on the map display, and thumbnails 103 of the video captured during the drive are also shown. These thumbnails 103 were captured by the drive recorder 6. The screen shown in Figure 9(b) is displayed using drive recording data transmitted from the content management server 1.

[0088] During a drive, users often take photos and videos using the camera 81 on their information terminal 8. For example, they might take pictures of scenery, food at a restaurant, or family and friends accompanying them while taking a break during their drive. Linking these photos and videos to detailed drive information and allowing them to be viewed together allows users to more deeply appreciate their travel memories.

[0089] In this embodiment, after the drive is completed, users can freely add and record photos and videos they have taken along with the drive information. Users can record memorable photos and videos, such as scenery, food, and souvenir photos taken during their trip, linked to the drive information. This allows users to more deeply relive their travel memories when viewing the detailed drive information. Furthermore, since the footage outside the car during a drive basically consists only of the road and surrounding scenery, it tends to be similar unless it includes landmarks or other symbolic elements of the location. In this respect as well, the ability for users to record photos and videos they have taken linked to the drive information enhances the usability of this system and is therefore desirable.

[0090] (Example of first action) Figure 10 is a flowchart showing a first operation example of this embodiment. When a user wants to add photos and videos they have taken to the drive information, they launch the mobile app on the information terminal 8 and select the drive list. Then, the drive list shown in Figure 9(a) is displayed on the screen of the information terminal 8 (S11).

[0091] The user selects a drive from the drive list screen to which they want to add photos and videos (S12). For example, they touch area 101 which indicates a drive. Then, the information terminal 8 displays a drive details screen for the drive selected by the user, as shown in Figure 11(a) (the same screen as Figure 9(b)) (S13).

[0092] The user selects the photos and videos they want to add to their records. For example, in the example screen shown in Figure 11(a), an "Add" button 104 is displayed at the bottom. When this "Add" button 104 is pressed, a list of photos and videos taken with the information terminal 8 is displayed. From this list, the user selects the photos and videos they want to add to their records, for example, the food photos shown in Figure 11(b). The photos and videos selected by the user are uploaded to the content management server 1 (S14).

[0093] The content management server 1 associates and records uploaded photos and videos with the drive whose drive details are displayed on the information terminal 8. Then, it displays the photos and videos selected by the user as thumbnails on the drive details screen. As a result, an image like the one shown in Figure 11(c) is displayed on the information terminal (S15). Thumbnail 105 is the photo or video selected by the user. At this time, the position in which thumbnail 105 is displayed can be determined, for example, by referring to the metadata attached to the photo or video selected by the user, and based on information such as the shooting location and time contained in this metadata, as well as the driving trajectory data.

[0094] Figure 12 shows an example of content information stored in the content management unit 3. In the example in Figure 12, the content and information of the video data stored in the first drive D1 and the second drive D2 are shown. In Figure 12, the actual storage location of each video data is shown in "Path". In Figure 12, in the first drive D1, video data transmitted from the information terminal 8 is added and stored as video data No. 14. In the second drive D2, still image data transmitted from the information terminal 8 is added and stored as video data No. 22. Thumbnails are then generated for video data No. 14 and 22, respectively.

[0095] When the user views the drive details screen where photos and videos have been added, they return the information terminal screen to the drive list screen (S16).

[0096] This feature allows users to record memorable photos and videos of scenery, food, and souvenirs taken during their trip, linking them to their driving information.

[0097] (Example of second action) Figure 13 is a flowchart showing a second operation example of this embodiment. When a user wants to add photos and videos they have taken to the drive information, they first launch the mobile app on the information terminal 8 and select the menu to add photos and videos (S21). When this menu is selected, a list of photos and videos taken with the information terminal 8 is displayed on the screen. From this list, the user selects the photo or video they want to add, for example, the photo of food shown in Figure 11(b). The photo or video selected by the user is uploaded to the content management server 1 (S22).

[0098] On the content management server 1, an application module 15 with a "drive search function" identifies the drive corresponding to the uploaded photos and videos (S23). This identification is performed, for example, by referring to the metadata attached to the uploaded photos and videos and comparing the shooting location and shooting time information contained in the metadata with the drive information of each drive. Once the corresponding drive is identified, the drive details of that drive are sent to the information terminal 8. The transmitted drive details are displayed on the screen of the information terminal 8 (S24).

[0099] The user confirms that the drive details displayed on the screen belong to the drive to which they want to add photos and videos. Upon user confirmation, the content management server 1 associates and records the uploaded photos and videos to the drive whose drive details are displayed on the information terminal 8. Then, the user's selected photos and videos are displayed as thumbnails on the drive details screen (S25). As a result, an image like the one shown in Figure 11(c) is displayed on the screen of the information terminal 8. Thumbnail 105 is the photo or video selected by the user. At this time, the position in which thumbnail 105 is displayed can be determined, for example, based on the shooting location and shooting time information included in the metadata attached to the photo or video selected by the user.

[0100] After the user confirms the drive details screen where photos and videos have been added, they exit the Add Photos / Videos menu (S26).

[0101] This functionality allows users to record memorable photos and videos—such as landscapes, food, and souvenirs—taken during their trip, linking them to their drive information. Furthermore, the system automatically identifies the drive corresponding to the photos and videos the user wants to add, making the process easier.

[0102] (Other examples) The actions described above are performed on a drive that has just finished. However, these actions can also be performed on a drive in progress. For example, photos and videos taken during a break in the drive can be linked to the current drive and added to the recording.

[0103] Furthermore, it may be possible to perform the same operations from a different information device, such as a home PC, rather than from the information terminal that took the photos and videos. In this case, the user would need to transfer or upload the photos and videos they wish to record from the information terminal to the other information device in advance.

[0104] <Other Embodiments> The technologies disclosed herein are not limited to the embodiments described above and may be substituted insofar as they do not depart from the spirit of the claims.

[0105] The vehicle may be a bicycle, motorbike, railway car, electric scooter, rickshaw, etc. When the vehicle is a bicycle, motorbike, or railway car, the device capable of acquiring location information is, for example, a drive recorder. When the vehicle is an electric scooter, bicycle, or motorbike, the linking of the mobile terminal and the drive recorder can be done in the same manner as in the embodiments described above. When the vehicle is a railway car, the railway company approves the user registration when linking the mobile terminal and the drive recorder. When the vehicle is a rickshaw, for example, the camera placed on the vehicle is attached to the rickshaw. The device that acquires the vehicle's positioning data is not particularly limited. For example, it may be attached to the rickshaw, or it may also be an information terminal held by the rickshaw driver. In this case, the positioning data acquired by the positioning function of the information terminal held by the rickshaw driver may be used as the vehicle's positioning data. Alternatively, the information terminal held by a passenger riding in the rickshaw may also serve as the device that acquires the vehicle's positioning data.

[0106] Furthermore, as a reference example, the technology disclosed herein can also be applied to aircraft. When applied to aircraft, the device capable of acquiring location information is a flight recorder. In linking a mobile terminal with a drive recorder, user registration approval is performed by the airline. In addition, as another reference example, the technology disclosed herein can be applied to privately operated vessels such as pleasure boats. For vessels, the system of this embodiment configured for vehicles can be easily adapted by replacing maps with nautical charts and the driving trajectory with a navigation trajectory.

[0107] The embodiments described above are merely illustrative and should not be interpreted as limiting the scope of this disclosure. The scope of this disclosure is defined by the claims, and any variations or modifications within the equivalent scope of the claims are all within the scope of this disclosure. [Industrial applicability]

[0108] The technology disclosed herein is useful for a drive recording and display service that utilizes an in-vehicle device equipped with a camera. [Explanation of symbols]

[0109] 1. Content Management Server 6. Dashcam 8. Information terminals 61 Camera 81 Camera M Drive Recording and Display System Q Vehicle

Claims

1. A drive record display system that displays records of driving using a vehicle, A device mounted on the aforementioned vehicle, having a first camera and capable of acquiring the vehicle's location information, It has a second camera and is connected to an information terminal held by the user of the vehicle, The system comprises a server capable of communicating with the aforementioned information terminal, The aforementioned server, The device acquires the first video data captured by the first camera and the location information acquired by the device. Based on the location information, drive information is generated, and drive recording data is generated for each drive, which includes the generated drive information and at least a portion of the first video data and is viewable by the information terminal. The second video data captured by the second camera of the information terminal is acquired. The second video data is added to the drive recording data of the drive corresponding to the acquired second video data. Drive record display system.

2. In the drive recording display system according to claim 1, The aforementioned drive information includes the drive trajectory, the drive's starting point, and the drive's destination. The drive recording data includes drawing data of the driving trajectory on the map display and a thumbnail of the first video data. Drive record display system.

3. In the drive recording display system according to claim 1, The second video data includes data on the date and time of shooting and the location of shooting as metadata. The aforementioned server, The drive corresponding to the second video data is identified by referring to the metadata and the drive information of each drive. Drive record display system.

4. A drive record display method that displays a record of driving using a vehicle, The vehicle is equipped with a device having a first camera and capable of acquiring the vehicle's location information. The user of the aforementioned vehicle has an information terminal equipped with a second camera, A server capable of communicating with the aforementioned information terminal, The device acquires the first video data captured by the first camera and the location information acquired by the device. Based on the location information, drive information is generated, and drive recording data is generated for each drive, which includes the generated drive information and at least a portion of the first video data and is viewable by the information terminal. The second video data captured by the second camera of the information terminal is acquired. The second video data is added to the drive recording data of the drive corresponding to the acquired second video data. How to display driving records.

Images