Drive record display system and drive record display method

Abstract

In a system that records, edits, and provides video data from a drive to a user, the display format of information related to the drive should be made easy for the user to understand. [Solution] Server 1 generates the driving trajectory based on the location information acquired by device 6 and thumbnails of the video data captured by camera 61 of device 6 as first drive display data, and generates driving information as second drive display data. Information terminal 8 receives the first and second drive display data from server 1, and on the display screen, draws the driving trajectory 101 represented by the first drive display data on the map data, pastes the thumbnail 102 and displays it on the lower layer L1, and displays the driving information indicated by the second drive display data on the upper layer L2.

Description

Technical Field

[0001] The technology disclosed herein relates to a drive recording display system that records video data in a drive or the like, performs editing processing, and provides it to users.

Background Art

[0002] In recent years, it has become common to share memories of a drive or the like through digital data. As a method of leaving memories of a drive, there is a method of creating an electronic album using image data captured during the drive.

[0003] Patent Document 1 discloses a memory sharing system for vehicles that can more easily share memory information among members who went on a trip together. In this system, an in-vehicle device cooperates with a user's mobile terminal brought into the vehicle. Then, the in-vehicle device and the mobile terminal acquire information representing memories during the trip, and the integrated memory information is transmitted from the in-vehicle device to the server. The integrated memory information is distributed to the mobile terminal.

[0004] Patent Document 2 discloses a shooting information browsing system in which a server collects and stores shooting location data of shootings by in-vehicle cameras implemented in a plurality of vehicles, and makes the collection and storage results viewable on a terminal device.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, the vehicle memory sharing system disclosed in Patent Document 1 requires the in-vehicle unit to have functions such as linking with the user's mobile terminal brought into the vehicle and integrating memory information acquired by the in-vehicle unit and the mobile terminal. Therefore, in order to realize the system, there is a problem that an in-vehicle unit with complex functions must be installed in the vehicle. Furthermore, since the functions can only be realized with a specific in-vehicle unit that is compatible with this system, it is difficult to change the in-vehicle unit.

[0007] Furthermore, in the photographic information viewing system described in Patent Document 2, the vehicle's trajectory, information related to the vehicle's movement, and photographic data are displayed on a single screen on the terminal device. When such a display method is used on an information terminal such as a smartphone, it becomes very difficult for the user to see.

[0008] The technology disclosed herein was developed in view of these points, and its purpose is to make the display format of information about the drive more user-friendly in a system that records, edits, and provides video data from the drive to the user. [Means for solving the problem]

[0009] To solve the aforementioned problems, the technology disclosed herein provides a drive record display system that displays a record of driving using a vehicle, comprising: a device mounted on the vehicle, having a camera and capable of acquiring the vehicle's location information; an information terminal owned by the vehicle's user; and a server capable of communicating with the information terminal, wherein the server acquires video data captured by the camera of the device and the location information acquired by the device, generates data representing the vehicle's driving trajectory based on the location information for the drive, generates a thumbnail from the video data, and displays the generated data representing the driving trajectory and the thumbnail. The system generates first drive display data for the drive in question, and generates second drive display data for the drive in question, including driving distance, driving time, departure point, and arrival point. The system transmits the first and second drive display data to the information terminal. The information terminal receives the first and second drive display data from the server, and on the display screen, it draws the driving trajectory represented by the first drive display data on the map data, pastes the thumbnail onto it, displays it on the lower layer, and displays the drive information indicated by the second drive display data on the upper layer which is superimposed on a part of the lower layer.

[0010] According to this configuration, the drive record display system comprises a device mounted in the vehicle, equipped with a camera and capable of acquiring vehicle location information, an information terminal held by the vehicle user, and a server capable of communicating with the information terminal. The server generates first drive display data, which consists of data representing the driving trajectory based on the location information acquired by the device and thumbnails generated from video data captured by the device's camera. The server also generates second drive display data, which includes drive information such as the departure and arrival points. The information terminal receives the first and second drive display data from the server and, on the display screen, draws the driving trajectory represented by the first drive display data on map data, pastes a thumbnail onto it, displays it on the lower layer, and displays the drive information indicated by the second drive display data on an upper layer superimposed on a portion of the lower layer. As a result, on the information terminal's display screen, the overall picture of the drive, including the driving trajectory, is displayed on the lower layer, and drive information such as the departure and arrival points is displayed on the upper layer, making the display of drive-related information on the information terminal easy for the user to understand.

[0011] Furthermore, the information terminal may be configured to allow the user to increase or decrease the area in which the upper layer is displayed through user operation.

[0012] This allows users to manipulate the area where the upper layer is displayed, making the drive information on the information terminal easier to view.

[0013] Furthermore, the information terminal may be configured to allow users to input comments on the upper layer.

[0014] This allows users to enter comments on the information terminal screen.

[0015] To solve the aforementioned problems, the technology disclosed herein provides a drive record display method for displaying a record of a drive using a vehicle, wherein the vehicle is equipped with a camera and a device capable of acquiring the vehicle's location information, the user of the vehicle has an information terminal, and a server capable of communicating with the information terminal acquires video data captured by the camera of the device and location information acquired by the device, generates data representing the vehicle's driving trajectory based on the location information for the drive, generates a thumbnail from the video data, and displays the generated driving trajectory data and thumbnail, The system generates first drive display data for the drive in question, and generates second drive display data for the drive in question, including driving distance, driving time, departure point, and arrival point. The first and second drive display data are transmitted to the information terminal. The information terminal receives the first and second drive display data from the server, and on the display screen, it draws the driving trajectory represented by the first drive display data on the map data, pastes the thumbnail onto it, and displays it on the lower layer. On the upper layer, which is superimposed on a part of the lower layer, the system displays the drive information indicated by the second drive display data.

[0016] In this configuration, the drive record display method involves a vehicle equipped with a camera and a device capable of acquiring vehicle location information, and the vehicle user having an information terminal. A server that can communicate with the information terminal generates first drive display data, which includes data representing the driving trajectory based on the location information acquired by the device, and a thumbnail generated from video data captured by the device's camera. The server also generates second drive display data, which includes drive information such as the departure and arrival points. The information terminal receives the first and second drive display data from the server and, on the display screen, draws the driving trajectory represented by the first drive display data on the map data, pastes the thumbnail onto it, and displays it on the lower layer. On the upper layer, which is superimposed on a part of the lower layer, the drive information indicated by the second drive display data is displayed. As a result, on the information terminal's display screen, the overall picture of the drive, including the driving trajectory, is displayed on the lower layer, and drive information such as the departure and arrival points is displayed on the upper layer, making the display of drive-related information on the information terminal easy for the user to understand. [Effects of the Invention]

[0017] As explained above, the technology disclosed herein allows for a system that records, edits, and provides video data from a drive to a user, making the display format of information about the drive more user-friendly. [Brief explanation of the drawing]

[0018] [Figure 1] Conceptual diagram showing an example of the overall configuration of a drive recording and display system according to an exemplary embodiment. [Figure 2] Schematic diagram of the vehicle [Figure 3] Schematic diagram of the device management server [Figure 4] Diagram showing the configuration of the content management server. [Figure 5] Schematic diagram of the information terminal [Figure 6] A diagram showing an example of the screen display of an information terminal. [Figure 7]Flowchart for determining the start and end of drive [Figure 8] Flowchart showing an example of acquisition processing and storage processing of video data [Figure 9] (a) and (b) are examples of screen displays of the information terminal in the embodiment [Figure 10] Operation examples of the information terminal and the content management server in the embodiment

Mode for Carrying Out the Invention

[0019] Hereinafter, exemplary embodiments will be described in detail with reference to the drawings.

[0020] Note that the following embodiments are exemplary, and there is no intention to limit the content of the present disclosure by the presence or absence of description, the exemplified numerical values, etc. Also, the components constituting the systems, servers, etc. described in this specification, and the functions realized by the components such as the functional blocks and modules described in this specification can be implemented in circuitry or processing circuitry including general-purpose processors such as CPU (Central Processing Unit) and GPU (Graphics Processing Unit), specific-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), conventional circuits, and / or combinations thereof. The processor includes transistors and other circuits and is regarded as circuitry or processing circuitry. The processor may be configured to execute a program stored in the memory. The memory includes, for example, volatile memory such as RAM and non-volatile semiconductor memory such as ROM and flash memory.

[0021] (1) Overall configuration of the drive recording display system Figure 1 is a conceptual diagram showing an example of the overall configuration of a drive recording and display system M according to an embodiment. The drive recording and display system M comprises a vehicle Q including equipment with a camera, an information terminal 8, a device management server 5, and a content management server 1.

[0022] In Figure 1, vehicle Q is equipped with a drive recorder 6, which has a camera 61. The drive recorder 6 captures and records video data Dv (including the first video data Dv1 described later) of the area around vehicle Q while it is in motion using the camera 61. The video data Dv is recorded periodically during driving, or when an event such as an impact occurs. The recorded video data Dv includes still images and videos (including those shot in time lapse). The drive recorder 6 is a communication-type drive recorder and communicates with a device management server 5 for drive recorders. The drive recorder 6 transmits the recorded video data Dv to the device management server 5, for example, periodically, or at any arbitrary time.

[0023] In this embodiment, the drive recorder 6 has a positioning system 65 (see Figure 2) that acquires the location information of the vehicle Q. The positioning system acquires positioning data Dp. Details of the positioning data Dp will be described later.

[0024] Information terminal 8 is a terminal device carried by user U who is driving vehicle Q or user U who is a passenger in vehicle Q. Information terminal 8 has a camera 81. Information terminal 8 records second video data Dv2 using the camera 81.

[0025] The device management server 5 and the content management server 1 can communicate with each other. The content management server 1 corresponds to the server in this disclosure.

[0026] The device management server 5 receives the video data Dv captured by the drive recorder 6's camera from the drive recorder 6 and temporarily stores it.

[0027] The content management server 1 receives and saves the first video data Dv1 from the device management server 5, which is video data captured by the drive recorder 6's camera 61. The content management server 1 edits the saved first video data Dv1 according to instructions from user U via the information terminal 8, creates, for example, drive memory information, and sends it to the information terminal 8. The content management server 1 may also use the second video data Dv2 sent by user U from the information terminal 8 to edit the memory information. The first video data Dv1 will be described later.

[0028] (2) Vehicles As shown in Figure 2, vehicle Q is equipped with a typical in-vehicle system 7. The drive recorder 6 was retrofitted to vehicle Q by user U.

[0029] The drive recorder 6 is an example of a device capable of acquiring location information of vehicle Q. The drive recorder 6 comprises a camera 61, an input unit 62, an output unit 63, a communication interface 64, a positioning system 65, and a storage unit 66. At predetermined intervals, the drive recorder 6 acquires positioning data Dp using the positioning system 65 and takes pictures of a predetermined range using the camera 61, and transmits the acquired positioning data Dp and captured video data Dv to the device management server 5. The drive recorder 6 may be in the form of a dedicated drive recorder device for the purpose of acquiring driving log data, or it may be in the form of an information terminal such as a smartphone with a mobile application for acquiring driving logs (hereinafter referred to as "mobile app") installed and the information terminal used for drive recorder purposes.

[0030] Camera 61 includes a front camera that captures a predetermined range in front of the vehicle when the housing of the drive recorder 6 (not shown) is installed in a predetermined position on the vehicle (installed state). Alternatively, camera 61 may include a rear camera that captures a predetermined range behind the vehicle when the drive recorder 6 is installed, or it may include an interior camera that captures the interior of the vehicle Q to acquire interior video data. The video data Dv captured by camera 61 is linked to the positioning data Dp acquired by the positioning system 65 and stored in the storage unit 66. Camera 61 corresponds to the first camera.

[0031] The input unit 62 accepts various types of input. Specifically, the input unit 62 includes, for example, a microphone for acquiring audio from inside the vehicle and an operation reception unit for receiving input operations from user U. Examples of the operation reception unit include a push-button switch and a touch-sensor touch panel. It is also possible to receive input operations from user U via voice recognition through the microphone. In other words, the microphone may also serve as the operation reception unit. Furthermore, in this disclosure, the microphone located inside the vehicle may be a "microphone that is standardly installed in the vehicle as part of the in-vehicle system 7", a "microphone of an aftermarket device such as a drive recorder 6 that is retrofitted to the vehicle Q", or a "microphone provided on an information terminal 8 that is brought into the vehicle by user U while driving".

[0032] The output unit 63 is used to output various types of information. Specifically, the output unit 63 may include, for example, a display panel that displays output information on a display screen, a speaker that outputs audio information, or a liquid crystal display unit that presents information by turning on or off.

[0033] The communication interface 64 is a circuit that is connected to the internet and communicates with the device management server 5 via the internet. In this example, the communication interface 64 does not communicate with the content management server 1. Positioning data Dp acquired by the positioning system 65 and video data Dv captured by the camera 61 are exchanged with the device management server 5 via the communication interface 64. The positioning data Dp and video data Dv are transmitted to the device management server 5 at predetermined intervals. The predetermined interval is not particularly limited, but for example, it is 1 second.

[0034] The positioning system 65 can use a conventionally known configuration (e.g., GPS), receive radio waves from satellites, and acquire positioning data Dp based on those satellite radio waves. The positioning data Dp may be stored in the storage unit 66 on its own, or it may be stored in the storage unit 66 in association with first video data Dv1 captured by the camera 61.

[0035] The memory unit 66 is composed of, for example, volatile memory such as RAM, and non-volatile memory such as ROM or flash memory. The memory unit 66 stores the program for operating the drive recorder 6. In addition, the memory unit 66 temporarily records positioning data Dp acquired by the positioning system 65 and video data Dv captured by the camera 61.

[0036] The in-vehicle system 7 includes an input unit 71, an output unit 72, a communication interface 73, a sensor group 74, a controller group 75, an actuator group 76, and a storage unit 77.

[0037] The input unit 71 accepts various types of input. Specifically, the input unit 71 includes, for example, a microphone for acquiring sound inside the vehicle cabin and an operation reception unit for receiving user U input operations. Examples of the operation reception unit include a push-button switch and a touch-sensor touch panel. Alternatively, user U input operations may be accepted via voice recognition through the microphone. In other words, the microphone may also function as the operation reception unit.

[0038] The output unit 72 is used to output various types of information. Specifically, the output unit 72 may include, for example, a display panel that displays output information on a display screen, a speaker that outputs audio information, or a liquid crystal display unit that presents information by turning on / off.

[0039] The communication interface 73 is a circuit or device connected to the internet for communicating with other computers, servers, information terminals, etc. In this embodiment, the communication interface 73 does not communicate with any of the device management server 5, drive recorder 6, or content management server 1.

[0040] The sensor group 74 includes a camera 741 positioned on the vehicle and a positioning system 742 used for autonomous driving or driver assistance. The camera 741 includes an external camera that captures a predetermined area outside the vehicle and an internal camera that captures the interior of the vehicle. Although not shown, the external camera is composed of multiple cameras, and the combined shooting ranges of each camera are arranged to capture a 360-degree area around the vehicle.

[0041] The controller group 75 consists of various controllers that control the in-vehicle system. The controller group 75 consists of, for example, ECUs (Electronic Control Units) located in various places on the vehicle Q, and controls the actuator group 76 based on inputs from the input unit 71 and the sensor group 74. The controller group 75 may also include ADAS (Advanced driver-assistance systems) 751 that assist in driving the vehicle Q. The actuator group 76 includes drive system actuators related to the driving of the vehicle Q, braking system actuators related to braking the vehicle Q, and steering system actuators related to steering the vehicle Q.

[0042] The memory unit 77 includes, for example, memory and storage. The memory includes, for example, volatile memory such as RAM and non-volatile memory such as ROM and flash memory. The storage includes, for example, magnetic recording media such as HDD (Hard Disk Drive), semiconductor recording media such as SDD (Solid State Drive), and optical recording media such as DVD. Positioning data Dp acquired by the positioning system 742 and video data Dv captured by the camera 741 are temporarily recorded in the memory unit 77.

[0043] (3) Device management server As shown in Figure 3, the device management server 5 comprises a communication interface 51, a content management unit 52, and storage 53. The device management server 5 acquires content data Dc (including positioning data Dp and video data Dv) acquired by the managed drive recorder 6 through data linkage. Furthermore, for drive recorders 6 that are linked with the content management server 1, the device management server 5 also links the content data Dc acquired from the drive recorder 6 with the content management server 1. The device management server 5 is a cloud-type server that receives and manages event notifications or content data Dc from the vehicle Q.

[0044] The communication interface 51 is connected to the internet and communicates with the content management server 1 and the drive recorder 6 via the internet. Data exchange with the drive recorder 6 and data exchange with the content management server 1 are performed via the communication interface 51.

[0045] The content management unit 52 manages the content data Dc in the storage 53. Specifically, it saves the content data Dc received from the managed drive recorder 6 to the storage 53. Figure 3 shows an example where content data Dc is received from each of the vehicles Q (Q1, Q2, Q3, ...) and saved to the storage areas 53a, 53b, and 53c of the storage 53, respectively. It also retrieves the content data Dc to be transmitted from the storage 53 and sends it to the content management server 1 via the communication interface 51.

[0046] (4) Content management server As shown in Figure 4, the content management server 1 comprises a device gateway 2, a content manager 3, a backend for frontend 4 (BFF), a messaging basis 11, and a set of functions 14. The content management server 1 is a cloud-type server that receives and manages content data Dc.

[0047] (4-1) Device Gateway 2 is connected to a network such as the Internet and has a communication interface for communicating with other computers, servers, information terminals, etc. Device Gateway 2 communicates with Device Management Server 5.

[0048] When content data Dc is transmitted from a designated vehicle Q to the device management server 5, the device management server 5 transmits the content data Dc of that vehicle Q to the content management server 1. The content management server 1 receives the content data Dc via the device gateway 2. The device gateway 2 separately receives the positioning data Dp and the first video data Dv1 acquired by the drive recorder 6 of the vehicle Q.

[0049] Positioning data Dp is the positioning information of vehicle Q at a predetermined time. Positioning data Dp includes, for example, the date the positioning information was acquired (data acquisition date), the time the positioning information was acquired (data acquisition time), the location information of vehicle Q (e.g., latitude and longitude), the direction of travel information of vehicle Q, the direction and bearing information, the speed of vehicle Q, and some or all of the identification ID. The file format of positioning data Dp is not particularly limited, but for example, it is a text file conforming to the NMEA format.

[0050] The first video data Dv1 is video data captured by the drive recorder 6, in addition to video data captured by events such as impacts. In the drive recording display system M, the first video data Dv1 to be recorded is not particularly limited. The first video data Dv1 may also include audio. For example, the first video data Dv1 includes "video data captured by an onboard camera while the vehicle is in motion and linked to positioning data, capturing a predetermined range outside the vehicle (for example, a predetermined range in the direction of travel of the vehicle)", "video data of the outside or inside of the vehicle captured based on input received directly or indirectly from the driver or passenger", "video data of the outside or inside of the vehicle that is automatically captured when a specific location or condition is observed by a sensor, etc.", and "time-lapse video data of the outside of the vehicle captured semi-automatically and continuously". Examples of specific conditions include "a state where the atmosphere inside the vehicle is lively" and "a state where scenery such as tourist spots or landmark buildings are within the camera's frame". The first video data Dv1 may be linked to positioning data Dp. The first video data Dv1 and the positioning data Dp linked to that first video data Dv1 only need to be related to each other. For example, the positioning data Dp at the time the first video data Dv1 was taken can be said to be the positioning data Dp linked to the first video data Dv1.

[0051] (4-2) Content Management Department The content management unit 3 integrates the first video data Dv1 transmitted from the device management server 5 and the second video data Dv2 transmitted from the information terminal 8, stores them in storage, and manages them centrally. In this centralized management, the content management unit 3 manages "content files" and "content information" separately. Management here refers to enabling each application module 15 of the function group 14 and the information terminal 8 to search, refer to, and retrieve "content files" and "content information" based on predetermined conditions. "Content files" and "content information" are related to each other, for example, through a content ID. This configuration allows for centralized management of the first video data Dv1 and the second video data Dv2. For example, when a mobile application on the information terminal 8 accesses the content management unit 3 to view videos or still images taken while driving, searching becomes easier, and search speed and accuracy can be improved.

[0052] (4-3) BFF BFF4 functions as a data exchange interface with the mobile application implemented on the information terminal 8. Specifically, BFF4 has a communication interface that receives and processes commands from the mobile application implemented on user U's information terminal 8.

[0053] BFF4 executes processing in response to commands from the mobile app on the information terminal 8. For example, BFF4 receives commands from the mobile app on the information terminal 8 such as "request to view drive logs and videos / still images," "request to create a memory movie," and "instruction to edit drive logs." BFF4 executes processing in response to these commands. Specifically, when BFF4 receives a "request to view drive logs" from the information terminal 8, it notifies the application module 15, which has a "drive log viewing request function," receives the drive log created by the application module 15, and sends (outputs) it to the information terminal 8 (mobile app).

[0054] (4-4) Messaging infrastructure The messaging infrastructure 11 is a messaging system used for exchanging messages between application modules 15 included in the function group 14. The messaging infrastructure 11 is provided in accordance with the functions available in the function group 14 and is used by the application modules 15 that implement each of those functions.

[0055] (4-5) Functional Groups Function group 14 includes an application module 15 for realizing various functions and services (hereinafter collectively referred to as "functions"). The application module 15 includes memory for storing application programs for realizing various functions and services, and a processor for executing those application programs, and various functions are realized by executing the application programs. Examples of functions realized by function group 14 include, for example, "user information management function," "drive log management function," "video data management function," "vehicle driving trajectory display function," "memory movie creation and editing function," and "log card (corresponding to drive record image data) creation and editing function."

[0056] As described above, each application module 15 obtains topic information necessary for realizing its respective function from the messaging infrastructure 11. Various processing requests based on messages may be sent from the messaging infrastructure 11 to the target application module 15. Examples of processing requests include, for example, "a request to create a new record to the drive log management function when a drive start event occurs" and "an instruction to import the drive end point to the drive log management function when a drive end event occurs".

[0057] The application module 15 for displaying the driving trajectory identifies the position of vehicle Q from the acquired positioning data Dp and connects the previously acquired position of vehicle Q with the currently acquired position of vehicle Q with a straight line. The starting point of the driving trajectory is the point where the content management server 1 determines that the drive has started. The ending point of the driving trajectory is the point where the content management server 1 determines that the drive has ended. In addition, the application module 15 for generating thumbnails generates thumbnails to be displayed together with the driving trajectory from the first video data Dv1 and the second video data Dv2, which are centrally managed by the content management unit 3. The driving trajectory and thumbnails are updated at least as long as the vehicle Q is being driven, and the updated data is transmitted to the information terminal 8 and displayed on the information terminal 8 together with map data. This improves the real-time display of the vehicle's driving trajectory and thumbnails during driving.

[0058] Content management server 1 may be configured to automatically link predetermined data using technologies such as API integration or SFTP integration. While a detailed explanation is omitted as it is outside the scope of the disclosed technology, content management server 1 consists of hardware such as a processor, memory, and storage, and software such as control programs and data implemented in memory. Each function of content management server 1 is realized by the execution of programs stored in memory by a processor such as a CPU or GPU. Note that content management server 1 is not limited to a cloud server; an on-premises server may also be used.

[0059] (5) Information terminals As shown in Figure 5, the information terminal 8 includes a camera 81, a communication interface 82, an input unit 83, an output unit 84, and a storage unit 86.

[0060] Camera 81 is a camera mounted on the main body of the information terminal 8. It captures optical images and generates second video data Dv2. The second video data Dv2 is not particularly limited. The second video data Dv2 may be a still image or a video. Camera 81 acquires the second video data Dv2 upon request from user U. Multiple cameras 81 may be provided on the information terminal 8. Camera 81 corresponds to the second camera.

[0061] The communication interface 82 is connected to the internet and communicates with the content management server 1. The communication interface 82 transmits the second video data Dv2 and positioning data Dp captured by the camera 81 to the content management server 1. The communication interface 82 transmits commands from the mobile application to the content management server 1. The communication interface 82 obtains the location information of vehicle Q from the content management server 1 at predetermined intervals. The predetermined interval is not particularly limited, but for example, it is 1 second. It receives memory information transmitted from the content management server 1.

[0062] The input unit 83 receives various inputs from user U. The input unit 83 includes a push-button switch, a touch-sensor touch panel, and a microphone for acquiring user U's voice.

[0063] The output unit 84 is used to output various types of information. Examples of the output unit 84 include a display panel that displays output information on a display screen and a speaker that outputs audio information. If the input unit 83 is composed of a touch panel, the output unit 84 and the input unit 83 are composed of the same panel. The output unit 84 reflects the location information and driving trajectory of vehicle Q received from the content management server 1 into map data and displays it together with the map image. The output unit 84 also displays thumbnails transmitted from the content management server 1 along with the driving trajectory. Since the display data transmitted from the content management server 1 is updated while vehicle Q is being driven, the display of the driving trajectory and thumbnails on the information terminal 8 is updated in real time while driving. The output unit 84 displays memory information received from the content management server 1. The method of acquiring map data is not particularly limited. For example, location information can be provided to a platform that provides digital maps, and map data corresponding to the provided location information can be acquired.

[0064] Figure 6 shows an example of a screen displayed on the output unit 84 of the information terminal 8. In the example screen in Figure 6, the date and time of the drive, the driving trajectory of vehicle Q, and either the first video data Dv1 or the second video data Dv2 are displayed.

[0065] The storage unit 86 is composed of, for example, volatile memory such as RAM, and non-volatile semiconductor memory such as ROM or flash memory. The storage unit 86 stores the second video data Dv2 and positioning data Dp acquired by the camera 81. The second video data Dv2 and positioning data Dp stored in the storage unit 86 are transmitted to the content management server 1 via the communication interface 82.

[0066] (6) Operation of the drive recording and display system The drive record display system M operates by downloading a mobile application for using the drive record display system M to the information terminal 8 and operating the mobile application on the information terminal 8.

[0067] As part of the initial setup of the mobile app, user U pre-registers information identifying the drive recorder 6 installed in their vehicle Q (manufacturer, model, ID, name of vehicle Q, etc.) with the content management server 1 via the information terminal 8. The content management server 1 identifies the device management server 5 for the drive recorder of that manufacturer and model, and communicates with this device management server 5. The content management server 1 receives the first video data Dv1 recorded by the drive recorder with the ID registered by user U from the device management server 5. The received first video data Dv1 is then edited and processed to create, for example, drive memory information, and sent to user U's information terminal 8.

[0068] Furthermore, when user U changes the drive recorder 6 in their vehicle Q, they register information identifying the new drive recorder 6 (manufacturer, model, ID, name of vehicle Q, etc.) with the content management server 1 via the information terminal 8. The content management server 1 then identifies the device management server 5 for the new drive recorder and communicates with this device management server 5. The content management server 1 receives the first video data Dv1 recorded by the drive recorder with the ID registered by user U from the device management server 5. The received first video data Dv1 is then edited and processed to create, for example, drive memory information, and sent to user U's information terminal 8.

[0069] Next, the sequence of operations of the drive recording display system M from drive start to drive end will be explained with reference to the flowcharts in Figures 7 and 8.

[0070] (6-1) Start and end of drive Figure 7 describes the operation and processing related to drive start and drive end. Figure 7 describes the operation and processing on content management server 1. In Figure 7, the initial state is after the drive end determination, and content management server 1 has not received positioning data Dp.

[0071] When the vehicle Q is powered on, power is supplied to the drive recorder 6, and the drive recorder 6 turns on. The drive recorder 6 starts recording with the camera 61 and acquiring positioning data Dp with the positioning system 65. When the drive recorder 6 is powered on, the drive recorder 6 and the device management server 5 begin coordinating. The drive recorder 6 exchanges data with the device management server 5, sharing the first video data Dv1 recorded by the camera 61 and the content data Dc, which includes the positioning data Dp acquired by the positioning system 65.

[0072] When the drive recorder 6 and the device management server 5 are linked, the device management server 5 and the content management server 1 begin to link. When the device management server 5 receives content data Dc (including first video data Dv1 and positioning data Dp) from the drive recorder 6, which is the target of data exchange with the content management server 1, it sends an event notification to the content management server 1.

[0073] In step S11, when the content management server 1 receives positioning data Dp from the device management server 5, in step S12, the content management server 1 determines that a drive has started. The positioning data Dp includes positioning data Dp associated with the first video data Dv1.

[0074] Next, in step S13, the content management server 1 determines whether the state in which positioning data Dp has not been received at the device gateway 2 (hereinafter referred to as "non-reception state") has exceeded a predetermined time. If the duration of the non-reception state is less than or equal to the predetermined time (NO), the content management server 1 determines in step S15 that the drive is continuing. On the other hand, if the duration of the non-reception state is longer than or equal to the predetermined time (YES), the content management server 1 determines in step S14 that the drive has ended.

[0075] The "predetermined period" for determining the occurrence of drive start and drive end events is not particularly limited. The "predetermined period" may be set in advance, or it may be configured to be set and changed externally. In addition, the content management server 1 may allow user U to set or change the "predetermined period".

[0076] (6-2) Acquisition and storage of video data Figure 8 is a flowchart showing an example of the video data acquisition and saving process. This flowchart is executed repeatedly as long as the drive is running.

[0077] In step S21, when the content management server 1 receives an event notification from the device management server 5, in step S22, it requests the device management server 5 to acquire the first video data Dv1.

[0078] Upon receiving an acquisition request from the content management server 1, the device management server 5 transmits the first video data Dv1 to the content management server 1. In step S23, the content management server 1 receives the transmitted first video data Dv1. At this time, the positioning data Dp and the first video data Dv1 associated with the positioning data Dp are received separately. Note that the timing at which the positioning data Dp and the first video data Dv1, which are associated with each other, are received is not required to be the same; they may be received at different times. The first video data Dv1 received by the device gateway 2 is transmitted to the content management unit 3, and the positioning data Dp is transmitted to the event hub 12 as topic information Dm.

[0079] In step S24, the content management server 1 determines whether the received first video data Dv1 is normal driving video (e.g., time-lapse video) taken during normal vehicle driving, or event video taken at a different time from normal driving video based on a predetermined input.

[0080] If the first video data Dv1 is normal driving video data, the content management server 1 proceeds to step S25. In step S25, the content management server 1 assigns a content ID to the first video data Dv1. The content ID of the first video data Dv1 is sent to the event hub 12 as topic information Dm. The content management server 1 also performs a virus check on the first video data Dv1 to confirm the data's security. Once the data's security is confirmed, the first video data Dv1 is saved to the storage of the content management unit 3. The content management server 1 also executes a process to acquire content information. The acquired content information is saved to the storage of the content management unit 3, linked to the content ID.

[0081] If the first video data Dv1 is event video data, the content management server 1 proceeds to step S26. In step S26, the same processing as in step S25 is performed. In the next step S27, the content management server 1 creates additional content based on the first video data Dv1. The additional content is not particularly limited, but for example, it may be a thumbnail of the first video data Dv1. The path to the additional content and the video data of the additional content are stored in the storage of the content management unit 3. Note that the processing in steps S26 and S27 may be executed in parallel.

[0082] <Screen display overlaying map and driving information> In the drive record display system M described above, when the drive ends, the content management server 1 generates drive information. The drive information is based on the location information contained in the positioning data Dp.

[0083] Drive information includes, for example, the date and time of the drive, the start location (departure point), the end location (destination point), the drive trajectory, the drive duration, the drive distance, and the name of the vehicle Q used for the drive. The start location of the drive is the location indicated by the latitude and longitude when the content management server 1 determines that the drive has started. The end location of the drive is the location indicated by the latitude and longitude when the content management server 1 determines that the drive has ended. The drive trajectory is the path taken by vehicle Q from the start location to the end location. The drive duration is the time from when the drive is determined to have started until when it is determined to have ended. As mentioned above, the start and end of a drive are determined by whether or not positioning data Dp containing the location information of vehicle Q is acquired, so the drive duration can be said to be drive information based on location information. Also, since positioning data Dp includes data that identifies which vehicle is located, the name of vehicle Q can be said to be drive information based on location information. Note that the end location of the drive does not necessarily coincide with the destination of the drive.

[0084] The content management server 1 generates drive record data viewable by the information terminal 8 for each drive. The drive record data includes text data and image data. For example, the content management server 1 generates text data for the date of the drive, the start location of the drive, the end location of the drive, the driving time, and the driving distance. For the driving trajectory, the content management server 1 generates image data, for example, a map display of the actual driving trajectory. It also generates thumbnails from the first video data Dv1 and the second video data Dv2 and pastes them onto the image data of the driving trajectory.

[0085] Users can view drive record data generated by content management server 1 using the mobile application on information terminal 8.

[0086] Figure 9 shows an example of the screen display of the information terminal 8 in this embodiment. In the screen shown in Figure 9(a), the lower layer L1 displays an image showing the driving trajectory 101 of vehicle Q on map data as driving information. In addition to the driving trajectory 101 of vehicle Q, thumbnails 102 of photos and videos taken by the drive recorder 6 are pasted onto the lower layer L1.

[0087] Then, an image showing detailed drive information is displayed on the upper layer L2, superimposed on a portion of the lower layer L1. As shown in Figure 9(b), the area of ​​the upper layer L2 displayed on the information terminal 8 screen increases or decreases according to the user's operation. In Figure 9(b), the upper layer L2 is pulled up to the top of the screen by the user's scroll operation.

[0088] The upper layer L2 displays drive information, including date, origin, destination, driving time, and distance traveled. Additionally, display area L21 shows some of the photos and videos captured by the drive recorder 6. This photo / video display area L21 can be scrolled horizontally by the user. The upper layer L2 also includes an area L22 where the user can write comments.

[0089] When a screen like the one in Figure 9 is displayed on the information terminal 8, the user can view an overall picture, including the drive trajectory 101, on the lower layer L1 within the limited screen space of the information terminal 8, and view detailed information about the drive on the upper layer L2. In addition, the user can pull up the upper layer L2 to the top of the screen and view photos and videos taken during the drive at location L21. Furthermore, the user can write comments at location L22 on the upper layer L2.

[0090] To display a screen like the one in Figure 9, the information terminal 8 and the content management server 1 perform operations as shown in Figure 10, for example.

[0091] First, the user performs an operation on the mobile app of the information terminal 8 to request to view drive details. In response to the user's operation, the mobile app of the information terminal 8 sends a command to the content management server 1 requesting to view drive details (S11).

[0092] In the content management server 1, the BFF 4 receives a "request to view drive details" command from the mobile application of the information terminal 8 via the communication interface 41 (S12). The command processing unit 42 then executes the processing corresponding to the "request to view drive details" command. That is, it notifies the application module 15, which has the "drive details generation function" in the function group 15.

[0093] Upon receiving the notification, the application module 15 generates data representing the driving trajectory of the drive and thumbnails of video data captured during the drive as first drive display data (S13). This first drive display data is the data displayed on the lower layer L1 in the drive details screen of the mobile application of the information terminal 8.

[0094] Furthermore, the application module 15 generates drive information for the drive as second drive display data (S14). This drive information includes data such as date, departure point, destination point, driving time, and driving distance. This second drive display data is the data displayed on the upper layer L2 in the drive details screen of the mobile application of the information terminal 8. The second drive display data may also include some of the photos and videos taken by the drive recorder 6.

[0095] In the BFF4 of the content management server 1, the instruction processing unit 42 receives the generated first and second drive display data from the application module 15. The communication interface 41 transmits the first and second drive display data to the mobile application of the information terminal 8 (S15).

[0096] The mobile application on the information terminal 8 receives the first and second drive display data transmitted from the content management server 1 (S16). It then draws the driving trajectory 101 of the drive indicated by the first drive display data on the map data and pastes a thumbnail 102 of the video data included in the first drive display data onto the lower layer L1 and displays it (S17). The mobile application also displays the drive information included in the second drive display data on the upper layer L2 (S18).

[0097] As described above, according to this embodiment, the content management server 1 generates first drive display data for a drive, which includes data representing the driving trajectory based on positioning data Dp acquired by the drive recorder 6, and a thumbnail generated from video data captured by the drive recorder 6's camera 61. The content management server 1 also generates second drive display data, which includes drive information such as the departure point and destination. The information terminal 8 receives the first and second drive display data from the content management server 1 and, on its display screen, draws the driving trajectory 101 represented by the first drive display data on the map data, pastes a thumbnail 102 onto it, and displays it on the lower layer L1. Then, on the upper layer L2, which is superimposed on a part of the lower layer L1, it displays the drive information indicated by the second drive display data. As a result, on the display screen of the information terminal 8, the overall picture of the drive, including the driving trajectory 101, is displayed on the lower layer L1, and drive information such as the departure point and destination is displayed on the upper layer L2, making the display of drive-related information on the information terminal 8 easy for the user to understand.

[0098] (Other examples) In the example above, the area of ​​the upper layer L2 on the display screen of the information terminal 8 is assumed to be expandable or contractible, but the area of ​​the upper layer L2 may also be fixed. Furthermore, although the area of ​​the upper layer L2 is assumed to expand or contract vertically on the display screen of the information terminal 8, the form of expansion or contraction of the area is not limited to this.

[0099] Furthermore, in addition to photos and videos taken by the drive recorder 6, photos and videos taken by the information terminal 8 may also be displayed at display location L21 in Figure 9(b). These photos and videos may be arranged, for example, in chronological order. When a user selects photos and videos taken by the information terminal 8 and presses the "Add" button, their thumbnails may be displayed on the map data of the lower layer L1. That is, the photos and videos selected by the user are uploaded to the content management server 1. The content management server 1 associates and records the uploaded photos and videos with the drive displayed on the information terminal 8. Then, the photos and videos selected by the user are displayed as thumbnails on the map data of the lower layer L1. At this time, the position where the thumbnails are displayed can be determined, for example, by referring to the metadata of the photos and videos selected by the user and based on information such as the shooting location and time included in this metadata, as well as the driving trajectory data.

[0100] Alternatively, in the screen shown in Figure 9(a), when the user presses the "Add" button on the upper layer L2, the upper layer L2 may automatically pull up to transition to the state shown in Figure 9(b). This allows the user to instantly view the photos and videos that can be added as candidates at the display area L21.

[0101] Alternatively, in the screen shown in Figure 9(a), a "Notes" button may be added to the upper layer L2. When the user presses the "Notes" button, the upper layer L2 may automatically pull up, transitioning to the state shown in Figure 9(b). This would allow the user to immediately enter a comment at location L22.

[0102] <Other Embodiments> The technologies disclosed herein are not limited to the embodiments described above and may be substituted insofar as they do not depart from the spirit of the claims.

[0103] The vehicle may be a bicycle, motorbike, railway car, electric scooter, rickshaw, etc. When the vehicle is a bicycle, motorbike, or railway car, the device capable of acquiring location information is, for example, a drive recorder. When the vehicle is an electric scooter, bicycle, or motorbike, the linking of the mobile terminal and the drive recorder can be done in the same manner as in the embodiments described above. When the vehicle is a railway car, the railway company approves the user registration when linking the mobile terminal and the drive recorder. When the vehicle is a rickshaw, for example, the camera placed on the vehicle is attached to the rickshaw. The device that acquires the vehicle's positioning data is not particularly limited. For example, it may be attached to the rickshaw, or it may also be an information terminal held by the rickshaw driver. In this case, the positioning data acquired by the positioning function of the information terminal held by the rickshaw driver may be used as the vehicle's positioning data. Alternatively, the information terminal held by a passenger riding in the rickshaw may also serve as the device that acquires the vehicle's positioning data.

[0104] Furthermore, as a reference example, the technology disclosed herein can also be applied to aircraft. When applied to aircraft, the device capable of acquiring location information is a flight recorder. In linking a mobile terminal with a drive recorder, user registration approval is performed by the airline. In addition, as another reference example, the technology disclosed herein can be applied to privately operated vessels such as pleasure boats. For vessels, the system of this embodiment configured for vehicles can be easily adapted by replacing maps with nautical charts and the driving trajectory with a navigation trajectory.

[0105] The embodiments described above are merely illustrative and should not be interpreted as limiting the scope of this disclosure. The scope of this disclosure is defined by the claims, and any variations or modifications within the equivalent scope of the claims are all within the scope of this disclosure. [Industrial applicability]

[0106] The technology disclosed herein is useful for drive recording and display systems. [Explanation of symbols]

[0107] 1. Content Management Server 6. Dashcam (device) 8. Information terminals 61 Camera 101 Travel Path 102 Thumbnails Q Vehicle L1 Lower Layer L2 Upper Layer

Claims

1. A drive record display system that displays records of driving using a vehicle, A device mounted on the aforementioned vehicle, having a camera and capable of acquiring the location information of the aforementioned vehicle, The information terminal held by the user of the aforementioned vehicle, The system comprises a server capable of communicating with the aforementioned information terminal, The aforementioned server, The device acquires video data captured by the camera and location information acquired by the device. Regarding the drive, data representing the vehicle's driving trajectory is generated based on the location information, and a thumbnail is generated from the video data. The generated data representing the driving trajectory and the thumbnail are then generated as first drive display data for that drive. Drive information, including distance traveled, travel time, departure point, and arrival point, is generated as second drive display data. The first and second drive display data are transmitted to the information terminal. The aforementioned information terminal is The server receives the first and second drive display data, On the display screen, the driving trajectory represented by the first drive display data is drawn on the map data, the thumbnail is pasted onto it, and it is displayed on the lower layer. The upper layer, which is superimposed on a portion of the lower layer, displays the drive information indicated by the second drive display data. Drive record display system.

2. In the drive recording display system according to claim 1, The aforementioned information terminal is configured to allow the user to increase or decrease the area in which the upper layer is displayed through user operation. Drive record display system.

3. In the drive recording display system according to claim 1, The aforementioned information terminal is configured in the upper layer to allow users to input comments. Drive record display system.

4. A drive record display method that displays a record of driving using a vehicle, The vehicle is equipped with a camera and a device capable of acquiring the vehicle's location information. The user of the aforementioned vehicle has an information terminal, A server capable of communicating with the aforementioned information terminal, The device acquires video data captured by the camera and location information acquired by the device. Regarding the drive, data representing the vehicle's driving trajectory is generated based on the location information, and a thumbnail is generated from the video data. The generated data representing the driving trajectory and the thumbnail are then generated as first drive display data for that drive. Drive information, including distance traveled, travel time, departure point, and arrival point, is generated as second drive display data. The first and second drive display data are transmitted to the information terminal. The aforementioned information terminal, The server receives the first and second drive display data, On the display screen, the driving trajectory represented by the first drive display data is drawn on the map data, the thumbnail is pasted onto it, and it is displayed on the lower layer. The upper layer, which is superimposed on a portion of the lower layer, displays the drive information indicated by the second drive display data. How to display driving records.

Images