An electronic device, a server that converts a still image into a moving image, a program, etc.

Abstract

To provide an electronic apparatus which transmits video information and has realizability and effectivity for creating a moving image, and a server, a program, etc.SOLUTION: The electronic apparatus comprises: a communication processing unit; and a control unit which processes picture information captured with a camera and transmits a plurality of still pictures via the communication processing unit. The control unit controls the transmission of the still pictures based on a traveling speed of a vehicle and / or a transmission state of the still pictures.SELECTED DRAWING: Figure 11

Description

Technical Field

[0001] The present invention relates to an electronic device, a server that converts a still image transmitted from the electronic device into a moving image, a program that enables a computer to realize the functions of the electronic device and the server, and the like.

Background Art

[0002] In recent years, in fields such as IoT (Internet of Things), as a communication function, LPWA (Low Power Wide Area) or LPWAN (Low Power Wide Area Network), which is called low-power wide-area (hereinafter referred to as LPWA), has been used. LPWA enables relatively long-distance communication with relatively small power by using relatively low frequencies.

[0003] As the background art in this technical field, there is Japanese Unexamined Patent Application Publication No. 2020-088803 (Patent Document 1). This publication describes that "a sensor, a transmitter of a low-power wide-area communication method for transmitting sensor data, a receiver of a low-power wide-area communication method for receiving sensor data, and a remote monitoring and control device for receiving sensor data from the receiver are provided, and the remote monitoring and control device associates sensor data with sensor data acquisition information and stores them in a sensor data storage unit, determines a specific state based on the sensor data stored in the sensor data storage unit, and acquires a video file corresponding to the sensor data acquisition location indicated by the sensor data acquisition information associated with the sensor data determined to be in the specific state from a video file storage device." (See the abstract).

[0004] Also, as the background art of this technical field, there is Japanese Patent Application Laid-Open No. 2019-129328 (Patent Document 2). This publication describes that "the high-definition video generation device 20 includes an original video reception unit 21 that receives an original video 1 which is a high-quality video, a low-bitrate video reception unit 24 that receives a low-bitrate video 2, a learning model generation unit 22 that generates a learning model M using the original video 1 received by the original video reception unit 21 as teacher data, and a high-definition video generation unit 25 that generates a high-definition video 3 from the low-bitrate video 2 received by the low-bitrate video reception unit 24 using the learning model M corresponding to the low-bitrate video 2 among the learning models M generated by the learning model generation unit 22." (See the abstract).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] The present invention provides a practical and effective electronic device, a server, a program, etc. for transmitting video information and creating videos.

[0007] Note that the applicant also intends to obtain rights through divisional applications, amendments, etc. for configurations aimed at achieving the effects produced by the components of the configurations disclosed in this specification, drawings, etc. For example, the problems obtained by reinterpreting the parts described as "can" or "is possible" in this specification as "is a problem" are disclosed in this specification. The problems are described as independent ones, and the applicant also intends to obtain rights through divisional applications, amendments, etc. alone for the configurations for solving each problem. Even if the problems are implicitly understood from the description of the specification, the applicant intends to use part of the configuration described in this specification as the scope of claims through amendment or divisional application. In addition, the applicant also discloses the configurations for solving the problems obtained by combining these independent problems and intends to obtain rights.

Means for Solving the Problems

[0008] To solve the above problems, for example, the configuration described in the scope of claims is adopted. This application includes a plurality of means for solving the above problems. If an example is given, it is an electronic device including a communication processing unit and a control unit that processes the image information captured by a camera and transmits a plurality of still images via the communication processing unit, wherein the control unit controls the transmission of the plurality of still images based on at least one of the running state of the vehicle and the transmission state of the still images.

Effects of the Invention

[0009] The present invention provides a practical and effective electronic device, a server, a program, etc. for transmitting video information and creating moving images.

[0010] Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments. Note that the effects of the invention of the present application are not limited to this, and the effects achieved by the components of the configuration disclosed in this specification and the drawings are also disclosed. The applicant also has the intention of obtaining rights through divisional applications, amendments, etc. for the configurations that achieve such effects. For example, in this specification, the parts described as "~ can be" or "~ is possible" are descriptions that clearly indicate the achieved effects. Even if there is no description such as "~ can be" or "~ is possible", there are parts that indicate the effects. Further, even without such descriptions, there are effects grasped by the configuration.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. It should be noted that the following embodiments are one of the embodiments providing the present invention, and the content of the present invention of the present application should not be construed as being limited based on the following description. Figure 1 is an example of the configuration of the overall drive recorder (DVR) management system 1. The DVR management system 1 includes a drive recorder 101, a management server 102, a user terminal 103, a data center 104, and a management terminal 105, each of which is connected via a network. Note that the network can be either wired or wireless, and each terminal can transmit and receive information via the network. The network is not limited to the Internet and may be a combination of networks with different protocols.

[0013] The drive recorder 101 is a device, for example, installed in an automobile or the like, and records the video during driving. The drive recorder 101 is an example of an in-vehicle device. However, the installation target is not limited to automobiles, and any moving body on which the drive recorder 101 can be installed is acceptable. For example, automobiles, buses, trucks, motorcycles, etc. can be targets. Also, for example, vehicles of transportation means such as trains, monorails, and linear motor cars can also be installation targets. The management server 102 is a server that stores and manages various information such as video information and status information acquired by the drive recorder 101. Each information acquired by the drive recorder 101 may be directly transmitted from the drive recorder 101 to the management server 102, or may be transmitted to the management server 102 via the data center 104.

[0014] The user terminal 103 is a terminal used by a user who drives an automobile, for example. The user terminal 103 is connected to the drive recorder 101 via an in-vehicle network such as a mobile data communication network or the Wi-Fi (registered trademark) of an automobile, and can operate the drive recorder 101. Also, the user terminal 103 is connected to the management server 102 via these networks and can display various information stored in the management server 102.

[0015] The data center 104 receives various information acquired by the drive recorder 101 and transmits it to the management server 102. For example, the data center 104 is connected to the drive recorder 101 via a LPWA (Low Power Wide Area) network, receives various information transmitted from the drive recorder 101 via the LPWA network, and transmits the received various information to the management server 102 via the Internet. Note that LPWA wireless communication can perform low-power and long-distance communication. On the other hand, there may be an issue with the restriction on the amount of data that can be transmitted per unit time. For example, if a large amount of data is to be transmitted, the communication delay may be large. As LPWA, for example, at least one of "ELTRES", "LoRa", "LoRaWAN", "LTE-MTC", "NB-IoT", "NB-Fi Protocol", "RPMA", "GreenOFDM", "DASH7", "RPMA", "Wi-SUN", "EnOcean Long Range", "Weightless-P", "SIGFOX", "LTE Cat.0", "LTE Cat.M1", etc. may be used. The management terminal 105 is a terminal that operates and manages the management server 102.

[0016] Each terminal and the management server 102 of the DVR management system 1 may be, for example, a mobile terminal (portable terminal) such as a smartphone, a tablet, a mobile phone, a personal digital assistant (PDA), etc., or a wearable terminal such as a glasses type, a wristwatch type, a clothing type, etc. Further, it may be a stationary or portable computer, or a server arranged on a cloud or a network. Further, as a function, it may be a VR (Virtual Reality) terminal, an AR (Augmented Reality) terminal, or an MR (Mixed Reality) terminal. Alternatively, it may be a combination of a plurality of these terminals. For example, a combination of one smartphone and one wearable terminal can function logically as one terminal. Also, it may be an information processing terminal other than these.

[0017] Each terminal and management server 102 of the DVR management system 1 includes a processor that executes an operating system, applications, programs, etc., a main memory device such as a RAM (Random Access Memory), an auxiliary storage device such as an IC card, a hard disk drive, an SSD (Solid State Drive), a flash memory, a communication control unit such as a network card, a wireless communication module, a mobile communication module, an input device such as a touch panel, a keyboard, a mouse, an input by voice input, a movement detection by imaging of a camera unit, and an output device such as a monitor or a display. Note that the output device may be a device or a terminal that transmits information for output to an external monitor, display, printer, device, etc.

[0018] The main memory device stores various programs, applications, etc. (referred to as modules). By the processor executing these programs and applications, each functional element of the entire system is realized. Note that these modules may be implemented in hardware by integration or the like. Also, each module may be an independent program or application, or may be implemented in the form of some sub-programs, functions, etc. in one integrated program or application.

[0019] In this specification, each module is described as the subject that performs processing. However, actually, the processor that processes various programs, applications, etc. (modules) executes the processing. The auxiliary storage device stores various databases (DBs). A "database" is a functional element (storage unit) that stores a data set so as to be able to respond to any data operation (for example, extraction, addition, deletion, overwrite, etc.) from a processor or an external computer. The implementation method of the database is not limited. For example, it may be a database management system, spreadsheet software, or a text file such as XML or JSON.

[0020] Figure 2 is an example of the connection configuration of the management server 102. In this connection configuration example, the drive recorder 101 transmits various types of information to the management server 102 via the data center 104. The drive recorder 101 transmits various types of information of the drive recorder 101 to the data center 104 through a wireless communication network such as an LPWA network. The customer-side management server 202 receives and stores various types of information from the data center 104 via the Internet using a protocol such as MQTT (Message Queue Telemetry Transport). The management terminal 105 and the user terminal 103 access the customer-side management server 202 via the Internet or a mobile communication network, and receive and display various types of information.

[0021] Also, the same function can be performed via the service provider-side management server 201 of the service provider who manages the DVR management system 1. Various types of information of the drive recorder 101 once stored in the service provider-side management server 201 are transmitted to the management terminal 105 and the user terminal 103 via the customer-side management server 203 or directly. The various types of information transmitted and received may be encoded and decoded by each server. Note that the configuration may be such that the drive recorder 101 directly transmits various types of information to the management server 102 instead of via the data center 104.

[0022] Figure 3 is an example of the hardware configuration of the drive recorder 101. The camera 304 captures images in the front, rear, or 360 degrees, and generates video information such as videos and still images. Note that by installing an AI processing function that performs pre-processing on the image sensor of the camera 304, the following outputs can be performed together with or instead of the generation of normal captured images. The AI processing function has a function of performing image recognition based on the video captured by the camera 304. · Output the object from the image as metadata. ·Outputs an image in a format such as YUV or RGB by an ISP (Image Signal Processor). ·Outputs an image with only a specific area cut out.

[0023] By performing preprocessing on the image sensor side in this way, it becomes possible to reduce the amount of data and perform real-time tracking of an object by high-speed AI processing. In particular, there is an advantage in that the amount of data (i.e., the communication volume) from the drive recorder 101 to the management server 102 can be reduced. This is because, even when transmitting data via an LPWA network as in the present embodiment, it is expected that data regarding the AI processing function can be transmitted, for example, with low latency.

[0024] The acceleration sensor 305 detects acceleration and generates acceleration information. For example, when a rapid change in acceleration occurs in response to an impact, sudden steering, sudden braking, etc., the DVR controller 303 detects the occurrence of an event. By analyzing the change in this acceleration information, it is possible to infer the situation of an accident or the like that has occurred in an automobile or the like. The SD card 306 is a storage device that records video information and various types of information. Note that it is not limited to an SD card, and any configuration that can store data may be used, but a flash memory with high resistance to vibration and shock is preferable for in-vehicle use.

[0025] The DVR controller 303 is connected to the camera 304, the acceleration sensor 305, the SD card 306, and other sensors (not shown), and controls these. Further, based on the information received from these elements, the DVR controller 303 generates status information indicating the state of the DVR controller 303. Also, the DVR controller 303 transmits the generated status information to the microcomputer 301. The microcomputer 301 performs the following various processes. ·Controls the communication processing unit 302. ·Notifies the DVR controller 303 of the position information acquired by the communication processing unit 302. ·Convert the status information obtained from the DVR controller 303 and the position information obtained from the communication processing unit 302 into, for example, 16-byte data. ·Perform sleep control when the secondary battery is driving. ·Execute the test mode during manufacturing and testing. The microcomputer 301 is programmable, and a program for executing various processes as described above is stored in the microcomputer 301. By executing this program with the processing unit of the microcomputer 301, the above-described various processes are realized. In this embodiment, the processing unit of this microcomputer 301 or the microcomputer 301 itself may also be referred to as a control unit.

[0026] The communication processing unit 302 transmits the information of the drive recorder 101 to the management server 102 directly or via the data center 104 via the wireless communication antenna 308. The communication processing unit 302 performs wireless communication using the LPWA method, which is an example of narrowband communication. Hereinafter, although described as the LPWA method, it may be replaced with other narrowband communication methods or communication methods other than narrowband communication. The communication processing unit 302 also has a position information acquisition unit implemented by an integrated circuit or the like, and acquires position information based on signals received from a GNSS antenna 307 corresponding to a GNSS (Global Navigation Satellite System) such as GPS (Global Positioning System). The position information acquisition unit can also acquire time information, moving speed information, and moving direction information.

[0027] Note that the communication processing unit 302 may not have a configuration with a position information acquisition unit, and the GNSS antenna 307 may be connected to the microcomputer 301, and these may have a configuration with a position information acquisition unit. Also, the position information acquisition unit may exist independently and be connected to the microcomputer 301 or the communication processing unit 302. Also, the time information is acquired by the position information acquisition unit, but the microcomputer 301 may have a configuration for acquiring position information.

[0028] The drive recorder 101 has a power supply 311 connected to the vehicle power supply 310. The power supply 311 supplies the power received (or sometimes referred to as power reception) from the vehicle power supply 310 to the DVR controller 303 and also charges the secondary battery 312. The secondary battery 312 is a rechargeable battery such as a lithium-ion battery, nickel-cadmium battery, nickel-metal hydride battery, lead-acid battery, etc. However, a non-rechargeable primary battery may also be used. When the accessory power supply (ACC power supply) is turned off, or when the power supply from the vehicle power supply 310 to the power supply 311 of the drive recorder 101 is interrupted due to a failure such as a disconnection or disengagement of wiring caused by an accident, or when the power supply 311 fails and power supply is impossible, the secondary battery transmits (or sometimes referred to as power transmission or power supply) power to the microcomputer 301, communication processing unit 302, etc., and maintains at least some functions of the drive recorder 101.

[0029] The supply voltage from the vehicle power supply 310 of an automobile is generally 12V. The power supplied from the power supply 311 or the secondary battery 312 is stepped down to a voltage of, for example, 2V via the DC / DC converter 313 and supplied to the microcomputer 301 and the communication processing unit 302 to drive them. Note that the DVR controller 303, microcomputer 301, and communication processing unit 302 can each be implemented by an integrated circuit (chip) such as an SoC (System on a chip), and a configuration in which a plurality of these are integrated into one chip may also be used. In particular, in this embodiment, since the microcomputer 301 and the communication processing unit 302 operate in cooperation, they may be implemented together on one chip.

[0030] In this embodiment, the microcomputer 301 and the communication processing unit 302 are configured such that the power supply from the DVR controller 303 is disconnected, and position information and various information can be transmitted to the management server 102 only by power supply from the secondary battery to the microcomputer 301 and the communication processing unit 302, without starting the DVR controller 303, and power consumption can be suppressed. In particular, the drive recorder 101 can perform low-power and long-distance communication by performing LPWA-based wireless communication. Therefore, while maximizing the duration of the secondary battery in the drive recorder 101, the status information of the drive recorder 101 can be transmitted to the remote management server 102. Also, since the drive recorder 101 transmits status information to the management server 102 using LPWA-based communication, it can be said that it is desirable to perform communication at high frequencies or short intervals while suppressing the data volume. Therefore, the status information transmitted from the drive recorder 101 to the management server 102 is also based on the findings of the inventors of the present application, taking into account the characteristics of LPWA-based wireless communication and circumstances such as the priority of the information to be transmitted to the management server 102.

[0031] In addition, by configuring the communication processing unit 302 to include a position information acquisition unit and a GNSS antenna 307, it is possible to acquire position information and speed information only by driving the low-power communication processing unit 302 without starting the DVR controller 303. Also, by suppressing the drive voltage of the microcomputer 301 and the communication processing unit 302 to a low voltage such as 2V, for example, compared to the drive voltage of the drive recorder 101, which is 12V, for example, the power consumption used for generating and transmitting the DVR status information 600 can be kept low.

[0032] By mounting a small-capacity rechargeable secondary battery as in this embodiment and notifying the management server 102 of the status of the drive recorder 101 when the secondary battery is driving, even when the user intentionally disconnects the power of the drive recorder 101 and drives, a notification can be sent to the administrator. Also, even when the main power supply does not turn on due to a failure, a failure notification can be sent. Also, theft can be tracked.

[0033] Figure 4 is an example of the hardware configuration of the management server 102. The management server 102 is configured by a server arranged, for example, on the cloud. The main memory device 401 stores programs and applications such as the DVR management module 410 and the video information management module 411. By executing these programs and applications, the processor 403 realizes each functional element of the management server 102.

[0034] The DVR management module 410 acquires DVR status information indicating the status of the drive recorder 101 from the drive recorder 101, decodes it, stores it in the DVR status information 600 of the auxiliary storage device 402, and manages it. Also, the DVR management module 410 transmits some or all of the information obtained by decoding the DVR status information 600 to the user terminal 103 or the management terminal 105 by push distribution or in response to requests from the user terminal 103 or the management terminal 105.

[0035] The video information management module 411 acquires the video information stored in the drive recorder 101 from the drive recorder 101, stores it in the video information 420 of the auxiliary storage device 402, and manages it. Also, the video information management module 411 transmits the video information 420 to these terminals by push distribution or in response to requests from the user terminal 103 or the management terminal 105. The moving image creation module 412 performs a process of integrating a plurality of still images received from the drive recorder 101 or the like to create a moving image. The auxiliary storage device 402 stores the DVR status information 600, the video information 420, and the like.

[0036] FIG. 5 is an example of the hardware configuration of the user terminal 103. The user terminal 103 is composed of terminals such as smartphones, tablets, notebook PCs, and desktop PCs. The main memory device 501 stores programs and applications such as the management server cooperation module 510 and the DVR cooperation module 511. By executing these programs and applications, the processor 503 realizes each functional element of the user terminal 103.

[0037] The management server cooperation module 510 cooperates with the management server 102, acquires the DVR status information 600 and video information 420 stored in the management server 102, and displays this information on an output device 505 such as a display. In addition, the management server cooperation module 510 can cooperate with the management server 102 to perform various settings of the management server 102. When the drive recorder 101 and the user terminal 103 are connected by an in-vehicle network such as Wi-Fi (registered trademark), the DVR cooperation module 511 operates the drive recorder 101 and performs settings of the drive recorder 101.

[0038] Note that the management terminal 105 can have the same configuration as the user terminal 103. The DVR cooperation module and the management server cooperation module stored in the main storage device can operate the drive recorder 101 and the management server 102 and perform their settings.

[0039] Figure 6 is an example of the data configuration of the DVR status information 600. The DVR status information 600 is a diagram for explaining the data configuration when the microcomputer 301 of the drive recorder 101 transmits the status information acquired from the DVR controller 303 and the position information acquired from the position information acquisition unit to the management server 102. The DVR status information 600 may also be referred to as status information regarding the drive recorder 101.

[0040] In this embodiment, the microcomputer 301 generates 16-byte data of the DVR status information 600, the communication processing unit 302 stores it in the payload, and transmits it to the management server 102 based on the LPWA protocol. Note that the protocol and the like used for transmission are not limited to LPWA, and other communication methods and protocols may also be used.

[0041] Item 610 indicates the item of DVR status information 600 to be transmitted. Type1 (620), Type2 (630), and Type3 (640) indicate three types of transmitted data contents, and the numbers described in each table represent the data bit lengths. When the numbers of the data bit lengths of each Type are totaled, the data length becomes 128 bits, that is, 16 bytes. In addition, as Type4 (not shown), there is a free format preliminary data configuration other than those described in item 610.

[0042] In this embodiment, when the vehicle power supply 310 is supplied, the communication processing unit 302 transmits the DVR status information 600 to the management server 102 at a 1-minute cycle. The microcomputer 301 acquires position information, speed information, information indicating the recording state, etc. at three different times during this one cycle, stores the position information, etc. for these three time points in the 16-byte payload, and can send the information to the management server 102 in one transmission process. Type1 is a format for transmitting position information, etc. at one time point. Type2 is a format for collectively transmitting position information, etc. at two time points. Type3 is a format for collectively transmitting information, etc. at three time points.

[0043] Referring to FIGS. 1 and 3 again, one embodiment will be described in detail. FIG. 1 shows a drive recorder 101 as a specific example of the embodiment. However, this embodiment is not limited to the drive recorder 101. For example, this embodiment can be implemented as other in-vehicle devices such as a detector (radar). Broadly speaking, this embodiment can relate to an electronic device that acquires video information. Hereinafter, as an example of the electronic device according to this embodiment, the in-vehicle device drive recorder 101 will be described.

[0044] The drive recorder 101 uses a camera (imaging unit) 304 to acquire video information of the surrounding scenery (e.g., the front, rear, side, diagonally forward, diagonally backward of the vehicle, or any angular range up to 360° around the vehicle) while the vehicle is in motion or stationary. The video information or image information is captured in a video format or a plurality of still image formats.

[0045] The drive recorder 101 is an example of an electronic device incorporating the camera 304. For example, the drive recorder 101 can incorporate a camera so as to capture the surrounding scenery and record (save) the video information thereof. On the other hand, for example, when the electronic device is implemented as a detector (radar), an externally attached camera electronic device, or a car navigation system, it can communicate or connect to an external camera so as to acquire video information captured by other in-vehicle devices.

[0046] The camera 304 generates a video format file and acquires a still image captured from the file. Alternatively, the camera 304 may directly capture a still image (photo) from the beginning and acquire the still image. The still image is, for example, in the JPEG format, but other formats are also possible. The video is, for example, in the MPEG format (MPEG2, MPEG4, etc.), but other formats are also possible. Since a video can be understood as a series of still images arranged in time series, hereinafter, the still image will be described as an example of the image information captured by the camera 304. Note that the camera 304 is not limited by differences in the size of the camera (changes such as small or large), differences in the color of the still images captured by the camera (changes such as black and white or full color), or differences in the imaging method of the camera (changes such as normal imaging or infrared imaging).

[0047] The drive recorder 101 preferably uses an LPWA communication processing unit 302 to transmit the data of the captured still images to an external management server 102. By using LPWA, relatively low power enables relatively long-distance communication. On the other hand, LPWA has the aspect that the data transmission speed is slow and it is not suitable for exchanging a large amount of data. Therefore, while taking advantage of the strengths of LPWA, the drive recorder 101 performs special control to suppress its weaknesses. This control can be performed by, for example, the microcomputer 301 and / or the DVR controller 303, etc.

[0048] An electronic device (for example, the drive recorder 101) includes at least a communication processing unit 302 and a control unit (such as the microcomputer 301 and / or the DVR controller 303, etc.) that processes the image information captured by the camera 304 and transmits a plurality of still images via the communication processing unit 302. Hereinafter, the microcomputer 301 and / or the DVR controller 303, etc. are collectively referred to simply as the control unit 350. The still images are transmitted to an external management server 102. The control on the management server 102 side will be described later.

[0049] "Test of Video Transmission (Test 1)" In order to grasp the characteristics of transmitting data in the LPWA method, the applicant of the present application conducted the following Tests 1 to 4. First, the applicant conducted a test in which the drive recorder 101 transmitted the video stream captured by the camera 304 to an external management server 102 via the communication processing unit 302. As can be understood by those skilled in the art, in order to transmit the video stream in real time by the communication processing unit 302, it is necessary to stably transmit the video stream. Generally, in order to transmit a video stream in real time, it is necessary to transmit a video stream with a bandwidth of 300 Kbps or more (ideally about 1 Mbps).

[0050] Therefore, the applicant conducted a test in which a vehicle in actual running transmitted a video stream with a bandwidth of 300 Kbps captured by the drive recorder 101 via the communication processing unit 302 corresponding to LPWA Cat.M1. As a result, on the receiving side (the management server 102 in this embodiment), it was obtained that the playback screen deteriorated and disconnections occurred frequently.

[0051] FIG. 7 is a diagram schematically explaining the content of Test 1 above. Taking the horizontal axis as the time series, as shown within the ranges 710 and 720 enclosed by the square frames, a video stream with a bandwidth of 300 Kbps was captured and an attempt was made to transmit it. As illustrated in FIG. 7, the transmission speed changes under the influence of changes in the surrounding environment and the like. For the first video stream exemplified by reference numeral 710, when the transmission speed deteriorated, the file transfer failed and the transmission was disconnected (see reference numeral 712). When the connection was made again (see reference numeral 722), the second video stream 720, although it did not result in a disconnection of the transmission due to the deterioration of the transmission speed, caused image distortion (see reference numeral 724).

[0052] As described above, in order for the drive recorder 101 to transmit the video stream captured by the camera 304 in real time, a stable communication speed environment is essential. However, it has been found that there are practical problems in directly transmitting the video stream captured by the camera 304 according to the LPWA Cat.M1 specifications as of January 2021.

[0053] In relation to this, Patent Document 1 described above states that, "In the LPWA communication method used by the LPWA communication system 1, since sufficient communication capacity cannot be ensured, there is a problem that video data with a large data volume such as high-definition still images and videos cannot be transmitted in real time (see paragraph 0023).". That is, in Patent Document 1, since the video information captured by the in-vehicle camera cannot be sufficiently transmitted in the LPWA format, it is described that other means are used.

[0054] Also, Patent Document 2 described above states that a low-bitrate video is uploaded to the cloud in real time using narrowband communication such as LPWA. However, in Patent Document 2, it is necessary to first convert the original video captured by the in-vehicle camera into a low-bitrate video and then further convert this low-bitrate video into a high-definition video. The drive recorder is intended to provide evidentiary images in the event of a traffic accident. Therefore, it is not preferable that the original captured video and the finally obtained high-definition video cannot be strictly one-to-one corresponding.

[0055] "Test of Still Image Transmission (Test 2)" Next, the applicant conducted a test in which the drive recorder 101 transmitted a plurality of still images captured by the camera 304 to an external management server 102 via the communication processing unit 302. Specifically, the applicant conducted a test of transmitting a still image (JPEG file) of about 60 Kbytes captured by the drive recorder 101 in a vehicle during actual driving via Cat.M1 connected HTTP (TCP / IP). As a result, on the receiving side (management server 102), the transmission was not interrupted (or the frequency was extremely low), and a satisfactory result was obtained in terms of practical use.

[0056] FIG. 8 is a diagram schematically explaining the content of the above Test 2. Taking the horizontal axis as a time series, a plurality of still images 810 to 860 of about 60 Kbytes were captured and their transmission was attempted. When the transmission conditions were good, it was confirmed that transmission could be completed in about 3.5 seconds (see reference numeral 810). When the transmission conditions deteriorated, a delay in the transmission speed of about 8 seconds was observed (see reference numeral 830), but even in that case, the transmission was not interrupted. Thus, when actually driving and transmitting a plurality of still images, although the time until transmission was completed changed according to the line state, it was confirmed that the images could be surely transmitted to the receiving side (management server 102) without causing file damage or the like.

[0057] From the results of these two Tests 1 and 2, the effectiveness of transmitting a plurality of still images periodically captured by the camera 304 to the external management server 102 and then converting those still images into a moving image on the management server 102 side was confirmed, rather than directly transmitting the video stream captured by the camera 304.

[0058] The management server 102 arranges a plurality of still images in chronological order and converts them into a video. Since there are gaps between each still image (for example, from about 1 second to several seconds, for example, about 5 seconds), the smoothness as a video is lost. However, since the role required of the drive recorder 101 is to secure recorded images at the time of a traffic accident or to confirm the actual driving route, there is no major problem. Rather, by collecting non-sequential still images in chronological order and converting them into a video, the data capacity can be reduced as a whole.

[0059] "Test of Transmission of Still Images and Vehicle Information (Test 3)" Furthermore, the applicant conducted tests on the transmission of still images under different conditions. In this case, a test was conducted to separately transmit still images (data with a relatively large size) captured by the drive recorder 101 and vehicle information such as position information (data with a relatively small size) from a vehicle during actual driving to an external management server 102. The information transmitted from the drive recorder 101 to the external management server 102 includes not only still images captured by the camera 304, but also other vehicle information that requires real-time performance, such as G-sensor events detected by an acceleration sensor 305, vehicle position information received from a GNSS antenna 307, etc., and vehicle information such as the driving speed of the vehicle (here, character data) (see FIGS. 3 and 6).

[0060] Under a transmission situation different from Test 2 above, a Test 3 was conducted to transmit a plurality of still images (here, each still image has a size of about 50 Kbytes) captured by the camera 304 to an external management server 102 in LPWA format while the vehicle was in motion. As a result, it was confirmed that the time until the transmission of each still image was completed could vary from 3 seconds to more than ten seconds depending on the line state, and it was not always possible to transmit stably at a cycle of several seconds. In particular, it was confirmed that when the line state was poor, the transmission time became longer and there was a risk of a decrease in real-time performance.

[0061] On the other hand, vehicle information such as position information is data with a relatively small size compared to the case of still images, so the transmission speed is faster. In Test 3, in the case of vehicle information having a small data size composed of characters and numerical values, it was confirmed that the transmission was completed within several hundred milliseconds under the same conditions, and the real-time performance was good. Thus, in the case of vehicle information, since the data size is relatively small, it was confirmed that the probability of transmission failure is lower compared to still images.

[0062] Therefore, from the results of Tests 1 to 3 actually conducted, the effectiveness of separately transmitting the still images captured by the camera 304 and the vehicle information to the external management server 102 was confirmed. By separately transmitting the still images and the vehicle information, the individual data capacities transmitted by the communication processing unit 302 can be made as small as possible. Also, by focusing on the transmission speed of the vehicle information transmitted alone, it is possible to determine whether the transmission status of the communication processing unit 302 is good. For this purpose, it is possible to use a ping (pin or ping) for confirming the reachability of data, which is even smaller in size.

[0063] "Test of Reception Strength, Transmission Speed, and Reception Speed (Test 4)" Furthermore, the applicant conducted a test to actually measure the changes in the traveling speed, reception strength (RSRP: Reference Signal Receive Power), transmission speed, and reception speed of a vehicle during actual driving. FIG. 9 shows the changes in the communication speed (iperf every 30 seconds) corresponding to CAT.M1 during actual driving, which was implemented by the applicant.

[0064] The traveling speed of the vehicle during the test is shown at the bottom of FIG. 9 (see reference numeral 910). The traveling speed of the vehicle changes according to the surrounding conditions and the like, and accordingly, the reception strength (RSRP) shown second from the bottom also changes (see reference numeral 920). It was found that when the reception strength (RSRP) decreases, the reception speed (see reference numeral 930) shown second from the top and the transmission speed (see reference numeral 940) shown at the top also decrease. On the one hand, when the vehicle stops (travel speed is 0 km / h), it was found that the received signal strength (RSRP) stabilizes, and the transmission speed and reception speed also stabilize. Note that since the graph is the average of the transmission and reception speeds measured by the communication performance measurement tool Iperf for about 15 seconds, there may be a possibility that the speed is momentarily close to zero in the vicinity of 100 kbps or less.

[0065] Based on the results of the above Tests 1 to 4, in this embodiment, while taking advantage of the advantages of LPWA used in the communication processing unit 302, special measures are taken to suppress its disadvantages. That is, the control unit 350 of the drive recorder 101 does not transmit all the still images captured by the camera 304, but controls the transmission of a plurality of still images based on at least one of (a) the driving state of the vehicle and (b) the transmission state of the still images (here, selectively transmits a part of the plurality of still images).

[0066] Preferably, since vehicle information such as position information is relatively small in size, it may all be transmitted from the drive recorder 101 to the management server 102. Thereby, the time-series changes in the position and speed of the vehicle can be comprehensively recorded. On the other hand, for still images, depending on the driving state of the vehicle, etc., not all of them are necessarily required. The role required of the drive recorder 101 is to ensure recorded images in the event of a traffic accident, confirm the actual driving route, etc. Therefore, during normal driving when no traffic accident has occurred, etc., some of the still images can be thinned out.

[0067] "Control based on (a) the driving state of the vehicle" Preferably, when the control unit 350 of the drive recorder 101 detects that at least one of the vehicle speed being within a predetermined speed range or the change in the vehicle position being within a predetermined distance range, it selects a predetermined still image from among the plurality of captured still images and transmits it.

[0068] For example, in the case of a vehicle traveling at a relatively high legal speed (for example, the traveling speed of the vehicle is 60 km / h or more), it can be predicted that each still image captured by the camera 304 captures a different scenery. In this case, if a part of the still images is thinned out, the traveling route suddenly changes, which is likely to give the viewer a sense of discomfort. On the other hand, when the vehicle temporarily stops during traffic congestion or the like (for example, the traveling speed of the vehicle is 0 km / h or the change in the position of the vehicle is 0 km), each still image captured by the camera 304 may capture substantially the same scenery. In this case, even if a part of the plurality of still images is selectively excluded and the remaining still images are made into a moving image, the continuity of the scenery is relatively not lost, and it is less likely to give the viewer a sense of discomfort.

[0069] Generally, when the image quality is low, the reliability of each still image decreases, so there is a risk of reducing the number of still images recorded as evidence. However, as illustrated in FIG. 9, when the vehicle stops (here, the traveling speed is 0 km / h), the received signal strength (RSRP) is stable, and the transmission speed and the reception speed are stable. In this state, the image quality of the captured still image is relatively good (Test 4). Therefore, it can also be considered that the risk of thinning out a part of the plurality of still images captured by the vehicle during parking is smaller. This is because the necessity of preparing a spare is further reduced since the reliability of each still image is high.

[0070] Therefore, in one of the present embodiments, when the control unit 350 of the drive recorder 101 determines that the traveling state of the vehicle is in low-speed traveling or stopped, only a part of the plurality of images captured by the camera 304 is selected and transmitted to the external management server 102. The control unit 350 may determine that the traveling state of the vehicle is in low-speed traveling or stopped when the speed of the vehicle is within a predetermined speed range or within a predetermined movement range. The fact that the traveling state of the vehicle is in low-speed traveling may be, for example, that the speed of the vehicle is 2 km / h or less. The fact that the vehicle is stopped may be, for example, that the speed of the vehicle is 0 km / h. Hereinafter, this embodiment will be described with reference to FIGS. 10 to 15.

[0071] "Test of Still Image Transmission (Test 5)" From the above perspective, the applicant conducted a test of selectively transmitting a plurality of still images captured by the camera 304 in a vehicle during actual running. Referring to FIG. 10, the results of the above Test 5 are shown in a table.

[0072] In the leftmost column of FIG. 10, the order of each still image captured over time is shown. In this example, a total of 24 still images are captured. These 24 still images are specifically shown in FIGS. 11 to 13. As can be understood from FIGS. 11 to 13, this embodiment can be suitably applied to the confirmation of the running route of the vehicle and the like. In the second column from the left in FIG. 10, the file names of the still images are shown. For example, in the case of No. 1, a still image was captured at the time of 02:41:55 on December 17, 2020. By the control unit 350, in accordance with the shooting time, the file name is automatically selected as "20201217-024155". In this way, preferably, each still image is sent to the management server 102 with the shooting date and time as the file name. Therefore, the management server 102 can smoothly perform data management of each still image.

[0073] In the third and fourth columns from the left in FIG. 10, the latitude and longitude of the vehicle at the time when each still image was captured are shown. This information may be obtained, for example, based on a signal received from the GNSS antenna 307. In the fifth column from the left in FIG. 10, the speed of the vehicle at the time when each still image was captured is shown. For example, in the first example of No. 1, it is displayed as "21", which means 21 km / h (kilometers per hour). Note that the unit may also be meters per second in SI units. Alternatively, units such as miles per hour or feet per second may be used.

[0074] In the sixth column from the left in FIG. 10, the still image time difference is shown as two adjacent time intervals of each still image. In the example shown in FIG. 10, basically, each still image is taken at an interval of 5 seconds from each other. This interval is not necessarily always constant and may be changed to a longer 6 seconds, a shorter 4 seconds, or other intervals. In addition, it is possible to take each still image at different intervals. By recording these still image time differences, the relative relationship between these still images can be understood. For example, when a plurality of still images are put together in time series, they can be aligned according to the corresponding shooting times.

[0075] In the rightmost column of FIG. 10, for each still image, it is shown which still image was not transmitted by the control unit 350. Specifically, in the example of FIG. 10, focusing on the traveling speed of the vehicle, it can be seen that the still images of No. 7, No. 8, No. 10, No. 11, No. 13, No. 14, No. 16, No. 17, and No. 19 were not transmitted. Note that, as described above, the still images and the vehicle information (latitude, longitude, etc.) are transmitted separately from each other by the control unit 350.

[0076] Next, with reference to FIGS. 14 and 15, specifically, within the range indicated by reference numeral 1000 in FIG. 10, the case of selectively transmitting still images (No. 15 to No. 22) among a plurality of still images that meet a predetermined condition (No. 15, No. 18, No. 20, No. 21, No. 22) will be specifically described.

[0077] In the left column of FIGS. 14 and 15, each still image taken within the range from No. 15 to No. 22 in FIG. 10 is illustrated. Each still image is continuously taken at a predetermined interval (about 5 - second cycle) from top to bottom. On the right side of FIGS. 14 and 15, the case where a still image corresponding to a predetermined condition among the above - mentioned plurality of still images is selected is shown. The still images that were not selected (No. 16, No. 17, No. 19) are not transmitted to the management server 102. Here, for the sake of simplicity, it is referred to as "no image".

[0078] Specifically, within the range from No. 15 to No. 19, the vehicle is equivalent to the case where it stops at a speed of 0 km / h (or a position change of 0 m) while waiting for the signal to change at an intersection. Next, within the range from No. 20 to No. 22, the vehicle is equivalent to the case where it restarts, enters the intersection at a low speed, and makes a left turn.

[0079] When the vehicle stops, basically, there is little relative change between the vehicle and the surrounding scenery. However, the movement of other vehicles and people around is excluded. Therefore, when the control unit 350 of the drive recorder 101 determines that the running state of the vehicle is in low-speed running or stopped, among the plurality of images captured by the camera 304, only a part of the still images corresponding to predetermined conditions are selected and transmitted to the external management server 102. In this example, among the plurality of still images (No. 15 to No. 19), the still images (No. 15, No. 18) are selected and transmitted.

[0080] The still image corresponding to the above-mentioned predetermined condition is a still image selected every predetermined number of frames among the plurality of continuously captured still images. Preferably, the still image corresponding to the above-mentioned predetermined condition is at least one still image among at least three continuously captured still images. Thereby, the control unit 350 regularly thins out the still images for each group of a predetermined number of frames, so that the burden at the time of transmission can be reduced. In particular, when the communication processing unit 302 transmits data in the LPWA method, since the data capacity at the time of transmission becomes a problem, it is effective. Also, by deleting two out of three, the playback speed difference from the video during running can be eliminated, and the video data size can be effectively compressed.

[0081] More preferably, the still image corresponding to the above-mentioned predetermined condition is the latest one still image among the plurality of continuously captured still images. Or, the still image corresponding to the above-mentioned predetermined condition is the latest one still image among at least three continuously captured still images, and in the examples of FIGS. 14 and 15, one still image is transmitted every 15 seconds. Accordingly, the control unit 350 regularly performs thinning of still images for each group of a predetermined number of images. Therefore, when converting the selected still images into a moving image, it is possible to relatively suppress the unnaturalness caused by omission of still images. In general, an accident or the like may occur immediately after a vehicle waiting for a signal at an intersection starts to move. Therefore, by leaving the latest still image, the still image immediately after the vehicle starts to move is selected.

[0082] More preferably, the still image corresponding to the predetermined condition is the latest one among a plurality of continuously captured still images when the speed of the vehicle is continuously below a low speed. For example, in the embodiments shown in FIGS. 10 to 15, the predetermined condition is set as follows: "When the speed of the vehicle is continuously in a low speed state (for example, 2 km / h or less) for three consecutive files, the third file that is the latest in time is transmitted, and the remaining two files are not transmitted."

[0083] More preferably, the still image corresponding to the predetermined condition is a still image at or after the point in time when the start of movement of the vehicle is detected among a plurality of continuously captured still images. For example, in the example shown in FIGS. 10 to 15, at No. 20, the speed of the vehicle changes from 0 km / h to 7 km / h (that is, 2 km / h or more), and the start of movement of the vehicle is detected. Therefore, the still image of No. 20 corresponding to the point in time when the start of movement of the vehicle is detected or later is selected and transmitted.

[0084] It is generally said that traffic accidents are likely to occur immediately after a vehicle starts moving at an intersection, etc. By performing the above control, it is possible to preferentially extract a still image that captures the surrounding situation immediately after the vehicle starts moving. For example, the second image from the top in FIG. 15 (No. 20 in FIG. 10) records a vehicle entering an intersection and a bicycle or other vehicle attempting to cross the crosswalk at the intersection. Therefore, the results of Test 5 above show that this embodiment is suitable for a drive recorder 101 that is required to record evidence images in the event of a traffic accident, etc., because it not only periodically thins out still images but also preferentially extracts still images that capture changes in the surrounding situation.

[0085] As described above, the drive recorder 101 prioritizes the extraction of highly necessary images while periodically thinning out less necessary images, thereby reducing the overall required image capacity (transmission volume or recording volume) (see FIGS. 14 and 15). In actual test 5, it was confirmed that by performing the above control, data reduction of 20% or more was possible for driving in a city. Therefore, it was found that the drive recorder 101 is suitable for transmitting data using the LPWA system.

[0086] However, when an event related to a vehicle collision or the like is detected, the control unit 350 of the drive recorder 101 records or transmits all still images within a predetermined period of time. Such an event can be detected, for example, by a G sensor or the like. For example, in the column of still images shown on the left side of FIGS. 14 and 15, if an event related to a vehicle collision or the like occurs, all still images No. 15 to No. 22 may be selected and transmitted.

[0087] When an event related to a vehicle collision is detected, the control unit 350 of the drive recorder 101 may transmit all still images within a specified period of time, from a time point in the past prior to the time of detection to a time point in the future prior to the time of detection. For example, when an event related to a vehicle collision is detected, the control unit 350 of the drive recorder 101 may transmit all still images from a past time point retroactively from the detection time by a predetermined period (for example, several seconds, several tens of seconds, one minute, several minutes or more) to a time point after a predetermined period (for example, several seconds, several tens of seconds, one minute, several minutes or more) has elapsed from the detection time.

[0088] In this case, the still images not transmitted by the control unit 350 may not be immediately deleted, but may be temporarily stored in the transmission buffer and then deleted with a time difference (for example, several seconds, several tens of seconds, one minute, several minutes or more). This enables all still images within a period before and after the moment of the accident to be recorded when a traffic accident occurs.

[0089] Strictly speaking, the still image shooting time and the still image transmission time do not coincide. For example, the still image transmission time is slightly (about 1 second) later than the still image shooting time. Therefore, the control unit 350 may transmit the still image if the vehicle speed at the transmission time or the speed at the still image shooting time does not meet the condition of being low speed.

[0090] Furthermore, in the case where there is a slight delay (or the transmission is intentionally delayed) due to reasons such as internal processing time in the timing of still image shooting and transmission, and the stopped (or low speed) state can be confirmed by GNSS or vehicle speed pulse etc. at both timings, the control unit 350 may reduce the transmission data volume and lower the communication cost by discarding the still image transmission two out of three times.

[0091] Regarding the "predetermined speed range" Preferably, the "predetermined speed range" relates to low-speed driving or stopping. Compared to normal legal speeds (for example, about 60 km / h), when driving at low speeds, the surrounding scenery changes more slowly, which increases the tendency for duplicate data and unnecessary data to accumulate. In particular, when the drive recorder 101 communicates wirelessly using LPWA, it is preferable to reduce the burden on transmission. Furthermore, the incidence of traffic accidents has been decreasing in Japan in recent years. Therefore, video of vehicles traveling at low speeds is selectively recorded.

[0092] In the above example, the predetermined speed range is set when the vehicle is traveling at a low speed of 2 km / h or less or when the vehicle is stopped at 0 km / h. However, this embodiment is not limited to this speed range. The traveling speed of the vehicle varies depending on the surrounding conditions, etc. The upper limit of the predetermined speed range can be changed in various ways depending on the embodiment.

[0093] Preferably, the predetermined speed range should include a stopped state (for example, 0 km / h). Generally, a vehicle may come to a complete stop (0 km / h) temporarily while waiting for a traffic light to change at an intersection, while waiting for a train to pass at a railroad crossing, or during heavy traffic congestion. In such cases, it is believed that a preceding vehicle or a following vehicle cannot stop completely, and a rear-end collision is likely to occur.

[0094] Preferably, the predetermined speed range may be an extremely slow speed (several kilometers per hour or less, for example, 2 kilometers per hour or less). Generally, when a vehicle is traveling at an extremely slow speed, there are situations where the vehicle is forced to travel slowly in order to maintain an appropriate distance from the vehicle ahead, such as in a traffic jam. In such cases, it is believed that the driver is more likely to be distracted or inattentive to the road ahead due to, for example, the operation of a car navigation system or a mobile phone.

[0095] The predetermined speed range may also be slow driving (for example, 10 km / h or less). Generally, when a vehicle is traveling slowly (for example, 10 km / h or less), there are situations where caution is required when driving, such as when traveling on narrow roads or entering intersections. At this speed, it is said that the stopping distance of the vehicle can change by several meters depending on whether the driver is driving while anticipating danger.

[0096] Furthermore, the predetermined speed range may be for low-speed driving (for example, 30 km / h or less). Generally, it is said that traffic accidents are likely to occur when the vehicle is driving at a low speed (for example, 30 km / h or less). For example, in Japan, statistically, it is said that three-quarters of all traffic accidents occur at 30 km / h or less.

[0097] Therefore, regarding the driving speed of the vehicle, in view of the role of the drive recorder 101 that captures the recorded video at the time of a traffic accident, for "low-speed driving", the upper limit may be 30 km / h or less, may be 10 km / h or less, preferably may be several km / h or less (for example, 2 km / h or less), and the lower limit is preferably 0 km / h.

[0098] "Start of vehicle movement" As a trigger for detecting the start of the vehicle movement, a change in the vehicle speed or acceleration can be used. For example, the control unit 350 may determine that the vehicle has started to move when an acceleration exceeding a predetermined threshold value (for example, 0) is detected from the input signal from the vehicle speed sensor or acceleration sensor of the vehicle.

[0099] Also, for example, there may be a case where the control unit 350 calculates the vehicle speed based on the signal received by the GNSS antenna 307. In this case, depending on the reception environment of the signal, a positioning error may occur, and the vehicle speed may be calculated at a speed faster by the amount of this positioning error. For example, even if the actual vehicle speed is 0 km / h, it may be calculated as a speed other than 0 km / h. Therefore, the control unit 350 may determine a predetermined speed range in consideration of this positioning error. For example, the control unit 350 may also determine that the vehicle is stopped when the speed is 7 km / h or less or 8 km / h or less.

[0100] In addition, the control unit 350 can use the change in the position of the vehicle as a trigger for detecting the start of movement of the vehicle. For example, the control unit 350 may determine that the start of vehicle movement has occurred based on the acquired position information from the signal received by the GNSS antenna 307. For example, generally, the vehicle's latitude and longitude information from GPS may include an error of about 7 km or 8 km. Therefore, when a change in the vehicle's position exceeding 7 km or 8 km is detected, the control unit 350 may recognize it as the start of vehicle movement.

[0101] The change in the speed or acceleration of the vehicle and the change in the position of the vehicle may be used in combination. Although it is difficult to detect the start of vehicle movement with high accuracy based on the signal received by the GNSS antenna 307, the reliability can be improved by combining the speed or acceleration of the vehicle. For example, the control unit 350 may enhance the reliability of the information by comparing (mutually checking) the change in the vehicle's speed or acceleration over time with the change in the vehicle's position over time.

[0102] The trigger for detecting the start of movement of the vehicle is not limited to the case where the vehicle actually starts moving and at least the position changes. The trigger for detecting the start of movement of the vehicle may include, based on the signal sent from the in-vehicle device, the moment when the vehicle's state switches from the stopped state to the running state or the state immediately before that, even if the vehicle's position has not actually changed.

[0103] For example, when the vehicle is a multi-wheeled vehicle including a four-wheeled vehicle, the control unit 350 can use one or more of the accelerator pedal, brake pedal, clutch pedal, parking brake lever, AT change lever, and MT shift lever as a trigger for detecting the start of movement of the vehicle if a change in one or more of these operations can be supplied to the drive recorder 101 as a signal.

[0104] For example, generally, when restarting from a temporary stop at an intersection or the like, the brake pedal may be released from being depressed by a predetermined amount, and the amount of depression of the accelerator pedal of the vehicle may increase from zero to a predetermined amount. Therefore, the start of movement of the vehicle may be detected based on the amount of depression of the accelerator pedal and / or the brake pedal. When restarting on an inclined road or the like, the parking brake lever may be further operated. Therefore, the start of movement of the vehicle may be detected based on the operation of the parking brake lever.

[0105] In the case of an MT vehicle, when restarting from a temporary stop at an intersection or the like, in addition to the brake operation, a clutch operation and an operation of the MT shift lever may be performed. Therefore, the start of movement of the vehicle may be detected based on these operations. In the case of an AT vehicle, when restarting from a temporary stop at an intersection or the like, the AT change lever may be operated. Therefore, the start of movement of the vehicle may be detected based on the operation of the AT change lever.

[0106] Further, when the vehicle is a motorcycle, a sidecar, or a trike (three-wheeler), the control unit 350 can use one or more of the accelerator grip, the front-wheel brake lever, the rear-wheel brake lever, the clutch lever, and the shift pedal as a trigger for detecting the start of movement of the vehicle when a change in the operation of one or more of them can be supplied to the drive recorder 101 as a signal.

[0107] For example, generally, when restarting from a temporary stop at an intersection or the like, the grip amount of the accelerator grip of the vehicle may increase from zero to a predetermined amount, or the front-wheel brake lever and / or the rear-wheel brake lever may be released from being depressed by a predetermined amount. Therefore, the start of movement of the vehicle may be detected based on the accelerator grip, the front-wheel brake lever, and / or the rear-wheel brake lever. Unlike an AT vehicle, in the case of an MT vehicle, when restarting from a temporary stop at an intersection or the like, in addition to brake operation, clutch operation and shift pedal operation may be performed. Therefore, based on these operations, it may be possible to detect the start of the vehicle's movement.

[0108] In addition, the control unit 350 may detect a change in the state of temporary stop or restart of the vehicle based on a signal from any device mounted on the vehicle. For example, in the case of a vehicle equipped with an idling stop function that stops the engine during parking or waiting for a signal, if the control unit 350 can receive a signal indicating the start or end of the idling stop function, etc., it may detect the temporary stop or restart of the vehicle based on that signal.

[0109] Furthermore, for example, if the control unit 350 can receive a signal indicating that the steering wheel has been steered, it may detect the temporary stop or restart of the vehicle based on that signal. Furthermore, when the control unit 350 detects the restart of the vehicle based on the signal sent from the above vehicle-mounted device, it can transmit a still image at that point (without a time difference) or with a delay from that point (for example, with a time difference of 1 second or more or several seconds or more, etc.).

[0110] "(b) Control Based on the Transmission State of Still Images" Furthermore, in another embodiment, the control unit 350 of the drive recorder 101 selects only some of the still images captured by the camera 304 based on the transmission state of the still images and transmits them to the external management server 102. Hereinafter, this embodiment will be described with reference to FIGS. 16 to 20.

[0111] "Test of Still Image Transmission (Test 6)" Furthermore, the applicant conducted a test to control the transmission of a plurality of still images (here, selectively transmit) based on the transmission state of the still images. In FIGS. 16 and 17, the results of the above Test 6 are shown in a table. Although FIGS. 16 and 17 constitute one table as a whole, they are shown separately in two parts for ease of viewing. In FIGS. 18 to 20, three specific examples among this table are described in more detail.

[0112] In this embodiment, three types of data are transmitted from the drive recorder 101 to the management server 102 by the control unit 350. The first data is the captured still image (the image data with the largest capacity), the second data is the position information of the vehicle's latitude and longitude (the character data with relatively small capacity), and the third data is the ping for confirming the reachability of the data (the data with the smallest capacity).

[0113] In FIGS. 16 and 17, in the leftmost column, the order of the still images captured over time is shown. In this example, a total of 37 still images are captured. In the second column from the left, the file names of the still images are shown. In the third and fourth columns from the left, the latitude and longitude of the vehicle at the time when each still image is captured are shown. In the fifth column from the left, the speed of the vehicle at the time when each still image is captured is shown. In the sixth column from the left, the still image time difference is shown as the time interval between two adjacent still images.

[0114] In FIGS. 16 and 17, further, in the seventh column from the left, the response time of the ping for confirming the reachability of the data is shown. In the eighth column from the left, the server (management server 102) arrival time of the latitude and longitude data (vehicle information) is shown. In the ninth column from the left, the server (management server 102) arrival time of the still image is shown. In the rightmost column, the transmission / reception state of the still image data is shown.

[0115] Next, referring to FIGS. 18 to 20, specific examples of FIGS. 16 and 17 will be described. In this embodiment, the control unit 350 determines the transmission state of the communication processing unit 302 based on the number of still images transmitted and the response time from the transmission of the still image until the transmission is completed.

[0116] Referring to FIG. 18, the transmission state of the data within the range indicated by reference numeral 1500 in FIG. 16 is conceptually shown. FIG. 18 shows a case where the data transmission state is good, and the three types of data sets are each transmitted at a predetermined interval of 5 seconds. Ping transmits a message from the source drive recorder 101 to the destination management server 102 and measures the round-trip time (RTT, round-trip time) until the response returned by the destination arrives. Among the three types of data, ping has the smallest data capacity and can be transmitted most favorably.

[0117] Generally, under good transmission conditions, ping responds within several hundred milliseconds. For this reason, Ping is used objectively to confirm the communication state for verification purposes. Vehicle information (latitude and longitude data) requires less than 1 second until the management server 102 receives it because of its size of several hundred bytes. Still image data requires 3 to 4 seconds until the management server 102 receives it because of its file size of 50K to 60K bytes.

[0118] In the example of FIG. 18, the first ping requires 67.7 milliseconds for data transmission, and the next ping requires 124.3 milliseconds for data transmission. Since vehicle information has a larger data capacity than ping, it requires a longer transmission time. The first vehicle information requires less than 1 second, and the next vehicle information requires less than 1 second. Since the still image has the largest data capacity, it requires the longest transmission time. The first still image requires 4 seconds, and the next still image requires 4 seconds. Since the predetermined interval at which each set of data is transmitted is 5 seconds, the transmission of the data is completed within the predetermined interval without causing any delay, respectively. Note that the predetermined interval for each still image can be determined in consideration of the transmission time of a normal still image (for example, 5 seconds, but not limited to this).

[0119] Next, referring to FIG. 19, the transmission state of the data within the range indicated by reference numeral 1600 in FIG. 16 is conceptually shown. FIG. 19 shows a case where the transmission state has deteriorated. Similarly, it exemplifies a situation where three types of data sets are transmitted at a predetermined interval of 5 seconds. The first ping takes 133.1 milliseconds for data transmission, and the next ping takes more than 1 second for data transmission. That is, at the time of the No. 7 Ping, the communication state is good, but since the data arrival time is long, it can be inferred that the communication state has deteriorated immediately after the Ping transmission. At the time of No. 8, the poor communication state continues, and the Ping transmission time is longer. In this way, the control unit 350 can determine whether the transmission state is good or deteriorated by comparing the time required for ping transmission with a predetermined threshold value.

[0120] Preferably, the control unit 350 determines the transmission state of the communication processing unit based on the response time of data smaller in size than the still image, which is transmitted separately from the still image. However, this small data is not limited to ping. For example, the response time of vehicle information may be used to determine the transmission state.

[0121] As the transmission situation deteriorates, the vehicle information takes 1 second for the first vehicle information and 9 seconds for the next vehicle information. Since the predetermined interval at which each set of data is transmitted is 5 seconds, the second transmitted vehicle information cannot be completed within 5 seconds. Therefore, there is a risk of affecting the transmission of subsequent vehicle information. In particular, for still images, the first vehicle information takes 15 seconds, and the next vehicle information takes 21 seconds. Since the predetermined interval at which each data set is transmitted is 5 seconds, the still images transmitted in each case cannot be completely transmitted within 5 seconds. Therefore, there is a risk of affecting the transmission of subsequent still images. In this situation, it is feared that the effects of the deteriorated communication state will accumulate.

[0122] Next, referring to FIG. 20, control for improving data transmission under the deteriorated transmission situation illustrated in FIG. 19 will be described. FIG. 20 conceptually shows two ranges, specifically the reference numeral 1700 in FIG. 16 and the reference numeral 1710 in FIG. 17.

[0123] In FIG. 20, similar to FIG. 19, when two or more consecutive still images are being transmitted (transmission is not completed), the control unit 350 determines that the transmission state (line state) is not good (see reference numeral A in FIG. 20). In this case, the control unit 350 temporarily stops the transmission of the subsequent still images of the still image whose transmission time has been extended and stores that data in the storage device (memory) of the drive recorder 101 (see reference numeral B in FIG. 20). By doing this, the transmission of the still image with the largest data capacity is temporarily stopped, and overall, the burden of data transmission is reduced. As a result, it is possible to avoid the effects of the deteriorated transmission state from cumulatively affecting subsequent data transmission and contribute to the early improvement of the deteriorated transmission state.

[0124] Furthermore, the control unit 350 continuously monitors the number of still images being transmitted. When the number of still images being transmitted is 0 and the latest Ping response time becomes equal to or less than a predetermined threshold value (designated time) (for example, 900 milliseconds or less) (No. 25), it can be determined that the communication state has improved. In this case, the control unit 350 starts the sequential transmission of the still images stored in the memory for temporary storage (see reference numeral C in FIG. 20).

[0125] In the specific example of FIG. 20, calculating backward from the fact that the time until the management server 102 of the still image No. 9 is reached is 87 seconds, the transmission from the drive recorder 101 starts at the timing of the transmission of the data of No. 26. In the specific example of FIG. 20, since the speed is low after No. 26, still image transmission is not performed by the still image transmission thinning function (see FIG. 17). Therefore, after No. 26, since the state of waiting for the completion of still image transmission continues to be zero, the control unit 350 can sequentially transmit the still images stored in the temporary storage memory.

[0126] As described above, the control unit 350 of the drive recorder 101 controls the transmission of a plurality of still images captured by the camera 304 based on at least one of the running state of the vehicle and the still image transmission state. (a) As illustrated in FIGS. 10 to 15, the control unit 350 may control the transmission of a plurality of still images captured by the camera 304 based on the running state of the vehicle. (b) As illustrated in FIGS. 16 to 20, the control unit 350 may control the transmission of a plurality of still images captured by the camera 304 based on the still image transmission state. (a) and (b) may be performed separately or may be combined with each other.

[0127] The above controls (a) and (b) may be started at any timing during the running or stopping of the vehicle. Preferably, in a vehicle that has stopped with the engine turned off, the control may be started at the moment when the engine is started and the vehicle can run. In other words, the control may be started in synchronization with the control of the drive recorder 101. The above controls (a) and (b) may be ended at any timing during the running or stopping of the vehicle. Preferably, the control may be ended at the moment when the vehicle has stopped, the engine has been turned off, and the vehicle cannot run. In other words, the control may be ended in synchronization with the control of the drive recorder 101.

[0128] In at least one of the cases of 「(a) control based on the driving state of the vehicle」 and 「(b) control based on the transmission state of still images」, when the control unit 350 delays the transmission of a still image due to deterioration of the radio wave condition or the like, the control unit 350 temporarily stores the still image in the transmission buffer. Under this configuration, when the control unit 350 re-transmits a still image from the transmission buffer, it may dynamically change the priority of each still image so that the latest still image is always transmitted with the highest priority. Specifically, if still image A, still image B, and still image C are stored in the transmission buffer in order of the most recent shooting time, the transmission priority is higher in the order of the most recent shooting time, and here it is in the order of still image A, still image B, and still image C.

[0129] If the latest still image Z cannot be transmitted from this state due to reasons such as radio wave conditions, the control unit 350 stores the still image Z in the transmission buffer and changes the transmission priority to still image Z, still image A, still image B, and still image C. In this way, the control unit 350 performs a control of the last-in first-out method. Also, when the capacity of the transmission buffer reaches the upper limit, the control unit 350 may perform a process of deleting from the transmission buffer in order of the still images with the lowest priority. As a result, when the radio wave condition improves and the transmission of still images becomes possible, the latest still image is always transmitted. Therefore, even when remotely monitoring the images captured by the drive recorder 101 in real time, it is possible to easily check the latest situation.

[0130] Note that the number of cameras 304 used in the drive recorder 101 is not limited to one. For example, the drive recorder 101 can use at least one of a camera that captures the front of the vehicle, a camera that captures the rear of the vehicle, a camera that captures the side of the vehicle, a camera that captures the diagonally front of the vehicle, a camera that captures the diagonally rear of the vehicle, and a camera that captures in an arbitrary angle range up to 360° around the vehicle as the camera 304.

[0131] For example, the cameras 304 of the drive recorder 101 may be provided on the front and rear sides of the vehicle. In this case, the camera 304F on the front side of the vehicle and the camera 304B on the rear side of the vehicle may independently capture an image of the front side and an image of the rear side, respectively. These cameras 304F and 304B may synchronize the timing of capturing video images under the control of the same or different control units 350. When selecting an appropriate still image from among a plurality of still images captured by each camera based on the running state of the vehicle, etc. (controls a and b above), these cameras 304F and 304B may synchronously extract still images in front of and behind the vehicle based on the same trigger.

[0132] As described above, the drive recorder 101 can transmit original still images captured by the camera 304 to the management server 102 in the LPWA format. In the above embodiment, each still image is captured at 5-second intervals, but this interval can be changed in other embodiments in accordance with the data capacity of each still image captured by the camera 304 and future changes in communication technologies such as LPWA.

[0133] "Control on the Management Server 102 Side" Next, referring again to FIGS. 1 and 4, The control on the management server 102 side will be explained.

[0134] The management server 102 has at least a receiving unit (for example, the communication control unit 406 in FIG. 4) that receives a plurality of still images transmitted from the drive recorder 101 that controls the above-mentioned contents, and a moving image creating unit 412 (for example, the moving image creating module 412 executed by the processor 403 in FIG. 4) that combines the received plurality of still images to create a moving image. Note that the moving image creating unit 412 may be implemented as a sub-module of the video information management module 411.

[0135] For example, the management server 102 may perform a method including the following steps. First, as the first step, the communication control unit 406 of the management server 102 sequentially receives a plurality of still images transmitted from the drive recorder 101 side. Next, at this time, the communication control unit 406 or the moving image creation unit 412 of the management server 102 may integrate a plurality of still images transmitted at a predetermined time interval (for example, a time interval of several minutes, 10 minutes, 30 minutes, 1 hour or more) into one group. Alternatively, the communication control unit 406 or the moving image creation unit 412 of the management server 102 may integrate a plurality of still images transmitted in a predetermined number (for example, several images, 10 images, 30 images, 100 images, several hundred images, several thousand images or more) into one group.

[0136] Next, as the second step, the moving image creation unit 412 of the management server 102 applies an animation effect to each still image integrated into one group, in which a plurality of still images are sequentially switched and displayed over time like a flip comic, to create a moving image. At this time, as will be described later, the moving image may be created so as to exhibit a special effect. At this time, the created moving image may be stored on the storage device 401 or 402 of the management server 102. Next, the communication control unit 406 of the management server 102 transmits the created moving image to the user terminal 103.

[0137] For example, when a plurality of still images are transmitted from the drive recorder 101 side to the management server 102, on the management server 102 side, the plurality of still images are converted into a moving image composed of, for example, two still images per second. At this time, the moving image creation unit 412 of the management server 102 may perform control to reduce the data capacity used for creating the moving image. For example, on the management server 102 side, the still images (video information 420) transmitted from the drive recorder 101 are stored and managed in the auxiliary storage device 402. At this time, if data is continuously stored in the state of still images on the management server 102 side, the number of management data is large, so the capacity increases. Therefore, when the management server 102 converts a plurality of still images into a moving image, only the data of the moving image may be stored, and the data of the still images used for creating the moving image may be deleted from the auxiliary storage device 402.

[0138] When creating a video from a plurality of still images, the video creation unit 412 of the management server 102 can perform control to make the video easier to view. For example, on the management server 102 side, each received still image may be made into a video by aligning their respective time axes (according to the actual shooting time). However, in this case, since the still images are updated at intervals of several seconds (for example, 5 seconds), the video will be a stretched video and the video size will also increase. Therefore, when creating a video by integrating a plurality of received still images, the video creation unit 412 of the management server 102 may create a video that is played back faster than the actual time series. For example, when creating a video from each still image taken at 5 - second intervals, the video creation unit 412 may display two still images per second. This can improve the viewability of the video. In this case, a video taken at one frame per 5 seconds is converted into a video at two frames per 0.5 seconds with faster playback.

[0139] In addition, when creating a video from a plurality of still images, the video creation unit 412 of the management server 102 can perform control to make the video easier to view. Ideally, on the drive recorder 101 side, a plurality of still images are taken at regular time intervals. However, on the management server 102 side, since delays can occur in the time between still images due to line conditions, it is not necessarily a constant interval. If this deviation is too large, there is a risk of giving the viewer a sense of discomfort when playing the video.

[0140] For example, if the above - mentioned deviation is within the range of several seconds to more than ten seconds, even if the video is created and played back with a constant time axis as it is, it will not give the viewer a great sense of discomfort. When the vehicle is running, the video is constantly changing, so even if there is a slight deviation in the playback time axis, it will not give the viewer a great sense of discomfort. Also, in cases such as stopping while waiting for a signal or level crossing to change, or stopping in traffic congestion, since there are consecutive still images with little change, even if there is a deviation in the time axis, it will not give the viewer a sense of discomfort.

[0141] Therefore, when there are consecutive still images with little change, the video creation unit 412 can also create a video by thinning out the still images with little change. For example, when there are 9 consecutive still images waiting for a signal, the video creation unit 412 can create a video from 3 of them by deleting 2 out of every 3 still images. With such a configuration, the playback time of the video can be further compressed, and the time for checking the parts with little change during viewing can be saved.

[0142] On the other hand, when the deviation of the time axis occurs at a larger time interval exceeding a predetermined threshold (for example, when a deviation of 1 minute or more occurs), it may give the viewer a sense of discomfort. For example, depending on the surrounding situation of the traveling vehicle, such as inside a tunnel, the function of the communication processing unit 302 may deteriorate or temporarily stop functioning. Therefore, when creating a video from a plurality of received still images, if the still images have an interval exceeding a predetermined threshold (for example, 1 minute), the video creation unit 412 may create a video by duplicating the still images before this interval. Alternatively, the video creation unit 412 may create a video by generating new still images based on the still images before and after this interval. In this way, when the still images are continuously missing, they may be duplicated at a ratio corresponding to the missing time and inserted to create a video.

[0143] However, in the above case, the video creation unit 412 creates a video that displays information indicating that there is no original image data. For example, the video creation unit 412 may add a telop (character information, etc.) indicating that there is no image data to the location where the duplicated still image is inserted and display it together with the telop during video playback. Thereby, when the images taken by the drive recorder 101 are used later, the actually taken images and the images created to smooth the video creation can be clearly distinguished.

[0144] In addition, for the case when the vehicle is traveling at high speed, the video creation unit 412 may change the control compared to the case of low-speed traveling or stopping as described above. For example, assume that each still image is captured at 5 - second intervals. In this case, when the vehicle's traveling speed is about 100 km / h, if the captured images are made into a video, during video playback, it may be recognized by the viewer as a fast - forward playback state. Therefore, the video creation unit 412 may change the fast - forward frequency during low - speed driving and the fast - forward frequency during high - speed driving.

[0145] Generally, when continuously playing a plurality of captured still images to seemingly play them as a video, it is necessary to align the vehicle's moving distance with the playback time axis. However, when the traveling speed is high, if the still images are duplicated to increase the number and then made into a video, it has been found that during playback, (relatively) the driving video of that part seems to be slow. That is, although the actual traveling speed of the vehicle is relatively high, there is a possibility of giving the viewer a sense of discomfort as if the traveling speed in the reproduced video is slow. Therefore, when the vehicle is traveling at a low speed or stopped, the video creation unit 412, as described above, duplicates the still images, increases the number, and then creates a video. On the other hand, when the vehicle is traveling at a high speed, the control of duplicating the still images as described above and increasing the number before creating a video may be avoided.

[0146] In addition, the video creation unit 412 can perform control to enhance the convenience of the video when creating a video from a plurality of still images. That is, to the management server 102 side, vehicle information such as the vehicle's position information, in addition to a plurality of still images, is transmitted from the drive recorder 101 side. The still images and the vehicle information can each have information on the shooting time. Therefore, even if the management server 102 side receives the still images and the vehicle information separately, it is possible to link them to each other.

[0147] Therefore, on the management server 102 side, when the communication control unit 406 receives a plurality of still images and the vehicle position information at the time when the plurality of still images were created from the drive recorder 101, the moving image creation unit 412 may embed the vehicle position information in the moving image when integrating the received plurality of still images to create a moving image. As a result, on the user terminal 103 side that plays back the moving image, the vehicle position information may be synchronously displayed based on the moving image. For example, when creating a moving image from a plurality of still images, the moving image creation unit 412 embeds vehicle information as metadata in the moving image file. Thereby, when playing back the moving image using a dedicated application on the user terminal 103 side, the position information may be synchronously displayed on the map only with the moving image data.

[0148] Also, the moving image creation unit 412 may separately transmit the moving image created from the still images and the vehicle information to the user terminal 103 side. The vehicle information can be transmitted earlier and better compared to the moving image. Therefore, in the application software of the user terminal 103 that displays these information, the vehicle information such as the position information is preferentially displayed on the map, and the moving image is displayed (updated) as soon as the reception is completed, so that as a whole, the user can not feel the data transmission delay.

[0149] Furthermore, the moving image creation unit 412 may include the still image file name and the checksum value in the vehicle information. Thereby, the individual data stored on the management server 102 side may be synchronously displayed later, or the still image data that cannot be received or is broken for some reason may be skipped and displayed in the application software.

[0150] "Vehicle Position Estimation by Still Images" As described above, the management server 102 can receive vehicle information from the drive recorder 101 and know the latitude and longitude of the vehicle based on it. However, depending on the surrounding situation and communication situation of the running vehicle, etc., there may be a case where the vehicle position information cannot be sufficiently received. Therefore, the management server 102 may perform image analysis on the received still image to estimate the position of the vehicle.

[0151] Figures (A) and (B) of FIG. 21 are diagrams schematically showing an example in which the management server 102 performs image analysis on a still image. As illustrated by reference numeral 1810 in FIG. 21(A), on ordinary roads, there are road signs near intersections and the like. Also, on highways and the like, there are exit guide plates and the like. Usually, these road signs and the like are marked with character information (for example, Tokyo) 1810 indicating the name of a region such as a place name. Also, as illustrated by reference numeral 1820 in FIG. 21(A), usually, these road signs and the like are marked with character information (for example, 4 km) 1820 indicating the distance to a reference region in numbers. Also, usually, these road signs and the like are marked with character information (for example, Route 6 etc., not shown in the figure) indicating the road number of national roads, prefectural roads, etc. in numbers (not shown in the figure).

[0152] Therefore, the moving image creation unit 412 of the management server 102 may perform image analysis of the still image, particularly based on the character information 1810 and 1820. By performing image analysis on the still image based on such names or numbers of roads and regions, the position of the vehicle may be estimated. For example, the moving image creation unit 412 of the management server 102 performs a scan to extract text of characters or numbers from the still image. At this time, the moving image creation unit 412 may use an "OCR" function for recognizing and converting characters from the still image.

[0153] Usually, since the road information (for example, the name of a region, the number of a national road, etc.) used in road signs and the like is known, it is possible to comprehensively create a database of various information such as maps or road information in advance and store it in the storage device 401 or 402. Therefore, the moving image creation unit 412 performs optical character recognition on the image data and temporarily stores the character information in the storage device 401 or 402. Then, the position of the vehicle may be estimated by comparing the information with a database storing various information such as maps or road information.

[0154] Furthermore, the moving image creation unit 412 may perform image analysis of a still image based on a characteristic shape, figure, contour, color, or the like. For example, as illustrated by reference numeral 1830 in FIG. 21(A), there may be a case where two or more roads temporarily overlap in the vertical direction (such as a junction). In such a case, based only on the latitude and longitude information of the traveling vehicle, it may not be possible to determine which of the two vertically overlapping roads the corresponding vehicle is traveling on. In such a case, the moving image creation unit 412 may perform image analysis on the received still image and distinguish, for example, a case where there is some structure above the road as illustrated by reference numeral 1830 in FIG. 21(A) and a case where there is nothing above the road as illustrated by reference numeral 1840 in FIG. 21(B).

[0155] For example, the moving image creation unit 412 may perform image analysis on the received still image, particularly focusing on above the road, and determine whether a characteristic target such as some structure can be extracted based on changes in shape, contour, color, or the like (see FIGS. 21(A) and (B)). At this time, the moving image creation unit 412 may classify the image for each pixel or each set of pixels of the received still image. Then, by identifying the shape, color, or the like for each pixel or each set of pixels, image analysis may be performed to identify a characteristic target.

[0156] The moving image creation unit 412 may perform image analysis training on various images in advance by machine learning using AI. In particular, based on various specific examples, actual image analysis is performed to train to identify and extract object targets (characters, numbers, shapes, contours, lines, vertices, colors, etc.) from the images. The training is performed, for example, on images of hundreds, thousands, or more different object targets, and various results are accumulated. At that time, for example, image processing may be performed under various situations such as images of only roads, images of roads with multiple vehicles running, daytime road images, and nighttime road images, and statistical data may be calculated and generated for the results. After obtaining a high identification rate, for example, an identification rate of about 99% or about 99.9% or higher by training, the moving image creation unit 412 may actually perform image processing to identify object targets from the images.

[0157] In the training performed by the above-mentioned machine learning using AI, various image information may be used as teacher data. For example, a determination model may be generated by machine learning with the input being a still image and the output being information regarding the contours of characteristic objects, vertices, colors, characters, numbers, etc. By inputting a new still image into this determination model, characteristic object targets may be obtained as the output.

[0158] For example, the moving image creation unit 412 may perform image analysis training on still images inside various tunnels or above and below elevated roads in advance and accumulate the statistical information. For example, remember the shapes, colors, etc. of characteristic object targets inside various tunnels or on elevated roads. Then, when performing image analysis on a still image, the moving image creation unit 412 may specify the position of the vehicle based on the detection of those characteristic objects.

[0159] In this way, the moving image creation unit 412 may estimate the position from the received still images through image analysis and AI for vehicle positions within a tunnel or above or below an elevated road where GNSS cannot make a determination, and use it for vehicle management and driving record. Also, the moving image creation unit 412 may estimate the position from characters such as road signs and exit guidance boards within the tunnel. Further, for the elevated road, the moving image creation unit 412 may estimate whether the vehicle is traveling upward or downward from the presence or absence of structures at the upper part of the screen and road signs.

[0160] The moving image creation unit 412 of the management server 102 may use a combination of vehicle information (latitude, longitude, etc.) transmitted from the drive recorder 101 and object information extracted based on image analysis performed on the still images to identify the position of the vehicle in motion with higher accuracy. The information on this position may also be utilized when converting the still images into a moving image and displaying the created moving image on the user terminal.

[0161] As described above, the control of converting a plurality of still images into a moving image on the management server 102 side may be started at an arbitrary timing. For example, the control of converting a plurality of still images into a moving image on the management server 102 side may start the conversion after starting the reception of the still images or after the reception of the still images has been interrupted. In other words, the moving image creation unit 412 may start the control of converting the still images into a moving image after the reception of a series of still images is completed so as to combine the still images during all stages of the vehicle's travel into one moving image. This may, for example, use the moment when the reception of the still images is interrupted at intervals of one minute, several minutes, five minutes, ten minutes, or more as a trigger to determine the end of the vehicle's travel and start creating the moving image immediately after that timing.

[0162] Alternatively, the control of converting a plurality of still images into a video on the management server 102 side may start the video conversion after a predetermined period has elapsed after the reception of the still images transmitted from the running vehicle has started. For example, the video creation unit 412 starts video creation several minutes, 5 minutes, 10 minutes, 30 minutes, 1 hour, or several hours after starting the reception of the still images. Thereby, a video with a predetermined time interval or a predetermined size can be created. When the running of the vehicle ends in a short period, the video creation unit 412 may create one video, and when the running of the vehicle extends over a long period, the video creation unit 412 may continuously create a plurality of videos. The control of converting a plurality of still images into a video on the management server 102 side may end when the video is created and transmitted to the user terminal 103. Alternatively, the control of converting a plurality of still images into a video on the management server 102 side may end when the video is created and stored in the storage device on the management server 102 side.

[0163] As described above, the drive recorder 101 and the management server 102 have been described. Further, it may be implemented as a program that enables the control of the above content to be executed on the drive recorder 101 and the management server 102.

[0164] For example, this embodiment may be a computer program product for causing a computer to realize the control of transmitting still images and / or vehicle information of the content described above with reference to FIGS. 1, 3, 7 to 20 to the drive recorder 101. The computer program product may be implemented as a program or a function or a routine or an executable object.

[0165] Preferably, the computer program product includes program code that enables the execution of a method for controlling the drive recorder 101 including the above controls a and b. The program is stored in a storage device connected to a control unit such as the microcomputer 301 of the drive recorder 101 or the DVR controller 303. The storage device may be built into the drive recorder 101 or externally connected. The program controls the control unit of the drive recorder 101 so as to enable execution of control of transmission of still images.

[0166] Furthermore, it may be a computer program product for causing a computer to realize control of video conversion of the content described above with reference to FIGS. 1, 4, and 21 for the management server 102. The computer program product may be implemented as a program or a function or a routine or an executable object.

[0167] Preferably, the computer program product includes program code for enabling execution of a method for controlling the management server 102 including the above video conversion. The program is stored in a storage device (such as 401 or 402) connected to the video creation unit 412 of the management server 102. The storage device may be built into the management server 102 or externally connected. The program controls the video creation unit 412 of the management server 102 so as to enable execution of control of video conversion.

[0168] Furthermore, this embodiment may be a computer program product for causing a computer to realize control of video playback of the content described above with reference to FIG. 1 for the user terminal 103. The computer program product may be implemented as a program or a function or a routine or an executable object.

[0169] Preferably, the computer program product includes program code for enabling execution of a method for controlling the user terminal 103 including the above video playback. The program is stored in a video playback unit such as the processor 503 of the user terminal 103 or in a storage device (such as 501 or 502) connected thereto. The storage device may be built into the user terminal 103 or may be externally connected. The program controls the video playback unit of the user terminal 103 so as to be able to execute control of video playback.

[0170] Note that the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Also, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, for a part of the configuration of each embodiment, addition, deletion, or replacement with other configurations is possible.

[0171] Also, each of the above configurations, functions, processing units, processing means, etc. may be realized in hardware by designing a part or all of them, for example, by an integrated circuit. Further, each of the above configurations, functions, etc. may be realized in software by a processor interpreting and executing a program for realizing each function. Information such as a program, table, file, etc. for realizing each function can be placed in a memory, a recording device such as a hard disk, an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, a DVD. Also, control lines and information lines show those considered necessary for explanation, and not necessarily all control lines and information lines are shown on the product. In practice, it may be considered that almost all configurations are interconnected.

[0172] A computer program product such as computer program means can be implemented as a memory card, a USB stick, a CD-ROM, a DVD, or a file downloadable from a server in a network. For example, such a file may be provided by transferring a file including the computer program product from a wireless communication network.

[0173] It should be noted that the scope of the present invention is not limited to the configurations explicitly described in the specification or limited ones. The combinations of various aspects of the present invention disclosed in this specification are also included within its scope. Among the present invention, the configuration for which a patent is sought has been specified in the appended claims. However, even if a configuration is not currently specified in the claims, the configuration disclosed in this specification has the intention of being the claims in the future.

[0174] The invention of the present application is not limited to the configurations described in the above-described embodiments. The constituent elements of the above-described embodiments and modified examples may be arbitrarily selected and combined. Also, any constituent element of each embodiment or modified example may be arbitrarily combined with any constituent element described in the means for solving the invention or a constituent element embodying any constituent element described in the means for solving the invention. Regarding these, there is also the intention of obtaining rights in the amendment or divisional application of the present application, etc. Even if there is a description such as "in the case of ~" or "when ~", it is not described as a configuration limited to that case or that time. The configurations other than these cases and times are also disclosed and have the intention of obtaining rights. Also, the parts with sequential descriptions are not limited to this order. The configurations with some parts deleted or the order changed are also disclosed and have the intention of obtaining rights.

[0175] The above-described embodiments related to various controls implemented for the drive recorder (electronic device or in-vehicle device) 101, the management server 102, and the user terminal 103 have been described. These inventions can be arbitrarily combined. Also, any configuration can be extracted from these inventions and the extracted configurations can be combined. The applicant of the present application intends to obtain rights for inventions including these configurations. Also, even if there is a description such as "in the case of ~" or "when ~", it is not described as a configuration limited to that case or that time. These show examples of better configurations, and the applicant also intends to obtain rights for configurations other than these cases or times. Also, the order described in the sequential descriptions is not limited to this order. The present application also discloses configurations in which some parts are deleted or the order is changed, and the applicant intends to obtain rights for such configurations.

[0176] Also, by changing to a design registration application, the applicant intends to obtain rights for the overall design or partial design. Although the drawings depict the entire device in solid lines, the drawings include not only the overall design but also partial designs claimed for a part of the device. For example, not only can a part of the members of the device be a partial design, but the drawings also include a partial design that includes a part of the device regardless of the members. As a part of the device, it may be a part of the members of the device or a part of those members. The applicant intends to obtain rights for partial designs in which any part of the solid-line portions of the drawings is changed to a dashed-line portion, as well as for the overall design. Also, with respect to the modules, members, parts, etc. inside the housing of the device, those shown in the drawings are all independently subject to transactions, and similarly, the applicant intends to change to a design registration application and obtain rights for them.

[0177] Note that the above-described embodiments disclose at least the following configurations. (1) A communication processing unit, A control unit that processes image information captured by a camera and transmits a plurality of still images via the communication processing unit, and The control unit is an electronic device that controls the transmission of the plurality of still images based on at least one of the driving state of the vehicle and the transmission state of the still images. By configuring in this way, in order to transmit the still images, compared with the case of directly transmitting the video stream, the burden on the electronic device side during transmission can be reduced. Or, by controlling the transmission of the still images based on the driving state of the vehicle or the transmission state of the still images, the still images can be transmitted under suitable conditions. (2) Regarding the driving state of the vehicle, the control unit When at least one of the vehicle speed being within a predetermined speed range or the change in the vehicle position being within a predetermined distance range is detected, among the plurality of captured still images, a predetermined still image may be selected and transmitted. By configuring in this way, compared with the case of transmitting all the still images, the burden on the electronic device side during transmission can be reduced. At this time, by paying attention to the change in the driving speed or position of the vehicle to thin out the still images, it is possible to preferentially leave the still images suitable for video creation and thin out the images. (3) The predetermined still image may be a still image selected every predetermined number among the plurality of continuously captured still images. By configuring in this way, in order to periodically thin out the still images for each group, it is possible to avoid the situation where the still images are thinned out locally in a grouped manner. (4) The predetermined still image may be the latest one still image among the plurality of continuously captured still images. By configuring in this way, in order to periodically thin out the still images for each head of the group, video creation can be performed regularly. (5) The predetermined still image may be the still image at the time when the start of movement of the vehicle is detected or the subsequent still images. By configuring in this way, it is possible to record the surrounding conditions when the vehicle that has temporarily stopped at an intersection or the like starts moving again. (6) The driving state of the vehicle is a change in the speed or acceleration of the vehicle, a change in the position of the vehicle, or when the vehicle is a multi-wheel vehicle including a four-wheel vehicle, one or more operations of an accelerator pedal, a brake pedal, a clutch pedal, a parking brake lever, an AT change lever, and an MT shift lever, and may be determined based on at least one of them. By configuring in this way, it is possible to detect the start of movement of the vehicle and transmit a still image based on the information. In addition, by using a combination of changes in speed, position, etc., the reliability of the detection result can be improved. (7) When an event related to a collision of the vehicle is detected, the control unit may transmit all still images within a predetermined period including the predetermined still image. By configuring in this way, in the event of a traffic accident or the like, it is possible to comprehensively record the original still images taken by the in-vehicle camera for the periods before and after that. (8) The predetermined speed may be the speed during low-speed driving of the vehicle or the speed when the vehicle is stopped. By configuring in this way, as a result of the change in the driving position of the vehicle becoming relatively slow, when continuously shooting similar landscapes, it is possible to thin out the still images. In addition, when the reception strength (RSRP), transmission speed, and reception speed are in a relatively stable state and the need to secure a spare decreases, it is possible to thin out the still images. (9) The control unit may determine the transmission state of the communication processing unit based on the number of still images transmitted and the response time from the transmission of the still image until the transmission is completed. By configuring in this way, it is possible to monitor the change in the communication state of the vehicle during driving depending on the location where the vehicle is driving, the speed during driving, etc. Therefore, it is possible to transmit a still image under a relatively good communication situation. (10) The control unit may determine the transmission state of the communication processing unit based on the response time of data that is transmitted separately from the still image and is smaller in size than the still image. By configuring in this way, before transmitting a still image with a large data capacity, based on a ping or vehicle information or the like with a small data capacity, the communication state of the vehicle during travel can be determined in a relatively short time. (11) When the response times of two consecutive still images exceed a predetermined threshold, the control unit may save the subsequent still image in the storage device without transmitting it. By configuring in this way, under a deteriorated transmission situation, it is possible to avoid the continuous accumulation of the influence caused by the delay in transmitting still images by always continuing to transmit still images. (12) When the number of still images being transmitted is 0 and the response time of data that is transmitted separately from the still image and is smaller in size than the still image is below a predetermined threshold, the control unit may transmit the still image saved in the storage device. By configuring in this way, it is possible to transmit all the selected still images while avoiding the continuous accumulation of the influence caused by the delay in transmitting still images. (13) The communication processing unit may transmit data by an LPWA (Low Power Wide Area) method. By configuring in this way, by using a relatively low frequency, communication over a relatively long distance can be performed with relatively small power. (14) The electronic device may be a drive recorder. By configuring in this way, the state of the surroundings during travel can be recorded using the video captured and recorded by the drive recorder. (15) [ A receiving unit that receives the plurality of still images transmitted from the electronic device, A moving image creation unit that integrates the received plurality of still images to create a moving image, And a management server having the above. By configuring in this way, on the side of the electronic device, it is possible to reduce the burden of sending still images and eliminate the burden of creating videos. Furthermore, on the side of the management server, since it has relatively high processing specifications, it is possible to create a video based on the transmitted still images. For example, on the side of the management server, compared with the case of creating a video on the side of the electronic device, it is possible to perform more advanced video creation such as changing the time series of the video or creating a video that supplements the still images. (16) When creating the video by integrating the received plurality of still images, the video creation unit may create a video that is faster than the actual time series. By configuring in this way, it is possible to avoid giving an impression of slowdown to the viewer during video playback, and create a video that can quickly perform confirmation of driving records and the like. (17) When creating the video from the received plurality of still images, if the time interval between the plurality of still images has an interval exceeding a predetermined threshold, the video may be created by replicating the still image before the interval, or by generating a new still image based on the still images before and after the interval. By configuring in this way, it is possible to create a video that can avoid giving an unnatural impression to the viewer during video playback due to the occurrence of missing still images. (18) When the video creation unit replicates a still image or generates a new still image, the video creation unit may create the video with information indicating that there is no original image data displayed. By configuring in this way, it is possible to create a video that can clearly distinguish and use the actually captured still images from those that are not during a traffic accident or when confirming a driving route. (19) The receiving unit receives the plurality of still images and the position information of the vehicle at the time when the plurality of still images are created from the electronic device. When the moving image creation unit integrates the received plurality of still images to create a moving image, the vehicle position information may be embedded in the moving image so that the vehicle position information can be synchronously displayed based on the moving image. With such a configuration, when playing a moving image on the user terminal side, the vehicle position can be known using only the moving image. (20) A program executed on at least one computer, which may be a program that enables the execution of the above method. With such a configuration, by providing a program to an existing electronic device or management server, the above control can be performed.

Explanation of Reference Numerals

[0178] 101…Drive recorder (electronic device or in-vehicle device), 102…Management server, 103…User terminal, 301…Microcomputer, 302…Communication processing unit, 303…DVR controller (control unit), 304…Camera (imaging unit), 350…Control unit

Claims

1. An electronic device mounted on a moving body, comprising: a communication processing unit; a control unit that processes image information captured by a camera and transmits a plurality of still images via the communication processing unit; The control unit has a function of controlling the transmission of the plurality of still images based on the traveling state of the moving body. When it is detected that the speed of the moving body is within a predetermined speed range, the control unit has a function of selecting and transmitting a predetermined still image among the plurality of captured still images. The predetermined speed is the speed during low-speed travel of the moving body or the speed when the moving body is stopped.

2. An electronic device mounted on a moving body, comprising: a communication processing unit; a control unit that processes image information captured by a camera and transmits a plurality of still images via the communication processing unit; The control unit has a function of controlling the transmission of the plurality of still images based on the transmission state of the still images. The control unit has a function of determining the transmission state of the communication processing unit based on the number of still images transmitted and the response time from the transmission of the still images until the transmission is completed.

3. An electronic device mounted on a moving body, comprising: a communication processing unit; a control unit that processes image information captured by a camera and transmits a plurality of still images via the communication processing unit; The control unit has a function of controlling the transmission of the plurality of still images based on the transmission state of the still images. The control unit has a function of determining the transmission state of the communication processing unit based on the response time of data smaller in size than the still images, which is transmitted separately from the still images.

4. An electronic device mounted on a moving body, comprising: a communication processing unit; a control unit that processes image information captured by a camera and transmits a plurality of still images via the communication processing unit; The control unit has a function of controlling the transmission of the plurality of still images based on the transmission state of the still images. When the response times of two consecutive still images exceed a predetermined threshold, the control unit has a function of storing the subsequent still images in a storage device without transmitting them.

5. When the number of still images being transmitted is 0 and the response time of data smaller in size than the still images, which is transmitted separately from the still images, is below a predetermined threshold, the control unit has a function of transmitting the still images stored in the storage device. The electronic device according to Claim 4.

6. ​ ​ ​ The electronic device according to any one of claims 1 to 5, wherein the communication processing unit has a function of transmitting data by an LPWA (Low Power Wide Area) method.

7. The electronic device according to any one of claims 1 to 6, wherein the electronic device is a drive recorder.

8. A management server, comprising: a communication processing unit; a control unit that processes image information captured by a camera and transmits a plurality of still images via the communication processing unit; and a reception unit that receives the plurality of still images transmitted from the electronic device mounted on the moving body, wherein the control unit has a function of controlling the transmission of the plurality of still images based on at least one of a traveling state of the moving body and a transmission state of the still images. a moving image creation unit that integrates the received plurality of still images to create a moving image. A management server having the above components.

9. A management server, comprising: a reception unit that receives the plurality of still images transmitted from the electronic device according to any one of claims 1 to 7; a moving image creation unit that integrates the received plurality of still images to create a moving image. A management server having the above components.

10. The management server according to claim 8 or 9, wherein the moving image creation unit creates a moving image that is faster than the actual time series when integrating the received plurality of still images to create the moving image.

11. When creating the moving image from the received plurality of still images, if the time interval between the plurality of still images has an interval exceeding a predetermined threshold, the management server according to claim 8 or 9, wherein the moving image is created by replicating the still image before the interval or generating a new still image based on the still images before and after the interval.

12. The management server according to claim 11, wherein the moving image creation unit creates the moving image in which information indicating that there is no original image data is displayed when the still image is replicated or the new still image is generated.

13. The reception unit receives the plurality of still images and the position information of the moving body at the time when the plurality of still images are created from the electronic device, and when the moving image creation unit integrates the received plurality of still images to create a moving image, the position information of the moving body is embedded in the moving image, so that the position information of the moving body can be synchronously displayed based on the moving image. The management server according to claim 8 or 9.

14. ​ ​ A program for causing a control unit of an electronic device to realize the function according to any one of claims 1 to 7.

15. A program for causing a management server to function as the receiving unit and the moving image creation unit according to any one of claims 8 to 13.

Images