Electronic apparatus, server to animate still picture, program and the like

Abstract

To provide an electronic apparatus which transmits video information and has realizability and effectivity for creating a moving image, and a server, a program, etc.SOLUTION: The electronic apparatus comprises: a communication processing unit; and a control unit which processes picture information captured with a camera and transmits a plurality of still pictures via the communication processing unit. The control unit controls the transmission of the still pictures based on a traveling speed of a vehicle and / or a transmission state of the still pictures.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to an electronic device, a server that animates still images transmitted from the electronic device, and a program that enables a computer to implement the functions of the electronic device and the server. [Background technology]

[0002] In recent years, low-power wide area networks (hereinafter referred to as LPWA) known as LPWA (Low Power Wide Area) or LPWAN (Low Power Wide Area Network) have come into use as communication functions in fields such as IoT (Internet of Things). LPWA uses relatively low frequencies, enabling communication over relatively long distances with relatively little power.

[0003] Japanese Patent Application Laid-Open Publication No. 2020-088803 (Patent Document 1) is a background technology in this technical field. This publication states that "a remote monitoring control device is provided with a sensor, a transmitter using a low-power wide-area communication method that transmits sensor data, a receiver using a low-power wide-area communication method that receives the sensor data, and a remote monitoring control device that receives the sensor data from the receiver, wherein the remote monitoring control device associates the sensor data with sensor data acquisition information and stores them in a sensor data storage unit, determines a specific state based on the sensor data stored in the sensor data storage unit, and acquires from a video file storage unit a video file corresponding to a sensor data acquisition location indicated by the sensor data acquisition information associated with the sensor data determined to be in the specific state" (see Abstract).

[0004] Furthermore, Japanese Patent Laid-Open Publication No. 2019-129328 (Patent Document 2) is a background technology in this technical field. This publication states that "a high-definition video generation device 20 includes an original video receiving unit 21 that receives an original video 1, which is a high-definition video, a low-bit-rate video receiving unit 24 that receives a low-bit-rate video 2, a learning model generation unit 22 that generates a learning model M using the original video 1 received by the original video receiving unit 21 as training data, and a high-definition video generation unit 25 that generates a high-definition video 3 from the low-bit-rate video 2 received by the low-bit-rate video receiving unit 24 using the learning model M corresponding to the low-bit-rate video 2 from the learning model M generated by the learning model generation unit 22" (see Abstract). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2020-088803 [Patent Document 2] Japanese Patent Application Publication No. 2019-129328 Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention provides a feasible and effective electronic device, server, program, etc. for transmitting video information and creating moving images.

[0007] The applicant intends to obtain rights to configurations that achieve the effects achieved by the components disclosed in this specification and drawings, etc., through divisional applications, amendments, etc. For example, this specification discloses problems in which phrases such as "can" and "is possible" are read as "the problem is." Each problem is described as an independent one, and the applicant intends to obtain rights to the configurations that solve each problem separately through divisional applications, amendments, etc. Even if the problem is implicitly understood from the description in the specification, the applicant intends to include part of the configuration described in this specification in the scope of the patent claim through amendments or divisional applications. The applicant also discloses configurations that solve problems that combine these independent problems, and the applicant intends to obtain rights to them. [Means for solving the problem]

[0008] In order to solve the above problem, for example, the configuration described in the claims is adopted. The present application includes a plurality of means for solving the above problem, and one example thereof is an electronic device including a communication processing unit and a control unit that processes image information captured by a camera and transmits a plurality of still images via the communication processing unit, wherein the control unit controls the transmission of the plurality of still images based on at least one of a vehicle running state and a transmission state of the still images. [Effects of the Invention]

[0009] The present invention provides a feasible and effective electronic device, server, program, etc. for transmitting video information and creating moving images.

[0010] Problems, configurations, and effects other than those described above will become clear from the following description of the embodiments. Note that the effects of the present invention are not limited to these, and the effects achieved by the configurations disclosed in the present specification and drawings are also disclosed, and the applicant intends to obtain rights to the configurations that achieve these effects through divisional applications, amendments, etc. For example, in this specification, phrases such as "can" and "is possible" are descriptions that clearly indicate the effects achieved, and there are also parts that demonstrate the effects even without the phrases "can" and "is possible." Furthermore, there are effects that can be understood from the configurations even without such phrases. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 shows an example of the overall configuration of a DVR management system 1. [Figure 2] FIG. 2 shows an example of the connection configuration of the management server 102. [Figure 3] FIG. 3 shows an example of the hardware configuration of the drive recorder 101. [Figure 4] FIG. 4 shows an example of the hardware configuration of the management server 102. [Figure 5] FIG. 5 shows an example of the hardware configuration of the user terminal 103. [Figure 6] FIG. 6 shows an example of the data structure of DVR status information 600. [Figure 7] Figure 7 is a conceptual diagram of the video stream sent to the management server during actual driving. [Figure 8] Figure 8 is a conceptual diagram of multiple still images sent to the management server during actual driving. [Figure 9] FIG. 9 is a graph showing changes in driving speed, reception strength (RSRP), transmission speed, and reception speed during actual driving. [Figure 10] Figure 10 shows an example of the data structure when vehicle information and some of the still images are sent to the management server during actual driving. [Figure 11] Figure 11 shows examples of still images taken during actual driving of No. 1 to No. 8 in Figure 10. [Figure 12] Figure 12 shows examples of still images taken during actual driving of No. 9 to No. 16 in Figure 10. [Figure 13] Figure 13 shows examples of still images taken during actual running of No. 17 to No. 24 in Figure 10. [Figure 14] FIG. 14 shows examples of still images taken during actual driving of No. 15 to No. 18 in FIG. 10 and still images extracted from them. [Figure 15] FIG. 15 shows examples of still images taken during actual driving from No. 19 to No. 22 in FIG. 10, and still images extracted from them. [Figure 16] FIG. 16 shows an example of the data structure when ping, vehicle information, and some of multiple still images are sent to the management server during actual driving. [Figure 17] Figure 17, following on from Figure 16, shows an example of the data structure when pings, vehicle information, and some of the still images are sent to the management server during actual driving. [Figure 18] FIG. 18 shows an example of the transmission speed of a set of ping, vehicle information, and still image when the transmission status is good. [Figure 19] FIG. 19 shows an example of the transmission speed of a set of ping, vehicle information, and still image when the transmission condition deteriorates. [Figure 20] FIG. 20 shows an example of the transmission speed of a set of ping, vehicle information, and still images, in the state shown in FIG. 19, where control is performed to improve the transmission status of still images in particular. [Figure 21] FIG. 21 is a conceptual diagram showing an example of image analysis performed on a still image in (A) and (B). DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment shown below is one embodiment of the present invention, and the content of the present invention should not be interpreted as being limited based on the following description. 1 shows an example of the overall configuration of a drive recorder (DVR) management system 1. The DVR management system 1 includes a drive recorder 101, a management server 102, a user terminal 103, a data center 104, and a management terminal 105, all of which are connected via a network. The network may be wired or wireless, and each terminal can send and receive information via the network. The network is not limited to the Internet, and may be a combination of networks with different protocols.

[0013] The drive recorder 101 is a device that is mounted on, for example, an automobile and records video images while driving. The drive recorder 101 is an example of an in-vehicle device. However, the installation target is not limited to automobiles, and may be any moving body in which the drive recorder 101 can be installed. For example, the installation target may be automobiles, buses, trucks, motorcycles, etc. Furthermore, the installation target may also be transportation vehicles such as trains, monorails, linear motor cars, etc. The management server 102 is a server that stores and manages various information such as video information and status information acquired by the drive recorder 101. The information acquired by the drive recorder 101 may be configured to be transmitted directly from the drive recorder 101 to the management server 102, or may be configured to be transmitted to the management server 102 via a data center 104.

[0014] The user terminal 103 is a terminal used by, for example, a user who drives a car. The user terminal 103 is connected to the drive recorder 101 via a mobile data communication network or an in-vehicle network such as Wi-Fi (registered trademark) of the car, and can operate the drive recorder 101. Furthermore, the user terminal 103 is connected to the management server 102 via these networks and can display various information stored in the management server 102 .

[0015] The data center 104 receives various information acquired by the drive recorder 101 and transmits it to the management server 102. For example, the data center 104 is connected to the drive recorder 101 via an LPWA (Low Power Wide Area) network, receives various information transmitted from the drive recorder 101 via the LPWA network, and transmits the received information to the management server 102 via the Internet. LPWA wireless communication is capable of low power consumption and long-distance communication. However, limitations on the amount of data that can be transmitted per unit time can be an issue. For example, transmitting large amounts of data can result in significant communication delays. Examples of LPWA technologies that can be used include at least one of "ELTRES," "LoRa," "LoRaWAN," "LTE-MTC," "NB-IoT," "NB-Fi Protocol," "RPMA," "GreenOFDM," "DASH7," "RPMA," "Wi-SUN," "EnOcean Long Range," "Weightless-P," "SIGFOX," "LTE Cat.0," and "LTE Cat.M1." The management terminal 105 is a terminal that operates and manages the management server 102 .

[0016] Each terminal of the DVR management system 1 and the management server 102 may be, for example, a portable terminal (mobile terminal) such as a smartphone, tablet, mobile phone, or personal digital assistant (PDA), or a wearable terminal such as glasses, a wristwatch, or clothing. They may also be stationary or portable computers, or servers located on the cloud or a network. In terms of functionality, they may be VR (Virtual Reality) terminals, AR (Augmented Reality) terminals, or MR (Mixed Reality) terminals. Or, they may be a combination of multiple of these terminals. For example, a combination of one smartphone and one wearable terminal can logically function as one terminal. Other information processing terminals may also be used.

[0017] Each terminal and management server 102 of the DVR management system 1 includes a processor that executes an operating system, applications, programs, etc., a main storage device such as RAM (Random Access Memory), an auxiliary storage device such as an IC card, hard disk drive, SSD (Solid State Drive), flash memory, etc., a communication control unit such as a network card, wireless communication module, or mobile communication module, input devices such as a touch panel, keyboard, mouse, voice input, and input based on motion detection from image capture by a camera unit, and an output device such as a monitor or display. Note that the output device may also be a device or terminal that transmits information to be output to an external monitor, display, printer, device, etc.

[0018] The main memory stores various programs and applications (called modules), and the processor executes these programs and applications to realize the various functional elements of the overall system. These modules may be implemented in hardware, such as by integration. Each module may be an independent program or application, or may be implemented as a subprogram or function within a single integrated program or application.

[0019] In this specification, each module is described as the entity (subject) that performs the processing, but in reality, the processing is carried out by a processor that processes various programs, applications, etc. (modules). Various databases (DB) are stored in the auxiliary storage device. A "database" is a functional element (storage unit) that stores a set of data so that it can accommodate any data manipulation (e.g., extraction, addition, deletion, overwriting, etc.) from a processor or an external computer. The method of implementing the database is not limited, and may be, for example, a database management system, spreadsheet software, or a text file such as XML or JSON.

[0020] FIG. 2 shows an example of the connection configuration of the management server 102. In this connection configuration example, the drive recorder 101 transmits various information to the management server 102 via the data center 104 . The drive recorder 101 transmits various types of information about the drive recorder 101 to a data center 104 via a wireless communication network such as an LPWA network. The customer-side management server 202 receives and stores various types of information from the data center 104 via the Internet using a protocol such as MQTT (Message Queue Telemetry Transport). The management terminal 105 and the user terminal 103 access the customer-side management server 202 via the Internet or a mobile communication network, receive and display various types of information.

[0021] The same function can also be performed via the service provider's management server 201 of the service provider that manages the DVR management system 1. Once various information about the drive recorder 101 is stored in the service provider's management server 201, it is sent to the management terminal 105 or the user terminal 103 via the customer's management server 203 or directly. The various information transmitted and received may be encoded and decoded by each server. It should be noted that the drive recorder 101 may be configured to transmit various pieces of information directly to the management server 102 rather than via the data center 104 .

[0022] FIG. 3 shows an example of the hardware configuration of the drive recorder 101. The camera 304 captures images of the front, rear, or 360 degrees, and generates video information such as moving images and still images. By equipping the image sensor of the camera 304 with an AI processing function that performs preprocessing, the following output can be generated in addition to or instead of the normal generation of captured images. The AI ​​processing function has the function of performing image recognition based on the video captured by the camera 304. -Export objects from images as metadata. - Outputs images in ISP (Image Signal Processor) output format, such as YUV or RGB. - Output an image cropped from a specific area.

[0023] By performing preprocessing on the image sensor side in this way, it is possible to reduce the amount of data and enable real-time tracking of objects using high-speed AI processing. In particular, there is an advantage in that the amount of data (i.e., communication volume) sent from the drive recorder 101 to the management server 102 can be reduced. This is because, even when data is sent via an LPWA network as in this embodiment, it is expected that data related to the AI ​​processing function can be sent with low latency, for example.

[0024] The acceleration sensor 305 detects acceleration and generates acceleration information. For example, if a sudden change in acceleration occurs due to an impact, sudden steering, sudden stopping, etc., the DVR controller 303 detects the occurrence of an event. By analyzing this change in acceleration information, it is possible to infer the circumstances of an accident, such as an automobile accident. The SD card 306 is a storage device for recording video information and various other information. Note that the SD card is not limited to an SD card and any other configuration that can store data will do, but a flash memory with high vibration resistance is preferable for in-vehicle use.

[0025] The DVR controller 303 is connected to and controls the camera 304, acceleration sensor 305, SD card 306, and other sensors (not shown). Based on information received from these elements, the DVR controller 303 generates status information indicating the state of the DVR controller 303. The DVR controller 303 also transmits the generated status information to the microcomputer 301. The microcomputer 301 performs the following various processes. Controls the communication processing unit 302. The communication processing unit 302 notifies the DVR controller 303 of the acquired location information. The status information acquired from the DVR controller 303 and the position information acquired from the communication processing unit 302 are converted into, for example, 16-byte data. - Performs sleep control when powered by secondary battery. - Execute test mode during manufacturing and testing. The microcomputer 301 is programmable, and stores programs for executing the various processes described above. The various processes described above are realized by the execution of these programs by a processing unit of the microcomputer 301. In this embodiment, the processing unit of the microcomputer 301 or the microcomputer 301 itself may also be referred to as a control unit.

[0026] The communication processing unit 302 transmits information from the drive recorder 101 to the management server 102 via a wireless communication antenna 308, either directly or via the data center 104. The communication processing unit 302 performs wireless communication using the LPWA method, which is an example of narrowband communication. Although the LPWA method is described below, this may be replaced with other methods of narrowband communication, or may be replaced with a communication method other than narrowband communication. The communication processing unit 302 also has a location information acquisition unit implemented as an integrated circuit or the like, and acquires location information based on a signal received from a GNSS antenna 307 compatible with a GNSS (Global Navigation Satellite System) such as a GPS (Global Positioning System). The location information acquisition unit can also acquire time information, moving speed information, and moving direction information.

[0027] It should be noted that the communication processing unit 302 does not necessarily have to have the location information acquisition unit, and the GNSS antenna 307 may be connected to the microcomputer 301, and these may have the location information acquisition unit. Alternatively, the location information acquisition unit may be configured to exist independently and be connected to the microcomputer 301 or the communication processing unit 302 . Furthermore, although the time information is acquired by the location information acquisition unit, the microcomputer 301 may be configured to acquire the location information.

[0028] The drive recorder 101 has a power supply 311 connected to a vehicle power supply 310. The power supply 311 supplies power received (or sometimes referred to as power reception) from the vehicle power supply 310 to the DVR controller 303 and also charges the secondary battery 312. The secondary battery 312 is a rechargeable battery such as a lithium ion battery, a nickel-cadmium battery, a nickel-metal hydride battery, or a lead-acid battery. However, a non-rechargeable primary battery may also be used. When the accessory power supply (ACC power supply) is turned off, or when the supply of power from the vehicle power supply 310 to the power supply 311 of the drive recorder 101 is interrupted due to a fault such as a broken wire or a disconnected wire caused by an accident, or when the power supply 311 is unable to supply power due to a malfunction or the like, the secondary battery transmits (or may also be called power transmission or power supply) power to the microcomputer 301, the communication processing unit 302, etc., thereby maintaining at least some of the functions of the drive recorder 101.

[0029] The voltage supplied from the vehicle power supply 310 of an automobile is generally 12 V. The power supplied from the power supply 311 or the secondary battery 312 is stepped down to a voltage of, for example, 2 V via a DC / DC converter 313 and supplied to the microcomputer 301 and the communication processing unit 302 to drive them. The DVR controller 303, microcomputer 301, and communication processing unit 302 can each be implemented as an integrated circuit (chip) such as an SoC (System on a chip), or a configuration in which a plurality of these are implemented together on a single chip is also possible. In particular, in this embodiment, the microcomputer 301 and communication processing unit 302 operate in cooperation with each other, so they may be implemented together on a single chip.

[0030] In this embodiment, the microcomputer 301 and the communication processing unit 302 are configured so that their power supply is separated from the DVR controller 303, and by simply supplying power from the secondary battery to the microcomputer 301 and the communication processing unit 302, location information and various other information can be sent to the management server 102 without starting the DVR controller 303, thereby reducing power consumption. In particular, the drive recorder 101 can perform long-distance communication with low power consumption by performing LPWA wireless communication, which allows the drive recorder 101 to transmit status information of the drive recorder 101 to the remote management server 102 while extending the life of the secondary battery in the drive recorder 101 as much as possible. Furthermore, since the drive recorder 101 transmits status information to the management server 102 using LPWA communication, it is desirable to perform communication frequently or at short intervals while suppressing the amount of data. Therefore, the status information that the drive recorder 101 sends to the management server 102 is also based on the knowledge of the inventors of the present application and takes into consideration factors such as the characteristics of LPWA wireless communication and the priority of the information to be sent to the management server 102.

[0031] Furthermore, by configuring the communication processing unit 302 to include a location information acquisition unit and a GNSS antenna 307, it becomes possible to acquire location information and speed information by simply driving the low-power communication processing unit 302 without starting the DVR controller 303. Furthermore, by keeping the drive voltage of the microcontroller 301 and the communication processing unit 302 at a low voltage, such as 2V, compared to the drive voltage of the drive recorder 101, which is, for example, 12V, the power consumption used to generate and transmit the DVR status information 600 can be kept low.

[0032] By installing a small-capacity rechargeable secondary battery as in this embodiment and notifying the management server 102 of the state of the drive recorder 101 when it is powered by the secondary battery, it is possible to notify the administrator even if the user intentionally drives with the drive recorder 101 powered off. Also, it is possible to notify the administrator of a malfunction even when the main body cannot be turned on due to a malfunction. It is also possible to track theft.

[0033] FIG. 4 shows an example of the hardware configuration of the management server 102. The management server 102 is configured, for example, by a server placed on a cloud. The main memory device 401 stores programs and applications such as a DVR management module 410 and a video information management module 411, and the processor 403 executes these programs and applications to realize each functional element of the management server 102.

[0034] The DVR management module 410 acquires DVR status information indicating the status of the drive recorder 101 from the drive recorder 101, decodes it, and stores and manages it in the DVR status information 600 of the auxiliary storage device 402. The DVR management module 410 also transmits some or all of the information obtained by decoding the DVR status information 600 to the user terminal 103 or the management terminal 105 via push distribution or in response to a request from the user terminal 103 or the management terminal 105.

[0035] The video information management module 411 acquires the video information stored in the drive recorder 101 from the drive recorder 101, stores it in the video information 420 in the auxiliary storage device 402, and manages it. The video information management module 411 also transmits video information 420 to the user terminal 103 or the management terminal 105 by push distribution or in response to a request from these terminals. The moving image creation module 412 performs processing to create a moving image by integrating a plurality of still images received from the drive recorder 101 or the like. The auxiliary storage device 402 stores the DVR status information 600, the video information 420, and the like.

[0036] FIG. 5 shows an example of the hardware configuration of the user terminal 103. The user terminal 103 is configured as a terminal such as a smartphone, a tablet, a notebook PC, or a desktop PC. The main memory device 501 stores programs and applications such as a management server collaboration module 510 and a DVR collaboration module 511, and the processor 503 executes these programs and applications to realize each functional element of the user terminal 103.

[0037] The management server cooperation module 510 cooperates with the management server 102, acquires the DVR status information 600 and the video information 420 stored in the management server 102, and displays this information on an output device 505 such as a display. Furthermore, the management server cooperation module 510 cooperates with the management server 102 and can perform various settings for the management server 102 . The DVR link module 511 operates the drive recorder 101 and configures the drive recorder 101 when the drive recorder 101 and the user terminal 103 are connected via an in-vehicle network such as Wi-Fi (registered trademark).

[0038] The management terminal 105 can also be configured in the same way as the user terminal 103, and can operate the drive recorder 101 and management server 102 and configure them using a DVR collaboration module and a management server collaboration module stored in the main memory device.

[0039] FIG. 6 shows an example of the data structure of DVR status information 600. DVR status information 600 is a diagram that explains the data structure when the microcomputer 301 of the drive recorder 101 transmits status information obtained from the DVR controller 303 and location information obtained from the location information acquisition unit to the management server 102. The DVR status information 600 may also be referred to as status information regarding the drive recorder 101.

[0040] In this embodiment, the microcomputer 301 generates 16-byte data of the DVR status information 600, and the communication processing unit 302 stores it in the payload and transmits it to the management server 102 based on the LPWA protocol. The protocol used for transmission is not limited to LPWA, and other communication methods and protocols may be used.

[0041] Item 610 indicates the item of the DVR status information 600 to be transmitted. Type 1 (620), Type 2 (630), and Type 3 (640) indicate three types of transmitted data content, and the numbers listed in each table indicate the data bit length. The total number of data bit lengths for each Type is 128 bits, or 16 bytes. In addition, there is a free format spare data configuration other than that described in item 610, called Type 4 (not shown).

[0042] In this embodiment, when the vehicle power supply 310 is being supplied, the communication processing unit 302 transmits the DVR status information 600 to the management server 102 at one-minute intervals. The microcomputer 301 acquires information such as location information, speed information, and recording status at three different times during this one interval, stores the location information for these three points in time in a 16-byte payload, and can send the information to the management server 102 in a single transmission process. Type 1 is a format for transmitting location information etc. at one point in time. Type 2 is a format for transmitting location information etc. at two points in time together. Type 3 is a format for transmitting information etc. at three points in time together.

[0043] Referring again to FIGS. 1 and 3, one embodiment will be described in detail. 1 shows a drive recorder 101 as a specific example of the embodiment. However, the embodiment is not limited to the drive recorder 101. For example, the embodiment can be implemented as other in-vehicle devices such as a detector (radar). In a broad sense, the embodiment can relate to an electronic device that acquires video information. Hereinafter, a drive recorder 101, which is an in-vehicle device, will be described as an example of an electronic device according to this embodiment.

[0044] The drive recorder 101 acquires video information of the surrounding scenery (for example, the front, rear, side, diagonally forward, diagonally backward, or any angle range up to 360° around the vehicle) while the vehicle is traveling or stopped, using a camera (photographing unit) 304. The video information or image information is captured in the form of a moving image or multiple still images.

[0045] The drive recorder 101 is an example of an electronic device that incorporates a camera 304. For example, the drive recorder 101 can incorporate a camera to capture images of the surrounding scenery and record (save) the video information. On the other hand, if the electronic device is implemented as a detector (radar), an electronic device with an external camera, or a car navigation system, it can communicate with or connect to an external camera to acquire video information captured by other in-vehicle devices.

[0046] The camera 304 generates a file in video format and acquires captured still images from the file. Alternatively, the camera 304 may capture still images (photographs) from the beginning and acquire the still images directly. The still images may be in JPEG format, for example, but may be in other formats. The video may be in MPEG format (MPEG2, MPEG4, etc.), for example, but may be in other formats. Note that a video can be understood as a plurality of still images arranged in chronological order, and therefore, still images will be described below as an example of image information captured by the camera 304. The camera 304 is not limited by differences in the size of the camera (small, large, etc.), the color of the still image captured by the camera (black and white, full color, etc.), or the camera's shooting method (normal shooting, infrared shooting, etc.).

[0047] The drive recorder 101 preferably uses an LPWA communication processing unit 302 to transmit data of captured still images to an external management server 102. The use of LPWA enables relatively long-distance communication with relatively low power consumption. On the other hand, LPWA has the disadvantage that the data transmission speed is slow and it is not suitable for exchanging large amounts of data. Therefore, the drive recorder 101 performs special control to take advantage of the advantages of LPWA while suppressing its disadvantages, and this control can be performed by, for example, the microcomputer 301 and / or the DVR controller 303, etc.

[0048] An electronic device (for example, drive recorder 101) includes at least a communication processing unit 302 and a control unit (microcomputer 301 and / or DVR controller 303, etc.) that processes image information captured by a camera 304 and transmits multiple still images via the communication processing unit 302. Hereinafter, the microcomputer 301 and / or DVR controller 303, etc. will be collectively referred to simply as control unit 350. The still image is transmitted to an external management server 102, and the control on the management server 102 side will be described later.

[0049] "Video transmission test (Test 1)" In order to understand the characteristics of transmitting data using the LPWA method, the applicant carried out the following tests 1 to 4. First, the applicant conducted a test in which the drive recorder 101 transmitted a video stream captured by the camera 304 to the external management server 102 via the communication processing unit 302. As would be understandable to a person skilled in the art, in order to transmit a video stream in real time using the communication processing unit 302, the video stream needs to be transmitted stably. Generally, in order to transmit a video stream in real time, it is necessary to transmit a video stream with a bandwidth of 300 Kbps or more (ideally, about 1 Mbps).

[0050] Therefore, the applicant conducted a test in which a video stream with a bandwidth of 300 Kbps captured by the drive recorder 101 was transmitted from a vehicle in motion via the communication processing unit 302 compatible with LPWA Cat.M1. As a result, degradation of the playback screen and frequent disconnections were observed on the receiving side (the management server 102 in this embodiment).

[0051] FIG. 7 is a diagram for explaining the outline of the contents of the above-mentioned test 1. With the horizontal axis representing time series, a video stream with a bandwidth of 300 Kbps was captured and attempted to be transmitted, as shown within the rectangular areas 710 and 720. As illustrated in FIG. 7, the transmission speed varies depending on the surrounding environment. When the transmission speed of the first video stream, illustrated by reference numeral 710, deteriorated, the file transfer failed and the transmission was interrupted (see reference numeral 712). When the connection was retried (see reference numeral 722), the deterioration in the transmission speed of the second video stream 720 did not result in a transmission interruption, but the image became distorted (see reference numeral 724).

[0052] As described above, a stable communication speed environment is essential for the drive recorder 101 to transmit the video stream captured by the camera 304 in real time. However, it has been found that the LPWA Cat.M1 specifications as of January 2021 pose practical issues when transmitting the video stream captured by the camera 304 as is.

[0053] In relation to this, Patent Document 1 states that "the LPWA communication method used by the LPWA communication system 1 cannot ensure sufficient communication capacity, and therefore has the problem of not being able to transmit large amounts of video data, such as high-resolution still images and videos, in real time (see paragraph 0023)." In other words, Patent Document 1 states that because video information captured by an in-vehicle camera cannot be transmitted sufficiently in LPWA format, other means must be used.

[0054] Furthermore, Patent Document 2 discloses uploading low-bit-rate video to the cloud in real time using narrowband communication such as LPWA. However, Patent Document 2 requires converting the original video captured by the in-vehicle camera into low-bit-rate video and then converting this low-bit-rate video into high-definition video. The purpose of a drive recorder is to provide evidence images in the event of a traffic accident. Therefore, it is undesirable that the captured original video and the resulting high-definition video cannot be strictly matched one-to-one.

[0055] "Test of sending still images (Test 2)" Next, the applicant conducted a test in which the drive recorder 101 transmitted a plurality of still images captured by the camera 304 to the external management server 102 via the communication processing unit 302. Specifically, the applicant conducted a test in which still images (JPEG files) of approximately 60 Kbytes captured by the drive recorder 101 were transmitted via Cat. M1 connection HTTP (TCP / IP) in an actual traveling vehicle. As a result, the transmission was not interrupted (or the frequency of interruptions was extremely low) on the receiving side (management server 102), and satisfactory results were obtained from a practical standpoint.

[0056] FIG. 8 is a diagram for explaining the outline of the contents of the above-mentioned test 2. With the horizontal axis representing time series, multiple still images 810 to 860 of approximately 60 Kbytes were captured and transmitted. It was confirmed that transmission was possible in approximately 3.5 seconds when transmission conditions were good (see reference numeral 810). When transmission conditions deteriorated, a delay of approximately 8 seconds in transmission speed was observed (see reference numeral 830), but even in this case, transmission was not interrupted. In this way, when multiple still images were transmitted while actually driving, it was confirmed that the time required to complete transmission varied depending on the line conditions, but the files could be reliably transmitted to the receiving side (management server 102) without causing file corruption or the like.

[0057] The results of these two tests 1 and 2 confirmed the effectiveness of transmitting multiple still images taken by the camera 304 periodically to the external management server 102, rather than simply transmitting the video stream captured by the camera 304, and then converting these still images into video on the management server 102 side.

[0058] The management server 102 aligns multiple still images in chronological order to create a moving image. If there is a gap between each still image (for example, about one second to several seconds, e.g., about five seconds), the smoothness of the moving image will be lost. However, this is not a major problem, since the role required of the drive recorder 101 is to secure recorded images in the event of a traffic accident, etc., and to confirm the actual driving route, etc. In fact, by collecting still images discontinuously in chronological order and creating a moving image, the overall data volume can be reduced.

[0059] "Test of sending still images and vehicle information (Test 3)" Furthermore, the applicant conducted a test of still image transmission under different conditions, in which still images (relatively large data size) captured by the drive recorder 101 and vehicle information (relatively small data size) such as location information were separately transmitted from a vehicle in actual motion to an external management server 102. The information transmitted from the drive recorder 101 to the external management server 102 includes not only still images captured by the camera 304, but also other vehicle information (here, text data) that similarly requires real-time performance, such as G-sensor events detected by an acceleration sensor 305, vehicle position information received from a GNSS antenna 307, and vehicle speed (see Figures 3 and 6).

[0060] Test 3 was conducted under different transmission conditions from Test 2 above. In this test, multiple still images (each still image was approximately 50 Kbytes in size) taken by camera 304 while the vehicle was traveling were transmitted to external management server 102 in LPWA format. As a result, it was confirmed that the time required to complete transmission of each still image varied from 3 seconds to over 10 seconds depending on the line conditions, and that stable transmission at intervals of several seconds was not always possible. It was also confirmed that when line conditions were particularly poor, the transmission time became longer, and there was a risk of a decrease in real-time performance.

[0061] In contrast, vehicle information such as location information is relatively small compared to still images, and therefore transmission speeds are faster. In Test 3, it was confirmed that, under the same conditions, transmission of vehicle information with a small data size consisting of characters and numbers was completed in several hundred milliseconds, demonstrating good real-time performance. In this way, it was confirmed that the probability of transmission failure is lower in the case of vehicle information compared to still images, because the data size is relatively small.

[0062] Therefore, the results of actual tests 1 to 3 confirmed the effectiveness of separately transmitting the still images captured by the camera 304 and the vehicle information to the external management server 102. By transmitting the still images and the vehicle information separately, the volume of each piece of data transmitted by the communication processing unit 302 can be minimized. Furthermore, by focusing on the transmission speed of the vehicle information transmitted independently, it is possible to determine whether the transmission status of the communication processing unit 302 is good or not. For this purpose, it is possible to use a ping (pin or ping) which is an even smaller signal to check the reachability of data.

[0063] "Test of reception strength, transmission speed and reception speed (Test 4)" Furthermore, the applicant conducted tests to measure changes in the running speed, reception strength (RSRP: Reference Signal Receive Power), transmission speed, and reception speed in a vehicle that was actually running. FIG. 9 shows the change in communication speed (iperf every 30 seconds) corresponding to CAT.M1 during actual driving, which was carried out by the applicant.

[0064] The vehicle speed during the test is shown at the bottom of Fig. 9 (see reference numeral 910). The vehicle speed changes depending on the surrounding conditions, etc., and the received signal strength (RSRP) shown second from the bottom changes accordingly (see reference numeral 920). It was found that when the reception strength (RSRP) decreases, the reception speed (see reference numeral 930) shown second from the top and the transmission speed (see reference numeral 940) shown at the top decrease. On the other hand, when the vehicle is stopped (travel speed is 0 km / h), the reception strength (RSRP) stabilizes, and the transmission and reception speeds become stable. Note that the graph shows the average sending and receiving speeds over approximately 15 seconds measured using the communication performance measurement tool Iperf, so there is a possibility that speeds below 100kbps may momentarily be close to zero.

[0065] Based on the results of the above tests 1 to 4, this embodiment makes special efforts to minimize the disadvantages while taking advantage of the advantages of the LPWA used in the communication processing unit 302. That is, the control unit 350 of the drive recorder 101 does not transmit all of the still images captured by the camera 304, but controls the transmission of multiple still images based on at least one of (a) the vehicle's running state and (b) the transmission state of the still images (here, some of the multiple still images are selectively transmitted).

[0066] Preferably, all vehicle information such as location information may be transmitted from the drive recorder 101 to the management server 102 because it is relatively small in size. This allows for comprehensive recording of time-series changes in the vehicle's location and speed. On the other hand, depending on the vehicle's driving conditions, not all still images are necessarily required. The role required of the drive recorder 101 is to secure recorded images in the event of a traffic accident, etc., and to confirm the actual driving route, etc., so some still images can be thinned out during normal driving when no traffic accidents or the like have occurred.

[0067] (a) Control based on vehicle driving conditions Preferably, when the control unit 350 of the drive recorder 101 detects at least one of the following: that the vehicle speed is within a predetermined speed range, or that the change in the vehicle's position is within a predetermined distance range, the control unit 350 selects and transmits a predetermined still image from among the multiple still images taken.

[0068] For example, when a vehicle is traveling at a relatively high legal speed (for example, the vehicle is traveling at a speed of 60 km / h or more), it can be predicted that each still image captured by camera 304 will capture a different scene. In this case, if some of the still images are thinned out, the traveling route will suddenly change, which is likely to give the viewer a sense of incongruity. On the other hand, when the vehicle is temporarily stopped due to traffic congestion or the like (for example, the vehicle's traveling speed is 0 km / h or the change in vehicle position is 0 km), the still images captured by camera 304 may each capture a substantially similar scene. In this case, even if some of the multiple still images are selectively removed and the remaining still images are animated, the continuity of the scenery is relatively not lost, and the viewer is unlikely to feel uncomfortable.

[0069] Generally, when image quality is low, the reliability of each individual still image decreases, so there is a risk in reducing the number of still images recorded as evidence. However, as shown in Figure 9, when the vehicle is stopped (here, the traveling speed is 0 km / h), the reception strength (RSRP) is stable, and the transmission and reception speeds are also stable. In this state, the image quality of the captured still images is relatively good (Test 4). Therefore, it can be considered that there is less risk in thinning out some of the multiple still images captured by a stopped vehicle. This is because the reliability of each still image is high, which further reduces the need to prepare spare images.

[0070] Therefore, in one embodiment, when the control unit 350 of the drive recorder 101 determines that the vehicle is traveling at a low speed or at a stop, it selects only some still images from the multiple images captured by the camera 304 and transmits them to the external management server 102. The control unit 350 may determine that the vehicle is traveling at a low speed or at a stop when the vehicle speed is within a predetermined speed range or a predetermined movement range. The vehicle's traveling at a low speed may be determined, for example, when the vehicle speed is 2 km / h or less. The vehicle's stop may be determined, for example, when the vehicle speed is 0 km / h. This embodiment will be described below with reference to FIGS.

[0071] "Test of sending still images (Test 5)" From this perspective, the applicant conducted a test in which a plurality of still images taken by the camera 304 were selectively transmitted from a vehicle in motion. Referring to FIG. 10, the results of Test 5 above are tabulated.

[0072] The leftmost column in Figure 10 shows the order in which each still image was captured over time. In this example, a total of 24 still images were captured. These 24 still images are specifically shown in Figures 11 to 13. As can be seen from Figures 11 to 13, this embodiment can be effectively applied to checking the route a vehicle is traveling, etc. The file names of the still images are shown in the second column from the left in Figure 10. For example, in Example No. 1, the still image was captured at 02:41:55 on December 17, 2020. The control unit 350 automatically selected the file name "20201217-024155" to match the capture time. In this way, preferably, each still image is sent to the management server 102 with the shooting date and time as the file name, which allows the management server 102 to smoothly manage the data of each still image.

[0073] The third and fourth columns from the left in Figure 10 show the latitude and longitude of the vehicle at the time each still image was taken. This information may be obtained, for example, based on signals received from the GNSS antenna 307. The fifth column from the left in Figure 10 shows the vehicle speed at the time each still image was captured. For example, the first example in No. 1 shows "21," which means 21 km / h (kilometers per hour). The unit may be meters per second, an SI unit. Alternatively, miles per hour or feet per second may be used.

[0074] The sixth column from the left in Figure 10 shows the still image time difference as the time interval between two adjacent still images. In the example shown in Figure 10, each still image was basically taken at an interval of 5 seconds. This interval is not necessarily constant and can be longer, such as 6 seconds, or shorter, such as 4 seconds. It is also possible to take each still image at different intervals. By recording these still image time differences, the relative relationship between these still images can be understood. For example, when organizing multiple still images into a chronological order, they can be aligned according to their corresponding shooting times.

[0075] The rightmost column in Fig. 10 shows which still images were not transmitted by the control unit 350. Specifically, in the example of Fig. 10, focusing on the vehicle's traveling speed, it can be seen that still images No. 7, No. 8, No. 10, No. 11, No. 13, No. 14, No. 16, No. 17, and No. 19 were not transmitted. As described above, the still image and the vehicle information (latitude, longitude, etc.) are transmitted separately by the control unit 350.

[0076] Next, referring to Figures 14 and 15, we will specifically explain the case where, among multiple still images (No. 15 to No. 22) within the range indicated by symbol 1000 in Figure 10, still images (No. 15, No. 18, No. 20, No. 21, No. 22) that meet specified conditions are selectively transmitted.

[0077] The left columns of Figures 14 and 15 show examples of still images taken within the range from No. 15 to No. 22 in Figure 10. The still images were taken continuously from top to bottom at a predetermined interval (approximately every 5 seconds). 14 and 15 show a case where a still image that meets a predetermined condition is selected from the plurality of still images. Still images that are not selected (No. 16, No. 17, No. 19) are not sent to the management server 102. For ease of explanation, they are referred to as "no image" here.

[0078] Specifically, the range from No. 15 to No. 19 corresponds to a situation where the vehicle is stopped at an intersection at 0 km / h (or with a position change of 0 m) while waiting for the traffic light to change. Next, the range from No. 20 to No. 22 corresponds to a situation where the vehicle has restarted, entered the intersection at a low speed, and is making a left turn.

[0079] When a vehicle is stopped, there is generally little relative change between the vehicle and the surrounding scenery. However, the movement of other vehicles and people in the vicinity is excluded. Therefore, when the control unit 350 of the drive recorder 101 determines that the vehicle is traveling at a low speed or is stopped, it selects only some still images that meet predetermined conditions from among the multiple images captured by the camera 304 and transmits them to the external management server 102. In this example, it selects and transmits still images (No. 15 and No. 18) from among the multiple still images (No. 15 to No. 19).

[0080] The still images that meet the predetermined conditions are still images that are selected every predetermined number of still images from a plurality of still images that are continuously shot. Preferably, the still image that meets the predetermined condition is at least one still image out of at least three still images that are taken consecutively. This allows the control unit 350 to regularly thin out still images for each group of a predetermined number of images, thereby reducing the burden on transmission. This is particularly effective when the communication processing unit 302 transmits data using the LPWA system, as the data volume during transmission is an issue. Also, by deleting two out of every three images, the difference in playback speed with the video of the vehicle running can be eliminated, effectively compressing the video data size.

[0081] More preferably, the still image that meets the predetermined condition is the most recent still image among a plurality of still images that have been taken consecutively. Alternatively, the still image that meets the above-mentioned specified conditions is the most recent still image among at least three still images taken in succession, and in the example of Figures 14 and 15, one still image is transmitted every 15 seconds. This allows the control unit 350 to more regularly thin out still images for each group of a predetermined number of images, so that when the selected still images are converted into a video, the unnaturalness caused by missing still images can be relatively reduced. Also, accidents and the like generally occur immediately after a vehicle waiting at a traffic light at an intersection starts to move. Therefore, by keeping the most recent still image, the still image immediately after the vehicle starts to move is selected.

[0082] More preferably, the still image that satisfies the predetermined condition is the most recent still image among a plurality of still images taken continuously when the vehicle speed is continuously at or below a low speed. For example, in the embodiment shown in Figures 10 to 15, the above-mentioned predetermined condition is set as follows: "If the vehicle speed is continuously low (for example, 2 km / h) or less for three consecutive files, the third file, which is the most recent in terms of time, is transmitted, and the remaining two files are not transmitted."

[0083] More preferably, the still image that satisfies the predetermined condition is a still image taken at or after the time when the vehicle starts to move, among a plurality of still images taken consecutively. For example, in the example shown in Figures 10 to 15, the vehicle speed changes from 0 km / h to 7 km / h (i.e., 2 km / h or more) in No. 20, and the start of vehicle movement is detected. Therefore, the still image No. 20, which corresponds to the time when the vehicle starts to move or after the time when it is detected, is selected and transmitted.

[0084] It is generally said that traffic accidents are likely to occur immediately after a vehicle starts moving at an intersection, etc. By performing the above control, it is possible to preferentially extract a still image that captures the surrounding situation immediately after the vehicle starts moving. For example, the second image from the top in FIG. 15 (No. 20 in FIG. 10) records a vehicle entering an intersection and a bicycle or other vehicle attempting to cross the crosswalk at the intersection. Therefore, the results of Test 5 above show that this embodiment is suitable for a drive recorder 101 that is required to record evidence images in the event of a traffic accident, etc., because it not only periodically thins out still images but also preferentially extracts still images that capture changes in the surrounding situation.

[0085] As described above, the drive recorder 101 prioritizes the extraction of highly necessary images while periodically thinning out less necessary images, thereby reducing the overall required image capacity (transmission volume or recording volume) (see FIGS. 14 and 15). In actual test 5, it was confirmed that by performing the above control, data reduction of 20% or more was possible for driving in a city. Therefore, it was found that the drive recorder 101 is suitable for transmitting data using the LPWA system.

[0086] However, when an event related to a vehicle collision or the like is detected, the control unit 350 of the drive recorder 101 records or transmits all still images within a predetermined period of time. Such an event can be detected, for example, by a G sensor or the like. For example, in the column of still images shown on the left side of FIGS. 14 and 15, if an event related to a vehicle collision or the like occurs, all still images No. 15 to No. 22 may be selected and transmitted.

[0087] When an event related to a vehicle collision is detected, the control unit 350 of the drive recorder 101 may transmit all still images within a specified period of time, from a time point in the past prior to the time of detection to a time point in the future prior to the time of detection. For example, when an event related to a vehicle collision is detected, the control unit 350 of the drive recorder 101 may transmit all still images from a past point in time that is a predetermined period (e.g., several seconds, several tens of seconds, one minute, several minutes, or more) before the time of detection to a future point in time that is a predetermined period (e.g., several seconds, several tens of seconds, one minute, several minutes, or more) after the time of detection.

[0088] In this case, still images that are not transmitted by the control unit 350 may not be deleted immediately, but may be temporarily stored in a transmission buffer and then deleted after a time lag (for example, several seconds, several tens of seconds, one minute, several minutes, or more). This makes it possible to record all still images taken during the period before and after the accident, including the moment of the accident, when a traffic accident occurs.

[0089] Strictly speaking, the still image capture time and the still image transmission time do not coincide. For example, the still image transmission time is slightly (about 1 second) later than the still image capture time. Therefore, the control unit 350 may transmit a still image if the vehicle speed at the transmission time or the speed at the still image capture time does not satisfy the low speed condition.

[0090] Furthermore, in this method, if there is a slight delay between the timing of still image capture and transmission due to internal processing time or the like (or transmission is intentionally delayed), and a stopped (or low-speed) state is confirmed by GNSS or vehicle speed pulses at both times, the control unit 350 may discard the transmission of the still image two out of three times, thereby reducing the amount of transmitted data and lowering communication charges.

[0091] Regarding "prescribed speed range" Preferably, the "predetermined speed range" relates to when the vehicle is traveling at low speed or when the vehicle is stationary. Compared to normal legal speeds (for example, about 60 km / h), when driving at low speeds, the surrounding scenery changes more slowly, which increases the tendency for duplicate data and unnecessary data to accumulate. In particular, when the drive recorder 101 communicates wirelessly using LPWA, it is preferable to reduce the burden on transmission. Furthermore, the incidence of traffic accidents has been decreasing in Japan in recent years. Therefore, video of vehicles traveling at low speeds is selectively recorded.

[0092] In the above example, the predetermined speed range is set when the vehicle is traveling at a low speed of 2 km / h or less or when the vehicle is stopped at 0 km / h. However, this embodiment is not limited to this speed range. The traveling speed of the vehicle varies depending on the surrounding conditions, etc. The upper limit of the predetermined speed range can be changed in various ways depending on the embodiment.

[0093] Preferably, the predetermined speed range should include a stopped state (for example, 0 km / h). Generally, a vehicle may come to a complete stop (0 km / h) temporarily while waiting for a traffic light to change at an intersection, while waiting for a train to pass at a railroad crossing, or during heavy traffic congestion. In such cases, it is believed that a preceding vehicle or a following vehicle cannot stop completely, and a rear-end collision is likely to occur.

[0094] Preferably, the predetermined speed range may be an extremely slow speed (several kilometers per hour or less, for example, 2 kilometers per hour or less). Generally, when a vehicle is traveling at an extremely slow speed, there are situations where the vehicle is forced to travel slowly in order to maintain an appropriate distance from the vehicle ahead, such as in a traffic jam. In such cases, it is believed that the driver is more likely to be distracted or inattentive to the road ahead due to, for example, the operation of a car navigation system or a mobile phone.

[0095] The predetermined speed range may also be slow driving (for example, 10 km / h or less). Generally, when a vehicle is traveling slowly (for example, 10 km / h or less), there are situations where caution is required when driving, such as when traveling on narrow roads or entering intersections. At this speed, it is said that the stopping distance of the vehicle can change by several meters depending on whether the driver is driving while anticipating danger.

[0096] Furthermore, the predetermined speed range may be low speed (for example, 30 km / h or less). Generally, it is said that traffic accidents are more likely to occur when a vehicle is traveling at low speed (for example, 30 km / h or less). For example, in Japan, statistically, three-quarters of all traffic accidents occur at speeds of 30 km / h or less.

[0097] Therefore, in consideration of the role of the drive recorder 101 in capturing recorded footage of traffic accidents, the upper limit of the vehicle's traveling speed "low speed traveling" may be 30 km / h or less, or 10 km / h or less, or preferably a few kilometers per hour or less (for example, 2 km / h or less), and the lower limit is preferably 0 km / h.

[0098] "Vehicle begins to move" A change in the speed or acceleration of the vehicle can be used as a trigger for detecting the start of vehicle movement. For example, the control unit 350 may determine that the vehicle has started to move when an acceleration of the vehicle that exceeds a predetermined threshold (e.g., 0) from 0 is detected based on an input signal from a speed sensor or an acceleration sensor of the vehicle.

[0099] Furthermore, for example, the control unit 350 may calculate the vehicle speed based on the signal received by the GNSS antenna 307. In this case, depending on the reception environment of the signal, a positioning error may occur, and the vehicle speed may be calculated as being faster by the amount of this positioning error. For example, even if the actual vehicle speed is 0 km / h, it may be calculated as a speed other than 0 km / h. Therefore, the control unit 350 may determine the predetermined speed range taking this positioning error into consideration. For example, the control unit 350 may determine that the vehicle is stopped when the speed is 7 km / h or less or 8 km / h or less.

[0100] Furthermore, the control unit 350 can use a change in the vehicle's position as a trigger for detecting the start of vehicle movement. For example, the control unit 350 may determine that the vehicle has started to move from position information acquired based on a signal received from the GNSS antenna 307. For example, generally, vehicle latitude and longitude information from a GPS may contain an error of about 7 km or 8 km. Therefore, the control unit 350 may recognize that the vehicle has started to move when a change in the vehicle's position of more than 7 km or 8 km is detected.

[0101] The change in the vehicle's speed or acceleration and the change in the vehicle's position may be used in combination. Although it is difficult to detect the start of vehicle movement with high accuracy based on the signal received from the GNSS antenna 307, combining the vehicle's speed or acceleration can improve reliability. For example, the control unit 350 may improve the reliability of the information by comparing (cross-checking) the change in the vehicle's speed or acceleration over time with the change in the vehicle's position over time.

[0102] The trigger for detecting the start of vehicle movement is not limited to when the vehicle actually starts moving and at least changes its position, but may include the moment when the vehicle state switches from a stopped state to a moving state or the state immediately before, based on a signal sent from an in-vehicle device, even if the position of the vehicle has not actually changed.

[0103] For example, in the case where the vehicle is a multi-wheeled vehicle including a four-wheeled vehicle, the control unit 350 can use one or more of the accelerator pedal, brake pedal, clutch pedal, parking brake lever, AT change lever, and MT shift lever as a trigger to detect the start of movement if changes in the operation of one or more of these can be supplied as signals to the drive recorder 101.

[0104] For example, when a vehicle restarts after a temporary stop at an intersection, etc., the brake pedal is generally released from a predetermined depression amount, and the depression amount of the accelerator pedal of the vehicle increases by a predetermined amount from zero. Therefore, the start of movement of the vehicle may be detected based on the depression amount of the accelerator pedal and / or the brake pedal. When restarting the vehicle on a slope, the parking brake lever may be further operated. Therefore, the start of vehicle movement may be detected based on the operation of the parking brake lever.

[0105] In the case of a manual transmission vehicle, when restarting from a temporary stop at an intersection, etc., in addition to braking, the clutch and manual shift lever may be operated. Therefore, the start of vehicle movement may be detected based on these operations. In the case of an automatic transmission vehicle, the automatic transmission shift lever may be operated when restarting the vehicle after a temporary stop at an intersection, etc. Therefore, the start of vehicle movement may be detected based on the operation of the automatic transmission shift lever.

[0106] In addition, if the vehicle is a two-wheeled vehicle, sidecar, or trike (three-wheeled vehicle), the control unit 350 can use one or more of the accelerator grip, front wheel brake lever, rear wheel brake lever, clutch lever, and shift pedal as a trigger to detect the start of movement, if changes in the operation of one or more of these can be supplied as signals to the drive recorder 101.

[0107] For example, when a vehicle starts moving again after a temporary stop at an intersection, etc., the amount of grip of the accelerator grip of the vehicle may generally increase by a predetermined amount from zero, or the front wheel brake lever and / or the rear wheel brake lever may be released from a predetermined amount of depression. Therefore, the start of movement of the vehicle may be detected based on the accelerator grip, the front wheel brake lever and / or the rear wheel brake lever. Unlike automatic transmission vehicles, manual transmission vehicles may require not only brake operation but also clutch and shift pedal operation when restarting from a temporary stop at an intersection, etc. Therefore, the start of vehicle movement may be detected based on these operations.

[0108] Additionally, the control unit 350 may detect a change in the state of the vehicle, such as temporary stopping or restarting, based on a signal from any device mounted on the vehicle. For example, in the case of a vehicle equipped with an idling stop function that stops the engine while the vehicle is parked or stopped, waiting at a traffic light, etc., if the control unit 350 can receive a signal indicating the start or end of the idling stop function, the control unit 350 may detect temporary stopping or restarting of the vehicle based on the signal.

[0109] Furthermore, when the control unit 350 can receive a signal indicating that the steering wheel has been turned, the control unit 350 may detect a temporary stop or restart of the vehicle based on the signal. Furthermore, when the control unit 350 detects that the vehicle has started moving again based on the signal sent from the above-mentioned in-vehicle device, the control unit 350 can transmit a still image at that time (without a time lag) or with a delay thereafter (for example, with a time lag of one second or more, or several seconds or more).

[0110] (b) Control based on the transmission status of still images Furthermore, in another embodiment, the control unit 350 of the drive recorder 101 selects only some of the still images from the multiple images captured by the camera 304 based on the transmission status of the still images and transmits them to the external management server 102. This embodiment will be described below with reference to FIGS.

[0111] "Test of sending still images (Test 6)" Furthermore, the applicant conducted a test to control (here, selectively transmit) the transmission of multiple still images based on the transmission status of the still images. The results of Test 6 are shown in the form of a table in Figures 16 and 17. Although Figures 16 and 17 together form one table, they are shown in two parts for ease of viewing. 18 to 20 provide a more detailed explanation of three specific examples of this table.

[0112] In this embodiment, the control unit 350 transmits three types of data from the drive recorder 101 to the management server 102. The first type of data is a captured still image (image data with the largest capacity), the second type of data is latitude and longitude position information of the vehicle (text data with a relatively small capacity), and the third type of data is a ping (data with the smallest capacity) for confirming data reachability.

[0113] 16 and 17, the leftmost column shows the order in which still images are taken over time. In this example, a total of 37 still images are taken. The second column from the left shows the filenames of the still images. The third and fourth columns from the left show the vehicle's latitude and longitude at the time each still image was taken. The fifth column from the left shows the vehicle's speed at the time each still image was taken. In the sixth column from the left, the still image time difference is shown as the time interval between two adjacent still images.

[0114] In addition, in FIGS. 16 and 17, the seventh column from the left shows the response time of a ping to check data reachability. The eighth column from the left shows the time it takes for the latitude and longitude data (vehicle information) to reach the server (management server 102). The ninth column from the left shows the time it takes for a still image to reach the server (management server 102). The rightmost column shows the transmission / reception status of still image data.

[0115] Next, a specific example of Figures 16 and 17 will be described with reference to Figures 18 to 20. In this embodiment, the control unit 350 determines the transmission status of the communication processing unit 302 based on the number of still images transmitted and the response time from transmission of the still images to completion of transmission.

[0116] Referring to FIG. 18, the transmission state of data within the range indicated by reference numeral 1500 in FIG. 16 is conceptually shown. FIG. 18 shows a case where the data transmission status is good, and each of the above three types of data sets is transmitted at a predetermined interval of five seconds. Ping sends a message from the drive recorder 101 as the sender to the management server 102 as the destination, and measures the round trip time (RTT) until the response returned from the destination. Of the above three types of data, ping has the smallest data volume and can therefore be transmitted most efficiently.

[0117] Generally, under good transmission conditions, a ping response will be returned within a few hundred milliseconds. For this reason, ping is used for verification purposes to objectively check the communication status. Since the vehicle information (latitude and longitude data) is several hundred bytes in size, it takes less than one second for the management server 102 to receive it. Since the still image data has a file size of 50K to 60K bytes, it takes 3 to 4 seconds for the management server 102 to receive it.

[0118] In the example of FIG. 18, the first ping takes 67.7 milliseconds to transmit data, and the second ping takes 124.3 milliseconds to transmit data. Vehicle information has a larger data volume than ping, and therefore takes longer to transmit; the first vehicle information takes less than one second, and the next vehicle information takes less than one second. Still images have the largest data volume and therefore require the longest transmission time, with the first still image taking 4 seconds, the next still image taking 4 seconds, and so on. The predetermined interval for transmitting each data set is 5 seconds, so that each data transmission is completed within the predetermined interval without causing any delay. The predetermined interval between each still image can be determined taking into consideration the normal transmission time of a still image (for example, 5 seconds, but is not limited to this).

[0119] Next, referring to FIG. 19, a conceptual diagram of the transmission state of data within the range indicated by reference numeral 1600 in FIG. 16 is shown. FIG. 19 shows a case where the transmission conditions have deteriorated, and similarly illustrates a situation where three types of data sets are transmitted at predetermined intervals of five seconds. The first ping took 133.1 milliseconds to send data, and the second ping took over a second to send data. That is, the communication state is good at the time of Ping No. 7, but the long data arrival time suggests that the communication state deteriorated immediately after the Ping was sent. At the time of No. 8, the communication state remains poor, and the Ping transmission time becomes even longer. In this way, the control unit 350 can determine whether the transmission state is good or deteriorated by comparing the time required to send the Ping with a predetermined threshold.

[0120] Preferably, the control unit 350 determines the transmission status of the communication processing unit based on the response time of data that is transmitted separately from the still image and is smaller in size than the still image, but this small data is not limited to Ping. For example, the response time of the vehicle information may be used to determine the transmission status.

[0121] As the transmission conditions worsen, the first vehicle information takes one second, and the next vehicle information takes nine seconds. Because the predetermined interval for transmitting each data set is five seconds, the second vehicle information to be transmitted cannot be completed within five seconds. This may affect the transmission of subsequent vehicle information. In particular, for still images, the first vehicle information takes 15 seconds, and the next vehicle information takes 21 seconds. Because the predetermined interval for transmitting each set of data is 5 seconds, none of the still images transmitted in each case can be completed within 5 seconds. This may affect the transmission of subsequent still images. In this situation, there is a concern that the impact of the deterioration of communication conditions will accumulate.

[0122] Next, with reference to FIG. 20, a control for improving data transmission under the deteriorated transmission conditions illustrated in FIG. 19 will be described. FIG. 20 conceptually shows two ranges, particularly the range 1700 in FIG. 16 and the range 1710 in FIG.

[0123] In FIG. 20, similarly to FIG. 19, when two or more consecutive still images are being transmitted (transmission is incomplete), the control unit 350 determines that the transmission state (line state) is not good (see symbol A in FIG. 20). In this case, the control unit 350 temporarily suspends the transmission of still images subsequent to the still image whose transmission time has been extended, and stores the data in the storage device (memory) of the drive recorder 101 (see symbol B in FIG. 20). This temporarily suspends the transmission of the still image with the largest data volume, thereby reducing the overall burden of data transmission. As a result, it is possible to prevent the cumulative effect of deteriorated transmission states from affecting subsequent data transmissions, and contribute to early improvement of deteriorated transmission states.

[0124] Furthermore, the control unit 350 continues to monitor the number of still images being transmitted, and when the number of still images being transmitted is zero and the latest Ping response time is equal to or less than a predetermined threshold (specified time) (for example, equal to or less than 900 milliseconds) (No. 25), it can determine that the communication condition has improved. In this case, the control unit 350 starts sequentially transmitting the still images stored in the temporary storage memory (see symbol C in FIG. 20).

[0125] In the specific example of FIG. 20, if we calculate backwards from the fact that it took 87 seconds for the still image No. 9 to reach the management server 102, transmission from the drive recorder 101 begins at the timing of transmission of data No. 26. In the specific example of Fig. 20, still images are not transmitted after No. 26 due to the low speed caused by the still image transmission thinning function (see Fig. 17). Therefore, after No. 26, the state of waiting for completion of still image transmission continues to be zero, so the control unit 350 can sequentially transmit the still images stored in the temporary storage memory.

[0126] As described above, the control unit 350 of the drive recorder 101 controls the transmission of a plurality of still images captured by the camera 304 based on at least one of the vehicle's running state and the still image transmission state. (a) As illustrated in FIGS. 10 to 15, the control unit 350 may control the transmission of a plurality of still images captured by the camera 304 based on the traveling state of the vehicle. (b) As illustrated in FIGS. 16 to 20, the control unit 350 may control the transmission of a plurality of still images taken by the camera 304 based on the transmission status of the still images. (a) and (b) may be performed separately or in combination with each other.

[0127] The above-mentioned controls (a) and (b) may be started at any timing while the vehicle is running or stopped. Preferably, the controls may be started at the moment when the engine of a stopped vehicle is started and the vehicle is ready to run after the engine is turned off. In other words, the controls may be started in synchronization with the control of the drive recorder 101. The above controls (a) and (b) may be terminated at any timing while the vehicle is running or stopped. Preferably, the control may be terminated the moment the vehicle stops, the engine is turned off, and the vehicle becomes unable to run. In other words, the control may be terminated in synchronization with the control of the drive recorder 101.

[0128] In at least one of the cases of "(a) control based on the vehicle's running state" and "(b) control based on the still image transmission state," when the control unit 350 delays the transmission of a still image due to deterioration of the radio wave conditions or the like, the control unit 350 temporarily stores the still image in the transmission buffer. With this configuration, when retransmitting a still image from the transmission buffer, the control unit 350 may dynamically change the priority of each still image so that the most recent still image is always transmitted with the highest priority. Specifically, if still image A, still image B, and still image C are stored in the transmission buffer in descending order of their shooting time, the transmission priority is highest in descending order of their shooting time, which in this case is still image A, still image B, and still image C.

[0129] If the latest still image Z cannot be transmitted from this state due to reasons such as radio wave conditions, the control unit 350 stores the still image Z in the transmission buffer and changes the transmission priority to still image Z, still image A, still image B, and still image C. In this way, the control unit 350 performs control on a last-in, first-out basis. Furthermore, when the upper limit of the transmission buffer capacity is reached, the control unit 350 may perform processing to discard still images from the transmission buffer in order of lowest priority. In this way, when the radio wave conditions improve and transmission of still images becomes possible, the latest still image is always transmitted. Therefore, even when images captured by the drive recorder 101 are monitored remotely in real time, the latest status can be easily confirmed.

[0130] The number of cameras 304 used in the drive recorder 101 is not limited to one. For example, the drive recorder 101 can use as the camera 304 at least one of a camera that takes pictures in front of the vehicle, a camera that takes pictures behind the vehicle, a camera that takes pictures to the side of the vehicle, a camera that takes pictures diagonally in front of the vehicle, a camera that takes pictures diagonally behind the vehicle, and a camera that takes pictures at any angle range up to 360° around the vehicle.

[0131] For example, the cameras 304 of the drive recorder 101 may be provided on the front and rear sides of the vehicle. In this case, the camera 304F on the front side of the vehicle and the camera 304B on the rear side of the vehicle may independently capture an image of the front side and an image of the rear side, respectively. These cameras 304F and 304B may synchronize the timing of capturing video images under the control of the same or different control units 350. When selecting an appropriate still image from among a plurality of still images captured by each camera based on the running state of the vehicle, etc. (controls a and b above), these cameras 304F and 304B may synchronously extract still images in front of and behind the vehicle based on the same trigger.

[0132] As described above, the drive recorder 101 can transmit original still images captured by the camera 304 to the management server 102 in the LPWA format. In the above embodiment, each still image is captured at 5-second intervals, but this interval can be changed in other embodiments in accordance with the data capacity of each still image captured by the camera 304 and future changes in communication technologies such as LPWA.

[0133] "Control on the Management Server 102 Side" Next, referring again to FIGS. 1 and 4, The control on the management server 102 side will be explained.

[0134] The management server 102 has at least a receiving unit (for example, the communication control unit 406 in FIG. 4) that receives a plurality of still images transmitted from the drive recorder 101 that controls the above-mentioned contents, and a moving image creating unit 412 (for example, the moving image creating module 412 executed by the processor 403 in FIG. 4) that combines the received plurality of still images to create a moving image. Note that the moving image creating unit 412 may be implemented as a sub-module of the video information management module 411.

[0135] For example, the management server 102 may perform a method including the following steps. First, as a first step, the communication control unit 406 of the management server 102 sequentially receives a plurality of still images transmitted from the drive recorder 101 side. Next, at this time, the communication control unit 406 or the video creation unit 412 of the management server 102 may combine into one group multiple still images transmitted at predetermined time intervals (for example, several minutes, 10 minutes, 30 minutes, one hour or more). Alternatively, the communication control unit 406 or the video creation unit 412 of the management server 102 may combine into one group multiple still images transmitted at predetermined numbers (for example, several, 10, 30, 100, hundreds, thousands or more).

[0136] Next, as a second step, the video creation unit 412 of the management server 102 applies an animation effect to each of the still images integrated into one group, in which multiple still images are sequentially switched over over time, like a flip book, to create a video. At this time, the video may be created to have a special effect, as described below. At this time, the created video may be stored in the storage device 401 or 402 of the management server 102. Next, the communication control unit 406 of the management server 102 transmits the created video to the user terminal 103 .

[0137] For example, when a plurality of still images are transmitted from the drive recorder 101 to the management server 102, the management server 102 converts the plurality of still images into a video consisting of, for example, two still images per second. At this time, the video creation unit 412 of the management server 102 may perform control to reduce the amount of data used for creating the video. For example, the management server 102 stores and manages still images (video information 420) transmitted from the drive recorder 101 in the auxiliary storage device 402. In this case, if the management server 102 stores the data in the form of still images one after another, the amount of data to be managed will be large. Therefore, when the management server 102 converts multiple still images into a video, it may store only the animated data and delete the data of the still images used in the video from the auxiliary storage device 402.

[0138] When creating a video from a plurality of still images, the video creation unit 412 of the management server 102 can perform control to make the video easier to view. For example, the management server 102 may animate each received still image by aligning their time axes (according to the actual shooting time). However, in this case, the still images are updated every few seconds (for example, every 5 seconds), resulting in a drawn-out video and a large video size. Therefore, when integrating multiple received still images to create a video, the video creation unit 412 of the management server 102 may create a video that is played back faster than the actual chronological order. For example, when animate each still image taken at 5-second intervals, the video creation unit 412 may display two still images per second. This improves the ease of viewing the video. In this case, a video shot at one frame every five seconds is converted into a fast-forwarded video at one frame every 0.5 seconds.

[0139] Additionally, when creating a video from a plurality of still images, the video creation unit 412 of the management server 102 can perform control to make the video easier to view. Ideally, the drive recorder 101 captures a plurality of still images at regular time intervals. However, on the management server 102 side, delays can occur between still images depending on the line conditions, so the intervals are not necessarily regular. If this difference is too large, there is a risk that viewers will feel uncomfortable when the video is played back.

[0140] For example, if the delay is within a range of several seconds to several tens of seconds, the viewer will not feel uncomfortable even if the video is animated and played back on a fixed time axis. Since the video is constantly changing when the vehicle is moving, even if there is a slight delay in the playback time axis, the viewer will not feel uncomfortable. Furthermore, when the vehicle is stopped while waiting for a traffic light or railroad crossing to change, or when the vehicle is stopped in a traffic jam, there are a series of still images with little change, so even if there is a delay in the time axis, the viewer will not feel uncomfortable.

[0141] Therefore, if there are a series of still images with little change, the video creation unit 412 can create a video by thinning out the still images with little change. For example, if there are nine consecutive still images of waiting at a traffic light, the video creation unit 412 can delete two out of every three of them and create a video from the three still images. By using this configuration, the playback time of the video can be further compressed, and the time spent checking parts with little change when watching can be saved.

[0142] On the other hand, if the time axis deviation occurs at a larger time interval than a predetermined threshold (for example, if a deviation of one minute or more occurs), viewers may feel uncomfortable. For example, depending on the surrounding conditions of the traveling vehicle, such as inside a tunnel, the function of the communication processing unit 302 may be reduced or may temporarily stop functioning. Therefore, when creating a video from a plurality of received still images, if the still images have an interval that exceeds a predetermined threshold (for example, one minute), the video creation unit 412 may create the video by duplicating the previous still image of this interval. Alternatively, the video creation unit 412 may create the video by generating a new still image based on the still images before and after this interval. In this way, when still images are continuously missing, they may be copied at a rate corresponding to the missing time and inserted to create a moving image.

[0143] However, in the above case, the video creation unit 412 creates a video that displays information indicating that the original image data does not exist. For example, the video creation unit 412 may add a caption (text information, etc.) indicating that there is no image data at the location where the copied still image is inserted, and display this caption together with the caption when the video is played back. This makes it possible to clearly distinguish between images that were actually captured and images that were created to facilitate video creation, when images captured by the drive recorder 101 are used later.

[0144] Additionally, the video creation unit 412 may change the control when the vehicle is traveling at high speed compared to when the vehicle is traveling at low speed or is stopped as described above. For example, suppose each still image is captured at an interval of 5 seconds. In this case, if the captured images are converted into a video when the vehicle is traveling at a speed of about 100 km / h, the viewer may perceive the video as being played back in fast-forward mode. Therefore, the video creation unit 412 may change the frequency of fast-forwarding when the vehicle is traveling at a low speed and the frequency of fast-forwarding when the vehicle is traveling at a high speed.

[0145] Generally, when a series of still images are played back consecutively to create the appearance of a moving image, it is necessary to align the distance traveled by the vehicle with the playback time axis. However, when the vehicle is traveling at a high speed, if the number of still images is increased and then the image is converted into a moving image, the corresponding portion of the moving image will appear to be moving relatively slowly when played back. In other words, even if the vehicle's actual traveling speed is relatively fast, the played back image may appear to be moving slowly, which may cause discomfort to the viewer. Therefore, when the vehicle is traveling at a low speed or stopped, the video creation unit 412 may copy still images and increase the number of images before creating a video, as described above, while when the vehicle is traveling at a high speed, the video creation unit 412 may avoid the control of copying still images and increasing the number of images before creating a video.

[0146] Additionally, when creating a video from multiple still images, the video creation unit 412 can perform control to improve the convenience of the video. That is, in addition to a plurality of still images, vehicle information such as vehicle position information is transmitted from the drive recorder 101 to the management server 102. The still images and vehicle information may each have information on the time of capture. Therefore, even if the management server 102 receives the still images and vehicle information separately, they can be linked together.

[0147] Therefore, on the management server 102 side, when the communication control unit 406 receives from the drive recorder 101 a plurality of still images and the vehicle's position information at the time the plurality of still images were created, the video creation unit 412 may embed the vehicle's position information in the video when integrating the received plurality of still images to create a video. This may enable the user terminal 103 that plays the video to display the vehicle's position information in synchronization with the video. For example, when creating a video from multiple still images, the video creation unit 412 embeds vehicle information as metadata in the video file. This allows the location information to be synchronized with the video data only and displayed on a map when the video is played back using a dedicated application on the user terminal 103 side.

[0148] Furthermore, the video creation unit 412 may transmit the video created from the still images and the vehicle information separately to the user terminal 103. Vehicle information can be transmitted faster and more efficiently than video. Therefore, the application software of the user terminal 103 that displays this information prioritizes displaying vehicle information such as location information on a map, and displays (updates) the video as soon as reception is complete, thereby preventing the user from feeling a delay in data transmission overall.

[0149] Furthermore, the video creation unit 412 may include a still image file name and a checksum value in the vehicle information, which allows individual data stored on the management server 102 side to be synchronized and displayed later, or still image data that cannot be received or is corrupted for some reason to be skipped and displayed by the application software.

[0150] "Vehicle position estimation using still images" As described above, the management server 102 receives vehicle information from the drive recorder 101 and can know the latitude, longitude, etc. of the vehicle based on the information. However, there may be cases where the location information of the vehicle cannot be received sufficiently depending on the surrounding conditions of the traveling vehicle, communication conditions, etc. Therefore, the management server 102 may perform image analysis on the received still image to estimate the location of the vehicle.

[0151] 21A and 21B are diagrams schematically illustrating an example in which the management server 102 performs image analysis on a still image. As shown by reference numeral 1810 in (A) of Fig. 21, on general roads, there are road signs near intersections, etc. On expressways, there are exit guide boards, etc. These road signs, etc., usually have text information (for example, Tokyo) 1810 indicating the name of a place or other local area. Furthermore, as shown by reference numeral 1820 in FIG. 21(A), these road signs and the like usually have text information 1820 indicating the distance to the reference area in numbers (for example, 4 km). Furthermore, these road signs and the like are usually written with text information (not shown) that indicates the road number of a national road, a prefectural road, or the like in numbers (for example, National Route 6, not shown).

[0152] Therefore, the video creation unit 412 of the management server 102 may perform image analysis of the still image, particularly based on the text information 1810 and 1820. The position of the vehicle may be estimated by performing image analysis on the still image based on the names or numbers of roads or areas. For example, the video creation unit 412 of the management server 102 scans the still image to extract text such as letters or numbers. At this time, the video creation unit 412 may use an "OCR" function to recognize and convert characters from the still image.

[0153] Normally, road information used on road signs, etc. (for example, area names and numbers for national highways, etc.) is known, so it is possible to comprehensively create a database of various information such as maps and road information in advance and store it in storage device 401 or 402. Therefore, the video creation unit 412 transcribes the image data and temporarily stores the text information in the storage device 401 or 402. The vehicle position may then be estimated by comparing the information with a database that stores various types of information such as maps or road information.

[0154] Furthermore, the video creation unit 412 may perform image analysis of the still image based on a characteristic shape, figure, contour, color, or the like. For example, there are cases where two or more roads temporarily overlap in the vertical direction (such as at a junction), as illustrated by reference numeral 1830 in (A) of Fig. 21. In such cases, it may not be possible to determine which of the two vertically overlapping roads the vehicle is traveling on, based only on the latitude and longitude information of the vehicle. In such a case, the video creation unit 412 may perform image analysis on the received still image to distinguish, for example, between a case where there is some kind of structure above the road, as illustrated by reference numeral 1830 in (A) of Figure 21, and a case where there is nothing above the road, as illustrated by reference numeral 1840 in (B) of Figure 21.

[0155] For example, the video creation unit 412 may perform image analysis on the received still image based on changes in shape, contour, color, etc., focusing particularly on the area above the road, to determine whether or not a characteristic target 1830 such as a structure can be extracted (see (A) and (B) of Figure 21). In this case, the video creation unit 412 may classify the received still images by pixel or pixel group, and then perform image analysis by identifying the shape or color of each pixel or pixel group to identify characteristic targets.

[0156] The video creation unit 412 may perform image analysis training on various images in advance using AI machine learning. In particular, based on various specific examples, actual image analysis is performed to train the system to identify and extract targets (characters, numbers, shapes, contours, lines, vertices, colors, etc.) from images. The training is performed on, for example, hundreds, thousands, or more images of different targets, and various results are accumulated. In this case, image processing may be performed under various conditions, such as images of roads alone, images of roads with multiple vehicles traveling, images of roads during the day, and images of roads at night, and statistical data may be calculated and generated from the results. After a high identification rate, for example, approximately 99%, 99.9%, or higher, is obtained through training, the video creation unit 412 may actually perform image processing to identify targets from images.

[0157] In the training performed by the AI ​​machine learning, various image information may be used as training data. For example, a determination model may be generated by machine learning, using a still image as input and outputting information such as the outline, vertices, colors, letters, and numbers of characteristic objects. A new still image may be input to this determination model, so that a characteristic target object may be obtained as output.

[0158] For example, the video creation unit 412 may perform image analysis training in advance on still images of various tunnels or the top and bottom of an elevated road, and accumulate statistical information about the training. For example, the video creation unit 412 may store the shapes, colors, and the like of characteristic objects in various tunnels or on an elevated road. Then, when performing image analysis on the still images, the video creation unit 412 may identify the position of the vehicle based on the detection of these characteristic objects.

[0159] In this way, the video creation unit 412 may estimate the position of a vehicle that cannot be determined by GNSS, such as inside a tunnel or above or below an elevated road, from received still images using image analysis and AI, and use the estimated position for vehicle management and driving records. Also, the video creation unit 412 may estimate the position inside a tunnel from text on road signs, exit guide boards, etc. Also, for an elevated road, the video creation unit 412 may estimate whether the vehicle is traveling above or below the road signs, based on the presence or absence of structures at the top of the screen.

[0160] The video creation unit 412 of the management server 102 may identify the position of a traveling vehicle with higher accuracy by combining the vehicle information (latitude, longitude, etc.) transmitted from the drive recorder 101 with target information extracted based on image analysis performed on the still image. This position information may be used when creating a video from the still image and displaying the created video on a user terminal.

[0161] As described above, the control of creating a video from a plurality of still images performed on the management server 102 side may be started at any timing. For example, the management server 102 may control the creation of a plurality of still images into a video after reception of the still images has started and then after the reception of the still images has stopped. In other words, the video creation unit 412 may start controlling the creation of a video after reception of a series of still images has been completed, so that all still images from the entire process of the vehicle's travel are compiled into a single video. This means that the moment when reception of still images stops for, for example, one minute, several minutes, five minutes, ten minutes, or more may be used as a trigger to determine that the vehicle has finished traveling, and video creation may start immediately after that timing.

[0162] Alternatively, the management server 102 may control the creation of a video from multiple still images after a predetermined period of time has elapsed since the start of reception of still images transmitted from a moving vehicle. For example, the video creation unit 412 may start creating a video several minutes, 5 minutes, 10 minutes, 30 minutes, 1 hour, or several hours after the start of reception of still images. This allows videos to be created at predetermined time intervals or with a predetermined size. The video creation unit 412 may create one video if the vehicle's travel is completed in a short period of time, and may create multiple videos in succession if the vehicle's travel is extended over a long period of time. The control of animation of a plurality of still images performed on the management server 102 side may end when the animation is created and transmitted to the user terminal 103. Alternatively, the control of animation of a plurality of still images performed on the management server 102 side may end when the animation is created and saved in a storage device on the management server 102 side.

[0163] The above has been a description of the drive recorder 101 and the management server 102. Furthermore, the above-described control may be implemented as a program that enables the drive recorder 101 and the management server 102 to execute the control.

[0164] For example, this embodiment may be a computer program product for causing a computer to implement the control of the transmission of still images and / or vehicle information of the content described above with reference to Figures 1, 3, and 7 to 20 to the drive recorder 101. A computer program product may be implemented as a program or feature or function or routine or executable object.

[0165] Preferably, the computer program product comprises program code enabling execution of a method for controlling the drive recorder 101 including the above-mentioned controls a and b. The program is stored in a storage device connected to a control unit such as the microcomputer 301 or the DVR controller 303 of the drive recorder 101. The storage device may be built into the drive recorder 101 or may be externally connected. The program controls the control unit of the drive recorder 101 so as to enable control of transmission of still images.

[0166] Furthermore, the present invention may be a computer program product for causing a computer to execute the control of animation of the contents described above with reference to Figures 1, 4 and 21 on the management server 102. A computer program product may be implemented as a program or feature or function or routine or executable object.

[0167] Preferably, the computer program product comprises program code enabling the execution of the method for controlling the management server 102 including the animation described above. The program is stored in a storage device (401 or 402, etc.) connected to the video creation unit 412 of the management server 102. The storage device may be built into the management server 102 or may be externally connected. The program controls the video creation unit 412 of the management server 102 so as to enable control of animation.

[0168] Furthermore, this embodiment may be a computer program product for causing a computer to implement the control of the playback of the video described above with reference to FIG. 1 on the user terminal 103. A computer program product may be implemented as a program or feature or function or routine or executable object.

[0169] Preferably, the computer program product comprises program code enabling the execution of the method for controlling a user terminal 103 including video playback as described above. The program is stored in a video playback unit such as processor 503 of user terminal 103 or in a storage device (such as 501 or 502) connected to the video playback unit. The storage device may be built into user terminal 103 or may be externally connected. The program controls the video playback unit of user terminal 103 so as to enable control of video playback.

[0170] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0171] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD. In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0172] A computer program product, such as a computer program means, may be embodied as a memory card, a USB stick, a CD-ROM, a DVD or as a file downloadable from a server in a network, for example such a file may be provided by transferring the file comprising the computer program product over a wireless communication network.

[0173] The scope of the present invention is not limited to the structures explicitly described in the specification, but also includes combinations of various aspects of the present invention disclosed herein. The structures of the present invention that are sought to be patented are specified in the appended claims, but it is the intention of the present inventors to claim structures disclosed in this specification in the future, even if they are not currently specified in the claims.

[0174] The present invention is not limited to the configurations described in the above-described embodiments. The components of each of the above-described embodiments and variations may be arbitrarily selected and combined. Furthermore, any component of each embodiment or variation may be arbitrarily combined with any component described in the Summary of the Invention or any component embodying any component described in the Summary of the Invention. The present invention also intends to obtain rights to these configurations through amendments or divisional applications of the present application. Even if a description is made of "in the case of..." or "when...," it is not intended to describe a configuration limited to that case or time. Configurations that are not limited to those cases or times are also disclosed, and the present invention intends to obtain rights to them. Furthermore, any descriptions that specify an order are not limited to this order. Configurations in which some parts are deleted or the order is changed are also disclosed, and the present invention intends to obtain rights to them.

[0175] The above describes embodiments relating to various controls performed by the drive recorder (electronic device or in-vehicle device) 101, management server 102, and user terminal 103. These inventions can be combined in any manner. Furthermore, any configurations may be extracted from these inventions and combined. The applicant of the present application intends to acquire rights to inventions including these configurations. Furthermore, even if a description is made of "in the case of..." or "when...," it is not intended to describe a configuration that is limited to that case or time. These are merely examples of better configurations, and the applicant intends to acquire rights to configurations that are not in these cases or times. Furthermore, parts that are described in order are not limited to this order. Configurations in which some parts are deleted or the order is changed are also disclosed, and the applicant intends to acquire rights to them.

[0176] Furthermore, we intend to obtain rights to the overall design or partial design by converting the application to a design registration. The drawings depict the entire device in solid lines, but they also include partial designs claimed for parts of the device. For example, a partial design can be a partial design for a part of the device, or a part of that part. A partial design can be a part of the device, or a part of that part. We intend to obtain rights not only for the overall design, but also for partial designs in which any part of the solid line portion of the drawing is drawn as a broken line. Furthermore, all of the modules, components, and parts inside the device's casing, shown in the drawings, are subject to independent commerce, and we intend to similarly obtain rights by converting the application to a design registration.

[0177] The above-described embodiment discloses at least the following configurations. (1) a communication processing unit; a control unit that processes image information captured by the camera and transmits a plurality of still images via the communication processing unit; Equipped with The control unit controls the transmission of the plurality of still images based on at least one of a vehicle running state and a still image transmission state. This configuration reduces the burden on the electronic device during transmission because still images are transmitted, compared to when a video stream is directly transmitted. Alternatively, still images can be transmitted under optimal conditions by controlling the transmission of still images based on the vehicle's running state and the transmission state of the still images. (2) The control unit, regarding the running state of the vehicle, When it is detected that the speed of the vehicle is within a predetermined speed range or that the change in the position of the vehicle is within a predetermined distance range, a predetermined still image may be selected from the plurality of captured still images and transmitted. This configuration reduces the burden on the electronic device during transmission compared to when all still images are transmitted. In this case, by thinning out still images while paying attention to changes in the vehicle's traveling speed or position, still images suitable for creating a video can be preferentially retained and thinned out. (3) The predetermined still images may be still images selected every predetermined number of the plurality of still images captured continuously. With this configuration, still images are periodically thinned out for each group, so that it is possible to avoid localized culling of still images. (4) The predetermined still image may be the most recent still image among the plurality of still images captured in succession. With this configuration, still images are periodically thinned out at the beginning of each group, so that moving images can be generated regularly. (5) The predetermined still image may be a still image taken at or after the start of movement of the vehicle is detected. By configuring in this way, it is possible to record the surroundings when a vehicle that has been temporarily stopped at an intersection or the like starts moving again. (6) The running state of the vehicle is a change in the speed or acceleration of the vehicle; a change in the position of the vehicle; or In the case where the vehicle is a multi-wheeled vehicle including a four-wheeled vehicle, operation of one or more of an accelerator pedal, a brake pedal, a clutch pedal, a parking brake lever, an AT change lever, and a MT shift lever; The determination may be based on at least one of the following: This configuration allows the system to detect when a vehicle starts moving and transmit still images based on that information. Furthermore, by combining this information with changes in speed and position, the reliability of the detection results can be improved. (7) The control unit may transmit all still images within a predetermined period, including the predetermined still image, when an event related to a collision of the vehicle is detected. By configuring in this way, in the event of a traffic accident or the like, it is possible to comprehensively record original still images taken by the vehicle-mounted camera for the period before and after the accident. (8) The predetermined speed may be a speed when the vehicle is traveling at a low speed or a speed when the vehicle is stopped. This configuration allows for the still images to be thinned out when the vehicle's running position changes relatively slowly and similar scenes are continuously captured. Also, when the reception strength (RSRP), transmission speed, and reception speed are relatively stable and the need for spare images decreases, the still images can be thinned out. (9) The control unit may determine the transmission state of the communication processing unit based on the number of transmitted still images and a response time from transmission of the still image to completion of transmission. This configuration makes it possible to monitor changes in the communication status of a moving vehicle depending on the location and speed of the vehicle, etc. This allows still images to be transmitted under relatively good communication conditions. (10) The control unit may determine the transmission state of the communication processing unit based on a response time of data that is transmitted separately from the still image and has a size smaller than the still image. By configuring in this manner, the communication status of a moving vehicle can be determined in a relatively short time based on pings or vehicle information, which have a small data capacity, before transmitting a still image, which has a large data capacity. (11) When the response time of two consecutive still images exceeds a predetermined threshold, the control unit may store the subsequent still images in a storage device without transmitting the subsequent still images. With this configuration, it is possible to avoid the cumulative effect of delays in still image transmission caused by constantly transmitting still images under worsening transmission conditions. (12) The control unit may transmit the still image stored in the storage device when the number of still images being transmitted is 0 and the response time of data smaller in size than the still image that is transmitted separately from the still image is below a predetermined threshold. With this configuration, all selected still images can be transmitted while avoiding the cumulative effect of delays in transmitting still images. (13) The communication processing unit may transmit data in a low power wide area (LPWA) system. With this configuration, by using a relatively low frequency, communication can be performed over a relatively long distance with a relatively small amount of power. (14) The electronic device may be a drive recorder. With this configuration, the surroundings while driving can be recorded using the video captured and recorded by the drive recorder. (15) a receiving unit that receives the plurality of still images transmitted from the electronic device; a video creation unit that combines the received still images to create a video; A management server having: This configuration reduces the burden on the electronic device of sending still images and eliminates the burden of creating videos. Furthermore, since the management server has relatively high processing specifications, it can create videos based on the transmitted still images. For example, the management server can create more advanced videos, such as videos with altered timelines or videos that supplement still images, compared to when videos are created on the electronic device. (16) When creating the moving image by integrating the received still images, the moving image creation unit may create a moving image that is played back faster than an actual chronological order. By configuring in this way, it is possible to avoid giving the viewer an impression that the video is drawn out when played back, and to create a video that allows the viewer to quickly check the driving record, etc. (17) When creating the video from the received still images, if a time interval between the still images exceeds a predetermined threshold, the video creation unit: The moving image may be created by duplicating still images from before the interval, or by generating new still images based on still images from before and after the interval. By configuring in this way, it is possible to create a moving image that can avoid giving an unnatural impression to the viewer when the moving image is played back due to the occurrence of missing still images. (18) When a still image is duplicated or a new still image is generated, the video creation section may create the video displaying information indicating that the original image data does not exist. By configuring in this way, it is possible to create a video that can be used when a traffic accident occurs or when checking a driving route, by clearly distinguishing between still images that were actually taken and those that were not. (19) the receiving unit receives, from the electronic device, the plurality of still images and position information of the vehicle at the time when the plurality of still images were created; When the video creation unit creates a video by integrating the received multiple still images, it may embed the vehicle's location information in the video, thereby enabling the vehicle's location information to be displayed in synchronization with the video. With this configuration, when the video is played back on the user terminal side, the vehicle position can be known using only the video. (20) A program running on at least one computer, It may also be a program that enables the above method to be executed. With this configuration, the above control can be performed by providing a program to an existing electronic device or a management server. [Explanation of symbols]

[0178] 101... drive recorder (electronic device or in-vehicle device), 102... management server, 103... user terminal, 301... microcomputer, 302... communication processing unit, 303... DVR controller (control unit), 304... camera (photographing unit), 350... control unit

Claims

1. Electronic equipment mounted on a mobile vehicle, Communication processing unit, A camera that captures images from the front, rear, or 360 degrees and generates video information such as videos and still images, Equipped with, The camera is equipped with an AI processing function that performs preprocessing on the camera's image sensor. The AI ​​processing function has the function of performing image recognition based on the video captured by the camera. Along with generating a normal captured image, or instead, outputting an image cropped to show only a specific region. An electronic device characterized by the following features.

2. The aforementioned AI processing function outputs the target object from the image as metadata. The electronic device according to claim 1.

3. The communication processing unit transmits the image or metadata extracted from the specific region to the management server. The electronic device according to claim 1 or 2.

4. The communication processing unit transmits data related to the AI ​​processing function via the LPWA network. The electronic device according to any one of claims 1 to 3.

Images